This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.

Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd

If tests have a TestMain method the os import is unused and
breaks building the test.

Test: gotestmain/testmain_test.go
Change-Id: Ia46cdd0df71b0fc7a53d08644d220ecfd779b2ff

Add globbing to filesystem mocking so that more code can be tested
against the mock.  Also moves the filesystem mock to pathtools,
and renames pathtools.GlobWithExcludes to pathtools.Glob, replacing
the existing pathtools.Glob.

Test: blueprint tests
Change-Id: I722df8121bc870c4a861d7c249245c57dcb607be

Add globbing with dependency checking to blueprint.  Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.

Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files.  The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints".  Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them.  The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.

Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.

Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6

Other than cleaning up the code, the only change in behavior is to run
the docs build in parallel with the primary builder, saving about a
second on a Soong bootstrap (10s -> 9s).

Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440

Soong (and soon, Blueprint) embeds the results of filesystem globs into
the build.ninja, and uses a helper tool to detect when the glob changes
and we need to rebuild the build.ninja file. This is more flexible than
listing the affected directories in the depfiles, since it can check to
see if a file we actually cared about was added, instead of re-running
anytime any file was added/removed (which happens on atomic file
modifications as well).

My recent bootstrap simplification broke this, since the helper rules
are in the main build.ninja, but I removed the ability to regenerate
that file from itself. So keep the current model, but add a rule into
the primary and main stages that allow themselves to re-run their
generator and write out a new build.ninja file if necessary. The actual
build rules of the generator aren't necessary, since we already built
them in the previous stage.

Change-Id: Ib51245920b2ec3ee5306c0c269361a5a8733caa8

This would have been a case for implicit outputs, but these just aren't
needed anymore.

Added in Ninja 1.7, for outputs that will not show up in $out.

Blueprint properties that end up as command line arguments need to be
both ninja and shell escaped.  Provide helpers that primary builders can
use to appropriately escape them.

Change-Id: Ifd697d87edb1c6f0a910377835c391bbe8f95b42

tl;dr: Read if you don't use the wrapper or use SKIP_NINJA

Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.

But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.

Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.

In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.

The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.

The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.

Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747