On Wed, Sep 16, 2015 at 1:33 AM, Mike Hommey <[email protected]> wrote:
> Hi, > > Following is a proposal for short term wins for Firefox front end > developers. > > The starting point is that there are too many things to change in the > recursive make build system to allow for something like this to work > properly for many different use cases. This also starts from the fact > that, well, we've kept the recursive make build system use the same > model it's been using for, well, close to 20 years. That model is > flawed, and I thought it was about time to do something about it, while > addressing real needs at the same time. > > My proposal is to start a new build backend from scratch, with the aim > to grow it into a complete solution. Build backend means I'm not talking > about replacing moz.build files with something new. I'm talking about > the build system they're munged into. > > It would initially piggy back on the existing build system, because > there are things that aren't exposed in moz.build yet that are > necessary. It would also, initially, not care about the entire build, > and even less about things that are outside of Firefox (tests, addons, > etc.) or compilation. > > My initial work is this patch: > https://pastebin.mozilla.org/8846539 > > (Note: it requires the patches from bug 1204712, bug 1204719 and bug > 1204715, all landed on mozilla-inbound, as of writing) > > What it allows is to: > - Run `mach configure` with a mozconfig with > ac_add_options --disable-compile-environment > - Download and unpack a nightly > - Use toolkit/mozapps/installer/unpack.py to explode its omni.ja files > - Move that unpacked nightly in $objdir/dist/bin (for mac, that involves > more fiddling, because dist/bin is a somewhat flattened version of the > .app directory) > - Ensure the files in $objdir/dist/bin are older than the source > files. > - Run `mach build-backend -b FasterMake` > - Run `mach build faster` > > After the last command, $objdir/dist/bin should contain a bastardized > Firefox, with xul, js, etc. coming from the source tree, and the > remainder still being there from the original nightly. > > `mach run` should work with that. > > Note: removing files is not supported in this initial implementation, so > removing files will still require manual intervention. Editing files > #include'd from others should work properly. > > From there, changes to js, xul, images, etc. only require running `mach > build faster`, and on my machine, that takes something between 3 and 4 > seconds. Future iterations should allow subsecond updates. > > Other changes would require a full `mach build`, without > --disable-compile-environment. > > Anyways, as you can see with the list of steps above, this requires a > lot of manual work at the moment, so my proposal extends to streamlining > this process. > > - Extend mach artifact to handle any type of Firefox build (not only > Firefox for Android), and to unpack builds completely (not just .so > files), and doubly (by which I mean, unpacking omni.ja as well, with > the code used by toolkit/mozapps/installer/unpack.py) > > - Extend the build system to allow to apply multiple backends at the > same time. `mach build-backend -b FasterMake` only spends 0.05s on my > machine generating its build system, but still spends 4s that are > already spent during configure to read and analyze moz.build files. > That's also true of other backends, and it would be better to allow > people to add their prefered extra backends to their mozconfig. > (and they all rely on the recursive make backend, so that one can be > kept always on for now). It would also help make some of those extra > backends enabled by default depending on what's built. > > - Ensure --disable-compile-environment works with all the above, but I > don't think there is actually anything to do for this. As a matter > of fact, I've tested it, and it seems to work well enough (and that > lead me to recently fix bug 1203851) > > - Make mac builds fill dist/Nightly.app directly instead of filling > dist/bin (or fill dist/bin as if it were dist/Nightly.app), because > having to run all of > > https://dxr.mozilla.org/mozilla-central/source/browser/app/Makefile.in?offset=0#95-109 > at the end of every build is a pain point. This would have benefits > for other kinds of builds, too. > > With those four above adressed, we can have a better experience for > Firefox frontend developers in a very short time. The last point is > likely to take some time, so it might make sense to replace it with an > extension to the first point, where mach artifact would flatten the > build to make it look like dist/bin, and the FasterMake backend would > invoke the right rules from browser/app/Makefile.in. > > The only downside I can see to this plan is that since the new backend > would not be exercized on automation, it would have chances of breaking. > Fair enough, but the same can be said of all the other backends. > > On the other hand, I can see many benefits: > - The new backend would not be exercized on automation. While it can be > seen as a downside, it is also a *huge* benefit, because it breaks all > the limits that come from ensuring all of automation works. Not having > to care about l10n is what makes the PoC work so well. > - As mentioned above, short term win for frontend developers > - Other use cases can plug on top of it. We could quickly add tests, for > example, and this is where the piggy-backing happens: we can just > invoke small parts of the current build system when it makes sense. > - It sets things in motion so that a future with non-make based build > backends can actually happen (like, replacing make with ninja or tup > in this new backend would be fairly easy). > - It allowed me to spot some of the shortcomings of the current > moz.build model with the emitter and helped get a clearer vision of > where we should be going on the long term (we can discuss this later). > See for instance the horror show in FasterMakeBackend.consume_object. > 6 slithly different ways to be notified of files to install or > preprocess, and that's only for things relevant to Firefox. The almost > 1:1 mapping between moz.build variables and emitted objects is not > helpful. > > I'm almost ready to send that patch to review. At this point, I just > want to add a few comments to fastermake.py before doing so. > > Thoughts on the overall proposal? > I am generally thrilled to see these ideas and the code that's already started to hit my review queue! One thing I am sensitive to is developer ergonomics and the intuitiveness of the solution. I want `mach build` to be sufficient to get the most ideal build possible. No `mach build binaries`. No `mach build faster`. Just a single command that "does the right thing." Existing solutions for Fennec and your proposal above seem to fall a little short of this, requiring people run special build backends and special, largely undocumented targets to `mach build`. It's inevitable for the short term future that we won''t have a build backend that can evaluate the entire DAG as one unit and have it complete in <1s for no-op or light builds. So, build/make targets that perform a subset of the build (like "binaries" and "faster") will be necessary. However, I think there is room to provide a bit more intelligence. One of the ideas that came out of the developer productivity meetup last week was the idea of a `mach config` or `mach create-environment` command that configures a new build environment. This is logically a tuple of (srcdir, objdir, build configuration). An interactive wizard would ask you what you would like to build and it would write out optimal build configuration files for you. The build tools (mach) would keep better state and would read these additional settings from the build configuration files. We could even throw in some filesystem watching to intelligently build the necessary subsystem. So e.g. `mach build` would automatically invoke `make binaries` if C++ files changed and `mach build` would invoke `make faster` if JS changed. These details are all flexible. The takeaway is the build system should be simple and fast without requiring additional pieces of knowledge like the "binaries" and "faster" targets.
_______________________________________________ dev-builds mailing list [email protected] https://lists.mozilla.org/listinfo/dev-builds
