> Building mozilla-central has gotten noticeably slower. Yep. A bit over two years ago I started doing frequent browser builds for the first time; previously I'd mostly just worked with the JS shell. I was horrified by the ~25 minutes it took for a clobber build. I got a new Linux64 box and build times dropped to ~12 minutes, which made a *large* difference to my productivity.
On that same machine, I'm now back to ~25 minutes again. I've assumed it's due to "more code", specifically: 1. more code in the repository; 2. more code generated explicitly (e.g. dom bindings); 3. more code generated implicitly (i.e. templates). I don't know the relative impacts, though 1 is clearly a big part of it. Even worse, link times are through the roof. I was thrilled when, two years ago, I switched from ld to gold and linking time plummeted. The first link after rebooting was always slow, but I could link libxul back then in about 9 seconds. I haven't measured recently but I'm certain it's now *much* higher. Even the JS shell, which used to take hardly any time to link, now takes 10s or more; enough that I often switch to doing something else while waiting. If I could speed up any part of the builds, it would be linking. Waiting a long time to test a one file change sucks. > # Header dependency hell I've recently done a bunch of work on improving the header situation in SpiderMonkey. I can break it down to two main areas. == MINIMIZING #include STATEMENTS == There's a clang tool called "include-what-you-use", a.k.a. "IWYU" (http://code.google.com/p/include-what-you-use/). It tells you exactly which headers should be included in all your files. I've used it to minimize #includes somewhat already (https://bugzilla.mozilla.org/show_bug.cgi?id=634839) and I plan to do some more Real Soon Now (https://bugzilla.mozilla.org/show_bug.cgi?id=888768). There are still a couple of hundred unnecessary #include statements in SpiderMonkey. (BTW, SpiderMonkey has ~280 .cpp files and ~370 .h files.) IWYU is great, because it's really hard to figure this stuff out manually. It's also not perfect; about 5% of its suggestions are simply wrong, i.e. it says you can remove a #include that you can't. Also, there are often project-specific idioms that it doesn't know about -- there were several, but the one I remember off the top of my head is that it was constantly suggesting I remove "mozilla/StandardInteger.h" and add <stdint.h> (thankfully that's not an issue any more :) There are pragmas that you can annotate your source with, but I found them to be not always work as advertised and not really worth the effort. Although IWYU basically works, it feels a bit like software that doesn't get much maintenance. I haven't been doing rigorous measurements, but I think that these IWYU-related improvements don't do much for clobber builds, but can help significantly with partial rebuilds. It also just feels good to make these improvements. IWYU tells you the #includes that are unnecessary; it also tells you which ones are missing, i.e. which ones are being #included indirectly through another header. I've only bothered removing #includes because adding the missing ones doesn't feel worthwhile. Sometimes this means that when you remove an unnecessary |#include "a.h"|, you have to add a |#include "b.h"| because b.h was being pulled in only via a.h. Not a big deal. Relatedly, jorendorff wrote a python script that identifies cycles in header dependencies and diagnosed a cycle in SpiderMonkey that involved *11* header files. He and I broke that cycle in a series of 29 patches in https://bugzilla.mozilla.org/show_bug.cgi?id=872416, https://bugzilla.mozilla.org/show_bug.cgi?id=879831 https://bugzilla.mozilla.org/show_bug.cgi?id=886205. Prior to the last 9 patches, if you touched vm/Stack-inl.h and rebuilt, you'd rebuild 125 .cpp files. After these patches landed, it dropped to 30. The cycle-detection script has been incorporated into the |make check-style| target that is about to land in https://bugzilla.mozilla.org/show_bug.cgi?id=880088. I've also done various bits of refactoring with an eye towards simplifying the header dependencies. https://bugzilla.mozilla.org/show_bug.cgi?id=880041 is one example. These kinds of things can interact well with IWYU -- you do a clean-up, then run IWYU to find all the #includes that are no longer necessary. Gregory suggested that headers aren't something that the build config group can tackle, and I agree. Modifying #include statements en masse is much easier if you have some familiarity with the code. You need a sense of which headers should include which others, and often you have to move code around. So I encourage people to use IWYU on parts of the code they are familiar with. (Aryeh did this with editor/ in https://bugzilla.mozilla.org/show_bug.cgi?id=772807.) I should also note that this work is pretty tedious. There's lots of waiting for compilation, lots of try server runs to pick up bustage in other configurations, and then when you land you'll break additional configurations that aren't tested on try server (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=866916). That's life. == PROPER #ifndef WRAPPERS == There's one lower-hanging piece of fruit relating to headers. Many headers get #included multiple times per .cpp file. For example, in SpiderMonkey, jsapi.h gets #included literally dozens of times per .cpp file. So compilers optimize for this. With GCC and clang, if you have a canonical #ifndef wrapper in a header file, e.g.: #ifndef FOO_H__ ... #endif the compiler will avoid re-including the header if FOO_H__ is defined. But for this to work the #ifndef wrapper must have the above form (|#if defined(FOO_H__)| is also allowed, and there can't be any tokens before or afterwards (but whitespace and comments are ok). (See http://gcc.gnu.org/onlinedocs/cpp/Once_002dOnly-Headers.html#Once_002dOnly-Headers and http://gcc.gnu.org/onlinedocs/cppinternals/Guard-Macros.html for details.) SpiderMonkey had ~30 header files that violated this, most of them by doing something like this: #if !defined(jsion_baseline_frame_inl_h__) && defined(JS_ION) ... #endif I fixed these in https://bugzilla.mozilla.org/show_bug.cgi?id=881579. Unlike the #include minimization, these don't require domain-specific expertise and are easy to fix. They are also easy to find. clang and GCC have a -H option which tells you exactly which header files get read during compilation. IIRC, GCC even tells you (at the bottom of the output) which of these files are read multiple times because they violate the standard #ifndef wrapper form. (More generally, if you want to take any kind of measurements about header file inclusions, you should really base it off |gcc -H| or |clang -H|; if you try it any other way -- e.g. counting #includes statically via a script -- you'll probably get wildly incorrect numbers because you won't account for this optimization.) Infuriatingly, on both my Linux and Mac machines, a number of the system headers violate these forms, and so get read many more times than necessary. For SpiderMonkey, IIRC some system headers get included ~5,000 times for 280 .cpp files :( (Another annoying Linux thing is that we have these Mozilla-specific $OBJDIR/dist/system_wrappers_js/*.h that just #include a system header. I don't entirely understand what they're for, but they don't have a #ifndef wrapper and so also get included many times per .cpp file. I tried adding a #ifndef wrapper but got bustage I didn't understand. At least these files are tiny.) I'm not sure how MSVC handles #ifndef wrappers. Someone told me that it needs |#pragma once| to avoid the multiple includes, but I don't know if that's true. > One of our Q3 goals is to replace the "export tier" with something more > efficient. I just want to say that I appreciate all this effort going into the build system and think it's thoroughly worthwhile. Thanks! Nick _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
