(Cross posting. Please reply to dev.builds.)
I've noticed an increase in the number of complaints about the build
system recently. I'm not surprised. Building mozilla-central has gotten
noticeably slower. More on that below. But first, a request.
Many of the complaints I've heard have been from overhearing hallway
conversations, noticing non-directed complaints on IRC, having 3rd
parties report anecdotes, etc. *Please, please, please voice your
complaints directly at me and the build peers.* Indirectly complaining
isn't a very effective way to get attention or to spur action. I
recommend posting to dev.builds so complaints and responses are public
and easily archived. If you want a more personal conversation, just get
in contact with me and I'll be happy to explain things.
Anyway, on to the concerns.
Builds are getting slower. http://brasstacks.mozilla.com/gofaster/#/ has
high-level trends for our automation infrastructure. I've also noticed
my personal machines taking ~2x longer than they did 2 years ago.
Unfortunately, I can't give you a precise breakdown over where the
increases have been because we don't do a very good job of recording
these things. This is one reason why we have better monitoring on our Q3
goals list.
Now, on to the reasons why builds are getting slower.
# We're adding new code at a significant rate.
Here is a breakdown of source file types in the tree by Gecko version.
These are file types that are directly compiled or go through code
generation to create a compiled file.
Gecko 7: 3359 C++, 1952 C, 544 CC, 1258 XPIDL, 110 MM, 195 IPDL
Gecko 14: 3980 C++, 2345 C, 575 CC, 1268 XPIDL, 272 MM, 197 IPDL, 30 WebIDL
Gecko 21: 4606 C++, 2831 C, 1392 CC, 1295 XPIDL, 292 MM, 228 IPDL, 231
WebIDL
Gecko 25: 5211 C++, 3029 C, 1427 CC, 1268 XPIDL, 262 MM, 234 IPDL, 441
WebIDL
That nets totals of:
7: 7418
14: 8667
21: 10875
25: 11872
As you can see, we're steadily adding new source code files to the tree.
mozilla-central today has 60% more source files than Gecko 7! If you
assume number of source files is a rough approximation for compile time,
it's obvious why builds are getting slower: we're building more.
As large new browser features like WebRTC and the ECMAScript
Internationalization API continue to dump hundreds of new source files
in the tree, build times will increase. There's nothing we can do about
this short of freezing browser features. That's not going to happen.
# Header dependency hell
We have hundreds of header files that are included in hundreds or even
thousands of other C++ files. Any time one of these widely-used headers
changes, the object files get invalidated by the build system
dependencies and we have to re-invoke the compiler. This also likely
invalidates ccache, so it's just like a clobber build.
No matter what we do to the build backend to make clobber builds faster,
header dependency hell will continue to undermine this progress for
dependency builds.
I don't believe the build config group is in a position to tackle header
dependency hell at this time. We are receptive to good ideas and will
work with people to land patches. Perhaps an ad-hoc group of Platform
developers can band together to address this?
# Increased reliance on C++ language features
I *suspect* that our increased reliance on C++ language features such as
templates and new C++11 features is contributing to slower build times.
It's been long known that templates and other advanced language features
can "blow up" the compiler if used in certain ways. I also suspect that
modern C++11 features haven't been optimized to the extent years-old C++
features have been. Combine this with the fact compilers are working
harder than ever to optimize code and it wouldn't surprise me if a CPU
cycle invested in the compiler isn't giving the returns it used to.
I would absolutely love for a compiler wizard to sit down and "profile"
Gecko C++ in Clang, GCC, and MSVC. If there are things we can do to our
source or to the compilers themselves to make things faster, that could
be a huge win.
Like dependency hell, I don't believe the build config group will tackle
this any time soon.
# Clobbers are more frequent and more annoying
Clobbers are annoying. It annoys me every time I see the CLOBBER file
has been updated. I won't make excuses for open bugs on known
required-clobber issues: we should fix them all.
I suspect clobbers have become more annoying in recent months because
overall build times have increased. If builds only took 5 minutes, I'm
not sure the cries would be as loud. That's no excuse for not fixing it,
however. Please continue to loudly complain every time there is a clobber.
# Slowness Summary
There are many factors contributing to making the build system slower. I
would argue that the primary contributors are not within the control of
the build config group. Instead, the fault lives with all the compiled
code (mainly C++) in the tree. I'm not trying to point fingers or
deflect blame. But if I had to give one answer for "why have the builds
gotten slower," that answer is "we're compiling a lot more code."
Since we can't change that, the next question becomes "what are we doing
to make building faster." The answer: lots of things.
# Building faster
One of our Q3 goals is to replace the "export tier" with something more
efficient. More on tiers at [1]. This should make builds faster,
especially on pymake. Just earlier this week we made WebIDL and XPIDL
code generation concurrent. Before, they executed serially, failing to
utilize multiple CPU cores. Next steps are XPIDL code gen, installing
headers, and preprocessing. This is all tracked in bug 892644.
We're also slowly working towards moving all the data that describes how
to invoke the compiler into moz.build files. Once that data is there, we
will be able to experiment with precompiled headers and other compiler
techniques to make compilation faster. This will also enable us to put
all the compiler rules in a single make file. This will enable scaling
out to as many cores as you have available. I'm optimistic we'll have
this by the end of 2013.
I concede that the rate of progress is slower than I would like. I would
halve the build times for everyone tomorrow if I could! Unfortunately,
this isn't an overnight project. It will take time. We're starting to
turn the corner and realize benefits of moving data to moz.build files.
Real gains should be seen over the next few weeks and months. Hopefully
the gains will be able to outpace the setbacks from all the new code
landing in the tree. Please stick with us. Until then, please continue
to complain about the build system. If you have time, please stop in
#build and ask how you can help. You might be surprised what you could do!
[1]
https://developer.mozilla.org/en-US/docs/How_Mozilla%27s_build_system_works#Build_Tiers
(Please reply to dev.builds.)
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
