First of all, I'd like to thank you and the rest of the build peers for
your tireless efforts!
On 2013-08-02 5:13 PM, Gregory Szorc wrote:
(Cross posting. Please reply to dev.builds.)
Sorry, but cross-posting to both lists. I don't think most of the
people interested in this conversation are on dev.builds (I am, FWIW.)
I've noticed an increase in the number of complaints about the build
system recently. I'm not surprised. Building mozilla-central has gotten
noticeably slower. More on that below. But first, a request.
Many of the complaints I've heard have been from overhearing hallway
conversations, noticing non-directed complaints on IRC, having 3rd
parties report anecdotes, etc. *Please, please, please voice your
complaints directly at me and the build peers.* Indirectly complaining
isn't a very effective way to get attention or to spur action. I
recommend posting to dev.builds so complaints and responses are public
and easily archived. If you want a more personal conversation, just get
in contact with me and I'll be happy to explain things.
This is fair, but really the builds getting slower is so obvious that I
would be surprised if none of the build config peers have noticed it in
their daily work. :-)
Builds are getting slower. http://brasstacks.mozilla.com/gofaster/#/ has
high-level trends for our automation infrastructure. I've also noticed
my personal machines taking ~2x longer than they did 2 years ago.
Unfortunately, I can't give you a precise breakdown over where the
increases have been because we don't do a very good job of recording
these things. This is one reason why we have better monitoring on our Q3
goals list.
My anecdotal evidence also matches the 2x slower metric.
Now, on to the reasons why builds are getting slower.
# We're adding new code at a significant rate.
Here is a breakdown of source file types in the tree by Gecko version.
These are file types that are directly compiled or go through code
generation to create a compiled file.
Gecko 7: 3359 C++, 1952 C, 544 CC, 1258 XPIDL, 110 MM, 195 IPDL
Gecko 14: 3980 C++, 2345 C, 575 CC, 1268 XPIDL, 272 MM, 197 IPDL, 30 WebIDL
Gecko 21: 4606 C++, 2831 C, 1392 CC, 1295 XPIDL, 292 MM, 228 IPDL, 231
WebIDL
Gecko 25: 5211 C++, 3029 C, 1427 CC, 1268 XPIDL, 262 MM, 234 IPDL, 441
WebIDL
That nets totals of:
7: 7418
14: 8667
21: 10875
25: 11872
As you can see, we're steadily adding new source code files to the tree.
mozilla-central today has 60% more source files than Gecko 7! If you
assume number of source files is a rough approximation for compile time,
it's obvious why builds are getting slower: we're building more.
As large new browser features like WebRTC and the ECMAScript
Internationalization API continue to dump hundreds of new source files
in the tree, build times will increase. There's nothing we can do about
this short of freezing browser features. That's not going to happen.
Hmm. I'm not sure if the number of source files is directly correlated
to build times, but yeah there's clearly a trend here!
# Header dependency hell
We have hundreds of header files that are included in hundreds or even
thousands of other C++ files. Any time one of these widely-used headers
changes, the object files get invalidated by the build system
dependencies and we have to re-invoke the compiler. This also likely
invalidates ccache, so it's just like a clobber build.
No matter what we do to the build backend to make clobber builds faster,
header dependency hell will continue to undermine this progress for
dependency builds.
I don't believe the build config group is in a position to tackle header
dependency hell at this time. We are receptive to good ideas and will
work with people to land patches. Perhaps an ad-hoc group of Platform
developers can band together to address this?
I have been playing with an idea in my head about this. What if we had
a list of the most popular headers in our tree, and we looked through
them and tried to cut down the number of #includes in the headers? That
should help create more isolated sub-graphs and hopefully help with
breaking the most severe dependency chains.
Writing a tool to spit out this information should be fairly easy.
# Increased reliance on C++ language features
I *suspect* that our increased reliance on C++ language features such as
templates and new C++11 features is contributing to slower build times.
It's been long known that templates and other advanced language features
can "blow up" the compiler if used in certain ways. I also suspect that
modern C++11 features haven't been optimized to the extent years-old C++
features have been. Combine this with the fact compilers are working
harder than ever to optimize code and it wouldn't surprise me if a CPU
cycle invested in the compiler isn't giving the returns it used to.
I would absolutely love for a compiler wizard to sit down and "profile"
Gecko C++ in Clang, GCC, and MSVC. If there are things we can do to our
source or to the compilers themselves to make things faster, that could
be a huge win.
Like dependency hell, I don't believe the build config group will tackle
this any time soon.
The problem of C++ template instantiation is quite well known. That is
hard to avoid in the compiler because the language mandates the compiler
to do the work. But it's possible that we've been added template idioms
that are excessively expensive to compile. It would be really nice if
we had a way of detecting spikes in our build times and see if any of
them were caused by this. But I do not expect us to be able to win much
here, as the problem could well be death by a thousand cuts.
But I'm not convinced at all about the C++11 features contributing to
this. I cannot think of any reason at all why that should be the case
for the things that we've started to use. Do you have any evidence to
implicate some of those features?
# Clobbers are more frequent and more annoying
Clobbers are annoying. It annoys me every time I see the CLOBBER file
has been updated. I won't make excuses for open bugs on known
required-clobber issues: we should fix them all.
I suspect clobbers have become more annoying in recent months because
overall build times have increased. If builds only took 5 minutes, I'm
not sure the cries would be as loud. That's no excuse for not fixing it,
however. Please continue to loudly complain every time there is a clobber.
This should be relatively "easy" to address (compared to the other
things that we can do, of course). I assert that every time we touch
the CLOBBER file, it's because the build system could not figure out the
dependencies properly. Fortunately we can easily log the CLOBBER file
and go back in time and find all of the patches that included CLOBBER
modifications and debug the build dependency issues. Has there been any
effort to address these issues by looking at the testcases that we have
in form of patches?
# Slowness Summary
There are many factors contributing to making the build system slower. I
would argue that the primary contributors are not within the control of
the build config group. Instead, the fault lives with all the compiled
code (mainly C++) in the tree. I'm not trying to point fingers or
deflect blame. But if I had to give one answer for "why have the builds
gotten slower," that answer is "we're compiling a lot more code."
Since we can't change that, the next question becomes "what are we doing
to make building faster." The answer: lots of things.
Every time that we don't utilize 100% of our cores during the build
process, that's an unnecessary slowdown. That consistently wastes a lot
of time during every build, and it also means that we can't address this
by getting more powerful machines. :(
# Building faster
One of our Q3 goals is to replace the "export tier" with something more
efficient. More on tiers at [1]. This should make builds faster,
especially on pymake. Just earlier this week we made WebIDL and XPIDL
code generation concurrent. Before, they executed serially, failing to
utilize multiple CPU cores. Next steps are XPIDL code gen, installing
headers, and preprocessing. This is all tracked in bug 892644.
Out of curiosity, why was the export tier the fist target for this? I
may lack context here, but the slowest tier that we have is the platform
libs tier. Wouldn't focusing on that have given us the biggest possible
bang for the buck?
We're also slowly working towards moving all the data that describes how
to invoke the compiler into moz.build files. Once that data is there, we
will be able to experiment with precompiled headers and other compiler
techniques to make compilation faster. This will also enable us to put
all the compiler rules in a single make file. This will enable scaling
out to as many cores as you have available. I'm optimistic we'll have
this by the end of 2013.
In your recent dev-builds post
<https://groups.google.com/forum/#!topic/mozilla.dev.builds/QkzCD8h9OjM>
you estimated that this conversion is going to take about another year.
Have your estimates changed since then?
I concede that the rate of progress is slower than I would like. I would
halve the build times for everyone tomorrow if I could! Unfortunately,
this isn't an overnight project. It will take time. We're starting to
turn the corner and realize benefits of moving data to moz.build files.
Real gains should be seen over the next few weeks and months. Hopefully
the gains will be able to outpace the setbacks from all the new code
landing in the tree. Please stick with us. Until then, please continue
to complain about the build system. If you have time, please stop in
#build and ask how you can help. You might be surprised what you could do!
For those of us not following the Build Config component, can you please
list the steps that you're planning to go through in order to achieve
these gains? The things that I've heard people talk about is
derecursifying the export tier and the moz.build conversion, but it's
not clear to me what the prioritization looks like, and what do we
expect to achieve with each piece of work...
Thanks a lot!
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
