On Wed, Nov 26, 2014 at 08:48:05AM -0800, Gregory Szorc wrote: > On 11/25/14 10:50 PM, Andreas Gal wrote: > > > >Would it make sense to check in some of the libraries we build that we very > >rarely change, and that don’t have a lot of configure dependencies people > >twiddle with? (icu, pixman, cairo, vp8, vp9). This could speed up build > >times in our infrastructure and for developers. This doesn’t have to be in > >mozilla-central. mach could pick up a matching binary for the current > >configuration from github or similar. Has anyone looked into this? > > Let me rephrase this request: you are asking for a cache of binary artifacts > for the build. > > Yes, this is critically important for developer productivity. > > Yes, it has been looked at extensively. > > Yes, it is achievable. > > But, it is a lot of work and the historical low engineering investment in > the build system has prevented this from coming to fruition thus far. > > Some background. > > There are 2 ways you can build your cache: high-level or low-level. > > In the low-level approach, you effectively have a globally distributed > ccache. This is what Mike Hommey has built in sscache. It's what release > automation uses. It even works on Windows. The low-level approach performs > per-object lookup when the build system is ready to produce that object. > Since the build system e.g. produces .o files then .so files, you must first > obtain cached values for the intermediate objects, then you move on to the > final objects. This is the nature of a low-level cache.
Actually, it doesn't have to be this way, and there are things in my sccache TODO list that would make it not need to download the intermediate objects unless linking requires them. > In the high-level approach, you recognize what the final output is and jump > straight to fetching that. e.g. if all you really need is libxul, you'll > fetch libxul.so. None of this intermediary .o files foo. > > Different audiences benefit from the different approaches. > > Firefox desktop, Fennec, and FxOS developers benefit mostly from a > high-level approach, as they don't normally care about changing C++. They > can jump straight to the end without paying a penalty of dealing with > intermediaries. > > Gecko/C++ developers care about the low-level approach, as they'll be > changing C++ things that invalidate the final output, so they'll be fetching > intermediate objects out of necessity. > > Implementing an effective cache either way relies on several factors: > > * For a high-level cache, a build system capable of skipping intermediates > to fetch the final entity (notably *not* make). > * Consistent build environments across release automation and developer > machines (otherwise the binaries are different and you sacrifice cache hit > rate or "accuracy"). > * People having fast internet connections to the cache (round trips don't > take longer than building locally). > * Fixing C++ header dependency hell so when C++ developers change something > locally, it doesn't invalidate the world, causing excessive cache misses and > local computation. > * Writing to a globally distributed cache that is also read by release > automation has some fun security challenges. > * Having a database to correlate source tree state with build artifacts *or* > a build system that is able to compute the equivalent DAG to formulate a > cache key (something we can't do today). Does the audience that needs high-level cache actually need a cache /that/ accurate? I wager using nightly builds would work for the vast majority of cases. We "just" need a build mode that doesn't build code, which we kinda sorta have, but it's broken at the moment (bug 1063880). Mike _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
