Re: unified build mode and memory consumption
On 11/30/13, 11:16 AM, Gregory Szorc wrote: On 11/30/13, 3:24 AM, smaug wrote: (Before using unified mode I didn't have mem usage problems with gcc and there was never any significant difference in build times comparing to clang.) I filed bug 944842 to have mach report swapping during builds. Presumably the warning message will include tips on how to address it. As of inbound just now, `mach build` will now print overall swap activity at the end of every build (assuming psutil is available). If your swap activity is beyond a hard-coded threshold, you get a big PERFORMANCE WARNING at the end of the build. We probably got the threshold wrong for a large audience, so disagreeing with the alert is likely well-placed. We're very interested in feedback to help tune the heuristic. Please let us know via bugs, #build, or dev-builds what your experience is. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: unified build mode and memory consumption
Gregory Szorc wrote: On 11/30/13, 11:16 AM, Gregory Szorc wrote: On 11/30/13, 3:24 AM, smaug wrote: (Before using unified mode I didn't have mem usage problems with gcc and there was never any significant difference in build times comparing to clang.) I filed bug 944842 to have mach report swapping during builds. Presumably the warning message will include tips on how to address it. As of inbound just now, `mach build` will now print overall swap activity at the end of every build (assuming psutil is available). Is there a prebuilt version of psutil for Windows? The virtualenv builder appears to be looking for VC9. (At least, it completely fails to find the compiler that mozillabuild has already selected.) -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: unified build mode and memory consumption
On 12/12/13, 6:14 AM, Neil wrote: Gregory Szorc wrote: On 11/30/13, 11:16 AM, Gregory Szorc wrote: On 11/30/13, 3:24 AM, smaug wrote: (Before using unified mode I didn't have mem usage problems with gcc and there was never any significant difference in build times comparing to clang.) I filed bug 944842 to have mach report swapping during builds. Presumably the warning message will include tips on how to address it. As of inbound just now, `mach build` will now print overall swap activity at the end of every build (assuming psutil is available). Is there a prebuilt version of psutil for Windows? The virtualenv builder appears to be looking for VC9. (At least, it completely fails to find the compiler that mozillabuild has already selected.) psutil should build on Windows without VC9 as of bug 910487. You can read that bug to learn all about the joys of binary Python packages on Windows. We could possibly ship a pre-built psutil wheel package as part of MozillaBuild or allow the build system to fetch one from a public server. Please file a bug if you really want psutil on Windows and we'll see what we can do. However, psutil doesn't report paging on Windows, so the swapping warning will never display there. I'm not sure if this is a missing feature from psutil or if the underlying Windows APIs just don't support reporting paging sufficiently. I apologize for forgetting to mention this in my last post. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: unified build mode and memory consumption
Gregory Szorc wrote: There are also plans to offer a build mode for non-Gecko developers that downloads a pre-built libxul so these developers don't have to spend many minutes building Gecko. Isn't it already possible to build both Firefox and Thunderbird against the libxul SDK? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: unified build mode and memory consumption
On 11/30/13, 7:06 PM, Neil wrote: Gregory Szorc wrote: There are also plans to offer a build mode for non-Gecko developers that downloads a pre-built libxul so these developers don't have to spend many minutes building Gecko. Isn't it already possible to build both Firefox and Thunderbird against the libxul SDK? Yes. ac_add_options --with-libxul-sdk=/path/to/xulrunner-sdk But, I'm willing to bet most people don't know about this feature and I doubt it is highly used (again, lack of good data). Furthermore, you probably want a tool to download the most appropriate xulrunner for the current revision you are building, otherwise you may hit issues with out of date Gecko, etc. This is perfect fodder for a higher-level building tool like mach. I also believe a soft blocker is a sane API for querying release data: it's very difficult to correlate a Mercurial changeset to build ID because the data is exposed as 50+MB compressed JSON files [1] or you must resort to scraping an FTP server or you must access the LDAP-requiring build-api [2] (which doesn't really get what you want easily). We really need a sane API to perform certain kinds of look-ups. Hopefully Treeherder will provide that web service. [1] https://secure.pub.build.mozilla.org/builddata/buildjson/ [2] https://secure.pub.build.mozilla.org/buildapi/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
unified build mode and memory consumption
Hi all and FYI unified build mode has increased memory usage of building with gcc significantly. On my laptop (8 gig mem) I started to see some swapping, and because of that build times with unified mode weren't that much better than before. But now, finally there is a use case for clang - it uses a lot less memory than gcc, and unified+clang+gold debug clobber build is 16 mins comparing to the old ~30mins! (Before using unified mode I didn't have mem usage problems with gcc and there was never any significant difference in build times comparing to clang.) -Olli ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: unified build mode and memory consumption
On 11/30/13, 3:24 AM, smaug wrote: Hi all and FYI unified build mode has increased memory usage of building with gcc significantly. On my laptop (8 gig mem) I started to see some swapping, and because of that build times with unified mode weren't that much better than before. But now, finally there is a use case for clang - it uses a lot less memory than gcc, and unified+clang+gold debug clobber build is 16 mins comparing to the old ~30mins! (Before using unified mode I didn't have mem usage problems with gcc and there was never any significant difference in build times comparing to clang.) I filed bug 944842 to have mach report swapping during builds. Presumably the warning message will include tips on how to address it. There are tentative plans to build reporting of system resource usage and mach usage patterns to Mozilla - in a privacy aware manner of course. This will help us identify what kind of build environments exist in the wild and will help us automatically select optimal build options for different classes of build environments. There are also plans to offer a build mode for non-Gecko developers that downloads a pre-built libxul so these developers don't have to spend many minutes building Gecko. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
