Re: mach has landed
On 5/28/13 7:28 AM, xunxun wrote: 于 2012/9/27 9:12, Gregory Szorc 写道: The next time you pull mozilla-central, you'll find a new tool in the root directory: mach $ ./mach build Well, when I try mach build on Win8 x64 and use VC2010 Chinese Edition compiler, its output information (Chinese information) will be messy: make build doesn't have the messy characters. How can I solve that? Please file a detailed bug if you are seeing this on the tip of mozilla-central. There is a long and painful history of dealing with Unicode in the build system and mach. I *thought* we had things in a good state. Perhaps not :/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
于 2012/9/27 9:12, Gregory Szorc 写道: The next time you pull mozilla-central, you'll find a new tool in the root directory: mach $ ./mach build Well, when I try mach build on Win8 x64 and use VC2010 Chinese Edition compiler, its output information (Chinese information) will be messy: make build doesn't have the messy characters. How can I solve that? Thanks. -- Best Regards, xunxun ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Thu, Oct 4, 2012 at 12:17 PM, Benjamin Smedberg wrote: > On 10/4/2012 12:11 PM, Gregory Szorc wrote: >> >> >> #1 is a larger problem with no easy solution. Without significantly >> altering how make/pymake work, about the best we can do is parse the process >> output of the build and try to print something intelligent if the build >> failed. Unfortunately, this is far from robust because parallel make >> sometimes interleaves output from multiple processes on stdout. > > There are a couple ways to approach the problem. The pymake-specific way is > to redirect output from commands to a temporary file/pipe and feed that back > to the driver in a more intelligent way. This is not trivial but probably > not super-hard. You'd end up with a lot of possibilities there too, such as > "make silently *except* when an error occurs, in which case dump the full > log". Tup uses the temporary file method, which seems to work pretty well. Though I found the output more readable when the command string is displayed after the command finishes, so instead of: print "Running gcc foo.c..." run 'gcc foo.c' > tmpfile dump tmpfile it's more like: run 'gcc foo.c' > tmpfile print "Ran gcc foo.c" dump tmpfile It ends up displaying the output under a banner for each job, so it is easy to see where the error messages belong. The downside is it's not immediately obvious which jobs are active since nothing is displayed until the process finishes. Maybe there's a better way to handle that, though :) -Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
Nicholas Nethercote wrote: On Thu, Oct 4, 2012 at 10:18 AM, Justin Lebar wrote: 1) Build errors are hard to identify with make. Parallel execution can make them even harder to track down. Poor output from invoked processes is also a problem. I have a script [1] which works well enough for my purposes in the normal Mozilla build (I haven't tried it with mach). It highlights stderr in red and summarizes all the errors after make finishes so you don't have to go searching for them. I redirect output to a file and then search for the first "error:" string, but I'm kind of low-tech like that. I run a parallel make, then if that fails I run a serial make in the deepest folder that I can identify (not always easy with parallel make, but better than doing a top-level serial make). If there are too many errors to deal with then (except on Windows) I run make from inside vim. -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Thu, Oct 4, 2012 at 10:18 AM, Justin Lebar wrote: >> 1) Build errors are hard to identify with make. Parallel execution can make >> them even harder to track down. Poor output from invoked processes is also a >> problem. > > I have a script [1] which works well enough for my purposes in the > normal Mozilla build (I haven't tried it with mach). It highlights > stderr in red and summarizes all the errors after make finishes so you > don't have to go searching for them. I redirect output to a file and then search for the first "error:" string, but I'm kind of low-tech like that. N ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
> 1) Build errors are hard to identify with make. Parallel execution can make > them even harder to track down. Poor output from invoked processes is also a > problem. I have a script [1] which works well enough for my purposes in the normal Mozilla build (I haven't tried it with mach). It highlights stderr in red and summarizes all the errors after make finishes so you don't have to go searching for them. It's possible for races to cause this tool to output too much information or to cause weird interleavings of error messages, but in practice that almost never happens to me. -Justin [1] https://github.com/jlebar/conf/blob/master/bin/rse Usage: rse make -sj16 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On 10/4/2012 12:11 PM, Gregory Szorc wrote: #1 is a larger problem with no easy solution. Without significantly altering how make/pymake work, about the best we can do is parse the process output of the build and try to print something intelligent if the build failed. Unfortunately, this is far from robust because parallel make sometimes interleaves output from multiple processes on stdout. There are a couple ways to approach the problem. The pymake-specific way is to redirect output from commands to a temporary file/pipe and feed that back to the driver in a more intelligent way. This is not trivial but probably not super-hard. You'd end up with a lot of possibilities there too, such as "make silently *except* when an error occurs, in which case dump the full log". The other way to do this is to build the log redirection into the individual commands (compilation) which are most common. This would work for standard `make`. Probably you really want a combination of those, since pymake already does compilation using a native command (IIRC) so you can do the pymake redirection in that command and the gmake redirection using some shell scripting structure. --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On 10/3/12 5:22 PM, Honza Bambas wrote: I just hit some compilation error caused by some recent patch in conjunction with my build config and mach has just failed with a python trace back: There was no warning log as well. I was a bit hoping that mach would help with this (tracking build error in parallel builds), but it didn't. Would it possible to let mach track and show the actual error in the log somehow to make it simpler to find? There are two problems here: 1) Build errors are hard to identify with make. Parallel execution can make them even harder to track down. Poor output from invoked processes is also a problem. 2) mach prints an unnecessary trace back when processes exit with non-0 exit code. As Ehsan said, #2 is being tracked in bug 795427. #1 is a larger problem with no easy solution. Without significantly altering how make/pymake work, about the best we can do is parse the process output of the build and try to print something intelligent if the build failed. Unfortunately, this is far from robust because parallel make sometimes interleaves output from multiple processes on stdout. Anyway, we already have something similar that does log parsing: TBPL. If someone were to port TBPL's build output parsing logic to mach (actually the build logic mach uses is in python/mozbuild/), I think that would be an excellent addition to the tree! FWIW, mach uses a structured logging system under the hood [1]. So, once you parse build output into a data structure, that's capable of being written to a log file without losing its structured aspect. In theory, once mach is used on the buildbot machines, instead of recording whole log blobs and parsing every time they are requested, we could record the individual structured events. This opens up new avenues for indexing and querying build logs. Additionally, we could hook the log stream up to metlog [2] to enable interesting scenarios such as streaming build logs off of buildbot hosts in real-time. A brokering agent could convert streamed build logs to websocket streams in real-time (using something like Node.js perhaps) and now you have real-time viewing of logs via TBPL. Cool! Of course, all this can be done for test output too. In summary, mach and structured logging enable the build system and test runners to produce a more rich data stream (one that doesn't need to be parsed) and allow it to be consumed in new and novel ways. If you are in position to enact change, I'd love to talk to you about realizing some of the scenarios above. [1] https://github.com/mozilla/mozilla-central/blob/master/python/mozbuild/README.rst [2] https://github.com/mozilla-services/metlog-py ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
I believe this and other similar issues are being tracked by bug 795427.
Cheers,
Ehsan
On 2012-10-03 8:22 PM, Honza Bambas wrote:
So, another subject:
when I do a full build in parallel (win/pymake or using mach) and there
is some build error, it is always very hard to track the error down in
the log, it is almost always hidden among many other lines. I just hit
some compilation error caused by some recent patch in conjunction with
my build config and mach has just failed with a python trace back:
File "./mach", line 48, in
mach.run(sys.argv[1:])
File "c:\Mozilla\src\mozilla-central\python/mach\mach\main.py", line
146, in run
self._run(argv)
File "c:\Mozilla\src\mozilla-central\python/mach\mach\main.py", line
202, in _run
fn(**stripped)
File "c:\Mozilla\src\mozilla-central\python/mach\mach\build.py", line
45, in build
log=False, print_directory=False)
File
"c:\Mozilla\src\mozilla-central\python/mozbuild\mozbuild\base.py", line
256, in _run_make
fn(**params)
File
"c:\Mozilla\src\mozilla-central\python/mozbuild\mozbuild\base.py", line
279, in _run_command_in_srcdir
self._run_command(cwd=self.topsrcdir, **args)
File
"c:\Mozilla\src\mozilla-central\python/mozbuild\mozbuild\base.py", line
338, in _run_command
raise Exception('Process executed with non-0 exit code: %s' % args)
Exception: Process executed with non-0 exit code:
[u'c:/Mozilla/mozilla-build/msys/bin/sh.exe', u'-c',
u'c:/Mozilla/src/mozilla-central/build/pymake/make.py
f client.mk -j8 -s']
There was no warning log as well. I was a bit hoping that mach would
help with this (tracking build error in parallel builds), but it
didn't. Would it possible to let mach track and show the actual error
in the log somehow to make it simpler to find?
Otherwise, thanks for this work!
-hb-
On 9/27/2012 3:12 AM, Gregory Szorc wrote:
The next time you pull mozilla-central, you'll find a new tool in the
root directory: mach
$ ./mach build
0.09 /usr/bin/make -f client.mk -j8 -s
0.25 Adding client.mk options from
/Users/gps/src/services-central/.mozconfig:
0.25 MOZ_OBJDIR=$(TOPSRCDIR)/objdir
0.25 MOZ_MAKE_FLAGS=-j9 -s
0.33 TEST-PASS | check-sync-dirs.py |
/Users/gps/src/services-central/js/src/build <=
/Users/gps/src/services-central/build
0.34 Generating /Users/gps/src/services-central/configure using autoconf
1.55 cd /Users/gps/src/services-central/objdir
1.55 /Users/gps/src/services-central/configure
1.57 Adding configure options from
/Users/gps/src/services-central/.mozconfig:
1.57 --enable-application=browser
1.57 --enable-optimize
1.70 loading cache ./config.cache
> 335.19 309 compiler warnings present.
$ ./mach warnings-summary
>
10-Wlogical-op-parentheses
10-Wtautological-compare
13-Wsometimes-uninitialized
26-Wconversion
30-Wunused-variable
34-Wunused-function
34-Wdeprecated-declarations
47-Wtautological-constant-out-of-range-compare
67-Wunused-private-field
309Total
$ ./mach warnings-list
>
content/base/src/nsContentUtils.cpp:279:15 [-Wunused-private-field]
private field 'mKey' is not used
content/base/src/nsDocumentEncoder.cpp:2014:78
[-Wlogical-op-parentheses] '&&' within '||'
content/xbl/src/nsBindingManager.cpp:433:1
[-Wdelete-non-virtual-dtor] delete called on 'nsBindingManager' that
has virtual functions but non-virtual destructor
content/xbl/src/nsXBLProtoImpl.cpp:22:21 [-Wunused-private-field]
private field 'cx' is not used
content/xbl/src/nsXBLProtoImpl.cpp:23:13 [-Wunused-private-field]
private field 'versionBefore' is not used
$ ./mach mochitest-browser browser/base/content/test/
>
(Yes, you can tab complete test paths in your shell instead of going
through some silly environment variable dance).
If you get lost:
$ ./mach help
(Try doing that with make!)
Read more about mach at [1].
I want to stress that mach is still in its infancy. It is currently
only a fraction of what I want it to be. However, mach today is
infinitely more than what it was yesterday: non-existent. That's
progress. If you have a feature request or a work flow, please file a
bug. Even better, contribute a patch! You only need to know beginner's
Python to add new commands to mach.
I hope you enjoy using mach. I look forward to many new and exciting
features in the near future.
[1] http://gregoryszorc.com/blog/2012/09/26/mach-has-landed/
Gregory
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
So, another subject:
when I do a full build in parallel (win/pymake or using mach) and there
is some build error, it is always very hard to track the error down in
the log, it is almost always hidden among many other lines. I just hit
some compilation error caused by some recent patch in conjunction with
my build config and mach has just failed with a python trace back:
File "./mach", line 48, in
mach.run(sys.argv[1:])
File "c:\Mozilla\src\mozilla-central\python/mach\mach\main.py", line
146, in run
self._run(argv)
File "c:\Mozilla\src\mozilla-central\python/mach\mach\main.py", line
202, in _run
fn(**stripped)
File "c:\Mozilla\src\mozilla-central\python/mach\mach\build.py", line
45, in build
log=False, print_directory=False)
File
"c:\Mozilla\src\mozilla-central\python/mozbuild\mozbuild\base.py", line
256, in _run_make
fn(**params)
File
"c:\Mozilla\src\mozilla-central\python/mozbuild\mozbuild\base.py", line
279, in _run_command_in_srcdir
self._run_command(cwd=self.topsrcdir, **args)
File
"c:\Mozilla\src\mozilla-central\python/mozbuild\mozbuild\base.py", line
338, in _run_command
raise Exception('Process executed with non-0 exit code: %s' % args)
Exception: Process executed with non-0 exit code:
[u'c:/Mozilla/mozilla-build/msys/bin/sh.exe', u'-c',
u'c:/Mozilla/src/mozilla-central/build/pymake/make.py
f client.mk -j8 -s']
There was no warning log as well. I was a bit hoping that mach would
help with this (tracking build error in parallel builds), but it
didn't. Would it possible to let mach track and show the actual error
in the log somehow to make it simpler to find?
Otherwise, thanks for this work!
-hb-
On 9/27/2012 3:12 AM, Gregory Szorc wrote:
The next time you pull mozilla-central, you'll find a new tool in the
root directory: mach
$ ./mach build
0.09 /usr/bin/make -f client.mk -j8 -s
0.25 Adding client.mk options from
/Users/gps/src/services-central/.mozconfig:
0.25 MOZ_OBJDIR=$(TOPSRCDIR)/objdir
0.25 MOZ_MAKE_FLAGS=-j9 -s
0.33 TEST-PASS | check-sync-dirs.py |
/Users/gps/src/services-central/js/src/build <=
/Users/gps/src/services-central/build
0.34 Generating /Users/gps/src/services-central/configure using autoconf
1.55 cd /Users/gps/src/services-central/objdir
1.55 /Users/gps/src/services-central/configure
1.57 Adding configure options from
/Users/gps/src/services-central/.mozconfig:
1.57 --enable-application=browser
1.57 --enable-optimize
1.70 loading cache ./config.cache
> 335.19 309 compiler warnings present.
$ ./mach warnings-summary
>
10-Wlogical-op-parentheses
10-Wtautological-compare
13-Wsometimes-uninitialized
26-Wconversion
30-Wunused-variable
34-Wunused-function
34-Wdeprecated-declarations
47-Wtautological-constant-out-of-range-compare
67-Wunused-private-field
309Total
$ ./mach warnings-list
>
content/base/src/nsContentUtils.cpp:279:15 [-Wunused-private-field]
private field 'mKey' is not used
content/base/src/nsDocumentEncoder.cpp:2014:78
[-Wlogical-op-parentheses] '&&' within '||'
content/xbl/src/nsBindingManager.cpp:433:1
[-Wdelete-non-virtual-dtor] delete called on 'nsBindingManager' that
has virtual functions but non-virtual destructor
content/xbl/src/nsXBLProtoImpl.cpp:22:21 [-Wunused-private-field]
private field 'cx' is not used
content/xbl/src/nsXBLProtoImpl.cpp:23:13 [-Wunused-private-field]
private field 'versionBefore' is not used
$ ./mach mochitest-browser browser/base/content/test/
>
(Yes, you can tab complete test paths in your shell instead of going
through some silly environment variable dance).
If you get lost:
$ ./mach help
(Try doing that with make!)
Read more about mach at [1].
I want to stress that mach is still in its infancy. It is currently
only a fraction of what I want it to be. However, mach today is
infinitely more than what it was yesterday: non-existent. That's
progress. If you have a feature request or a work flow, please file a
bug. Even better, contribute a patch! You only need to know beginner's
Python to add new commands to mach.
I hope you enjoy using mach. I look forward to many new and exciting
features in the near future.
[1] http://gregoryszorc.com/blog/2012/09/26/mach-has-landed/
Gregory
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Tue, Oct 2, 2012 at 7:12 PM, Jeff Gilbert wrote: > As an additional data point, my experience is that the interactivity of my > machine is not noticeably impacted when I overcommit with -j12 on my > 4core/8thread i7 windows machine. Right, I think we've discussed this issue enough. Everyone with strong opinions about this can simply change the default. Good work, Gregor. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
As an additional data point, my experience is that the interactivity of my machine is not noticeably impacted when I overcommit with -j12 on my 4core/8thread i7 windows machine. Task manager shows the cores often pegged at 100%, but the machine basically behaves normally. Neither is my Ubuntu 8core/16thread/-j16 (sometimes -j24) linux box noticeably affected when it's fully pegged. Clearly it depends what you're trying to do, but for my uses, I just spin up a build and do other things normally until it's done, regardless of how over-committed it is. - Original Message - From: "Karl Tomlinson" To: dev-platform@lists.mozilla.org Sent: Tuesday, October 2, 2012 3:05:55 PM Subject: Re: mach has landed On Fri, 28 Sep 2012 11:44:56 -0700, Gary Kwong wrote: >> http://blog.johnford.org/new-mac-builders-ssds-j-settings/ > > Quoted from that blog: > > "I did find that it is better to set the -j setting too high than > it is to set it too low." > > -Gary Better in terms of build time, which is the right metric for a dedicated build machine. But bear in mind that a developer is usually trying to use the machine at the same time as doing a build. Adding more context switching load to the system without significantly improving build times is not necessarily a win for the developer. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Fri, 28 Sep 2012 11:44:56 -0700, Gary Kwong wrote: >> http://blog.johnford.org/new-mac-builders-ssds-j-settings/ > > Quoted from that blog: > > "I did find that it is better to set the -j setting too high than > it is to set it too low." > > -Gary Better in terms of build time, which is the right metric for a dedicated build machine. But bear in mind that a developer is usually trying to use the machine at the same time as doing a build. Adding more context switching load to the system without significantly improving build times is not necessarily a win for the developer. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On 09/28/2012 08:49 AM, Mike Hommey wrote: On Fri, Sep 28, 2012 at 05:45:00PM +0200, Mike Hommey wrote: On Fri, Sep 28, 2012 at 05:34:09PM +0200, Honza Bambas wrote: On 9/28/2012 12:58 PM, Mike Hommey wrote: On Fri, Sep 28, 2012 at 06:45:24AM -0400, Benoit Jacob wrote: 2012/9/28 Aryeh Gregor : On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: I actually held out on you with the initial landing of mach: there is more advanced tree building code in the pipes, complete with progress indicators. However, getting it reviewed is a challenge because we want the build system integration to be right. When that lands, mach will be smart enough to automatically define make flags optimal for your machine. e.g. -j == # of cores. So, you get optimal/parallel builds with no configuration necessary. This will all be configurable, of course. Is -j equal to number of cores really optimal? I've always been told that it's better to set it to more like twice the number of cores, because some fraction of threads will normally be stalled on I/O and you don't want cores idle. Depending on various factors, this can be offset by other factors like cores competing for limited resources such as CPU cache. On my core i7 with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More generally while building a C++ project (C++ is computationally expensive to compile) you're not very likely to be IO bound, especially with a SSD. A C project might be different. On my core i7, -j12 is fastest. bigger values are marginally slower but they are all faster than -j8. Mike Win7 x64 / i7@3.9GHz / 8 cores / 12GB@1600MHz / SSD mirrored, warm clobbered build of debug desktop browser: -j12: 19m00s exactly -j9: 18m45s exactly Using more processes then cores is counter productive. I checked this 2 years ago already when build took just 13 minutes (we grow!!). Conclusion: YMMV, most likely, depending on OS ; I'm on Linux. IOW, it's better to make it configurable. And on OSX, the same is true http://blog.johnford.org/new-mac-builders-ssds-j-settings/ We distcc users want it higher too. In fact, I recommend using something crazy like -j20 and using the DISTCC_HOSTS syntax to limit the number of concurrent jobs on a per-host basis. It may even make sense when you're only using localhost, since then the limit is global across multiple source tree compiles. That's theoretical, though; I haven't actually tested that out. echo "localhost/8" > ~/.distcc/hosts # or export DISTCC_HOSTS="localhost/8" make -j20 My current hosts file has "localhost/6 10.0.0.99/12". I have 8 virtual cores. I limit to 6 jobs when I have a distcc server available, to keep the build from messing up the responsiveness of my desktop. cgroups would probably work better for that, honestly. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
Win7 x64 / i7@3.9GHz / 8 cores / 12GB@1600MHz / SSD mirrored, warm clobbered build of debug desktop browser: -j12: 19m00s exactly -j9: 18m45s exactly Using more processes then cores is counter productive. I checked this 2 years ago already when build took just 13 minutes (we grow!!). Conclusion: YMMV, most likely, depending on OS ; I'm on Linux. IOW, it's better to make it configurable. And on OSX, the same is true http://blog.johnford.org/new-mac-builders-ssds-j-settings/ Mike Quoted from that blog: "I did find that it is better to set the -j setting too high than it is to set it too low." -Gary ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On 9/28/2012 8:49 AM, Mike Hommey wrote: On Fri, Sep 28, 2012 at 05:45:00PM +0200, Mike Hommey wrote: On Fri, Sep 28, 2012 at 05:34:09PM +0200, Honza Bambas wrote: On 9/28/2012 12:58 PM, Mike Hommey wrote: On Fri, Sep 28, 2012 at 06:45:24AM -0400, Benoit Jacob wrote: 2012/9/28 Aryeh Gregor : On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: I actually held out on you with the initial landing of mach: there is more advanced tree building code in the pipes, complete with progress indicators. However, getting it reviewed is a challenge because we want the build system integration to be right. When that lands, mach will be smart enough to automatically define make flags optimal for your machine. e.g. -j == # of cores. So, you get optimal/parallel builds with no configuration necessary. This will all be configurable, of course. Is -j equal to number of cores really optimal? I've always been told that it's better to set it to more like twice the number of cores, because some fraction of threads will normally be stalled on I/O and you don't want cores idle. Depending on various factors, this can be offset by other factors like cores competing for limited resources such as CPU cache. On my core i7 with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More generally while building a C++ project (C++ is computationally expensive to compile) you're not very likely to be IO bound, especially with a SSD. A C project might be different. On my core i7, -j12 is fastest. bigger values are marginally slower but they are all faster than -j8. Mike Win7 x64 / i7@3.9GHz / 8 cores / 12GB@1600MHz / SSD mirrored, warm clobbered build of debug desktop browser: -j12: 19m00s exactly -j9: 18m45s exactly Using more processes then cores is counter productive. I checked this 2 years ago already when build took just 13 minutes (we grow!!). Conclusion: YMMV, most likely, depending on OS ; I'm on Linux. IOW, it's better to make it configurable. And on OSX, the same is true http://blog.johnford.org/new-mac-builders-ssds-j-settings/ Well, we can all agree that on 99.9% of multi core machines, -jN>1 is faster than the implied -j1 default and that the typical naive first-time builder will see faster build times out of the gate if a somewhat reasonable -jN default is chosen for them. If this reduces complaints of "I have an i7 and compiling Firefox took over an hour, OMGWTFBBQ?!" then we have won, IMO. Power users continue to have the ability to fine-tune their -jN value at their discretion, of course. Nobody loses. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Fri, Sep 28, 2012 at 05:45:00PM +0200, Mike Hommey wrote: > On Fri, Sep 28, 2012 at 05:34:09PM +0200, Honza Bambas wrote: > > On 9/28/2012 12:58 PM, Mike Hommey wrote: > > >On Fri, Sep 28, 2012 at 06:45:24AM -0400, Benoit Jacob wrote: > > >>2012/9/28 Aryeh Gregor : > > >>>On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: > > I actually held out on you with the initial landing of mach: there is > > more > > advanced tree building code in the pipes, complete with progress > > indicators. > > However, getting it reviewed is a challenge because we want the build > > system > > integration to be right. When that lands, mach will be smart enough to > > automatically define make flags optimal for your machine. e.g. -j == # > > of > > cores. So, you get optimal/parallel builds with no configuration > > necessary. > > This will all be configurable, of course. > > >>>Is -j equal to number of cores really optimal? I've always been told > > >>>that it's better to set it to more like twice the number of cores, > > >>>because some fraction of threads will normally be stalled on I/O and > > >>>you don't want cores idle. > > >>Depending on various factors, this can be offset by other factors like > > >>cores competing for limited resources such as CPU cache. On my core i7 > > >>with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More > > >>generally while building a C++ project (C++ is computationally > > >>expensive to compile) you're not very likely to be IO bound, > > >>especially with a SSD. A C project might be different. > > >On my core i7, -j12 is fastest. bigger values are marginally slower > > >but they are all faster than -j8. > > > > > >Mike > > > > Win7 x64 / i7@3.9GHz / 8 cores / 12GB@1600MHz / SSD mirrored, warm > > clobbered build of debug desktop browser: > > -j12: 19m00s exactly > > -j9: 18m45s exactly > > > > Using more processes then cores is counter productive. I checked > > this 2 years ago already when build took just 13 minutes (we > > grow!!). > > Conclusion: YMMV, most likely, depending on OS ; I'm on Linux. IOW, it's > better to make it configurable. And on OSX, the same is true http://blog.johnford.org/new-mac-builders-ssds-j-settings/ Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Fri, Sep 28, 2012 at 05:34:09PM +0200, Honza Bambas wrote: > On 9/28/2012 12:58 PM, Mike Hommey wrote: > >On Fri, Sep 28, 2012 at 06:45:24AM -0400, Benoit Jacob wrote: > >>2012/9/28 Aryeh Gregor : > >>>On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: > I actually held out on you with the initial landing of mach: there is more > advanced tree building code in the pipes, complete with progress > indicators. > However, getting it reviewed is a challenge because we want the build > system > integration to be right. When that lands, mach will be smart enough to > automatically define make flags optimal for your machine. e.g. -j == # of > cores. So, you get optimal/parallel builds with no configuration > necessary. > This will all be configurable, of course. > >>>Is -j equal to number of cores really optimal? I've always been told > >>>that it's better to set it to more like twice the number of cores, > >>>because some fraction of threads will normally be stalled on I/O and > >>>you don't want cores idle. > >>Depending on various factors, this can be offset by other factors like > >>cores competing for limited resources such as CPU cache. On my core i7 > >>with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More > >>generally while building a C++ project (C++ is computationally > >>expensive to compile) you're not very likely to be IO bound, > >>especially with a SSD. A C project might be different. > >On my core i7, -j12 is fastest. bigger values are marginally slower > >but they are all faster than -j8. > > > >Mike > > Win7 x64 / i7@3.9GHz / 8 cores / 12GB@1600MHz / SSD mirrored, warm > clobbered build of debug desktop browser: > -j12: 19m00s exactly > -j9: 18m45s exactly > > Using more processes then cores is counter productive. I checked > this 2 years ago already when build took just 13 minutes (we > grow!!). Conclusion: YMMV, most likely, depending on OS ; I'm on Linux. IOW, it's better to make it configurable. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On 9/28/2012 12:58 PM, Mike Hommey wrote: On Fri, Sep 28, 2012 at 06:45:24AM -0400, Benoit Jacob wrote: 2012/9/28 Aryeh Gregor : On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: I actually held out on you with the initial landing of mach: there is more advanced tree building code in the pipes, complete with progress indicators. However, getting it reviewed is a challenge because we want the build system integration to be right. When that lands, mach will be smart enough to automatically define make flags optimal for your machine. e.g. -j == # of cores. So, you get optimal/parallel builds with no configuration necessary. This will all be configurable, of course. Is -j equal to number of cores really optimal? I've always been told that it's better to set it to more like twice the number of cores, because some fraction of threads will normally be stalled on I/O and you don't want cores idle. Depending on various factors, this can be offset by other factors like cores competing for limited resources such as CPU cache. On my core i7 with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More generally while building a C++ project (C++ is computationally expensive to compile) you're not very likely to be IO bound, especially with a SSD. A C project might be different. On my core i7, -j12 is fastest. bigger values are marginally slower but they are all faster than -j8. Mike Win7 x64 / i7@3.9GHz / 8 cores / 12GB@1600MHz / SSD mirrored, warm clobbered build of debug desktop browser: -j12: 19m00s exactly -j9: 18m45s exactly Using more processes then cores is counter productive. I checked this 2 years ago already when build took just 13 minutes (we grow!!). -hb- ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Fri, Sep 28, 2012 at 06:45:24AM -0400, Benoit Jacob wrote: > 2012/9/28 Aryeh Gregor : > > On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: > >> I actually held out on you with the initial landing of mach: there is more > >> advanced tree building code in the pipes, complete with progress > >> indicators. > >> However, getting it reviewed is a challenge because we want the build > >> system > >> integration to be right. When that lands, mach will be smart enough to > >> automatically define make flags optimal for your machine. e.g. -j == # of > >> cores. So, you get optimal/parallel builds with no configuration necessary. > >> This will all be configurable, of course. > > > > Is -j equal to number of cores really optimal? I've always been told > > that it's better to set it to more like twice the number of cores, > > because some fraction of threads will normally be stalled on I/O and > > you don't want cores idle. > > Depending on various factors, this can be offset by other factors like > cores competing for limited resources such as CPU cache. On my core i7 > with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More > generally while building a C++ project (C++ is computationally > expensive to compile) you're not very likely to be IO bound, > especially with a SSD. A C project might be different. On my core i7, -j12 is fastest. bigger values are marginally slower but they are all faster than -j8. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
2012/9/28 Aryeh Gregor : > On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: >> I actually held out on you with the initial landing of mach: there is more >> advanced tree building code in the pipes, complete with progress indicators. >> However, getting it reviewed is a challenge because we want the build system >> integration to be right. When that lands, mach will be smart enough to >> automatically define make flags optimal for your machine. e.g. -j == # of >> cores. So, you get optimal/parallel builds with no configuration necessary. >> This will all be configurable, of course. > > Is -j equal to number of cores really optimal? I've always been told > that it's better to set it to more like twice the number of cores, > because some fraction of threads will normally be stalled on I/O and > you don't want cores idle. Depending on various factors, this can be offset by other factors like cores competing for limited resources such as CPU cache. On my core i7 with 8 logical cores and 8M cache and a SSD, -j8 is fastest. More generally while building a C++ project (C++ is computationally expensive to compile) you're not very likely to be IO bound, especially with a SSD. A C project might be different. Benoit > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
On Thu, Sep 27, 2012 at 6:08 PM, Gregory Szorc wrote: > I actually held out on you with the initial landing of mach: there is more > advanced tree building code in the pipes, complete with progress indicators. > However, getting it reviewed is a challenge because we want the build system > integration to be right. When that lands, mach will be smart enough to > automatically define make flags optimal for your machine. e.g. -j == # of > cores. So, you get optimal/parallel builds with no configuration necessary. > This will all be configurable, of course. Is -j equal to number of cores really optimal? I've always been told that it's better to set it to more like twice the number of cores, because some fraction of threads will normally be stalled on I/O and you don't want cores idle. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
Aside from Windows being busted right now (https://bugzilla.mozilla.org/show_bug.cgi?id=794729), pymake "just works." If you are on Windows, mach runs pymake. If you are elsewhere, mach runs GNU make. I actually held out on you with the initial landing of mach: there is more advanced tree building code in the pipes, complete with progress indicators. However, getting it reviewed is a challenge because we want the build system integration to be right. When that lands, mach will be smart enough to automatically define make flags optimal for your machine. e.g. -j == # of cores. So, you get optimal/parallel builds with no configuration necessary. This will all be configurable, of course. On 9/27/2012 8:50 AM, Honza Bambas wrote: This really sounds good! How is mach working on windows with pymake? -hb- On 9/27/2012 3:12 AM, Gregory Szorc wrote: The next time you pull mozilla-central, you'll find a new tool in the root directory: mach $ ./mach build 0.09 /usr/bin/make -f client.mk -j8 -s 0.25 Adding client.mk options from /Users/gps/src/services-central/.mozconfig: 0.25 MOZ_OBJDIR=$(TOPSRCDIR)/objdir 0.25 MOZ_MAKE_FLAGS=-j9 -s 0.33 TEST-PASS | check-sync-dirs.py | /Users/gps/src/services-central/js/src/build <= /Users/gps/src/services-central/build 0.34 Generating /Users/gps/src/services-central/configure using autoconf 1.55 cd /Users/gps/src/services-central/objdir 1.55 /Users/gps/src/services-central/configure 1.57 Adding configure options from /Users/gps/src/services-central/.mozconfig: 1.57 --enable-application=browser 1.57 --enable-optimize 1.70 loading cache ./config.cache > 335.19 309 compiler warnings present. $ ./mach warnings-summary > 10-Wlogical-op-parentheses 10-Wtautological-compare 13-Wsometimes-uninitialized 26-Wconversion 30-Wunused-variable 34-Wunused-function 34-Wdeprecated-declarations 47-Wtautological-constant-out-of-range-compare 67-Wunused-private-field 309Total $ ./mach warnings-list > content/base/src/nsContentUtils.cpp:279:15 [-Wunused-private-field] private field 'mKey' is not used content/base/src/nsDocumentEncoder.cpp:2014:78 [-Wlogical-op-parentheses] '&&' within '||' content/xbl/src/nsBindingManager.cpp:433:1 [-Wdelete-non-virtual-dtor] delete called on 'nsBindingManager' that has virtual functions but non-virtual destructor content/xbl/src/nsXBLProtoImpl.cpp:22:21 [-Wunused-private-field] private field 'cx' is not used content/xbl/src/nsXBLProtoImpl.cpp:23:13 [-Wunused-private-field] private field 'versionBefore' is not used $ ./mach mochitest-browser browser/base/content/test/ > (Yes, you can tab complete test paths in your shell instead of going through some silly environment variable dance). If you get lost: $ ./mach help (Try doing that with make!) Read more about mach at [1]. I want to stress that mach is still in its infancy. It is currently only a fraction of what I want it to be. However, mach today is infinitely more than what it was yesterday: non-existent. That's progress. If you have a feature request or a work flow, please file a bug. Even better, contribute a patch! You only need to know beginner's Python to add new commands to mach. I hope you enjoy using mach. I look forward to many new and exciting features in the near future. [1] http://gregoryszorc.com/blog/2012/09/26/mach-has-landed/ Gregory ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: mach has landed
This really sounds good! How is mach working on windows with pymake? -hb- On 9/27/2012 3:12 AM, Gregory Szorc wrote: The next time you pull mozilla-central, you'll find a new tool in the root directory: mach $ ./mach build 0.09 /usr/bin/make -f client.mk -j8 -s 0.25 Adding client.mk options from /Users/gps/src/services-central/.mozconfig: 0.25 MOZ_OBJDIR=$(TOPSRCDIR)/objdir 0.25 MOZ_MAKE_FLAGS=-j9 -s 0.33 TEST-PASS | check-sync-dirs.py | /Users/gps/src/services-central/js/src/build <= /Users/gps/src/services-central/build 0.34 Generating /Users/gps/src/services-central/configure using autoconf 1.55 cd /Users/gps/src/services-central/objdir 1.55 /Users/gps/src/services-central/configure 1.57 Adding configure options from /Users/gps/src/services-central/.mozconfig: 1.57 --enable-application=browser 1.57 --enable-optimize 1.70 loading cache ./config.cache > 335.19 309 compiler warnings present. $ ./mach warnings-summary > 10-Wlogical-op-parentheses 10-Wtautological-compare 13-Wsometimes-uninitialized 26-Wconversion 30-Wunused-variable 34-Wunused-function 34-Wdeprecated-declarations 47-Wtautological-constant-out-of-range-compare 67-Wunused-private-field 309Total $ ./mach warnings-list > content/base/src/nsContentUtils.cpp:279:15 [-Wunused-private-field] private field 'mKey' is not used content/base/src/nsDocumentEncoder.cpp:2014:78 [-Wlogical-op-parentheses] '&&' within '||' content/xbl/src/nsBindingManager.cpp:433:1 [-Wdelete-non-virtual-dtor] delete called on 'nsBindingManager' that has virtual functions but non-virtual destructor content/xbl/src/nsXBLProtoImpl.cpp:22:21 [-Wunused-private-field] private field 'cx' is not used content/xbl/src/nsXBLProtoImpl.cpp:23:13 [-Wunused-private-field] private field 'versionBefore' is not used $ ./mach mochitest-browser browser/base/content/test/ > (Yes, you can tab complete test paths in your shell instead of going through some silly environment variable dance). If you get lost: $ ./mach help (Try doing that with make!) Read more about mach at [1]. I want to stress that mach is still in its infancy. It is currently only a fraction of what I want it to be. However, mach today is infinitely more than what it was yesterday: non-existent. That's progress. If you have a feature request or a work flow, please file a bug. Even better, contribute a patch! You only need to know beginner's Python to add new commands to mach. I hope you enjoy using mach. I look forward to many new and exciting features in the near future. [1] http://gregoryszorc.com/blog/2012/09/26/mach-has-landed/ Gregory ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
mach has landed
The next time you pull mozilla-central, you'll find a new tool in the root directory: mach $ ./mach build 0.09 /usr/bin/make -f client.mk -j8 -s 0.25 Adding client.mk options from /Users/gps/src/services-central/.mozconfig: 0.25 MOZ_OBJDIR=$(TOPSRCDIR)/objdir 0.25 MOZ_MAKE_FLAGS=-j9 -s 0.33 TEST-PASS | check-sync-dirs.py | /Users/gps/src/services-central/js/src/build <= /Users/gps/src/services-central/build 0.34 Generating /Users/gps/src/services-central/configure using autoconf 1.55 cd /Users/gps/src/services-central/objdir 1.55 /Users/gps/src/services-central/configure 1.57 Adding configure options from /Users/gps/src/services-central/.mozconfig: 1.57 --enable-application=browser 1.57 --enable-optimize 1.70 loading cache ./config.cache > 335.19 309 compiler warnings present. $ ./mach warnings-summary > 10 -Wlogical-op-parentheses 10 -Wtautological-compare 13 -Wsometimes-uninitialized 26 -Wconversion 30 -Wunused-variable 34 -Wunused-function 34 -Wdeprecated-declarations 47 -Wtautological-constant-out-of-range-compare 67 -Wunused-private-field 309 Total $ ./mach warnings-list > content/base/src/nsContentUtils.cpp:279:15 [-Wunused-private-field] private field 'mKey' is not used content/base/src/nsDocumentEncoder.cpp:2014:78 [-Wlogical-op-parentheses] '&&' within '||' content/xbl/src/nsBindingManager.cpp:433:1 [-Wdelete-non-virtual-dtor] delete called on 'nsBindingManager' that has virtual functions but non-virtual destructor content/xbl/src/nsXBLProtoImpl.cpp:22:21 [-Wunused-private-field] private field 'cx' is not used content/xbl/src/nsXBLProtoImpl.cpp:23:13 [-Wunused-private-field] private field 'versionBefore' is not used $ ./mach mochitest-browser browser/base/content/test/ > (Yes, you can tab complete test paths in your shell instead of going through some silly environment variable dance). If you get lost: $ ./mach help (Try doing that with make!) Read more about mach at [1]. I want to stress that mach is still in its infancy. It is currently only a fraction of what I want it to be. However, mach today is infinitely more than what it was yesterday: non-existent. That's progress. If you have a feature request or a work flow, please file a bug. Even better, contribute a patch! You only need to know beginner's Python to add new commands to mach. I hope you enjoy using mach. I look forward to many new and exciting features in the near future. [1] http://gregoryszorc.com/blog/2012/09/26/mach-has-landed/ Gregory ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
