Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 1:26 PM, Paul Smith wrote: > On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote: >> As parallel make are becoming more and more popular, can make >> serialize the output of parallel make? >> >> Make can redirect every parallelly issued shell's output to an >> temporary file, and output the stored output serially, as if in a >> serial make. > > This would be a good thing, but as always the details are not quite so > trivial. > > We have to ensure that these temporary files are cleaned up properly, > even in the face of users ^C'ing their make invocations. We also need > to verify that whatever methods we use will work properly on Windows and > VMS and other operating systems make supports (where are their "/tmp" > equivalents?) > My suggestion is that you can implement it as an optional command line option(like -j), and on one or two primary platforms(Linux/Windows), instead of on all platforms. > And, what about stdout vs. stderr? Should we write both to the same > file? Then we lose the ability to do things like "make -j4 2>/dev/null" > since all output will be written to stdout (presumably). Or should we > keep two files per command, one for stdout and one for stderr? But > that's even worse since then when we printed it we'd have to print all > the stdout first then all the stderr, which could lose important > context. > The scenario like "make -j4 2>/dev/null" may be very rare, but scenario like "make -j4 2>&1 | tee output.txt" may be common. I think using make to parallelly build or test large software on multicore system is by far the most normal situation. User of make most need is : 1. utilize the computing power of multicore system using parallel make(-j) ; 2. easily analyze the output of the build or test, to precisely find the problem. When user of make provide the optional output-serializing command line option, he known what he need, he is responsible for the mess situation. > Then there's the possibility that some commands will behave differently > if they are writing to a TTY, then they will if they're writing to a > file. Do we not care about that, or do we try to do something crazy > with PTYs or similar (ugh!) > > And of course we have to have a guaranteed unique naming strategy in > case multiple instances of make are running on the same system at the > same time, maybe running the same makefiles and even building the same > targets. On POSIX systems we can use tmpfile() or mkstemp() or > something but other systems might need other measures. > > These are just some things I thought of off the top of my head. > > It certainly does not mean that it would not be a good thing to have > this ability though. > > -- > --- > Paul D. Smith Find some GNU make tips at: > http://www.gnu.org http://make.mad-scientist.net > "Please remain calm...I may be mad, but I am a professional." --Mad Scientist > > -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
Paul Smith wrote: On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote: As parallel make are becoming more and more popular, can make serialize the output of parallel make? Make can redirect every parallelly issued shell's output to an temporary file, and output the stored output serially, as if in a serial make. This would be a good thing, but as always the details are not quite so trivial. Aside from the difficulties outlined below, I just am not fond of having output batched up instead of appearing in realtime. It tends to complicate the polling logic too (though I guess in this case, you just have to cat the appropriate file[s] whenever a child process ends.) The scheme that I grew up with on Alliant Concentrix was just to prefix each output line with its job number "|xx|blah blah blah". It obviously requires a pipe for each child process' output, so that lines can be read by the parent make and then the prefix attached. In the original jobserver prototype I used unique bytes for each job token so that the token == the job ID, with an eye toward adding this support later. But that obviously limited it to supporting only 256 concurrent jobs, and these days we already get complaints that it's limited to only 4096. Using a pipe per job would likewise cut make's maximum job count in half (or worse, if using a separate stderr pipe). I still favor this latter approach because it keeps the output flowing in realtime, and its easy enough to use grep if you want to zero in on a single output stream. But the cost in resources will add up... We have to ensure that these temporary files are cleaned up properly, even in the face of users ^C'ing their make invocations. We also need to verify that whatever methods we use will work properly on Windows and VMS and other operating systems make supports (where are their "/tmp" equivalents?) And, what about stdout vs. stderr? Should we write both to the same file? Then we lose the ability to do things like "make -j4 2>/dev/null" since all output will be written to stdout (presumably). Or should we keep two files per command, one for stdout and one for stderr? But that's even worse since then when we printed it we'd have to print all the stdout first then all the stderr, which could lose important context. Then there's the possibility that some commands will behave differently if they are writing to a TTY, then they will if they're writing to a file. Do we not care about that, or do we try to do something crazy with PTYs or similar (ugh!) And of course we have to have a guaranteed unique naming strategy in case multiple instances of make are running on the same system at the same time, maybe running the same makefiles and even building the same targets. On POSIX systems we can use tmpfile() or mkstemp() or something but other systems might need other measures. These are just some things I thought of off the top of my head. It certainly does not mean that it would not be a good thing to have this ability though. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote: > As parallel make are becoming more and more popular, can make > serialize the output of parallel make? > > Make can redirect every parallelly issued shell's output to an > temporary file, and output the stored output serially, as if in a > serial make. This would be a good thing, but as always the details are not quite so trivial. We have to ensure that these temporary files are cleaned up properly, even in the face of users ^C'ing their make invocations. We also need to verify that whatever methods we use will work properly on Windows and VMS and other operating systems make supports (where are their "/tmp" equivalents?) And, what about stdout vs. stderr? Should we write both to the same file? Then we lose the ability to do things like "make -j4 2>/dev/null" since all output will be written to stdout (presumably). Or should we keep two files per command, one for stdout and one for stderr? But that's even worse since then when we printed it we'd have to print all the stdout first then all the stderr, which could lose important context. Then there's the possibility that some commands will behave differently if they are writing to a TTY, then they will if they're writing to a file. Do we not care about that, or do we try to do something crazy with PTYs or similar (ugh!) And of course we have to have a guaranteed unique naming strategy in case multiple instances of make are running on the same system at the same time, maybe running the same makefiles and even building the same targets. On POSIX systems we can use tmpfile() or mkstemp() or something but other systems might need other measures. These are just some things I thought of off the top of my head. It certainly does not mean that it would not be a good thing to have this ability though. -- --- Paul D. Smith Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net "Please remain calm...I may be mad, but I am a professional." --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: insufficient debug info from gnu-make
On Thu, 2010-07-29 at 21:09 -0700, Peter Lawrence wrote: > make[3]: *** No rule to make target `real-install-headers-tar'. Stop. This is printed when you've invoked make and the target you asked for on the command line cannot be created (for example you ran "make foo" but the makefile has no target "foo" defined). There's no line number or filename information to provide, here, because the request came from the command line, not from inside the makefile. > here is where the offending command originates: > > $(MAKE) real-$(INSTALL_HEADERS_DIR) DESTDIR=`pwd`/../gcc/ \ > libsubdir=. ; \ So the sub-make is the one that generated the error message, and as I mentioned it can't give you information about what line of the parent make's makefile was the problem (just like your C compiler has no way to tell you what line of the makefile it was invoked from). All that could be done in this case is that the parent make could be asked to show filename/linenumber information for the target that failed (that is, the error after the error you quote above). I'm certainly not asserting that GNU make couldn't do a better job giving error messages with more details regarding where the error happened, I'm just saying that this particular case is not quite so clear-cut as it may appear at first. -- --- Paul D. Smith Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net "Please remain calm...I may be mad, but I am a professional." --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
insufficient debug info from gnu-make
Sirs, in trying to debug a failing gcc build, it seems that the real trouble is that there doesn't seem to be any easy way to figure out what directory/file[line number] a make command comes from, for example I see as output: make[3]: *** No rule to make target `real-install-headers-tar'. Stop. and I cannot even tell what makefile is being processed in what directory, farther up in the captured output of make I see: /bin/sh ../../gcc-4.3.3/gcc/../move-if-change tmp-fixinc_list fixinc_list echo timestamp > s-fixinc_list rm -rf include-fixed; mkdir include-fixed chmod a+rx include-fixed and I grep for fixinc_list in */Makefile to find it is in gcc/ Makefile, OK, but I should not have to resort to such antics, make could be outputting this directory/file information as a normal operation. here is where the offending command originates: $(MAKE) real-$(INSTALL_HEADERS_DIR) DESTDIR=`pwd`/../gcc/ \ libsubdir=. ; \ as you can see it is virtually impossible for a reader that is not already intimately familiar with these specific makefiles to be able to read the "*** No rule to make target `real-install-headers-tar`. Stop." message and find the corresponding line "$(MAKE) real-$ (INSTALL_HEADERS_DIR) ..." all the macro usage makes it impossible to search for corresponding strings, the strings don't correspond unless you know all the macros by heart and do your own reverse- substitution. so the only recourse is directory/file[line] information about where the command came from... I seem to remember, perhaps it was Sun's make, that did output directory/file[line] information, and I am now feeling a strong longing to be back in that environment, we should be doing the same with gnu's make. -Peter Lawrence. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[RFC]serialize the output of parallel make?
As parallel make are becoming more and more popular, can make serialize the output of parallel make? Make can redirect every parallelly issued shell's output to an temporary file, and output the stored output serially, as if in a serial make. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
