Re: Fwd: [RFC]serialize the output of parallel make?
On Wed, Aug 4, 2010 at 10:41 AM, Eric Melski wrote: > ElectricAccelerator is a gmake replacement that does exactly this. I wrote > about this feature a while back: > > http://blog.electric-cloud.com/2008/12/01/untangling-parallel-build-logs/ > > You can read more about Accelerator on the blog, or here: > > http://www.electric-cloud.com/products/electricaccelerator.php > > Eric Melski > Architect > Electric Cloud, Inc. > http://blog.electric-cloud.com/ > Excellent ! -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
On Tue, Aug 3, 2010 at 2:51 PM, Tim Murphy wrote: > > Since some things happen at the same time there is no single "serial > order". The semaphore mechanism, forces one of the possible orders. > I'm not familiar with source code of make, but I believe the "serial order" of shells is determined by the dependence DAG, it may be unique for a given dependence DAG. Shells can be issued and completed at random order(only need satisfying the dependence relation). But make can print their outputs strictly in their "serial order". -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
On Mon, Aug 2, 2010 at 4:22 PM, Edward Welbourne wrote: >> If my guess is not wrong, the semaphore safeguard the consistency of >> output of one command, not the order of commands. > > well, with -j, commands are being run concurrently, so there *isn't* a > strict ordering of commands to "safeguard", although output shall be > delivered in roughly the order of completion of commands, with only > minor disturbances. > > Still, if target A is a prerequisite of B, the recipe to make A is > run, and must complete, before the recipe to make B will be initiated; > since the recipe for A ends with whatever is ensuring its output comes > out as an atom, A's output is produced before B's recipe is initiated, > so you can be sure they appear in the right order. So the only > ordering property among commands that actually matters *is* preserved. > This is not my ideal solution. My idea is to preserve the order of output of parallel make as if it is a "serial make". Modern CPU can issue multiple instructions simultaneously, but preserve the order of commit to program order. So the instruction level parallelism of CPU is transparent to programmer. What I want is transparent "parallel make". Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Fwd: [RFC]serialize the output of parallel make?
-- Forwarded message -- From: Chiheng Xu Date: Fri, Jul 30, 2010 at 6:02 PM Subject: Re: [RFC]serialize the output of parallel make? To: Tim Murphy On Fri, Jul 30, 2010 at 5:54 PM, Tim Murphy wrote: > Hi, > > The cost to the system of even starting a shell to invoke a recipe is > so huge compared to the time needed to reserve a semaphore that it is > insignificant in comparison. > > The amount of contention is limited by -j i.e. by how many processes > there are ( 2 * CPUs is usually considered reasonable) and by how long > the lock is held for which is basically about how long GNU make takes > to read the output from the process that currently has the lock. > Since modern computers have <1000s of CPUs the degree of contention is > not high and most of the cost of contention is something you pay for > no matter what method you use to descramble stuff. > > Our experience indicates that it performs very well. > If my guess is not wrong, the semaphore safeguard the consistency of output of one command, not the order of commands. -- Chiheng Xu Wuhan,China -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 6:01 PM, Howard Chu wrote: > Chiheng Xu wrote: >> >> On Fri, Jul 30, 2010 at 5:35 PM, Eli Zaretskii wrote: >>> >>> I asked for an example. Could you please show a "messy" output and >>> the output you'd like to have after "serialization"? >>> >>> TIA >>> >> >> serially make : execute A, B, C programs, they print: >> >> A: Hello, I'm A, I am from Earth. >> B: The moon is my home. >> C: Welcome to Mars, It's an amazing planet. >> >> parallely make : the output of A, B, C programs interleave : >> >> C: Welcome to >> B: The moon is my >> A: Hello, I'm A, I am from Earth.home.Mars, It's an amazing planet. > > This seems like quite an extreme example. stdout is line buffered by > default, so individual lines would get written atomically unless the > programs you're running are doing weird things with their output. In the > common case interleaving like this doesn't happen within lines, it only > happens between lines of multi-line output. stderr may skew things since > it's usually nonbuffered, but again, that's not the common case. > I use "make -j 4" to build and test gcc, the situation above is very common. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 5:35 PM, Eli Zaretskii wrote: > > I asked for an example. Could you please show a "messy" output and > the output you'd like to have after "serialization"? > > TIA > serially make : execute A, B, C programs, they print: A: Hello, I'm A, I am from Earth. B: The moon is my home. C: Welcome to Mars, It's an amazing planet. parallely make : the output of A, B, C programs interleave : C: Welcome to B: The moon is my A: Hello, I'm A, I am from Earth.home.Mars, It's an amazing planet. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 5:31 PM, Chiheng Xu wrote: > On Fri, Jul 30, 2010 at 5:27 PM, Tim Murphy wrote: > >> No, I understand better than you think as I do huge parallel builds every >> day. >> >> The shell wrapper buffers the recipe output and then grabs a semaphore >> before writing the output to it's stdout.. if another recipe has >> completed and is in the process of outputting to the stdout then it >> has to wait a few microseconds. > > The use of semaphore may impair performance. > And "serialzation" you mean is not the same as I mean. I believe Paul and Edward fully understand what I mean. As English is not my mother tongue, I don't known how to describe the concept of "serialization of output" more clearly. Sorry. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 5:27 PM, Tim Murphy wrote: > No, I understand better than you think as I do huge parallel builds every day. > > The shell wrapper buffers the recipe output and then grabs a semaphore > before writing the output to it's stdout.. if another recipe has > completed and is in the process of outputting to the stdout then it > has to wait a few microseconds. The use of semaphore may impair performance. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 4:53 PM, Eli Zaretskii wrote: > This is not to say that I think the original idea is easy to > implement. > > Actually, it is not entirely clear to me what is meant by > "serialization" in this context. Commands are invoked by Make in > parallel, and the output is produced in the order of their execution > (modulo buffering issues). What would serialization look like in this > context? Can someone show a simple example? > Parallelly invoked shells(and commands the shells invoke) may print to output randomly, render the output messy. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 4:40 PM, Tim Murphy wrote: > Hi, > > > You set talon as the shell for make and talon in turn runs whatever > the actual shell is but adds the serialisation. there are a lot of > other nice things you can do with this - e.g. measuring the execution > time of every build step so that you can see what affects performance. Sorry, I don't understand this. You probably misunderstand the meaning of "serialization of output" and "parallel make". -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 3:45 PM, Philip Guenther wrote: > On Thu, Jul 29, 2010 at 11:29 PM, Chiheng Xu wrote: > ... >> My suggestion is that you can implement it as an optional command line >> option(like -j), and on one or two primary platforms(Linux/Windows), >> instead of on all platforms. > > So, the complexity of both possibilities. You're writing, debugging, > and contributing this code yourself? > Nop, I'm not maintainer of make, just a user :) . > (I would hope that this wouldn't require any Linux-specific code; > perhaps you meant "POSIX & Windows"?) > Yes. > > ... >> The scenario like "make -j4 2>/dev/null" may be very rare, but >> scenario like "make -j4 2>&1 | tee output.txt" may be common. > > And what, exactly, are you suggesting that make do to reflect that > guess about usage patterns? > > > Philip Guenther > I mean, normal user of make does not differentiate stdout or stderr very seriously, they see them both as "output". They want serialized "output", whether or not it's stdout or stderr. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: [RFC]serialize the output of parallel make?
On Fri, Jul 30, 2010 at 1:26 PM, Paul Smith wrote: > On Fri, 2010-07-30 at 09:59 +0800, Chiheng Xu wrote: >> As parallel make are becoming more and more popular, can make >> serialize the output of parallel make? >> >> Make can redirect every parallelly issued shell's output to an >> temporary file, and output the stored output serially, as if in a >> serial make. > > This would be a good thing, but as always the details are not quite so > trivial. > > We have to ensure that these temporary files are cleaned up properly, > even in the face of users ^C'ing their make invocations. We also need > to verify that whatever methods we use will work properly on Windows and > VMS and other operating systems make supports (where are their "/tmp" > equivalents?) > My suggestion is that you can implement it as an optional command line option(like -j), and on one or two primary platforms(Linux/Windows), instead of on all platforms. > And, what about stdout vs. stderr? Should we write both to the same > file? Then we lose the ability to do things like "make -j4 2>/dev/null" > since all output will be written to stdout (presumably). Or should we > keep two files per command, one for stdout and one for stderr? But > that's even worse since then when we printed it we'd have to print all > the stdout first then all the stderr, which could lose important > context. > The scenario like "make -j4 2>/dev/null" may be very rare, but scenario like "make -j4 2>&1 | tee output.txt" may be common. I think using make to parallelly build or test large software on multicore system is by far the most normal situation. User of make most need is : 1. utilize the computing power of multicore system using parallel make(-j) ; 2. easily analyze the output of the build or test, to precisely find the problem. When user of make provide the optional output-serializing command line option, he known what he need, he is responsible for the mess situation. > Then there's the possibility that some commands will behave differently > if they are writing to a TTY, then they will if they're writing to a > file. Do we not care about that, or do we try to do something crazy > with PTYs or similar (ugh!) > > And of course we have to have a guaranteed unique naming strategy in > case multiple instances of make are running on the same system at the > same time, maybe running the same makefiles and even building the same > targets. On POSIX systems we can use tmpfile() or mkstemp() or > something but other systems might need other measures. > > These are just some things I thought of off the top of my head. > > It certainly does not mean that it would not be a good thing to have > this ability though. > > -- > ------- > Paul D. Smith Find some GNU make tips at: > http://www.gnu.org http://make.mad-scientist.net > "Please remain calm...I may be mad, but I am a professional." --Mad Scientist > > -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[RFC]serialize the output of parallel make?
As parallel make are becoming more and more popular, can make serialize the output of parallel make? Make can redirect every parallelly issued shell's output to an temporary file, and output the stored output serially, as if in a serial make. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
