Re: Fwd: [RFC]serialize the output of parallel make?
On Wed, Aug 4, 2010 at 10:41 AM, Eric Melski wrote: > ElectricAccelerator is a gmake replacement that does exactly this. I wrote > about this feature a while back: > > http://blog.electric-cloud.com/2008/12/01/untangling-parallel-build-logs/ > > You can read more about Accelerator on the blog, or here: > > http://www.electric-cloud.com/products/electricaccelerator.php > > Eric Melski > Architect > Electric Cloud, Inc. > http://blog.electric-cloud.com/ > Excellent ! -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: What I want is transparent "parallel make". Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. ElectricAccelerator is a gmake replacement that does exactly this. I wrote about this feature a while back: http://blog.electric-cloud.com/2008/12/01/untangling-parallel-build-logs/ You can read more about Accelerator on the blog, or here: http://www.electric-cloud.com/products/electricaccelerator.php Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: What I want is transparent "parallel make". Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. ElectricAccelerator is a gmake replacement that does exactly this. I wrote about this feature a while back: http://blog.electric-cloud.com/2008/12/01/untangling-parallel-build-logs/ You can read more about Accelerator on the blog, or here: http://www.electric-cloud.com/products/electricaccelerator.php Eric Melski Architect Electric Cloud, Inc. http://blog.electric-cloud.com/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
> Date: Tue, 3 Aug 2010 07:51:22 +0100 > From: Tim Murphy > Cc: e...@opera.com, bug-make@gnu.org > > mytarget: > ->command1 && > ->command2 && > ->command3 > > > Note that I'm using bash syntax here. On windows if you want to use > cmd.exe then good luck - I don't think it's really fit for purpose. cmd.exe supports the same `command1 && command2' semantics as does Bash, so there's no problem here and no need for any ``luck''. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: On Tue, Aug 3, 2010 at 2:51 PM, Tim Murphy wrote: Since some things happen at the same time there is no single "serial order". The semaphore mechanism, forces one of the possible orders. I'm not familiar with source code of make, but I believe the "serial order" of shells is determined by the dependence DAG, it may be unique for a given dependence DAG. Shells can be issued and completed at random order(only need satisfying the dependence relation). But make can print their outputs strictly in their "serial order". I'm trying very hard to only provide constructive comments in response to this thread, but frankly this is, in a word, stupid. If you want make's output to be in serial order, then don't use parallel make at all. The point to parallel make is that it allows jobs which have no ordering dependency to run in parallel. If you want their output to be fully serialized, then you will force make to wait for them to complete serially. Which automatically also means that make will have to maintain an arbitrarily large internal queue for all of the output, because given the unpredictable completion times of multiple jobs running concurrently, no output can be emitted until the slowest parallel job completes. In particular, if you have recursive makefiles, no parent make process can output anything at all until all of its submakes have completed, because no individual make process has enough knowledge about what the actual serial order is. Given that this discussion seems to have arisen due to the braindead stdio handling in Cygwin, it seems like any de-mangling of parallel make's output should be directed to the Cygwin libraries. In my experience Cygwin is too slow an environment to be useful anyway, which is why I use MSYS for Windows builds. But I have to admit, I only use it inside a single-core VirtualBox these days so I haven't looked at how parallel make behaves there. But the fact is all I/O in Cygwin is funneled through the Cygwin DLL, so there's no reason that it can't be fixed to not mingle/mangle lines from different processes together. But again, that's not gnu-make's problem, that's a Cygwin issue. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
On Tue, Aug 3, 2010 at 2:51 PM, Tim Murphy wrote: > > Since some things happen at the same time there is no single "serial > order". The semaphore mechanism, forces one of the possible orders. > I'm not familiar with source code of make, but I believe the "serial order" of shells is determined by the dependence DAG, it may be unique for a given dependence DAG. Shells can be issued and completed at random order(only need satisfying the dependence relation). But make can print their outputs strictly in their "serial order". -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Hi, Since some things happen at the same time there is no single "serial order". The semaphore mechanism, forces one of the possible orders. I forgot to say that for recipes with multiple commands you need to either use the new .ONESHELL target or do this kind of thing: mytarget:mytarget: ->command1 && \ ->command2 && \ ->command3 This causes them to be executed in a single shell invocation for which the output can be gathered together (I am using -> to indicate TAB) With .ONESHELL, as I understand it, you would not need the '\' characters to escape the end-of-line: mytarget: ->command1 && ->command2 && ->command3 Note that I'm using bash syntax here. On windows if you want to use cmd.exe then good luck - I don't think it's really fit for purpose. Regards, Tim On 3 August 2010 02:11, Chiheng Xu wrote: > On Mon, Aug 2, 2010 at 4:22 PM, Edward Welbourne wrote: >>> If my guess is not wrong, the semaphore safeguard the consistency of >>> output of one command, not the order of commands. >> >> well, with -j, commands are being run concurrently, so there *isn't* a >> strict ordering of commands to "safeguard", although output shall be >> delivered in roughly the order of completion of commands, with only >> minor disturbances. >> >> Still, if target A is a prerequisite of B, the recipe to make A is >> run, and must complete, before the recipe to make B will be initiated; >> since the recipe for A ends with whatever is ensuring its output comes >> out as an atom, A's output is produced before B's recipe is initiated, >> so you can be sure they appear in the right order. So the only >> ordering property among commands that actually matters *is* preserved. >> > > This is not my ideal solution. > > My idea is to preserve the order of output of parallel make as if it > is a "serial make". > > Modern CPU can issue multiple instructions simultaneously, but > preserve the order of commit to program order. So the instruction > level parallelism of CPU is transparent to programmer. > > What I want is transparent "parallel make". Make can issue multiple > shells simultaneously, but print their outputs in the same order as in > a serial make. > > > > > -- > Chiheng Xu > Wuhan,China > > ___ > Bug-make mailing list > Bug-make@gnu.org > http://lists.gnu.org/mailman/listinfo/bug-make > -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
On Mon, Aug 2, 2010 at 4:22 PM, Edward Welbourne wrote: >> If my guess is not wrong, the semaphore safeguard the consistency of >> output of one command, not the order of commands. > > well, with -j, commands are being run concurrently, so there *isn't* a > strict ordering of commands to "safeguard", although output shall be > delivered in roughly the order of completion of commands, with only > minor disturbances. > > Still, if target A is a prerequisite of B, the recipe to make A is > run, and must complete, before the recipe to make B will be initiated; > since the recipe for A ends with whatever is ensuring its output comes > out as an atom, A's output is produced before B's recipe is initiated, > so you can be sure they appear in the right order. So the only > ordering property among commands that actually matters *is* preserved. > This is not my ideal solution. My idea is to preserve the order of output of parallel make as if it is a "serial make". Modern CPU can issue multiple instructions simultaneously, but preserve the order of commit to program order. So the instruction level parallelism of CPU is transparent to programmer. What I want is transparent "parallel make". Make can issue multiple shells simultaneously, but print their outputs in the same order as in a serial make. -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
> 2x is too much. 1.5x has been the best in my experience, any more than that > and you're losing too much CPU to scheduling overhead instead of real work. > Any less and you're giving up too much in idle or I/O time. This depends a bit on whether you're using icecc or some similar distributed compilation system. I believe a better approach is to set a generous -j, such as twice the count of CPUs, but impose a load limit using -l, tuned rather more carefully. Scheduling overhead contributes to load, so is taken into account this way. Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
> If my guess is not wrong, the semaphore safeguard the consistency of > output of one command, not the order of commands. well, with -j, commands are being run concurrently, so there *isn't* a strict ordering of commands to "safeguard", although output shall be delivered in roughly the order of completion of commands, with only minor disturbances. Still, if target A is a prerequisite of B, the recipe to make A is run, and must complete, before the recipe to make B will be initiated; since the recipe for A ends with whatever is ensuring its output comes out as an atom, A's output is produced before B's recipe is initiated, so you can be sure they appear in the right order. So the only ordering property among commands that actually matters *is* preserved. Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Edward Welbourne wrote: 2x is too much. 1.5x has been the best in my experience, any more than that and you're losing too much CPU to scheduling overhead instead of real work. Any less and you're giving up too much in idle or I/O time. This depends a bit on whether you're using icecc or some similar distributed compilation system. I believe a better approach is to set a generous -j, such as twice the count of CPUs, but impose a load limit using -l, tuned rather more carefully. Scheduling overhead contributes to load, so is taken into account this way. Perhaps in a perfect world -l would be useful. In fact, since load averages are calculated so slowly, by the time your -l limit is reached the actual CPU load will have blown past it and your machine will be thrashing. That's the entire reason I came up with the -j implementation in the first place. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Fwd: [RFC]serialize the output of parallel make?
Chiheng Xu wrote: -- Forwarded message -- From: Chiheng Xu Date: Fri, Jul 30, 2010 at 6:02 PM Subject: Re: [RFC]serialize the output of parallel make? To: Tim Murphy On Fri, Jul 30, 2010 at 5:54 PM, Tim Murphy wrote: Hi, The cost to the system of even starting a shell to invoke a recipe is so huge compared to the time needed to reserve a semaphore that it is insignificant in comparison. The amount of contention is limited by -j i.e. by how many processes there are ( 2 * CPUs is usually considered reasonable) and by how long 2x is too much. 1.5x has been the best in my experience, any more than that and you're losing too much CPU to scheduling overhead instead of real work. Any less and you're giving up too much in idle or I/O time. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Fwd: [RFC]serialize the output of parallel make?
-- Forwarded message -- From: Chiheng Xu Date: Fri, Jul 30, 2010 at 6:02 PM Subject: Re: [RFC]serialize the output of parallel make? To: Tim Murphy On Fri, Jul 30, 2010 at 5:54 PM, Tim Murphy wrote: > Hi, > > The cost to the system of even starting a shell to invoke a recipe is > so huge compared to the time needed to reserve a semaphore that it is > insignificant in comparison. > > The amount of contention is limited by -j i.e. by how many processes > there are ( 2 * CPUs is usually considered reasonable) and by how long > the lock is held for which is basically about how long GNU make takes > to read the output from the process that currently has the lock. > Since modern computers have <1000s of CPUs the degree of contention is > not high and most of the cost of contention is something you pay for > no matter what method you use to descramble stuff. > > Our experience indicates that it performs very well. > If my guess is not wrong, the semaphore safeguard the consistency of output of one command, not the order of commands. -- Chiheng Xu Wuhan,China -- Chiheng Xu Wuhan,China ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
