Re: RFR: 8041151: More concurrent hgforest.
On Apr 22 2014, at 01:27 , Staffan Larsen staffan.lar...@oracle.com wrote: I ran the patch on OS X and it worked there too. Thanks! You mention “status code from subprocesses” which got me thinking of a problem I frequently run into when I have some uncommitted changes in one of the repos: ... .: searching for changes .: adding changesets .: adding manifests .: adding file changes .: added 1 changesets with 2 changes to 2 files (+1 heads) .: not updating: not a linear update .: (merge or update --check to force update) ... This output is frequently lost to me in all the other output, would it be possible to catch these errors and somehow highlight the failure at the end of the run? Unfortunately we don't buffer the output so there's no way at exit time to go back and gather the output. The good news is that the planned changes should be directly helpful for the case you are encountering. We/I hope to add rebase support to get_source.sh so that local changes whether unpushed changesets, mq patches or uncommitted changes will be rebased to the new tip whenever you pull. I guess I could go spelunking in the code to find out, but I’m too lazy and prefer to bother someone else. :) Not a problem. Mike Thanks, /Staffan On 19 apr 2014, at 01:21, Mike Duigou mike.dui...@oracle.com wrote: Hello all; This is an improvement to hgforest to increase it's concurrency behaviour. Currently hgforest.sh limits the rate at which it starts new sub-processes because it wants to limit the number of concurrent tasks. The naive approach it takes can cause unnecessary delays. For sequences of operations that are entirely local the overhead of waiting is significant. The revised implementation uses fifos for completion notification on capable platforms or compares started task count to completed task count in a shorter sleep loop. The intention is to use the enhanced concurrency to allow for a fancier get_source.sh that can handle rebasing for mq patches. This involves running a half dozen commands through hgforest. With my current in-development get_source.sh script changes these hgforest changes provide a 10X speedup. (4s vs 40s) The changeset also incorporates a build-dev suggested improvement to extra base repo url handling and other minor fixes (status code from subprocesses). JBSBUG: https://bugs.openjdk.java.net/browse/JDK-8041151 WEBREV: http://cr.openjdk.java.net/~mduigou/JDK-8041151/0/webrev/ I've so far tested it on successfully on Linux (Ubuntu 13.10) Solaris (10u9) and Cygwin x64 Cheers, Mike
Re: RFR: 8041151: More concurrent hgforest.
Seems to work for me too. Nice speedup! Looks good to me. /Erik On 2014-04-19 01:21, Mike Duigou wrote: Hello all; This is an improvement to hgforest to increase it's concurrency behaviour. Currently hgforest.sh limits the rate at which it starts new sub-processes because it wants to limit the number of concurrent tasks. The naive approach it takes can cause unnecessary delays. For sequences of operations that are entirely local the overhead of waiting is significant. The revised implementation uses fifos for completion notification on capable platforms or compares started task count to completed task count in a shorter sleep loop. The intention is to use the enhanced concurrency to allow for a fancier get_source.sh that can handle rebasing for mq patches. This involves running a half dozen commands through hgforest. With my current in-development get_source.sh script changes these hgforest changes provide a 10X speedup. (4s vs 40s) The changeset also incorporates a build-dev suggested improvement to extra base repo url handling and other minor fixes (status code from subprocesses). JBSBUG: https://bugs.openjdk.java.net/browse/JDK-8041151 WEBREV: http://cr.openjdk.java.net/~mduigou/JDK-8041151/0/webrev/ I've so far tested it on successfully on Linux (Ubuntu 13.10) Solaris (10u9) and Cygwin x64 Cheers, Mike
Re: RFR: 8041151: More concurrent hgforest.
I ran the patch on OS X and it worked there too. Thanks! You mention “status code from subprocesses” which got me thinking of a problem I frequently run into when I have some uncommitted changes in one of the repos: ... .: searching for changes .: adding changesets .: adding manifests .: adding file changes .: added 1 changesets with 2 changes to 2 files (+1 heads) .: not updating: not a linear update .: (merge or update --check to force update) ... This output is frequently lost to me in all the other output, would it be possible to catch these errors and somehow highlight the failure at the end of the run? I guess I could go spelunking in the code to find out, but I’m too lazy and prefer to bother someone else. :) Thanks, /Staffan On 19 apr 2014, at 01:21, Mike Duigou mike.dui...@oracle.com wrote: Hello all; This is an improvement to hgforest to increase it's concurrency behaviour. Currently hgforest.sh limits the rate at which it starts new sub-processes because it wants to limit the number of concurrent tasks. The naive approach it takes can cause unnecessary delays. For sequences of operations that are entirely local the overhead of waiting is significant. The revised implementation uses fifos for completion notification on capable platforms or compares started task count to completed task count in a shorter sleep loop. The intention is to use the enhanced concurrency to allow for a fancier get_source.sh that can handle rebasing for mq patches. This involves running a half dozen commands through hgforest. With my current in-development get_source.sh script changes these hgforest changes provide a 10X speedup. (4s vs 40s) The changeset also incorporates a build-dev suggested improvement to extra base repo url handling and other minor fixes (status code from subprocesses). JBSBUG: https://bugs.openjdk.java.net/browse/JDK-8041151 WEBREV: http://cr.openjdk.java.net/~mduigou/JDK-8041151/0/webrev/ I've so far tested it on successfully on Linux (Ubuntu 13.10) Solaris (10u9) and Cygwin x64 Cheers, Mike
Re: RFR: 8041151: More concurrent hgforest.
On 22 Apr 2014, at 09:10, Erik Joelsson erik.joels...@oracle.com wrote: Seems to work for me too. Nice speedup! Looks good to me. +1. Thanks for doing these improvements Mike. -Chris. /Erik On 2014-04-19 01:21, Mike Duigou wrote: Hello all; This is an improvement to hgforest to increase it's concurrency behaviour. Currently hgforest.sh limits the rate at which it starts new sub-processes because it wants to limit the number of concurrent tasks. The naive approach it takes can cause unnecessary delays. For sequences of operations that are entirely local the overhead of waiting is significant. The revised implementation uses fifos for completion notification on capable platforms or compares started task count to completed task count in a shorter sleep loop. The intention is to use the enhanced concurrency to allow for a fancier get_source.sh that can handle rebasing for mq patches. This involves running a half dozen commands through hgforest. With my current in-development get_source.sh script changes these hgforest changes provide a 10X speedup. (4s vs 40s) The changeset also incorporates a build-dev suggested improvement to extra base repo url handling and other minor fixes (status code from subprocesses). JBSBUG: https://bugs.openjdk.java.net/browse/JDK-8041151 WEBREV: http://cr.openjdk.java.net/~mduigou/JDK-8041151/0/webrev/ I've so far tested it on successfully on Linux (Ubuntu 13.10) Solaris (10u9) and Cygwin x64 Cheers, Mike
