Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-24 Thread Junio C Hamano
Stefan Beller sbel...@google.com writes: diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 3f10840..159ee36 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -11,6 +11,7 @@ #include exec_cmd.h #include streaming.h #include thread-utils.h +#include

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:44 PM, Jeff King p...@peff.net wrote: On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: The primary reason I suspect is because you sent to a wrong set of people. Submodule folks have largely been working in the scripted ones, and may not necessarily

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Junio C Hamano
Stefan Beller sbel...@google.com writes: +struct task_queue *create_task_queue(unsigned max_threads) +{ + struct task_queue *aq = xmalloc(sizeof(*aq)); + +#ifndef NO_PTHREADS + int i; + if (!max_threads) + aq-max_threads = online_cpus(); + else +

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Jeff King
On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: The primary reason I suspect is because you sent to a wrong set of people. Submodule folks have largely been working in the scripted ones, and may not necessarily be the ones who are most familiar with the run-command

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Junio C Hamano
Jeff King p...@peff.net writes: On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: The primary reason I suspect is because you sent to a wrong set of people. Submodule folks have largely been working in the scripted ones, and may not necessarily be the ones who are most

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Jeff King
On Fri, Aug 21, 2015 at 12:48:23PM -0700, Stefan Beller wrote: Before even looking at the implementation, my first question would be whether this pattern is applicable in several places in git (i.e., is it worth the extra complexity of abstracting out in the first place). I think there

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 1:47 PM, Junio C Hamano gits...@pobox.com wrote: I do not think we are on the same wavelength. What I meant was to do this: aq = xmalloc(...); set up _everything_ in aq and make it a consistent state; /* aq-first and aq-last are part of

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:05 PM, Junio C Hamano gits...@pobox.com wrote: Stefan Beller sbel...@google.com writes: This adds functionality to do work in parallel. The whole life cycle of such a thread pool would look like struct task_queue * tq = create_task_queue(32); // no of threads

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:51 PM, Jeff King p...@peff.net wrote: On Fri, Aug 21, 2015 at 12:48:23PM -0700, Stefan Beller wrote: Before even looking at the implementation, my first question would be whether this pattern is applicable in several places in git (i.e., is it worth the extra

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Junio C Hamano
Stefan Beller sbel...@google.com writes: This adds functionality to do work in parallel. The whole life cycle of such a thread pool would look like struct task_queue * tq = create_task_queue(32); // no of threads for (...) add_task(tq, process_one_item_function, item); //

Re: [RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-21 Thread Stefan Beller
On Fri, Aug 21, 2015 at 12:44 PM, Jeff King p...@peff.net wrote: On Fri, Aug 21, 2015 at 12:05:13PM -0700, Junio C Hamano wrote: The primary reason I suspect is because you sent to a wrong set of people. Submodule folks have largely been working in the scripted ones, and may not necessarily

[RFC PATCH 2/3] run-commands: add an async queue processor

2015-08-20 Thread Stefan Beller
This adds functionality to do work in parallel. The whole life cycle of such a thread pool would look like struct task_queue * tq = create_task_queue(32); // no of threads for (...) add_task(tq, process_one_item_function, item); // non blocking ... int ret =