Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[Bakul Shah]

Here's a slightly different way of looking at this:

It is known that if you have N servers, a single queue is better than
N separate queues, so as to avoid the situation where you have an
idle server with an empty queue and a waiting customer in a queue
for a busy server. So the only other task is how to rearrange the outputs
in the same order. An interface as follows may help:

type Any2SerialQ interface { // Any order in, serial order out
Put(n uint, item interface{})
Get() interface{}
}

Where Put() can be in any order (as dictated by n) but Get() is always
in sequence.

The peak buffer use depends on the laggard. If item N is taking a long
time and further K items have completed, their results must be held
until N finishes.

Note that the problem is somewhat analogous to reconstituting a TCP
stream when packets arrive out of order.

> On Jan 31, 2019, at 4:06 AM, roger peppe  wrote:
> 
> On Thu, 31 Jan 2019 at 00:06, Michael Jones  > wrote:
> note that my code sidesteps this problem generally
> 
> I'm not sure that that's true. Although your code mitigates the problem 
> somewhat,
> it's still possible for a one slow worker to block the others. You've added 
> 512*NumCPU/2
> buffer slots, but in general it's not possible to order results and provide 
> avoid unnecessary
> blocking without having N buffer slots. In your code, assume 4 CPUs and that 
> work items 0
> and 2 take 1s and all other work items take 1ms. If we've got 5000 items in 
> total,
> the total time taken will be 2.002498s instead of the ideal time (~1s+2499µs).
> 
> https://play.golang.org/p/5Ty6pgpmZ0w 
> 
> Here's a kind of hybrid approach. It still serializes, but it makes as good a 
> use of the buffer
> space as it can - it won't block until a slow item is at least bufSize items 
> behind the
> most recently processed item:
> 
> https://play.golang.org/p/PP9NSJuLeEK 
>  
> On Wed, Jan 30, 2019 at 3:48 PM roger peppe  > wrote:
> On Wed, 30 Jan 2019 at 20:07, 'Bryan Mills' via golang-nuts 
> mailto:golang-nuts@googlegroups.com>> wrote:
> The code to sequence the results using a channel is not much more verbose.
> 
> That not only avoids the library dependency, but also makes the peak memory 
> consumption for the results O(runtime.NumCPU()), instead of O(N) with the 
> number of tasks, and allows the output to be streamed instead of buffered to 
> a slice.
> 
> https://play.golang.org/p/zkBjxlcvESe 
> 
> Nice! In practice though, I've usually found that I do want to keep the 
> results around or I don't care about the order at all, so the parallel 
> package works OK.
> 
> I'd point out one down side to the sequencing approach - one very slow work 
> item can block the others. For example, NProc is 4, the first item takes 200 
> milliseconds to process and all the others take 1 millisecond, then the first 
> 4 workers have started, none more will be started until the first has, so the 
> overall time will be quite a bit longer (249ms) than if they were all allowed 
> to proceed irrespective of order (200ms).
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 
> 
> -- 
> Michael T. Jones
> michael.jo...@gmail.com 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[robert engels]

I think that level of imbalance is the design thought behind “work stealing 
queues” - which is what Go schedulers uses.. no ?

> On Jan 31, 2019, at 10:21 AM, Michael Jones  wrote:
> 
> Agree. I’ve never had that level of imbalance but yes. 
> 
> On Thu, Jan 31, 2019 at 4:06 AM roger peppe  > wrote:
> On Thu, 31 Jan 2019 at 00:06, Michael Jones  > wrote:
> note that my code sidesteps this problem generally
> 
> I'm not sure that that's true. Although your code mitigates the problem 
> somewhat,
> it's still possible for a one slow worker to block the others. You've added 
> 512*NumCPU/2
> buffer slots, but in general it's not possible to order results and provide 
> avoid unnecessary
> blocking without having N buffer slots. In your code, assume 4 CPUs and that 
> work items 0
> and 2 take 1s and all other work items take 1ms. If we've got 5000 items in 
> total,
> the total time taken will be 2.002498s instead of the ideal time (~1s+2499µs).
> 
> https://play.golang.org/p/5Ty6pgpmZ0w 
> 
> Here's a kind of hybrid approach. It still serializes, but it makes as good a 
> use of the buffer
> space as it can - it won't block until a slow item is at least bufSize items 
> behind the
> most recently processed item:
> 
> https://play.golang.org/p/PP9NSJuLeEK 
>  
> On Wed, Jan 30, 2019 at 3:48 PM roger peppe  > wrote:
> On Wed, 30 Jan 2019 at 20:07, 'Bryan Mills' via golang-nuts 
> mailto:golang-nuts@googlegroups.com>> wrote:
> The code to sequence the results using a channel is not much more verbose.
> 
> That not only avoids the library dependency, but also makes the peak memory 
> consumption for the results O(runtime.NumCPU()), instead of O(N) with the 
> number of tasks, and allows the output to be streamed instead of buffered to 
> a slice.
> 
> https://play.golang.org/p/zkBjxlcvESe 
> 
> Nice! In practice though, I've usually found that I do want to keep the 
> results around or I don't care about the order at all, so the parallel 
> package works OK.
> 
> I'd point out one down side to the sequencing approach - one very slow work 
> item can block the others. For example, NProc is 4, the first item takes 200 
> milliseconds to process and all the others take 1 millisecond, then the first 
> 4 workers have started, none more will be started until the first has, so the 
> overall time will be quite a bit longer (249ms) than if they were all allowed 
> to proceed irrespective of order (200ms).
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .
> 
> 
> -- 
> Michael T. Jones
> michael.jo...@gmail.com -- 
> Michael T. Jones
> michael.jo...@gmail.com 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[Michael Jones]

Sorry to hear that. Sharing code is like sharing mathematical knowledge—a
universal language that is naturally at home above petty squabbles. Alas,
the world’s issues spoil many pure things here and there.

On Sun, Jan 27, 2019 at 2:07 AM  wrote:

> when i open the link, it return:Unavailable For Legal Reasons
>
> Viewing and/or sharing code snippets is not available in your country for
> legal reasons. This message might also appear if your country is
> misdetected. If you believe this is an error, please file an issue
> .
>
> 在 2019年1月26日星期六 UTC+8上午6:00:12，Michael Jones写道：
>>
>> code snippet:
>> https://play.golang.org/p/pphzr6QKiQ3
>>
>> here I have an existing serial process (using channels in preparation)
>> for a run-time switch between serial and parallel evaluation. the snippet
>> is that switch.
>>
>> N desired to be used cpus
>> N worker goroutines
>> N work chan
>> N answer chan
>> distribute work to job%N
>> gather answers from job%N
>> results are faster and in same total order
>>
>> On Fri, Jan 25, 2019 at 12:57 PM robert engels 
>> wrote:
>>
>>> Also you need to do is read the results (each result object should have
>>> the task ID of the submission)from the channel, add the result to slice of
>>> results, when result count == task count, sort the results by task id, and
>>> output each one. Straightforward.
>>>
>>> A single Go routine reads from all of the “output channels “ (one per
>>> number of CPU), and builds the result set.
>>>
>>> On Jan 25, 2019, at 2:39 PM, twp...@gmail.com wrote:
>>>
>>> The output is a series of strings written to the console. I think this
>>> is independent of the problem description though: once you have either a
>>> slice of results in order, or a channel returning results in order, then
>>> the problem is solved.
>>>
>>> On Friday, January 25, 2019 at 9:36:53 PM UTC+1, twp...@gmail.com wrote:

 For task submission, I have a slice of tasks. The easy way to submit
 them is to spin up a goroutine that writes them sequentially to a channel
 and then closes the channel when there are no more tasks.

 On Friday, January 25, 2019 at 9:29:43 PM UTC+1, robert engels wrote:
>
> I think the reason you haven’t found a library, is that given the
> specifications, it is probably less than 100 lines of Go code to 
> accomplish.
>
> The important elements lacking in your specification though are “how
> are the tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the
> output presented” (e.g. to the console, each task decides, etc .)
>
> On Jan 25, 2019, at 1:04 PM, twp...@gmail.com wrote:
>
> Hi,
>
> I have a number of slow tasks that I want to run concurrently across
> runtime.NumCPU() workers in a single process. The tasks have a specific
> input order, but they are completely independent of each other and can
> execute in any order. I would like to print the output of each task in the
> same order as the input order of tasks.
>
> This can be implemented by including each task's index in the input
> order as it is distributed via a channel to the workers, and the final
> collection of results assembled using these task indexes before the 
> results
> are printed.
>
> Assumptions:
> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
> - Single Go process, i.e. I don't want/need a distributed system.
>
> This feels like it should be common problem and there's probably
> either a library or a standard Go pattern out there which can do it. My 
> web
> search skills didn't find such a library though. Do you know of one?
>
> Cheers,
> Tom
>
>
> Background info to avoid the XY problem :
> this is to make chezmoi  run
> faster. I want to run the doctor checks
> 
> (basically os.Exec'ing a whole load of binaries to get their versions)
> concurrently in the short term. In the long term I want to make chezmoi's
> apply concurrent, so it runs faster too. In the first case, the order
> requirement is because I want all users to see the output in the same 
> order
> so that it's easy to compare. In the second case, the order requirement
> comes because I need to ensure that parent directories are in the correct
> state before checking their children.
>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>>> --
>>> You received this message because you are subscribed to the 

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[mountainfpf]

when i open the link, it return:Unavailable For Legal Reasons

Viewing and/or sharing code snippets is not available in your country for 
legal reasons. This message might also appear if your country is 
misdetected. If you believe this is an error, please file an issue 
.

在 2019年1月26日星期六 UTC+8上午6:00:12，Michael Jones写道：
>
> code snippet:
> https://play.golang.org/p/pphzr6QKiQ3
>
> here I have an existing serial process (using channels in preparation) for 
> a run-time switch between serial and parallel evaluation. the snippet is 
> that switch.
>
> N desired to be used cpus
> N worker goroutines
> N work chan
> N answer chan
> distribute work to job%N
> gather answers from job%N
> results are faster and in same total order
>
> On Fri, Jan 25, 2019 at 12:57 PM robert engels  > wrote:
>
>> Also you need to do is read the results (each result object should have 
>> the task ID of the submission)from the channel, add the result to slice of 
>> results, when result count == task count, sort the results by task id, and 
>> output each one. Straightforward.
>>
>> A single Go routine reads from all of the “output channels “ (one per 
>> number of CPU), and builds the result set.
>>
>> On Jan 25, 2019, at 2:39 PM, twp...@gmail.com  wrote:
>>
>> The output is a series of strings written to the console. I think this is 
>> independent of the problem description though: once you have either a slice 
>> of results in order, or a channel returning results in order, then the 
>> problem is solved.
>>
>> On Friday, January 25, 2019 at 9:36:53 PM UTC+1, twp...@gmail.com wrote:
>>>
>>> For task submission, I have a slice of tasks. The easy way to submit 
>>> them is to spin up a goroutine that writes them sequentially to a channel 
>>> and then closes the channel when there are no more tasks.
>>>
>>> On Friday, January 25, 2019 at 9:29:43 PM UTC+1, robert engels wrote:

 I think the reason you haven’t found a library, is that given the 
 specifications, it is probably less than 100 lines of Go code to 
 accomplish.

 The important elements lacking in your specification though are “how 
 are the tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the 
 output presented” (e.g. to the console, each task decides, etc .)

 On Jan 25, 2019, at 1:04 PM, twp...@gmail.com wrote:

 Hi,

 I have a number of slow tasks that I want to run concurrently across 
 runtime.NumCPU() workers in a single process. The tasks have a specific 
 input order, but they are completely independent of each other and can 
 execute in any order. I would like to print the output of each task in the 
 same order as the input order of tasks.

 This can be implemented by including each task's index in the input 
 order as it is distributed via a channel to the workers, and the final 
 collection of results assembled using these task indexes before the 
 results 
 are printed.

 Assumptions:
 - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
 - Single Go process, i.e. I don't want/need a distributed system.

 This feels like it should be common problem and there's probably either 
 a library or a standard Go pattern out there which can do it. My web 
 search 
 skills didn't find such a library though. Do you know of one?

 Cheers,
 Tom


 Background info to avoid the XY problem : this 
 is to make chezmoi  run faster. I 
 want to run the doctor checks 
 
  
 (basically os.Exec'ing a whole load of binaries to get their versions) 
 concurrently in the short term. In the long term I want to make chezmoi's 
 apply concurrent, so it runs faster too. In the first case, the order 
 requirement is because I want all users to see the output in the same 
 order 
 so that it's easy to compare. In the second case, the order requirement 
 comes because I need to ensure that parent directories are in the correct 
 state before checking their children.

 -- 
 You received this message because you are subscribed to the Google 
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to golang-nuts...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.



>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[Michael Jones]

code snippet:
https://play.golang.org/p/pphzr6QKiQ3

here I have an existing serial process (using channels in preparation) for
a run-time switch between serial and parallel evaluation. the snippet is
that switch.

N desired to be used cpus
N worker goroutines
N work chan
N answer chan
distribute work to job%N
gather answers from job%N
results are faster and in same total order

On Fri, Jan 25, 2019 at 12:57 PM robert engels 
wrote:

> Also you need to do is read the results (each result object should have
> the task ID of the submission)from the channel, add the result to slice of
> results, when result count == task count, sort the results by task id, and
> output each one. Straightforward.
>
> A single Go routine reads from all of the “output channels “ (one per
> number of CPU), and builds the result set.
>
> On Jan 25, 2019, at 2:39 PM, twpa...@gmail.com wrote:
>
> The output is a series of strings written to the console. I think this is
> independent of the problem description though: once you have either a slice
> of results in order, or a channel returning results in order, then the
> problem is solved.
>
> On Friday, January 25, 2019 at 9:36:53 PM UTC+1, twp...@gmail.com wrote:
>>
>> For task submission, I have a slice of tasks. The easy way to submit them
>> is to spin up a goroutine that writes them sequentially to a channel and
>> then closes the channel when there are no more tasks.
>>
>> On Friday, January 25, 2019 at 9:29:43 PM UTC+1, robert engels wrote:
>>>
>>> I think the reason you haven’t found a library, is that given the
>>> specifications, it is probably less than 100 lines of Go code to accomplish.
>>>
>>> The important elements lacking in your specification though are “how are
>>> the tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the
>>> output presented” (e.g. to the console, each task decides, etc .)
>>>
>>> On Jan 25, 2019, at 1:04 PM, twp...@gmail.com wrote:
>>>
>>> Hi,
>>>
>>> I have a number of slow tasks that I want to run concurrently across
>>> runtime.NumCPU() workers in a single process. The tasks have a specific
>>> input order, but they are completely independent of each other and can
>>> execute in any order. I would like to print the output of each task in the
>>> same order as the input order of tasks.
>>>
>>> This can be implemented by including each task's index in the input
>>> order as it is distributed via a channel to the workers, and the final
>>> collection of results assembled using these task indexes before the results
>>> are printed.
>>>
>>> Assumptions:
>>> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
>>> - Single Go process, i.e. I don't want/need a distributed system.
>>>
>>> This feels like it should be common problem and there's probably either
>>> a library or a standard Go pattern out there which can do it. My web search
>>> skills didn't find such a library though. Do you know of one?
>>>
>>> Cheers,
>>> Tom
>>>
>>>
>>> Background info to avoid the XY problem : this
>>> is to make chezmoi  run faster. I
>>> want to run the doctor checks
>>> 
>>> (basically os.Exec'ing a whole load of binaries to get their versions)
>>> concurrently in the short term. In the long term I want to make chezmoi's
>>> apply concurrent, so it runs faster too. In the first case, the order
>>> requirement is because I want all users to see the output in the same order
>>> so that it's easy to compare. In the second case, the order requirement
>>> comes because I need to ensure that parent directories are in the correct
>>> state before checking their children.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 

*Michael T. jonesmichael.jo...@gmail.com *

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit 

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[robert engels]

Also you need to do is read the results (each result object should have the 
task ID of the submission)from the channel, add the result to slice of results, 
when result count == task count, sort the results by task id, and output each 
one. Straightforward.

A single Go routine reads from all of the “output channels “ (one per number of 
CPU), and builds the result set.

> On Jan 25, 2019, at 2:39 PM, twpa...@gmail.com wrote:
> 
> The output is a series of strings written to the console. I think this is 
> independent of the problem description though: once you have either a slice 
> of results in order, or a channel returning results in order, then the 
> problem is solved.
> 
> On Friday, January 25, 2019 at 9:36:53 PM UTC+1, twp...@gmail.com wrote:
> For task submission, I have a slice of tasks. The easy way to submit them is 
> to spin up a goroutine that writes them sequentially to a channel and then 
> closes the channel when there are no more tasks.
> 
> On Friday, January 25, 2019 at 9:29:43 PM UTC+1, robert engels wrote:
> I think the reason you haven’t found a library, is that given the 
> specifications, it is probably less than 100 lines of Go code to accomplish.
> 
> The important elements lacking in your specification though are “how are the 
> tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the output 
> presented” (e.g. to the console, each task decides, etc .)
> 
>> On Jan 25, 2019, at 1:04 PM, twp...@gmail.com <> wrote:
>> 
>> Hi,
>> 
>> I have a number of slow tasks that I want to run concurrently across 
>> runtime.NumCPU() workers in a single process. The tasks have a specific 
>> input order, but they are completely independent of each other and can 
>> execute in any order. I would like to print the output of each task in the 
>> same order as the input order of tasks.
>> 
>> This can be implemented by including each task's index in the input order as 
>> it is distributed via a channel to the workers, and the final collection of 
>> results assembled using these task indexes before the results are printed.
>> 
>> Assumptions:
>> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
>> - Single Go process, i.e. I don't want/need a distributed system.
>> 
>> This feels like it should be common problem and there's probably either a 
>> library or a standard Go pattern out there which can do it. My web search 
>> skills didn't find such a library though. Do you know of one?
>> 
>> Cheers,
>> Tom
>> 
>> 
>> Background info to avoid the XY problem : this is to 
>> make chezmoi  run faster. I want to run 
>> the doctor checks 
>> 
>>  (basically os.Exec'ing a whole load of binaries to get their versions) 
>> concurrently in the short term. In the long term I want to make chezmoi's 
>> apply concurrent, so it runs faster too. In the first case, the order 
>> requirement is because I want all users to see the output in the same order 
>> so that it's easy to compare. In the second case, the order requirement 
>> comes because I need to ensure that parent directories are in the correct 
>> state before checking their children.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com <>.
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[twpayne]

This is accurate for the details, but not accurate overall. Basically I 
want "parallel map". Input order defined. Element-wise transformations are 
independent. Output order must be the same as input order. If a worker 
stalls, I'm quite happy with the entire process stalling at the point.

On Friday, January 25, 2019 at 9:39:44 PM UTC+1, Jan Mercl wrote:
>
> On Fri, Jan 25, 2019 at 8:04 PM > wrote:
>
> IINM, concurrent _and_ order preserving are concepts mutually exclusive by 
> definition.
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[Jan Mercl]

On Fri, Jan 25, 2019 at 8:04 PM  wrote:

IINM, concurrent _and_ order preserving are concepts mutually exclusive by
definition.
-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[twpayne]

The output is a series of strings written to the console. I think this is 
independent of the problem description though: once you have either a slice 
of results in order, or a channel returning results in order, then the 
problem is solved.

On Friday, January 25, 2019 at 9:36:53 PM UTC+1, twp...@gmail.com wrote:
>
> For task submission, I have a slice of tasks. The easy way to submit them 
> is to spin up a goroutine that writes them sequentially to a channel and 
> then closes the channel when there are no more tasks.
>
> On Friday, January 25, 2019 at 9:29:43 PM UTC+1, robert engels wrote:
>>
>> I think the reason you haven’t found a library, is that given the 
>> specifications, it is probably less than 100 lines of Go code to accomplish.
>>
>> The important elements lacking in your specification though are “how are 
>> the tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the 
>> output presented” (e.g. to the console, each task decides, etc .)
>>
>> On Jan 25, 2019, at 1:04 PM, twp...@gmail.com wrote:
>>
>> Hi,
>>
>> I have a number of slow tasks that I want to run concurrently across 
>> runtime.NumCPU() workers in a single process. The tasks have a specific 
>> input order, but they are completely independent of each other and can 
>> execute in any order. I would like to print the output of each task in the 
>> same order as the input order of tasks.
>>
>> This can be implemented by including each task's index in the input order 
>> as it is distributed via a channel to the workers, and the final collection 
>> of results assembled using these task indexes before the results are 
>> printed.
>>
>> Assumptions:
>> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
>> - Single Go process, i.e. I don't want/need a distributed system.
>>
>> This feels like it should be common problem and there's probably either a 
>> library or a standard Go pattern out there which can do it. My web search 
>> skills didn't find such a library though. Do you know of one?
>>
>> Cheers,
>> Tom
>>
>>
>> Background info to avoid the XY problem : this 
>> is to make chezmoi  run faster. I 
>> want to run the doctor checks 
>> 
>>  
>> (basically os.Exec'ing a whole load of binaries to get their versions) 
>> concurrently in the short term. In the long term I want to make chezmoi's 
>> apply concurrent, so it runs faster too. In the first case, the order 
>> requirement is because I want all users to see the output in the same order 
>> so that it's easy to compare. In the second case, the order requirement 
>> comes because I need to ensure that parent directories are in the correct 
>> state before checking their children.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[twpayne]

For task submission, I have a slice of tasks. The easy way to submit them 
is to spin up a goroutine that writes them sequentially to a channel and 
then closes the channel when there are no more tasks.

On Friday, January 25, 2019 at 9:29:43 PM UTC+1, robert engels wrote:
>
> I think the reason you haven’t found a library, is that given the 
> specifications, it is probably less than 100 lines of Go code to accomplish.
>
> The important elements lacking in your specification though are “how are 
> the tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the 
> output presented” (e.g. to the console, each task decides, etc .)
>
> On Jan 25, 2019, at 1:04 PM, twp...@gmail.com  wrote:
>
> Hi,
>
> I have a number of slow tasks that I want to run concurrently across 
> runtime.NumCPU() workers in a single process. The tasks have a specific 
> input order, but they are completely independent of each other and can 
> execute in any order. I would like to print the output of each task in the 
> same order as the input order of tasks.
>
> This can be implemented by including each task's index in the input order 
> as it is distributed via a channel to the workers, and the final collection 
> of results assembled using these task indexes before the results are 
> printed.
>
> Assumptions:
> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
> - Single Go process, i.e. I don't want/need a distributed system.
>
> This feels like it should be common problem and there's probably either a 
> library or a standard Go pattern out there which can do it. My web search 
> skills didn't find such a library though. Do you know of one?
>
> Cheers,
> Tom
>
>
> Background info to avoid the XY problem : this is 
> to make chezmoi  run faster. I want 
> to run the doctor checks 
> 
>  
> (basically os.Exec'ing a whole load of binaries to get their versions) 
> concurrently in the short term. In the long term I want to make chezmoi's 
> apply concurrent, so it runs faster too. In the first case, the order 
> requirement is because I want all users to see the output in the same order 
> so that it's easy to compare. In the second case, the order requirement 
> comes because I need to ensure that parent directories are in the correct 
> state before checking their children.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Existing code for order-preserving concurrent work queue?

[robert engels]

I think the reason you haven’t found a library, is that given the 
specifications, it is probably less than 100 lines of Go code to accomplish.

The important elements lacking in your specification though are “how are the 
tasks submitted” (e.g. http? in code ?, etc.) , , and “how is the output 
presented” (e.g. to the console, each task decides, etc .)

> On Jan 25, 2019, at 1:04 PM, twpa...@gmail.com wrote:
> 
> Hi,
> 
> I have a number of slow tasks that I want to run concurrently across 
> runtime.NumCPU() workers in a single process. The tasks have a specific input 
> order, but they are completely independent of each other and can execute in 
> any order. I would like to print the output of each task in the same order as 
> the input order of tasks.
> 
> This can be implemented by including each task's index in the input order as 
> it is distributed via a channel to the workers, and the final collection of 
> results assembled using these task indexes before the results are printed.
> 
> Assumptions:
> - Small number of tasks (~10,000 max), i.e. this easily fits in memory.
> - Single Go process, i.e. I don't want/need a distributed system.
> 
> This feels like it should be common problem and there's probably either a 
> library or a standard Go pattern out there which can do it. My web search 
> skills didn't find such a library though. Do you know of one?
> 
> Cheers,
> Tom
> 
> 
> Background info to avoid the XY problem : this is to 
> make chezmoi  run faster. I want to run 
> the doctor checks 
> 
>  (basically os.Exec'ing a whole load of binaries to get their versions) 
> concurrently in the short term. In the long term I want to make chezmoi's 
> apply concurrent, so it runs faster too. In the first case, the order 
> requirement is because I want all users to see the output in the same order 
> so that it's easy to compare. In the second case, the order requirement comes 
> because I need to ensure that parent directories are in the correct state 
> before checking their children.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Existing code for order-preserving concurrent work queue?

[twpayne]

Hi,

I have a number of slow tasks that I want to run concurrently across 
runtime.NumCPU() workers in a single process. The tasks have a specific 
input order, but they are completely independent of each other and can 
execute in any order. I would like to print the output of each task in the 
same order as the input order of tasks.

This can be implemented by including each task's index in the input order 
as it is distributed via a channel to the workers, and the final collection 
of results assembled using these task indexes before the results are 
printed.

Assumptions:
- Small number of tasks (~10,000 max), i.e. this easily fits in memory.
- Single Go process, i.e. I don't want/need a distributed system.

This feels like it should be common problem and there's probably either a 
library or a standard Go pattern out there which can do it. My web search 
skills didn't find such a library though. Do you know of one?

Cheers,
Tom


Background info to avoid the XY problem : this is 
to make chezmoi  run faster. I want to 
run the doctor checks 

 
(basically os.Exec'ing a whole load of binaries to get their versions) 
concurrently in the short term. In the long term I want to make chezmoi's 
apply concurrent, so it runs faster too. In the first case, the order 
requirement is because I want all users to see the output in the same order 
so that it's easy to compare. In the second case, the order requirement 
comes because I need to ensure that parent directories are in the correct 
state before checking their children.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.