You deserve extra credit for getting the phrase "ouroboros data structure" in there. ;-)
On Thu, Jan 31, 2019 at 10:20 AM roger peppe <rogpe...@gmail.com> wrote: > On Thu, 31 Jan 2019 at 16:22, Michael Jones <michael.jo...@gmail.com> > wrote: > >> Agree. I’ve never had that level of imbalance but yes. >> > > When doing mathematical calculations, as I think your code was, I think > it's usual to have comparable costs for each item. By contrast, I'm usually > adding this kind of code to make concurrent network requests, and in that > scenario it's much more likely that some requests are orders of magnitude > slower than others. > > FWIW I was quite pleased with the `chan chan R` idea - I might use it for > real some time :) Come to think of it, it's slightly reminiscent of another > pattern I wrote about some years ago: > https://rogpeppe.wordpress.com/2009/12/01/concurrent-idioms-1-broadcasting-values-in-go-with-linked-channels/ > > cheers, > rog. > > >> On Thu, Jan 31, 2019 at 4:06 AM roger peppe <rogpe...@gmail.com> wrote: >> >>> On Thu, 31 Jan 2019 at 00:06, Michael Jones <michael.jo...@gmail.com> >>> wrote: >>> >>>> note that my code sidesteps this problem generally >>>> >>> >>> I'm not sure that that's true. Although your code mitigates the problem >>> somewhat, >>> it's still possible for a one slow worker to block the others. You've >>> added 512*NumCPU/2 >>> buffer slots, but in general it's not possible to order results and >>> provide avoid unnecessary >>> blocking without having N buffer slots. In your code, assume 4 CPUs and >>> that work items 0 >>> and 2 take 1s and all other work items take 1ms. If we've got 5000 items >>> in total, >>> the total time taken will be 2.002498s instead of the ideal time >>> (~1s+2499µs). >>> >>> https://play.golang.org/p/5Ty6pgpmZ0w >>> >>> Here's a kind of hybrid approach. It still serializes, but it makes as >>> good a use of the buffer >>> space as it can - it won't block until a slow item is at least bufSize >>> items behind the >>> most recently processed item: >>> >>> https://play.golang.org/p/PP9NSJuLeEK >>> >>> >>>> On Wed, Jan 30, 2019 at 3:48 PM roger peppe <rogpe...@gmail.com> wrote: >>>> >>>>> On Wed, 30 Jan 2019 at 20:07, 'Bryan Mills' via golang-nuts < >>>>> golang-nuts@googlegroups.com> wrote: >>>>> >>>>>> The code to sequence the results using a channel is not much more >>>>>> verbose. >>>>>> >>>>>> That not only avoids the library dependency, but also makes the peak >>>>>> memory consumption for the results O(runtime.NumCPU()), instead of O(N) >>>>>> with the number of tasks, and allows the output to be streamed instead of >>>>>> buffered to a slice. >>>>>> >>>>>> https://play.golang.org/p/zkBjxlcvESe >>>>>> >>>>> >>>>> Nice! In practice though, I've usually found that I do want to keep >>>>> the results around or I don't care about the order at all, so the parallel >>>>> package works OK. >>>>> >>>>> I'd point out one down side to the sequencing approach - one very slow >>>>> work item can block the others. For example, NProc is 4, the first item >>>>> takes 200 milliseconds to process and all the others take 1 millisecond, >>>>> then the first 4 workers have started, none more will be started until the >>>>> first has, so the overall time will be quite a bit longer (249ms) than if >>>>> they were all allowed to proceed irrespective of order (200ms). >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to golang-nuts+unsubscr...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- >>>> >>>> *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>* >>>> >>> -- >> >> *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>* >> > -- *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>* -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
