I agree. I meant that worker pools are especially useful when you can do cpu affinity - doesn’t apply to Go.
I think Go probably needs some idea of “capping” for cpu based workloads. You can cap in the local N CPUs , but in a larger app that has multiple parallel processing points you can easy create too many routines and then you are paying overhead switching costs for no reason. > On Dec 28, 2019, at 10:17 AM, Ian Lance Taylor <i...@golang.org> wrote: > > On Sat, Dec 28, 2019 at 6:11 AM Robert Engels <reng...@ix.netcom.com> wrote: >> >> Spinning up a Go routine when for each piece of work may be idiomatic but it >> is far from the best practice for many situations - mainly because of cpu >> cache invalidation. Most HPC systems create worker pools by type and then >> isolate them to cpus/cores - something you can’t do in Go. >> >> I believe there are outstanding proposals for grouping routines and core >> affinity. > > I think that today CPU cache invalidation is more or less independent > of spinning up separate goroutines. CPU cache invalidation is > something to consider for threads, but, as you say, goroutines have no > affinity to threads. Whether you use a worker pool or not, Go doesn't > currently give you any way to control how goroutines are assigned to > threads. So in general I don't see any reason to think that a worker > pool would be better or worse with regard to CPU cache invalidation. > > If Go introduces some way to associate goroutines with threads and > some way to associate threads with CPUs, that might well have an > effect on whether it is better to use a worker pool. > > Ian > > >> On Dec 28, 2019, at 7:02 AM, Agniva De Sarker <agniva.quicksil...@gmail.com> >> wrote: >> >> >>> (the task was to limit the whole thing to about 10% of cores) >> >> I still don't think you needed a worker pool here. Like OP mentioned above, >> you could just limit the number of goroutines executed to 10% of total cores. >> >> >> On Saturday, 28 December 2019 18:02:08 UTC+5:30, Chris Burkert wrote: >>> >>> There are Pros and Cons for everything in life. Some time ago I wrote a >>> database tool which does something per table where the runtime largely >>> depends on the table size. I started with one goroutine per table because >>> it was easy. But that put a high load on the database (the task was to >>> limit the whole thing to about 10% of cores) with out of memory situations. >>> So I switched to a worker pool which solved that. However now the overall >>> runtime was unpredictable. When the large tables were handled at the end >>> their runtime defined the overall runtime. So I to feed the pool with large >>> tables first. This again lead to out of memory situations so I reordered >>> the tables such that large tables are mixed with smaller tables. >>> >>> Brian Candler <b.ca...@pobox.com> schrieb am Sa. 28. Dez. 2019 um 11:09: >>>> >>>> On Friday, 27 December 2019 16:30:48 UTC, Bruno Albuquerque wrote: >>>>> >>>>> This might be useful too you, in any case: >>>>> >>>>> https://git.bug-br.org.br/bga/workerpool >>>>> >>>> >>>> I think the point from Bryan Mills' video is, "worker pool" is something >>>> of an anti-pattern in go. goroutines are so cheap that you might as well >>>> start a goroutine for each piece of work you have to do, and let it >>>> terminate when that piece of work is done. >>>> >>>> Apart from the startup cost, the other reason for having a "worker pool" >>>> is to limit the number of concurrent tasks being executed, and there are >>>> better ways of doing that in go (also shown in the video). >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to golan...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/f840beee-748f-42b6-809f-4c7505208aee%40googlegroups.com. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/ecddafbc-eb73-45e1-8a5a-f738e88c6821%40googlegroups.com. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/8C1153D5-25E7-4A06-8626-1FE61D9015D5%40ix.netcom.com. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcW9ORpo4dcUe6HYk1VS8Jp6J0EDpWAUXG4dHuYoUtbV7w%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/F7201C4F-D59C-4C26-A291-CFBB614903FC%40ix.netcom.com.
