Spinning up a Go routine when for each piece of work may be idiomatic but it is far from the best practice for many situations - mainly because of cpu cache invalidation. Most HPC systems create worker pools by type and then isolate them to cpus/cores - something you can’t do in Go.
I believe there are outstanding proposals for grouping routines and core affinity. > On Dec 28, 2019, at 7:02 AM, Agniva De Sarker <agniva.quicksil...@gmail.com> > wrote: > > > > (the task was to limit the whole thing to about 10% of cores) > > I still don't think you needed a worker pool here. Like OP mentioned above, > you could just limit the number of goroutines executed to 10% of total cores. > > >> On Saturday, 28 December 2019 18:02:08 UTC+5:30, Chris Burkert wrote: >> There are Pros and Cons for everything in life. Some time ago I wrote a >> database tool which does something per table where the runtime largely >> depends on the table size. I started with one goroutine per table because it >> was easy. But that put a high load on the database (the task was to limit >> the whole thing to about 10% of cores) with out of memory situations. So I >> switched to a worker pool which solved that. However now the overall runtime >> was unpredictable. When the large tables were handled at the end their >> runtime defined the overall runtime. So I to feed the pool with large tables >> first. This again lead to out of memory situations so I reordered the tables >> such that large tables are mixed with smaller tables. >> >> Brian Candler <b.ca...@pobox.com> schrieb am Sa. 28. Dez. 2019 um 11:09: >>>> On Friday, 27 December 2019 16:30:48 UTC, Bruno Albuquerque wrote: >>>> This might be useful too you, in any case: >>>> >>>> https://git.bug-br.org.br/bga/workerpool >>>> >>> >>> I think the point from Bryan Mills' video is, "worker pool" is something of >>> an anti-pattern in go. goroutines are so cheap that you might as well >>> start a goroutine for each piece of work you have to do, and let it >>> terminate when that piece of work is done. >>> >>> Apart from the startup cost, the other reason for having a "worker pool" is >>> to limit the number of concurrent tasks being executed, and there are >>> better ways of doing that in go (also shown in the video). >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to golan...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/f840beee-748f-42b6-809f-4c7505208aee%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/ecddafbc-eb73-45e1-8a5a-f738e88c6821%40googlegroups.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8C1153D5-25E7-4A06-8626-1FE61D9015D5%40ix.netcom.com.
