On Sat, Dec 28, 2019 at 6:11 AM Robert Engels <reng...@ix.netcom.com> wrote: > > Spinning up a Go routine when for each piece of work may be idiomatic but it > is far from the best practice for many situations - mainly because of cpu > cache invalidation. Most HPC systems create worker pools by type and then > isolate them to cpus/cores - something you can’t do in Go. > > I believe there are outstanding proposals for grouping routines and core > affinity.
I think that today CPU cache invalidation is more or less independent of spinning up separate goroutines. CPU cache invalidation is something to consider for threads, but, as you say, goroutines have no affinity to threads. Whether you use a worker pool or not, Go doesn't currently give you any way to control how goroutines are assigned to threads. So in general I don't see any reason to think that a worker pool would be better or worse with regard to CPU cache invalidation. If Go introduces some way to associate goroutines with threads and some way to associate threads with CPUs, that might well have an effect on whether it is better to use a worker pool. Ian > On Dec 28, 2019, at 7:02 AM, Agniva De Sarker <agniva.quicksil...@gmail.com> > wrote: > > > > (the task was to limit the whole thing to about 10% of cores) > > I still don't think you needed a worker pool here. Like OP mentioned above, > you could just limit the number of goroutines executed to 10% of total cores. > > > On Saturday, 28 December 2019 18:02:08 UTC+5:30, Chris Burkert wrote: >> >> There are Pros and Cons for everything in life. Some time ago I wrote a >> database tool which does something per table where the runtime largely >> depends on the table size. I started with one goroutine per table because it >> was easy. But that put a high load on the database (the task was to limit >> the whole thing to about 10% of cores) with out of memory situations. So I >> switched to a worker pool which solved that. However now the overall runtime >> was unpredictable. When the large tables were handled at the end their >> runtime defined the overall runtime. So I to feed the pool with large tables >> first. This again lead to out of memory situations so I reordered the tables >> such that large tables are mixed with smaller tables. >> >> Brian Candler <b.ca...@pobox.com> schrieb am Sa. 28. Dez. 2019 um 11:09: >>> >>> On Friday, 27 December 2019 16:30:48 UTC, Bruno Albuquerque wrote: >>>> >>>> This might be useful too you, in any case: >>>> >>>> https://git.bug-br.org.br/bga/workerpool >>>> >>> >>> I think the point from Bryan Mills' video is, "worker pool" is something of >>> an anti-pattern in go. goroutines are so cheap that you might as well >>> start a goroutine for each piece of work you have to do, and let it >>> terminate when that piece of work is done. >>> >>> Apart from the startup cost, the other reason for having a "worker pool" is >>> to limit the number of concurrent tasks being executed, and there are >>> better ways of doing that in go (also shown in the video). >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to golan...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/f840beee-748f-42b6-809f-4c7505208aee%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/ecddafbc-eb73-45e1-8a5a-f738e88c6821%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/8C1153D5-25E7-4A06-8626-1FE61D9015D5%40ix.netcom.com. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcW9ORpo4dcUe6HYk1VS8Jp6J0EDpWAUXG4dHuYoUtbV7w%40mail.gmail.com.
