RE: [go-nuts] Managing perpetually running goroutines

sanyia.saidova Wed, 01 Sep 2021 14:55:22 -0700

Thanks Bryan, especially for the code example.

Would using a sync.WaitGroup in place of the `done` channel in your example be overkill? Just exploring what’s possible here for my own education.

From: 'Bryan C. Mills' via golang-nuts
Sent: 31 August 2021 20:22
To: golang-nuts
Subject: Re: [go-nuts] Managing perpetually running goroutines

For the specific case of managing long-running polling- or pubsub-based tasks, I would suggest basically the same pattern that others on the thread have already given: use a `sync.Map` to associate each task ID with a small struct containing a `context.CancelFunc` and a channel that can be used to confirm when the task has actually stopped (https://play.golang.org/p/Gq3SXuPFDvi).

That way you can have the invariant that after a successful call to Stop, the worker has completely finished all of its work. That property is extremely useful, for example, in writing tests and benchmarks for your implementation: if you can guarantee that the workers have actually stopped by the time Stop returns, your tests can detect unexpected deadlocks and your benchmarks can measure a controlled, well-defined amount of work.

On Monday, August 30, 2021 at 7:52:59 AM UTC-4 Brian Candler wrote:

For that case, I'd suggest you pass a channel to each worker. It can either close the channel, or send a specific message down it, to indicate its termination.

This may also be an X-Y problem. Generally speaking, "worker pools" are an anti-pattern in Go. The startup and termination overhead of a goroutine is so tiny that often it's better just to start a goroutine when there's work to be done, and let it run to completion. The other reason you might want to have a worker pool is to limit concurrency, but there are better ways to do that in Go: e.g. you can have a buffered channel with N slots, which the goroutine writes to before starting work and reads from when it's finished. That will limit your concurrency to N tasks.

Bryan C Mills discusses this in detail in this excellent video on "Rethinking Classical Concurrency Patterns":
https://www.youtube.com/watch?v=5zXAHh5tJqQ
It took me several views with pauses to fully understand it, but it was worth the effort :-)

On Monday, 30 August 2021 at 12:29:26 UTC+1 sansaid wrote:
Appreciated, thanks Brian and Ian.

On the subject of sync.WaitGroups, I haven't tackled this yet for my use case, but will use the opportunity to brainstorm some options, if that's okay. I want to be able to terminate each worker independently of others (i.e. close on worker, wait for it to finish its tasks, but allow other workers to run if they haven't been signalled to stop). Should I create a wait group per worker and store that in a map? This sounds bloated to me, especially since it's always going to be a wait group of 1, but I haven't come across better alternatives yet (relatively new to concurrency myself).

If that's too broad a question, can come back once I've explored some options.

On Mon, 30 Aug 2021, 12:20 Brian Candler, <b.ca...@pobox.com> wrote:
Also: you can use the same context for all the workers (if that's appropriate); or if you want each worker to have a different context, you can make those contexts children of the same top-level context. In that case, you only need to cancel one context and all the workers will stop, without having to keep a map of them.

Do bear in mind that you may wish to wait for the workers to finish what they're currently doing, before the program exits. One way to do that is with a sync.WaitGroup, which you increment before starting each worker, and each worker decrements just before it terminates. Then in your main program you wait on the WaitGroup.

Typical pattern: https://play.golang.org/p/51oy7ySbL4T

On Monday, 30 August 2021 at 10:01:08 UTC+1 Ian Davis wrote:

On Sun, 29 Aug 2021, at 10:45 PM, sansaid wrote:
Hello,

Does anybody know of some reference code or common patterns I can use to keep track of worker goroutines? For context, I want a user to be able to issue a "stop" command using a CLI tool which will prompt my server to gracefully terminate one of my workers.

For example, when a user issues a "start" command through a CLI tool, this will signal my server to spawn a goroutine that perpetually runs and polls an HTTP endpoint until a user initiates a "stop" command. When the user issues a "stop" command through the same CLI tool, I want the server to be able to signal the goroutine to stop. Any reference code would be much appreciated!

One approach I thought of was by passing a `context.WithCancel()` and holding a reference to the cancel function in a global map (with the worker ID as keys). When the user issues a "stop" command against the worker ID, another function is executed which calls the context's cancel function. Example code below (appreciate this is horrendously breaking a lot of rules, but I want to focus only on the elements around passing the cancel function around - happy to discuss anything else that is of concern to someone though):

```
// invoked when a CLI start subcommand is issued to my main CLI tool
func (w *WorkerGroup) Start(ctx context.Context, id string) {
_, cancel := context.WithCancel(ctx)

w.Workers.Lock()
w.Workers[id] := cancel
w.Workers.Unlock()

go startWorker(id)
}

// invoked when a CLI stop subcommand is issued to my main CLI tool
func (w *WorkerGroup) Stop(id string) {
cancelWorker := w.Workers[id]

cancelWorker()
}
```

I have read that passing a context's cancel function around is a bad idea, but in an example like this, is it justified?

Passing the cancel function is fine and storing it for use later is fine too.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/fdff94c4-c0e4-4418-abb3-eba91298fbb7n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7246761d-28a2-420b-a04b-c869c85194ccn%40googlegroups.com.

RE: [go-nuts] Managing perpetually running goroutines

Reply via email to