When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

I'm writing a an ETL process to read event level data from a product database, transform / aggregate it and write to to an analytics data warehouse. I'm using clojure's core.async library to separate these process into concurrently executing components. Here's what the main part of my code look

Re: When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

I think your case is exactly where not to use go blocks. Stuff in go blocks is executed on a limited size thread pool so with enough blocking I/O in there you could in theory slow down async processing. The win in using ! in go blocks is that they don't block async threads _while they are waiting

Re: When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

go blocks, together with >!, !! wrote: > I think your case is exactly where not to use go blocks. Stuff in go > blocks is executed on a limited size thread pool so with enough blocking > I/O in there you could in theory slow down async processing. > > The win in using ! in go blocks is that they d

Re: When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

Couple of points: - If this piece of code is a performance bottleneck, benchmark whether go 'threads' or real threads better suits your needs. - Handle edge cases, e.g. how your code behave when the channels close, mainly if you have (while true ...) combo. - async/thread uses growing thread poo

Re: When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

On Thu, Jan 30, 2014 at 12:48 PM, Jozef Wagner wrote: > > go blocks, together with >!, threads and are not run in separate thread. There is no thread pool for go > blocks. > I thought that go blocks do run in a thread pool of size 42 + (2 * Num of processors). In the definition of the go macro t

Re: When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

To quote Jozef "it can happen all in one thread". This is somewhat true, there are some rare situations where this can happen, but it is fairly rare. Many times putting a value into a channel will mean that the callback on the other end of the channel needs to be dispatched. In that case Mauricio

Re: When should I use non-blocking >! / threads and blocking >!! / goroutines with clojure core.async

Thank you for the clarification. If I understand it correctly, these callbacks in case of go 'threads' resume the parked state machine, so if there is a blocking IO inside go block, it ends up waiting in one of these callbacks, thus tying up a thread from a fixed sized pool until IO operation u