Re: Custom core.async go threadpools? Using go parking for heavy calculation parallelism throughput?

2018-05-02 Thread Alex Miller
Rich has considered making some of the internal analysis stuff that is in the go macro available via the compiler (so it doesn't have to be re-built in go), but I don't think that includes anything related to channels or blocking takes/puts, unless I'm misremembering. You might want to look at

Re: Custom core.async go threadpools? Using go parking for heavy calculation parallelism throughput?

2018-05-02 Thread Leon Grapenthin
I remember a Rich Hickey talk on core.async where he mentioned building blocking takes/puts into the compiler, as a possible future extension, making the go macro obsolete. Is that on any roadmap? Tesser I have to look at again, it seemed to go into a similar direction. Fork/Join /w reducers is

Re: Custom core.async go threadpools? Using go parking for heavy calculation parallelism throughput?

2018-05-02 Thread Didier
This seems well suited for tesser https://github.com/aphyr/tesser/blob/master/README.markdown Or you could just look at using fold https://clojure.org/reference/reducers#_reduce_and_fold -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to th

Re: Custom core.async go threadpools? Using go parking for heavy calculation parallelism throughput?

2018-05-01 Thread Leon Grapenthin
Yeah, that goes in a direction I thought about after my post. I'm going to implement sth. like this and we will see how much code overhead is necessary. At the time it appears to me that being able to locally dedicate threads to the go mechanism would make this much easier (than a hand control

Re: Custom core.async go threadpools? Using go parking for heavy calculation parallelism throughput?

2018-05-01 Thread Justin Smith
Just a couple of small points (and not yet a full answer): > A node can obviously not pmap over all the child nodes (would spawn exponential amount of threads) pmap is not that naive, it uses a pool sized with the assumption that its work is CPU bound > (2) Made me wonder why I couldn't use the

Custom core.async go threadpools? Using go parking for heavy calculation parallelism throughput?

2018-05-01 Thread Leon Grapenthin
Recently I worked on an algorithm where a distributed tree is (sort of) flattened in a way that each node runs a commutative aggregation over all of its child nodes calculations. 1 A node can obviously not pmap over all the child nodes (would spawn exponential amount of threads). 2 If I want t

Re: Using go/

2015-07-30 Thread Martin Raison
Fair point, thanks a lot for the insight. Any pointers to a significant data-flow oriented clojure codebase would be awesome, because that's not something I see a lot in the wild, and I'm still trying to wrap my head around how to implement this on a very complex system. Le jeudi 30 juillet 201

Re: Using go/

2015-07-30 Thread Timothy Baldridge
My advice is to treat channel ops (!) as IO, which they are. In functional programming we try to stay away from doing IO deep down inside a call stack. So don't do that. Instead use async/pipeline functions and channel transducers to create pipelines and flow the data through them. A sort of anti-

Re: Using go/

2015-07-30 Thread Martin Raison
Hey Thomas, Thanks for the great feedback! A few clarifications below. > It should be noted that your async macro does in fact use the dispatcher > just like a normal go would, the only difference is that it will start > executing immediately in the current thread until the first "pause" inst

Re: Using go/

2015-07-30 Thread Thomas Heller
Hey, I made a similar suggestion (minus the ThreadLocal) a few weeks ago, although for slightly different reasons. [1] It should be noted that your async macro does in fact use the dispatcher just like a normal go would, the only difference is that it will start executing immediately in the cu

Using go/

2015-07-29 Thread Martin Raison
go blocks tend to spread in Clojure programs just like async/await in C#/Hack/Python, etc. The problem is that they aren't cheap. I was curious to know what you guys think of the following workaround: http://blog.martinraison.com/clojure/2015/07/27/clojure-core-async-go-blocks-everywhere.html (T