Re: Spread work onto multiple threads (in pure Clojure)

Sorry I meant "Fair enough ..." Cheers, Ivan. On 13 October 2011 12:41, Ivan Koblik wrote: > Fail enough. I guess, to allow nested spawning and avoid deadlocks, tasks > should finish without waiting for the result but spawn a new task similar to > itself that would check for the completion of

Re: Spread work onto multiple threads (in pure Clojure)

Fail enough. I guess, to allow nested spawning and avoid deadlocks, tasks should finish without waiting for the result but spawn a new task similar to itself that would check for the completion of the child tasks, repeating the cycle if necessary. Imagine that task t1 is monitored through a future

Re: Spread work onto multiple threads (in pure Clojure)

On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: > If it were implemented not by creating Java Threads for each task, but submitting a task to an ExecutorService... As far as I understand, that's easy if you're willing to use it asynchronously: Push tasks on the queue, let th

Re: Spread work onto multiple threads (in pure Clojure)

On Oct 12, 2011, at 4:41 PM, Avram wrote: > I haven't read the entire thread carefully, but have you considered > the "work" library as a potential > fit for what you are trying to do? I hadn't heard of it, but it does look promising and I will check it out. T

Re: Spread work onto multiple threads (in pure Clojure)

I haven't read the entire thread carefully, but have you considered the "work" library as a potential fit for what you are trying to do? Example from the github link: user> (require '[work.core :as work]) user> (work/map-work client/get ["http://clojure.org"; "h

Re: Spread work onto multiple threads (in pure Clojure)

> On Tue, Oct 11, 2011 at 10:55 AM, j-g-faustus > wrote: >> I expect it would be possible to nest it (possible as in "no exceptions or >> deadlocks"), but I can't see any scenario where you would want to - you >> would get an exponentially increasing number of threads. If 48 cores each >> sta

Re: Spread work onto multiple threads (in pure Clojure)

Lee Spector writes: Hi Lee, >> So indeed, one starts with the number of available processors + 2, >> and one single longer running task will wipe out any parallelism. :-( >> >> IMO, that's not what 99% of the users would expect nor want when >> calling (pmap foo coll). I'd vote for making pmap

Re: Spread work onto multiple threads (in pure Clojure)

One benefit would be convenience of enabling parallelism on nested data structures. One function at the top level could use parallelism, and the pieces, perhaps handled by separate functions, and perhaps nested several levels deep in function calls, could also use parallelism. If it were implemen

Re: Spread work onto multiple threads (in pure Clojure)

On Tuesday, October 11, 2011 3:55:09 AM UTC+2, Lee wrote: > > > Does your pmap-pool permit nesting? (That is, does it permit passing > pmap-pool a function which itself calls pmap-pool?). If so then that would > be a reason to prefer it over my pmapall. > I expect it would be possible to nest it

Re: Spread work onto multiple threads (in pure Clojure)

On Oct 11, 2011, at 7:56 AM, Tassilo Horn wrote: > So indeed, one starts with the number of available processors + 2, and > one single longer running task will wipe out any parallelism. :-( > > IMO, that's not what 99% of the users would expect nor want when calling > (pmap foo coll). I'd vote f

Re: Spread work onto multiple threads (in pure Clojure)

Andy Fingerhut writes: Hi Andy, > pmap will limit the maximum number of simultaneous threads. So will > the medusa library's medusa-pmap. > > The difference is that if one job early in the list takes, e.g., 100 > times longer than the next 20, and you have 4 cpus available, pmap > will start th

Re: Spread work onto multiple threads (in pure Clojure)

Just to throw in my two cents here: it would really be a shame for Clojure's pmap to continue to languish in the state it's currently in (i.e. nearly useless). Even though implementing an alternative using thread pools isn't too hard, it hurts the Clojure concurrency sales pitch-- I can't tell p

Re: Spread work onto multiple threads (in pure Clojure)

I'll post more on this later, but I wanted to point out one case where I found that pmap was not achieving the desired level of speedup (# of CPUs/cores) that you would initially expect, and it is not due to any reasons that I've posted about before. Imagine a 4-core CPU. There are 4 physical CPU

Re: Spread work onto multiple threads (in pure Clojure)

Interesting. I'll try some of your suggested tests to see if my pmapall all is behaving better than I thought. Does your pmap-pool permit nesting? (That is, does it permit passing pmap-pool a function which itself calls pmap-pool?). If so then that would be a reason to prefer it over my pmapal

Re: Spread work onto multiple threads (in pure Clojure)

I made an alternative implementation using a thread pool and a queue, based on the example at http://clojure.org/concurrent_programming In short, your pmapall and the pool-based implementation (below) both give approximately perfect scaling on my 4/8-core system (Intel i7 920 and HT). Both giv

Re: Spread work onto multiple threads (in pure Clojure)

On Oct 10, 2011, at 7:16 PM, Phil Hagelberg wrote: > What you're really looking for is pdoseq, right? Seems like futures > might be a better building-block for this, although again Clojure's > lack of flexibility over the thread pool could easily bite you here. No -- I want all of the returned va

Re: Spread work onto multiple threads (in pure Clojure)

On Mon, Oct 10, 2011 at 1:07 PM, Lee Spector wrote: > Here's my stab at doing this using agents: > > (defn pmapall >  "Like pmap but: 1) coll should be finite, 2) the returned sequence >   will not be lazy, 3) calls to f may occur in any order, to maximize >   multicore processor utilization, and

Re: Spread work onto multiple threads (in pure Clojure)

A weakness of my pmapall: # Which means, I think, that I can't call pmapall within a function that I pass to pmapall. Unfortunate. Is there a better way? -Lee PS to see these exceptions one must change the call to agent in my definition with something like #(agent % :error-handler (fn [agn

Re: Spread work onto multiple threads (in pure Clojure)

I think that the following partially answers my own question and that it provides a way to get decent multicore performance for collections of non-uniform but compute-intensive tasks through a simple, pmap-like interface. But I'm not sure if it's the best approach and I'd like some feedback. If

Re: Spread work onto multiple threads (in pure Clojure)

I've been playing with medusa and it sometimes does what I expect, but sometimes it's doing something strange and I'm wondering if someone can help me to do one specific medusa-like thing but more simply (and without the strangeness, which I haven't fully traced but I hope to avoid having to).

Re: Spread work onto multiple threads (in pure Clojure)

I don't know whether there are similar limitations of parallelism when launching threads via sends to agents. I haven't looked at that yet. If you have an example program you can share, preferably trimmed down to the core of the issue, I might be able to look at it. I only know about pmap perfor

Re: Spread work onto multiple threads (in pure Clojure)

Thanks for this info -- I didn't realize quite how pmap worked. I often launch parallel threads with pmap and have sometimes been puzzled by dips in processor utilization that I can't trace to memory resource contention, etc. I have similar issues sometimes when I launch parallel threads via

Re: Spread work onto multiple threads (in pure Clojure)

pmap will limit the maximum number of simultaneous threads. So will the medusa library's medusa-pmap. The difference is that if one job early in the list takes, e.g., 100 times longer than the next 20, and you have 4 cpus available, pmap will start the first (4+2)=6 in parallel threads, let jobs

Re: Spread work onto multiple threads (in pure Clojure)

if the list is quite large, is there anything preventing the option below from creating way too many threads? there is presumably an inflection point where this will cost you. pmap is trying to keep that in check, right? Kevin On Sep 21, 5:28 pm, Nathan Sorenson wrote: > Futures begin executing

Re: Spread work onto multiple threads (in pure Clojure)

Sorry for my delayed reply (busy day), Andy: Ill check Medusa, the number of threads in pmap does impose a limit but in my case I can live with that. Nathan: Yes the reason I wanted futures because of the blocking IO, I don't know why I figured out that futures are lazy, this indeed makes pmap re

Re: Spread work onto multiple threads (in pure Clojure)

On Wed, Sep 21, 2011 at 6:06 PM, ronen wrote: > (defn email-approved [approved] >  (doall (pmap deref (for [req approved] (future (email-request > req)) Wouldn't the following be close enough to what you want? (defn email-approved [approved] (doall (pmap email-request approved))) -- Sean

Re: Spread work onto multiple threads (in pure Clojure)

Futures begin executing their contents immediately, you don't have to deref them to trigger the side effects. (perhaps you were thinking of delay?) I'm assuming you are using futures because email-request is an io- blocking operation? The thing to note is that the body of a future automatically ru

Re: Spread work onto multiple threads (in pure Clojure)

pmap already uses future/deref in its implementation. When it does so, it limits the *maximum* parallelism possible to be at most (number-of-cpus + 2) threads running at a time, and depending upon the time taken by each of the objects you are mapping over, it can be less then that. I don't know i

Spread work onto multiple threads (in pure Clojure)

I was looking for a pure Clojure solution to run multiple non- dependant tasks on multiple threads, iv considered using Agent, Promises or Futures, yet the simplest cleanest succinct solution iv found is: (defn email-approved [approved] (doall (pmap deref (for [req approved] (future (email-reque