I haven't read the entire thread carefully, but have you considered the "work" library <https://github.com/getwoven/work> as a potential fit for what you are trying to do?
Example from the github link: user> (require '[work.core :as work]) user> (work/map-work client/get ["http://clojure.org"; "http:// github.com/richhickey/clojure"] 2) user> (doc work/map-work) ------------------------- work.core/map-work ([f xs threads]) like clojure's map or pmap, but takes a number of threads, executes eagerly, and blocks. Hope this is useful. -A On Oct 11, 12:51 pm, Lee Spector <lspec...@hampshire.edu> wrote: > > On Tue, Oct 11, 2011 at 10:55 AM, j-g-faustus <johannes.fries...@gmail.com> > > wrote: > >> I expect it would be possible to nest it (possible as in "no exceptions or > >> deadlocks"), but I can't see any scenario where you would want to - you > >> would get an exponentially increasing number of threads. If 48 cores each > >> start 48 threads, each of those threads start another 48 etc., it doesn't > >> take long before you have enough threads to bog down even the most > >> powerful server. > > >> But what would be the purpose of a nested "run this on all cores" > >> construct? You are already running on all cores, there are no spare > >> resources, so in terms of CPU time I can't see how it would differ from > >> merely having the pmapped function use a plain same-thread map? > > On Oct 11, 2011, at 2:07 PM, Andy Fingerhut wrote: > > > One benefit would be convenience of enabling parallelism on nested data > > structures. One function at the top level could use parallelism, and the > > pieces, perhaps handled by separate functions, and perhaps nested several > > levels deep in function calls, could also use parallelism. > > Or consider the following scenario (which is exactly what I was doing :-): > You want to produce the next generation of programs in an evolutionary > computation system from n parents, each of which may produce m offspring, > with n much larger than your number of cores (c). The runtimes for offspring > production may vary widely. So you do something like (pmapall reproduce > population) to maximize processor utilization among the parents, but late in > the process, when the number of parents who are still busy making offspring > is less than c, your cores start to go idle. Suppose you have just a few > remaining parents who are much slower than the others, and each has to chug > through m independent birth processes with cores sitting idle. If you could > nest calls to pmapall then those slowpokes would be farming some of their m > births out to the available cores. > > In this situation, at least, I don't care about the order in which the (* n > m) independent tasks are completed. I just want them all done, with as little > wasted CPU capacity as possible. And I don't want to somehow factor out all > of the births into one long queue that I pass to a single call to pmapall. > The natural decomposition of the problem is to say something like (pmapall > reproduce population) at the population level, something like (pmapall (fn > [_] (make-baby parent)) (range litter-size)) at the parent level, and ask the > platform to make the calls as aggressively as possible, reusing cores for > pending calls whenever possible. > > Calls to pmap can be nested and I *think* that they will actually do some of > this, if the sequencing of fast vs slow things is fortuitous. For example, I > *think* that in the some of the situations that have been described here, > where some cores are idle waiting for a slow, early computation in a sequence > to complete, a pmap from within the slow, early computation will use some of > those available cores. I'm not certain of this, however! And in any event > it's not ideal for the reasons that have been raised. > > -Lee -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
