On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: > If it were implemented not by creating Java Threads for each task, but submitting a task to an ExecutorService...
As far as I understand, that's easy if you're willing to use it asynchronously: Push tasks on the queue, let the worker threads deal with them whenever they get around to it, and never block waiting for results within a task. I.e. a task could submit nested tasks to the queue, but it couldn't use them to compute partial results for its own use. If you want synchronous behavior, there are only two ways that I know of to do that: Either disallow nesting, like (await agent), or use an unbounded thread pool, like pmap. If you allow unbounded nesting of "submit task and wait for the result" on a fixed size thread pool, I'm pretty sure you'll end up with deadlock. (Which is why agents disallow it, I assume.) Imagine a 1-thread ExecutorService where the task running on the single work thread submits a new task to the service and blocks until it receives a result: The only thread that can process that task just went to sleep, and it won't wake up until it receives a result that there are no threads left to compute. The issue is the same with more worker threads; except that it will work, more or less, as long as at least one thread is still awake. But the scenario where every worker thread submits-and-waits simultaneously is bound to happen at some point. With the disclaimer that I might be missing something, jf On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: > > One benefit would be convenience of enabling parallelism on nested data > structures. One function at the top level could use parallelism, and the > pieces, perhaps handled by separate functions, and perhaps nested several > levels deep in function calls, could also use parallelism. > > If it were implemented not by creating Java Threads for each task, but > submitting a task to an ExecutorService, then the actual number of active > Java Threads could be kept reasonably low (e.g. maybe 2 times the number of > physical CPU cores), whereas the number of parallel tasks the work is > divided into could be limited only by memory for storing the tasks scheduled > for future execution. > > Andy > > > On Tue, Oct 11, 2011 at 10:55 AM, j-g-faustus <johannes...@gmail.com>wrote: > >> On Tuesday, October 11, 2011 3:55:09 AM UTC+2, Lee wrote: >>> >>> >>> Does your pmap-pool permit nesting? (That is, does it permit passing >>> pmap-pool a function which itself calls pmap-pool?). If so then that would >>> be a reason to prefer it over my pmapall. >>> >> I expect it would be possible to nest it (possible as in "no exceptions or >> deadlocks"), but I can't see any scenario where you would want to - you >> would get an exponentially increasing number of threads. If 48 cores each >> start 48 threads, each of those threads start another 48 etc., it doesn't >> take long before you have enough threads to bog down even the most powerful >> server. >> >> But what would be the purpose of a nested "run this on all cores" >> construct? You are already running on all cores, there are no spare >> resources, so in terms of CPU time I can't see how it would differ from >> merely having the pmapped function use a plain same-thread map? >> >> There are no particular advantages to pmap-pool over pmapall that I can >> see, except that pmap-pool lets you control the number of threads more >> easily. (E.g. for debugging "where does it stop scaling linearly" problems.) >> I thought creating thousands of agents (as pmapall does) would be more >> expensive, so I tried an alternative, but in practice they seem to be >> equally fast; at least on this test and the kind of hardware I have access >> to. >> So pmapall wins by having fewer lines of code. >> >> >> jf >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
