Fail enough. I guess, to allow nested spawning and avoid deadlocks, tasks should finish without waiting for the result but spawn a new task similar to itself that would check for the completion of the child tasks, repeating the cycle if necessary.
Imagine that task t1 is monitored through a future f1, now it spawns child task t2 which is monitored through a future f2. t1 doesn't wait for t2 to complete, but reschedules a new task t3 that would check f2 for completion, and if f2 is done then sets f1 to done, otherwise reschedules itself as t4. Although I don't know how memory hungry it may get and what would be the impact on the length of the task queue. Maybe we could make the queue of limited length and if new task t2 gets rejected then it is performed within the task t1? Cheers, Ivan. On 13 October 2011 00:21, j-g-faustus <johannes.fries...@gmail.com> wrote: > On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: > > If it were implemented not by creating Java Threads for each task, but > submitting a task to an ExecutorService... > > As far as I understand, that's easy if you're willing to use it > asynchronously: Push tasks on the queue, let the worker threads deal with > them whenever they get around to it, and never block waiting for results > within a task. > I.e. a task could submit nested tasks to the queue, but it couldn't use > them to compute partial results for its own use. > > If you want synchronous behavior, there are only two ways that I know of to > do that: Either disallow nesting, like (await agent), or use an unbounded > thread pool, like pmap. > > If you allow unbounded nesting of "submit task and wait for the result" on > a fixed size thread pool, I'm pretty sure you'll end up with deadlock. > (Which is why agents disallow it, I assume.) > Imagine a 1-thread ExecutorService where the task running on the single > work thread submits a new task to the service and blocks until it receives a > result: The only thread that can process that task just went to sleep, and > it won't wake up until it receives a result that there are no threads left > to compute. > The issue is the same with more worker threads; except that it will work, > more or less, as long as at least one thread is still awake. But the > scenario where every worker thread submits-and-waits simultaneously is bound > to happen at some point. > > > With the disclaimer that I might be missing something, > > jf > > > On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: > >> One benefit would be convenience of enabling parallelism on nested data >> structures. One function at the top level could use parallelism, and the >> pieces, perhaps handled by separate functions, and perhaps nested several >> levels deep in function calls, could also use parallelism. >> >> If it were implemented not by creating Java Threads for each task, but >> submitting a task to an ExecutorService, then the actual number of active >> Java Threads could be kept reasonably low (e.g. maybe 2 times the number of >> physical CPU cores), whereas the number of parallel tasks the work is >> divided into could be limited only by memory for storing the tasks scheduled >> for future execution. >> >> Andy >> >> >> On Tue, Oct 11, 2011 at 10:55 AM, j-g-faustus <johannes...@gmail.com>wrote: >> >>> On Tuesday, October 11, 2011 3:55:09 AM UTC+2, Lee wrote: >>>> >>>> >>>> Does your pmap-pool permit nesting? (That is, does it permit passing >>>> pmap-pool a function which itself calls pmap-pool?). If so then that would >>>> be a reason to prefer it over my pmapall. >>>> >>> I expect it would be possible to nest it (possible as in "no exceptions >>> or deadlocks"), but I can't see any scenario where you would want to - you >>> would get an exponentially increasing number of threads. If 48 cores each >>> start 48 threads, each of those threads start another 48 etc., it doesn't >>> take long before you have enough threads to bog down even the most powerful >>> server. >>> >>> But what would be the purpose of a nested "run this on all cores" >>> construct? You are already running on all cores, there are no spare >>> resources, so in terms of CPU time I can't see how it would differ from >>> merely having the pmapped function use a plain same-thread map? >>> >>> There are no particular advantages to pmap-pool over pmapall that I can >>> see, except that pmap-pool lets you control the number of threads more >>> easily. (E.g. for debugging "where does it stop scaling linearly" problems.) >>> I thought creating thousands of agents (as pmapall does) would be more >>> expensive, so I tried an alternative, but in practice they seem to be >>> equally fast; at least on this test and the kind of hardware I have access >>> to. >>> So pmapall wins by having fewer lines of code. >>> >>> >>> jf >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com >>> >>> For more options, visit this group at >>> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en> >>> >> >> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
