Sorry I meant "Fair enough ..." Cheers, Ivan.
On 13 October 2011 12:41, Ivan Koblik <ivankob...@gmail.com> wrote: > Fail enough. I guess, to allow nested spawning and avoid deadlocks, tasks > should finish without waiting for the result but spawn a new task similar to > itself that would check for the completion of the child tasks, repeating the > cycle if necessary. > > Imagine that task t1 is monitored through a future f1, now it spawns child > task t2 which is monitored through a future f2. t1 doesn't wait for t2 to > complete, but reschedules a new task t3 that would check f2 for completion, > and if f2 is done then sets f1 to done, otherwise reschedules itself as t4. > > Although I don't know how memory hungry it may get and what would be the > impact on the length of the task queue. Maybe we could make the queue of > limited length and if new task t2 gets rejected then it is performed within > the task t1? > > Cheers, > Ivan. > > > > On 13 October 2011 00:21, j-g-faustus <johannes.fries...@gmail.com> wrote: > >> On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: >> > If it were implemented not by creating Java Threads for each task, but >> submitting a task to an ExecutorService... >> >> As far as I understand, that's easy if you're willing to use it >> asynchronously: Push tasks on the queue, let the worker threads deal with >> them whenever they get around to it, and never block waiting for results >> within a task. >> I.e. a task could submit nested tasks to the queue, but it couldn't use >> them to compute partial results for its own use. >> >> If you want synchronous behavior, there are only two ways that I know of >> to do that: Either disallow nesting, like (await agent), or use an unbounded >> thread pool, like pmap. >> >> If you allow unbounded nesting of "submit task and wait for the result" on >> a fixed size thread pool, I'm pretty sure you'll end up with deadlock. >> (Which is why agents disallow it, I assume.) >> Imagine a 1-thread ExecutorService where the task running on the single >> work thread submits a new task to the service and blocks until it receives a >> result: The only thread that can process that task just went to sleep, and >> it won't wake up until it receives a result that there are no threads left >> to compute. >> The issue is the same with more worker threads; except that it will work, >> more or less, as long as at least one thread is still awake. But the >> scenario where every worker thread submits-and-waits simultaneously is bound >> to happen at some point. >> >> >> With the disclaimer that I might be missing something, >> >> jf >> >> >> On Tuesday, October 11, 2011 8:07:16 PM UTC+2, Andy Fingerhut wrote: >> >>> One benefit would be convenience of enabling parallelism on nested data >>> structures. One function at the top level could use parallelism, and the >>> pieces, perhaps handled by separate functions, and perhaps nested several >>> levels deep in function calls, could also use parallelism. >>> >>> If it were implemented not by creating Java Threads for each task, but >>> submitting a task to an ExecutorService, then the actual number of active >>> Java Threads could be kept reasonably low (e.g. maybe 2 times the number of >>> physical CPU cores), whereas the number of parallel tasks the work is >>> divided into could be limited only by memory for storing the tasks scheduled >>> for future execution. >>> >>> Andy >>> >>> >>> On Tue, Oct 11, 2011 at 10:55 AM, j-g-faustus <johannes...@gmail.com>wrote: >>> >>>> On Tuesday, October 11, 2011 3:55:09 AM UTC+2, Lee wrote: >>>>> >>>>> >>>>> Does your pmap-pool permit nesting? (That is, does it permit passing >>>>> pmap-pool a function which itself calls pmap-pool?). If so then that would >>>>> be a reason to prefer it over my pmapall. >>>>> >>>> I expect it would be possible to nest it (possible as in "no exceptions >>>> or deadlocks"), but I can't see any scenario where you would want to - you >>>> would get an exponentially increasing number of threads. If 48 cores each >>>> start 48 threads, each of those threads start another 48 etc., it doesn't >>>> take long before you have enough threads to bog down even the most powerful >>>> server. >>>> >>>> But what would be the purpose of a nested "run this on all cores" >>>> construct? You are already running on all cores, there are no spare >>>> resources, so in terms of CPU time I can't see how it would differ from >>>> merely having the pmapped function use a plain same-thread map? >>>> >>>> There are no particular advantages to pmap-pool over pmapall that I can >>>> see, except that pmap-pool lets you control the number of threads more >>>> easily. (E.g. for debugging "where does it stop scaling linearly" >>>> problems.) >>>> I thought creating thousands of agents (as pmapall does) would be more >>>> expensive, so I tried an alternative, but in practice they seem to be >>>> equally fast; at least on this test and the kind of hardware I have access >>>> to. >>>> So pmapall wins by having fewer lines of code. >>>> >>>> >>>> jf >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clo...@googlegroups.com >>>> >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+u...@googlegroups.com >>>> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en> >>>> >>> >>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
