Sounds like a job for a future. Something like: (->> job-list (partition-in-sublists 4) (map #(future (do-job-on-sublist %))) (mapv deref))
This is untested and written on a phone, so might not even be syntactically correct, but the future calls will create new threds to execute the do-job functions on the sublists, and the deref call on a future is blocking. If the result from the futures is not important, I guess you could save some memory by using map and doall instead of mapv, though I doubt it would make any difference if you have only 4 sublists. Alternatively, have you looked at pmap ? On Wednesday, 17 September 2014, Beau Fabry <imf...@gmail.com> wrote: > We don't have streams of data here, the long running tasks have > side-effects. I would prefer to avoid adding another whole framework just > to run a few long running jobs in p//. > > I have a list of jobs to do, I'm partitioning that list up into 4 sub > lists to be worked through by 4 p// workers, I then want to block and wait > until all 4 workers have finished their tasks. > > On Wednesday, September 17, 2014 3:27:07 AM UTC+10, larry google groups > wrote: >> >> >> This does not look correct to me. Perhaps someone else has more insight >> into this. I am suspicious about 2 things: >> >> 1.) your use of doall >> >> 2.) your use of (thread) >> >> It looks to me like you are trying to hack together a kind of pipeline or >> channel. Clojure has a wealth of libraries that can handle that for you. >> The main thing you are trying to do is this: >> >> (long-running-widget-processor widget)) >> >> >> You go to some trouble to set up workers, all to ensure that >> long-running-widget-processor is handled in its own thread. >> >> I would suggest you look at Lamina: >> >> https://github.com/ztellman/lamina >> >> In particular, look at pipelines: >> >> https://github.com/ztellman/lamina/wiki/Pipelines >> >> >> >> >> >> On Friday, September 5, 2014 1:46:02 AM UTC-4, Beau Fabry wrote: >>> >>> Is the kinda ugly constant (doall usage a sign that I'm doing something >>> silly? >>> >>> (let [num-workers 4 >>> widgets-per-worker (inc (int (/ (count widgets) num-workers))) >>> bucketed-widgets (partition-all widgets-per-worker widgets) >>> workers (doall (map (fn [widgets] >>> (thread >>> (doseq [widget widgets] >>> (long-running-widget-processor widget)) >>> true)) >>> bucketed-widgets))] >>> (doall (map <!! workers))) >>> >>> https://gist.github.com/bfabry/ad830b1888e4fc550f88 >>> >>> All comments appreciated :-) >>> >>> Cheers, >>> Beau >>> >> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > <javascript:_e(%7B%7D,'cvml','clojure@googlegroups.com');> > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','clojure%2bunsubscr...@googlegroups.com');> > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','clojure%2bunsubscr...@googlegroups.com');>. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
