I have used pipeline before, and when working with channels it is quite convenient. But transducers are useful for many cases outside the use of channels and core.async. And pipeline cannot compose with transducers, afaik. So for the situation I describe, it seems to me that using pipeline would introduce even further complexity. I would start with comp and transducers, and wind up with a let, two local temporary channels, etc. etc. I don't see what pipeline would buy me that current core.async pmap would not already provide.
It's a bit of a simple case for parallelism, but then again so is current `pmap` behavior. Thanks, ~Timothy Dean On Fri, May 26, 2017 at 11:47 AM, Gary Trakhman <gary.trakh...@gmail.com> wrote: > Have you looked into core.async's 'pipeline' function for parallelism? > That was my first question when I saw this post. I'm not sure why there's > a need for pmap. > > On Fri, May 26, 2017 at 12:29 PM Timothy Dean <timothysd...@gmail.com> > wrote: > >> You can also get an indication this way, I think: >> >> user=> (defn f [x] (Thread/sleep 100) x) >> #'user/f >> user=> (time (doall (pmap f (range 32)))) >> "Elapsed time: 128.383719 msecs" >> user=> (time (doall (pmap f (range 33)))) >> "Elapsed time: 201.462131 msecs" >> user=> (time (doall (pmap f (range 64)))) >> "Elapsed time: 201.89915 msecs" >> user=> (time (doall (pmap f (range 65)))) >> "Elapsed time: 301.711263 msecs" >> user=> (time (doall (pmap f (range 96)))) >> "Elapsed time: 302.205999 msecs" >> user=> (time (doall (pmap f (range 97)))) >> "Elapsed time: 402.236708 msecs" >> >> Regarding motivation for having a pmap transducer, my own is mainly that >> it composes with other transducers. A major practical/implementation >> "triumph" of transducers may be the reduced garbage, but there is a benefit >> of simplified expression as well, since transducers encode transformation >> in a manner agnostic to many elements of the context of the transformation. >> In a project containing many transducers variously composed and consumed >> throughout, it may be that I wish to parallelize one portion of a >> transformation occurring in the middle of a composition of transducers. If >> so, I can't simply drop in clojure.core's pmap. >> >> Instead I now must -- in the specific context of whatever higher-order >> transducing function I happen to be using -- break the composed transducers >> apart, generate a transitory sequence, pmap over that, and then proceed >> with transforming the temporary sequence. That is, in this case I'm not >> concerned so much with the resource overhead of swapping a transducer for a >> non-transducer, but the expressive overhead. Composition is elegant. My >> solutions with clojure.core's pmap are clunky. I'm not ruling out PEBCAK >> error, but a pmap transducer seems to me the "obvious" solution preserving >> compositional elegance. >> >> For instance, a very simple example using transduce: >> >> (transduce (comp (remove bad) >> >> ... >> >> (map transform) >> >> ... >> >> (keep pred)) >> >> reducer >> >> init coll) >> >> >> must become something like: >> >> (->> coll >> >> (sequence (comp (remove bad) >> >> ...)) >> >> (pmap transform) >> >> (transduce (comp ... >> >> (keep pred)) >> >> reducer >> >> init)) >> >> But it could be as simple as: >> >> (transduce (comp (remove bad) >> >> ... >> >> (pmap transform) >> >> ... >> >> (keep pred)) >> >> reducer >> >> init coll) >> >> Does that make sense, or am I crazy? :) >> >> ~Timothy Dean >> >> On Friday, May 26, 2017 at 3:10:15 AM UTC-6, Matching Socks wrote: >>> >>> With the overhead of threading, isn't the triumph of a transducer (no >>> seq allocations) rather subtle in the case of pmap? >>> >>> >>> At any rate!, as a point of interest, since you mentioned a quirk of >>> pmap's thread usage: It apparently has to do with whether the input >>> sequence is a >>> chunked sequence or not. >>> >>> Comparing two possible inputs >>> >>> user> (def c (into [] (range 50))) ;; chunked >>> >>> user> (def n (into '() (range 50))) ;; not chunked >>> >>> With pmap invoking a function that just returns the thread's name (and >>> takes some time, so as to keep its assigned thread busy enough not to >>> accept more tasks right away) >>> >>> user> (defn f [x] (Thread/sleep 22) (.getName (Thread/currentThread))) >>> #'user/f >>> user> (count (distinct (pmap f c))) >>> 32 >>> user> (count (distinct (pmap f n))) >>> 7 >>> >>> The test is not perfect because it does not show that all 32 >>> threads were used *at once*. There were simply 32 distinct threads used >>> at one >>> point or another. Nonetheless the difference between 7 and 32 is >>> suggestive. >>> >>> In this environment there are 4 "processors", so 32 is indeed more than >>> you might want. >>> >>> The docstring about pmap could be clearer about this. >>> >>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to a topic in the > Google Groups "Clojure" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/clojure/vLZnWJCtL_U/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
