Sweet, thanks everyone for your responses! This was very helpful! r/foldcat did the trick. It looks like it's faster than pmap by about 10%. Also, I'm happy to have my sanity back.
Thank you! Daniel On Wednesday, July 15, 2015 at 12:38:39 AM UTC-4, James Reeves wrote: > > Reducers and pmap have different approaches to concurrency. > > pmap works a lot like map, except that instead of evaluating the seq one > item at a time, it spins up N threads which work on the next N items in > parallel. N is 2 + the number of CPU cores you have. > > The reducers library works in a rather different way. It's important to > understand that the reducers library is split into reduce functions and > combine functions. The idea is that a collection is split up into equal > parts, then given to reduce functions in separate threads. Finally, all the > separate results are stuck back again by folding over the combine function. > > If you're just using something like: > > (into [] (r/map blur (vec indexes)) > > Then you're only using the reduce function. The parallelism is initiated > by the fold, so instead you want: > > (r/foldcat (r/map blur (vec indexes)) > > By default, this will split the vector into pieces 512 in length, map > "blur" over each piece in parallel, then fold the pieces back together > again using the r/cat function, which is designed to be performant at > concatenating the pieces of a fold back together again. > > My guess is that for your particular use-case, pmap is going to be faster. > The reducers library shines when you start to get into situations that pmap > can't handle. > > - James > > On 15 July 2015 at 04:02, Daniel Higginbotham <nonrec...@gmail.com > <javascript:>> wrote: > >> I’ve been trying to better understand ways to increase the performance of >> Clojure programs and I’ve run into an interesting issue with using the >> reducers library. I’d love any help! The code I’m working on is at >> https://github.com/flyingmachine/quil-pix/tree/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc >> >> The issue I’m having is that reducers/map has the same performance as >> core/map, and it doesn’t seem like that should be the case. (The current >> code ( >> https://github.com/flyingmachine/quil-pix/blob/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc/src/quil_pix/blur.clj#L73) >> >> uses mapv, but I’ve also tried just map.) pmap is ~3x faster. >> >> Surely reducers/map should be faster, but it doesn’t seem like I’ve done >> something wrong? This is driving me crazy :) >> >> Daniel >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> <javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
