Re: Parallelising reduce-kv - r/fold into maps

[Wolodja Wentland]

On Tue, Oct 09, 2012 at 15:31 -0700, Jean Niklas L'orange wrote:
>On Tuesday, October 9, 2012 9:14:51 PM UTC+2, Wolodja Wentland wrote:
> 
>  --- snip ---
>  (reduce-kv
>   (fn [ret k v]
>    (assoc ret k (func-that-does-something-with v)))
>   some-map))
>  --- snip --
> 
>  I am using reducers in other places, but am not entirely sure how to
>  employ
>  them here. I would be grateful for any advice.
> 
>You could use merge as the combine function, and let (fn [ret [k
>v]] (assoc ret k (func-that-does-something-with v))) be the reducing
>function you pass to fold. Then it's at least parallelisable, though I'm
>unsure of its performance.

I played with the following implementation:

(defn update-vals
  [m f]
  (persistent!
(reduce-kv
  (fn [ret k v]
(assoc! ret k (f v)))
  (transient {})
  m)))

(defn pupdate-vals
  [m f]
  (r/fold
(r/monoid merge hash-map)
(fn [ret [k v]]
  (conj ret [k (f v)]))
(vec m)))

(defn foo
  [x]
  (Thread/sleep 0.2)
  (inc x))

but unfortunately update-vals still outperforms pupdate-vals even on large
maps. I am not sure what else I can do to speed up folding into maps
(as opposed to a single value), but I'll keep on experimenting.

user=> (def m (zipmap (range 10) (range 10)))
#'user/m
user=> (time (dotimes [_ 100] (pupdate-vals m inc)))
"Elapsed time: 8068.950282 msecs"
nil
user=> (time (dotimes [_ 100] (update-vals m inc)))
"Elapsed time: 1106.253256 msecs"
nil
user=> (time (dotimes [_ 100] (pupdate-vals m foo)))
"Elapsed time: 8890.445481 msecs"
nil
user=> (time (dotimes [_ 100] (update-vals m foo)))
"Elapsed time: 3954.917294 msecs"
-- 
Wolodja 

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature

Re: Parallelising reduce-kv

[Jean Niklas L'orange]

On Tuesday, October 9, 2012 9:14:51 PM UTC+2, Wolodja Wentland wrote:
>
> --- snip --- 
> (reduce-kv 
>  (fn [ret k v] 
>   (assoc ret k (func-that-does-something-with v))) 
>  some-map)) 
> --- snip -- 
>
> I am using reducers in other places, but am not entirely sure how to 
> employ 
> them here. I would be grateful for any advice. 
>

You could use merge as the combine function, and let (fn [ret [k v]] (assoc 
ret k (func-that-does-something-with v))) be the reducing function you pass 
to fold. Then it's at least parallelisable, though I'm unsure of its 
performance.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Parallelising reduce-kv

[Wolodja Wentland]

Hi all,

I've noticed that I wrote the following recently and wonder if this can be
done in a more succinct/idiomatic way and also if it can be parallelised:

--- snip ---
(reduce-kv
 (fn [ret k v]
  (assoc ret k (func-that-does-something-with v)))
 some-map))
--- snip --

I am using reducers in other places, but am not entirely sure how to employ
them here. I would be grateful for any advice.
-- 
Wolodja 

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature