So this turned out to be pretty easy. I've implemented a function called "foldable-seq" that takes a lazy sequence and turns it into something that can be folded in parallel. I've checked an example program that uses it to count words in a Wikipedia XML dump into GitHub:
https://github.com/paulbutcher/foldable-seq The code for foldable-seq is here: https://github.com/paulbutcher/foldable-seq/blob/master/src/wordcount/reducers.clj#L60 On my 4-core MacBook Pro, I see a 40 second runtime without parallel-seq, 13 seconds with. -- paul.butcher->msgCount++ Snetterton, Castle Combe, Cadwell Park... Who says I have a one track mind? http://www.paulbutcher.com/ LinkedIn: http://www.linkedin.com/in/paulbutcher MSN: [email protected] AIM: paulrabutcher Skype: paulrabutcher On 11 Mar 2013, at 13:38, Paul Butcher <[email protected]> wrote: > On 11 Mar 2013, at 11:00, Marko Topolnik <[email protected]> wrote: > >> The idea is to transform into a lazy sequence of eager chunks. That approach >> should work. > > Exactly. Right - I guess I should put my money where my mouth is and see if I > can get it working... > > -- > paul.butcher->msgCount++ > > Snetterton, Castle Combe, Cadwell Park... > Who says I have a one track mind? > > http://www.paulbutcher.com/ > LinkedIn: http://www.linkedin.com/in/paulbutcher > MSN: [email protected] > AIM: paulrabutcher > Skype: paulrabutcher > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
