Well it should be possible to implement a foldseq variant which takes a reducible collection as an input. This would speed things, as you don't create so much garbage with reducers. XML parser which produces reducible collection will be a bit harder :).
Anyway, I think the bottleneck in your code is at https://github.com/paulbutcher/parallel-word-count/blob/master/src/wordcount/core.clj#L9 Instead of creating new persistent map for each word, you should use a transient here. JW On Saturday, September 28, 2013 7:12:08 PM UTC+2, Paul Butcher wrote: > > On 28 Sep 2013, at 17:42, Jozef Wagner <jozef....@gmail.com <javascript:>> > wrote: > > I mean that you should forgot about lazy sequences and sequences in > general, if you want to have a cutting edge performance with reducers. > Example of reducible slurp, https://gist.github.com/wagjo/6743885 , does > not hold into the head. > > > OK - I buy that logic. But I'm unsure whether it can be applied to the > case that I'm interested in. > > Firstly, I'm dealing with an XML file, so I need to parse it. Right now > I'm using data.xml/parse to do so, which returns a lazy sequence. So, short > of writing my own XML parser, I have to go via a lazy sequence at some > point along the way? > > Secondly, the primary point of this is to exploit parallelism, not > reducers per-se. So supporting CollReduce isn't enough, I also need to > support CollFold. Which is exactly what foldable-seq does - I'm not sure > how that differs from what you're proposing? > > Am I missing something? > > -- > paul.butcher->msgCount++ > > Snetterton, Castle Combe, Cadwell Park... > Who says I have a one track mind? > > http://www.paulbutcher.com/ > LinkedIn: http://www.linkedin.com/in/paulbutcher > MSN: pa...@paulbutcher.com <javascript:> > AIM: paulrabutcher > Skype: paulrabutcher > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
