Re: Reducers exception when calling into

[Matt Grimm]

Thanks very much for the quick response. From your hints, I tried switching 
uses of into to r/foldcat and it worked. 

On Thursday, May 11, 2017 at 4:44:37 PM UTC-6, Alex Miller wrote:
>
> The symptoms sound like you are running into a case where there are 
> multiple protocol branches that apply - this is a case where the behavior 
> is undefined and can vary between JVM executions.
>
> Looks like it's going down the seq reduce path. I'm not at a computer to 
> check more precisely what the other preferred option might be. I think 
> there's actually an open ticket about a similar case.
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reducers exception when calling into

[Alex Miller]

The symptoms sound like you are running into a case where there are multiple 
protocol branches that apply - this is a case where the behavior is undefined 
and can vary between JVM executions.

Looks like it's going down the seq reduce path. I'm not at a computer to check 
more precisely what the other preferred option might be. I think there's 
actually an open ticket about a similar case.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reducers exception when calling into

[Matt Grimm]

Hi,

I've run into a bizarre issue with reducers that is consistently 
reproducible, but only on one set of systems. On a separate system with the 
same OS and java, the error is not reproducible. The exception messages 
seem to indicate that I've tried to access a reducible like a seq before 
doing something like `into', but that's not the case. In fact, the 
exception is thrown at the very line where I'm trying to call into on the 
reducible. 

The exception happens in a reducers chain like this:
(->> content
(r/map :vals)
r/flatten
(r/filter (comp #{1 2 3} :page-number))
(into []) ;; <-- exception occurs at this line 
count)

Stack trace:
https://gist.github.com/tkocmathla/2d4bc38c2ac2977b6c421fc92cd812ed

$ java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS" 

Thanks for any insight,
m.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers and sets/PersistentSet

[Alex Miller]

Get cracking! :)

On Tuesday, November 10, 2015 at 10:44:33 AM UTC-6, retnuH wrote:
>
> That's what I was thinking...
>
> I may try to put together a patch.
>
> But it may have to wait until after Clojure/Conj!  Still don't have 
> presentation finished...
>
> H
>
> On Tuesday, 10 November 2015 16:26:58 UTC, Alex Miller wrote:
>>
>> Persistent sets are really persistent maps under the hood, so in 
>> principle doesn't seem like there's any reason that shouldn't work.
>>
>> On Tuesday, November 10, 2015 at 10:23:19 AM UTC-6, retnuH wrote:
>>>
>>> Hi there, I was playing with reducers recently, and was wondering why 
>>> PersistentSets aren't foldable (i.e. don't take advantage of the Fork/Join 
>>> stuff?
>>>
>>> Is there some fundamental reason why it wouldn't work (or is especially 
>>> difficult), or just a case that no one has gotten around to implementing it?
>>>
>>> Cheers,
>>>
>>> H
>>>
>>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers and sets/PersistentSet

[retnuH]

That's what I was thinking...

I may try to put together a patch.

But it may have to wait until after Clojure/Conj!  Still don't have 
presentation finished...

H

On Tuesday, 10 November 2015 16:26:58 UTC, Alex Miller wrote:
>
> Persistent sets are really persistent maps under the hood, so in principle 
> doesn't seem like there's any reason that shouldn't work.
>
> On Tuesday, November 10, 2015 at 10:23:19 AM UTC-6, retnuH wrote:
>>
>> Hi there, I was playing with reducers recently, and was wondering why 
>> PersistentSets aren't foldable (i.e. don't take advantage of the Fork/Join 
>> stuff?
>>
>> Is there some fundamental reason why it wouldn't work (or is especially 
>> difficult), or just a case that no one has gotten around to implementing it?
>>
>> Cheers,
>>
>> H
>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers and sets/PersistentSet

[Alex Miller]

Persistent sets are really persistent maps under the hood, so in principle 
doesn't seem like there's any reason that shouldn't work.

On Tuesday, November 10, 2015 at 10:23:19 AM UTC-6, retnuH wrote:
>
> Hi there, I was playing with reducers recently, and was wondering why 
> PersistentSets aren't foldable (i.e. don't take advantage of the Fork/Join 
> stuff?
>
> Is there some fundamental reason why it wouldn't work (or is especially 
> difficult), or just a case that no one has gotten around to implementing it?
>
> Cheers,
>
> H
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reducers and sets/PersistentSet

[retnuH]

Hi there, I was playing with reducers recently, and was wondering why 
PersistentSets aren't foldable (i.e. don't take advantage of the Fork/Join 
stuff?

Is there some fundamental reason why it wouldn't work (or is especially 
difficult), or just a case that no one has gotten around to implementing it?

Cheers,

H

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers question

[Alan Busby]

On Wed, Jul 15, 2015 at 12:02 PM, Daniel Higginbotham <
nonrecurs...@gmail.com> wrote:

> Surely reducers/map should be faster, but it doesn’t seem like I’ve done
> something wrong? This is driving me crazy :)


It is much faster, and runs in parallel, if you use fold.

Try this;

(defn fold-into-vec [coll]
  "Provided a reducer, concatenate into a vector.
   Note: same as (into [] coll), but parallel."
  (r/fold (r/monoid into vector) conj coll))

So it should read as;
(defn reducer-test [nums]
   (fold-into-vec (r/map inc nums)))

Details here;
http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html

Hope this helps

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers question

[Daniel Higginbotham]

Sweet, thanks everyone for your responses! This was very helpful! r/foldcat 
did the trick. It looks like it's faster than pmap  by about 10%. Also, I'm 
happy to have my sanity back.

Thank you!
Daniel

On Wednesday, July 15, 2015 at 12:38:39 AM UTC-4, James Reeves wrote:
>
> Reducers and pmap have different approaches to concurrency.
>
> pmap works a lot like map, except that instead of evaluating the seq one 
> item at a time, it spins up N threads which work on the next N items in 
> parallel. N is 2 + the number of CPU cores you have.
>
> The reducers library works in a rather different way. It's important to 
> understand that the reducers library is split into reduce functions and 
> combine functions. The idea is that a collection is split up into equal 
> parts, then given to reduce functions in separate threads. Finally, all the 
> separate results are stuck back again by folding over the combine function.
>
> If you're just using something like:
>
>   (into [] (r/map blur (vec indexes))
>
> Then you're only using the reduce function. The parallelism is initiated 
> by the fold, so instead you want:
>
>   (r/foldcat (r/map blur (vec indexes))
>
> By default, this will split the vector into pieces 512 in length, map 
> "blur" over each piece in parallel, then fold the pieces back together 
> again using the r/cat function, which is designed to be performant at 
> concatenating the pieces of a fold back together again.
>
> My guess is that for your particular use-case, pmap is going to be faster. 
> The reducers library shines when you start to get into situations that pmap 
> can't handle.
>
> - James
>
> On 15 July 2015 at 04:02, Daniel Higginbotham  > wrote:
>
>> I’ve been trying to better understand ways to increase the performance of 
>> Clojure programs and I’ve run into an interesting issue with using the 
>> reducers library. I’d love any help! The code I’m working on is at 
>> https://github.com/flyingmachine/quil-pix/tree/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc
>>
>> The issue I’m having is that reducers/map has the same performance as 
>> core/map, and it doesn’t seem like that should be the case. (The current 
>> code (
>> https://github.com/flyingmachine/quil-pix/blob/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc/src/quil_pix/blur.clj#L73)
>>  
>> uses mapv, but I’ve also tried just map.) pmap is ~3x faster.
>>
>> Surely reducers/map should be faster, but it doesn’t seem like I’ve done 
>> something wrong? This is driving me crazy :)
>>
>> Daniel
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com 
>> 
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers question

[Erik Assum]

You might want to check out this talk by Leon Barrett at Clojure/West 2015

https://www.youtube.com/watch?v=BzKjIk0vgzE&index=2&list=PLZdCLR02grLrKAOj8FJ1GGmNM5l7Okz0a

Erik.

> On 15. jul. 2015, at 05.02, Daniel Higginbotham  
> wrote:
> 
> I’ve been trying to better understand ways to increase the performance of 
> Clojure programs and I’ve run into an interesting issue with using the 
> reducers library. I’d love any help! The code I’m working on is at 
> https://github.com/flyingmachine/quil-pix/tree/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc
> 
> The issue I’m having is that reducers/map has the same performance as 
> core/map, and it doesn’t seem like that should be the case. (The current code 
> (https://github.com/flyingmachine/quil-pix/blob/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc/src/quil_pix/blur.clj#L73)
>  uses mapv, but I’ve also tried just map.) pmap is ~3x faster.
> 
> Surely reducers/map should be faster, but it doesn’t seem like I’ve done 
> something wrong? This is driving me crazy :)
> 
> Daniel
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reducers question

[Francis Avila]

Reducers can perform fewer allocations than lazy seqs, but that does not 
automatically translate into a big speed difference, especially with your 
trivial example.

Try comparing chained lazy seq ops (map, mapcat, filter, take, drop, etc) to an 
equivalent chain of reducer calls: you may see a more pronounced speed 
difference.

Reducers can also be foldable (thus parallel) if the source data structure 
supports it but are not so automatically. Try r/foldcat or r/fold to possibly 
see pmap speeds.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: reducers question

[James Reeves]

Reducers and pmap have different approaches to concurrency.

pmap works a lot like map, except that instead of evaluating the seq one
item at a time, it spins up N threads which work on the next N items in
parallel. N is 2 + the number of CPU cores you have.

The reducers library works in a rather different way. It's important to
understand that the reducers library is split into reduce functions and
combine functions. The idea is that a collection is split up into equal
parts, then given to reduce functions in separate threads. Finally, all the
separate results are stuck back again by folding over the combine function.

If you're just using something like:

  (into [] (r/map blur (vec indexes))

Then you're only using the reduce function. The parallelism is initiated by
the fold, so instead you want:

  (r/foldcat (r/map blur (vec indexes))

By default, this will split the vector into pieces 512 in length, map
"blur" over each piece in parallel, then fold the pieces back together
again using the r/cat function, which is designed to be performant at
concatenating the pieces of a fold back together again.

My guess is that for your particular use-case, pmap is going to be faster.
The reducers library shines when you start to get into situations that pmap
can't handle.

- James

On 15 July 2015 at 04:02, Daniel Higginbotham 
wrote:

> I’ve been trying to better understand ways to increase the performance of
> Clojure programs and I’ve run into an interesting issue with using the
> reducers library. I’d love any help! The code I’m working on is at
> https://github.com/flyingmachine/quil-pix/tree/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc
>
> The issue I’m having is that reducers/map has the same performance as
> core/map, and it doesn’t seem like that should be the case. (The current
> code (
> https://github.com/flyingmachine/quil-pix/blob/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc/src/quil_pix/blur.clj#L73)
> uses mapv, but I’ve also tried just map.) pmap is ~3x faster.
>
> Surely reducers/map should be faster, but it doesn’t seem like I’ve done
> something wrong? This is driving me crazy :)
>
> Daniel
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

reducers question

[Daniel Higginbotham]

I’ve been trying to better understand ways to increase the performance of 
Clojure programs and I’ve run into an interesting issue with using the reducers 
library. I’d love any help! The code I’m working on is at 
https://github.com/flyingmachine/quil-pix/tree/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc
 
<https://github.com/flyingmachine/quil-pix/tree/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc>

The issue I’m having is that reducers/map has the same performance as core/map, 
and it doesn’t seem like that should be the case. (The current code 
(https://github.com/flyingmachine/quil-pix/blob/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc/src/quil_pix/blur.clj#L73
 
<https://github.com/flyingmachine/quil-pix/blob/4cce95390f5ac7a206cc14a8ec5a4a2492c813fc/src/quil_pix/blur.clj#L73>)
 uses mapv, but I’ve also tried just map.) pmap is ~3x faster.

Surely reducers/map should be faster, but it doesn’t seem like I’ve done 
something wrong? This is driving me crazy :)

Daniel

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: are there any real examples on Github of how to use reducers?

[Marshall Bockrath-Vandegrift]

piastkra...@gmail.com writes:

> Can anyone point me to a project on Github that is using Clojure 1.5
> reducerers?

The Parkour project provides an interface to Hadoop MapReduce primarily
in terms of tasks as functions over task input chunks as
clojure.core.reducers reducible collections:

https://github.com/damballa/parkour

-- 
Marshall Bockrath-Vandegrift 
Principal Software Engineer, Damballa R&D

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: are there any real examples on Github of how to use reducers?

[Max Countryman]

Kyle Kingsbury’s linearizability checker “Knossos” uses reducers. (See the 
core, history, and util namespaces.)

https://github.com/aphyr/knossos


> On May 21, 2015, at 19:55, piastkra...@gmail.com wrote:
> 
> 
> Can anyone point me to a project on Github that is using Clojure 1.5 
> reducerers? 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> <http://groups.google.com/group/clojure?hl=en>
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> <mailto:clojure+unsubscr...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout 
> <https://groups.google.com/d/optout>.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

are there any real examples on Github of how to use reducers?

[piastkrakow]

Can anyone point me to a project on Github that is using Clojure 1.5 
reducerers? 



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Reducers question

[Thomas Heller]

Cat is just missing a print-method entry.

Try (into [] (r/fold 1 r/cat r/append! [1 2 3])), the result is what you'd 
expect. The n parameter isn't actually about parallelism but partition 
size. Fork/Join will decide the parallelism. In the case of n=1 the input 
will be split into 3 partition of 1 (eg, [[1] [2] [3]]) and then handed off 
to reducers, the default of n=512 is larger than your input so you'll just 
use regular reduce.

/thomas

On Monday, February 9, 2015 at 10:38:59 AM UTC+1, Aaron Cohen wrote:
>
> I'm not sure if the 4-arity fold function is working as expected. My 
> understanding is that it should be equivalent to the 3-arity version, but 
> with specified parallelism.
>
> However:
>
> (r/fold r/cat r/append! [1 2 3]) => [1 2 3]
>
> (r/fold 1 r/cat r/append! [1 2 3]) => # clojure.core.reducers.Cat@4b2ae664>
>
> I don't actually understand how this is possible from reading the source 
> code.
>
> --Aaron
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reducers question

[Aaron Cohen]

I'm not sure if the 4-arity fold function is working as expected. My
understanding is that it should be equivalent to the 3-arity version, but
with specified parallelism.

However:

(r/fold r/cat r/append! [1 2 3]) => [1 2 3]

(r/fold 1 r/cat r/append! [1 2 3]) => #

I don't actually understand how this is possible from reading the source
code.

--Aaron

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Very strange behaviour in reducers

[Yves Parès]

Ok, but apparently I'm not the only one to be puzzled by this behaviour, 
which we may regard as an inconsistency.
Just to know, is it planned to be changed? (so that I can note it and 
remember it if I have an error that appears in this part of my code after a 
clojure upgrade)

Le mardi 14 janvier 2014 18:20:45 UTC+1, Filip Štaffa a écrit :
>
> Hi Yves,
>
> I guess it is because r/reduce calls r/reduce-kv (which is specific for 
> map), while r/fold it does not have such special handling
>
> Filip
>
> On Tuesday, January 14, 2014 4:43:10 PM UTC+1, Yves Parès wrote:
>>
>> Hello!
>> When mapping and reducing a map, it seems arguments are passed 
>> differently to the function that is mapped depending on whether you reduce 
>> with r/reduce or r/fold:
>>
>> (r/fold +
>> (r/map (fn [k v] (inc v)) {:a 4 :b 5}))
>> But:
>> (r/reduce +
>> (r/map (fn [[k v]] (inc v)) {:a 4 :b 5}))
>>
>> The function r/mapped is the same each time, but if the return value of 
>> r/map is further reduce with r/fold, it received 2 arguments (key and 
>> value). However, if it is reduced with r/reduce, the fn is called with ONE 
>> pair [k v].
>>
>> Is that normal?
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Very strange behaviour in reducers

[icamts]

Similar threads are

https://groups.google.com/forum/?hl=en#!topic/clojure/A1gW_BB_4MU

https://groups.google.com/forum/?hl=en#!topic/clojure-dev/scvyi2Cwk7g

Luca

Il giorno martedì 14 gennaio 2014 16:43:10 UTC+1, Yves Parès ha scritto:
>
> Hello!
> When mapping and reducing a map, it seems arguments are passed differently 
> to the function that is mapped depending on whether you reduce with 
> r/reduce or r/fold:
>
> (r/fold +
> (r/map (fn [k v] (inc v)) {:a 4 :b 5}))
> But:
> (r/reduce +
> (r/map (fn [[k v]] (inc v)) {:a 4 :b 5}))
>
> The function r/mapped is the same each time, but if the return value of 
> r/map is further reduce with r/fold, it received 2 arguments (key and 
> value). However, if it is reduced with r/reduce, the fn is called with ONE 
> pair [k v].
>
> Is that normal?
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Very strange behaviour in reducers

[Filip Štaffa]

Hi Yves,

I guess it is because r/reduce calls r/reduce-kv (which is specific for 
map), while r/fold it does not have such special handling

Filip

On Tuesday, January 14, 2014 4:43:10 PM UTC+1, Yves Parès wrote:
>
> Hello!
> When mapping and reducing a map, it seems arguments are passed differently 
> to the function that is mapped depending on whether you reduce with 
> r/reduce or r/fold:
>
> (r/fold +
> (r/map (fn [k v] (inc v)) {:a 4 :b 5}))
> But:
> (r/reduce +
> (r/map (fn [[k v]] (inc v)) {:a 4 :b 5}))
>
> The function r/mapped is the same each time, but if the return value of 
> r/map is further reduce with r/fold, it received 2 arguments (key and 
> value). However, if it is reduced with r/reduce, the fn is called with ONE 
> pair [k v].
>
> Is that normal?
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Very strange behaviour in reducers

[Yves Parès]

Hello!
When mapping and reducing a map, it seems arguments are passed differently 
to the function that is mapped depending on whether you reduce with 
r/reduce or r/fold:

(r/fold +
(r/map (fn [k v] (inc v)) {:a 4 :b 5}))
But:
(r/reduce +
(r/map (fn [[k v]] (inc v)) {:a 4 :b 5}))

The function r/mapped is the same each time, but if the return value of 
r/map is further reduce with r/fold, it received 2 arguments (key and 
value). However, if it is reduced with r/reduce, the fn is called with ONE 
pair [k v].

Is that normal?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: lazy seq from reducers with core.async

[Jozef Wagner]

Another nice abuse of core.async is for the implementation of 
multi-reducible collections. Here is the implementation of interleave 
accepting/producing reducibles. Slower than lazy seqs, but hey, you can 
feed it with any reducible coll. BTW anybody knows the status of multi 
reducibles? Are they being worked on?

 (defn interleave* [& colls]
(reify clojure.core.protocols.CollReduce
  (coll-reduce [this f1]
(clojure.core.protocols/coll-reduce this f1 (f1)))
  (coll-reduce [_ f1 init]
(let [NIL (Object.)
  encode-nil #(if (nil? %) NIL %)
  decode-nil #(if (identical? NIL %) nil %)
  send-fn (fn [coll]
(let [c (chan)]
  (thread (reduce (fn [r v] (>!! c (encode-nil v))) nil 
coll)
  (close! c)) 
  c))
  chans (doall (map send-fn colls))]
  (loop [val init
 s (cycle chans)]
(let [c (first s)
  x (https://gist.github.com/wagjo/6335189

JW

On Sunday, August 25, 2013 7:47:14 PM UTC+2, Jozef Wagner wrote:
>
> Hi,
>
> A distinctive feature of reducers is that "reducing is a one-shot thing". 
> The common understanding is that reducers are fast for cases where you want 
> to process whole collection at once, but for infinite and lazy seqs, you 
> have to use good old seqs.
>
> With core.async, it is now easy to create a transformation which produces 
> lazy seq from any reducible collection.
>
>   (defn lazy-seq*
> [reducible]
> (let [c (chan)
>   NIL (Object.)
>   encode-nil #(if (nil? %) NIL %)
>   decode-nil #(if (identical? NIL %) nil %)]
>   (thread
>(reduce (fn [r v] (>!! c (encode-nil v))) nil reducible)
>(close! c))
>   (take-while (complement nil?) (repeatedly #(decode-nil (  
>   (def s (lazy-seq* (clojure.core.reducers/map inc (range
>  
>   (first s)
>  
>   (take 100 s)
>
> This approach can be also extended to produce chunked seqs and chan buffer 
> can also be used to further tune the performance.
>
> JW
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: lazy seq from reducers with core.async

[Timothy Baldridge]

Although this also ties up a thread for every reduce. Not exactly efficient
IMO.

Timothy


On Sun, Aug 25, 2013 at 11:50 PM, Alan Busby  wrote:

>
> On Mon, Aug 26, 2013 at 1:37 PM, Timothy Baldridge 
> wrote:
>
>> Since reducers use fork-join pools,
>
>
> Reducers use multiple threads and fork-join pools when called with "fold"
> on vectors (anything else?), not "reduce".
> By making the single producer thread of the reducer block on writes, per
> Jozef's code, he's making it act like a lazy-seq.
>
> If fold was used, then something similar to the code snippet I provided
> may be more appropriate.
> It would really depend on the use-case though.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: lazy seq from reducers with core.async

[Alan Busby]

On Mon, Aug 26, 2013 at 1:37 PM, Timothy Baldridge wrote:

> Since reducers use fork-join pools,


Reducers use multiple threads and fork-join pools when called with "fold"
on vectors (anything else?), not "reduce".
By making the single producer thread of the reducer block on writes, per
Jozef's code, he's making it act like a lazy-seq.

If fold was used, then something similar to the code snippet I provided may
be more appropriate.
It would really depend on the use-case though.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: lazy seq from reducers with core.async

[Timothy Baldridge]

Although this looks like it might work, it's not exactly a good idea. Look
at what you're doing inside the reducer, a call to >!! will block a OS
level thread until someone takes from the channel. Since reducers use
fork-join pools, I wouldn't be surprised if it caused some serious
side-effects. Since reducing over things like vectors and maps happen in
parallel, you could end up blocking unknown numbers of threads until these
seqs get realized. So the pool may go out and create more threads, just to
have those blocked by this code.  Either that, or the fork/join pool will
just clog up, keeping other reducers from running properly.


Timothy Baldridge


On Sun, Aug 25, 2013 at 8:43 PM, Alan Busby  wrote:

> Here is something similar pulled from bagotricks "1.5.2", using Java's
> linked blocking queue;
>
> https://github.com/thebusby/bagotricks/blob/master/src/bagotricks.clj#L204-L238
>
> It uses fold instead of reduce to run in parallel, so has a slightly
> different use case than above.
>
>
> On Mon, Aug 26, 2013 at 10:14 AM, Mikera wrote:
>
>> Nice idea Jozef!
>>
>> Hmmm this is another example of why nil-as-end-of-channel is a
>> slightly problematic design decision for core.async: it makes this kind of
>> code much more fiddly.
>>
>>
>> On Monday, 26 August 2013 01:47:14 UTC+8, Jozef Wagner wrote:
>>>
>>> Hi,
>>>
>>> A distinctive feature of reducers is that "reducing is a one-shot
>>> thing". The common understanding is that reducers are fast for cases where
>>> you want to process whole collection at once, but for infinite and lazy
>>> seqs, you have to use good old seqs.
>>>
>>> With core.async, it is now easy to create a transformation which
>>> produces lazy seq from any reducible collection.
>>>
>>>   (defn lazy-seq*
>>> [reducible]
>>> (let [c (chan)
>>>   NIL (Object.)
>>>   encode-nil #(if (nil? %) NIL %)
>>>   decode-nil #(if (identical? NIL %) nil %)]
>>>   (thread
>>>(reduce (fn [r v] (>!! c (encode-nil v))) nil reducible)
>>>(close! c))
>>>   (take-while (complement nil?) (repeatedly #(decode-nil (>>
>>>
>>>   (def s (lazy-seq* (clojure.core.reducers/map inc (range
>>>
>>>   (first s)
>>>
>>>   (take 100 s)
>>>
>>> This approach can be also extended to produce chunked seqs and chan
>>> buffer can also be used to further tune the performance.
>>>
>>> JW
>>>
>>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: lazy seq from reducers with core.async

[Alan Busby]

Here is something similar pulled from bagotricks "1.5.2", using Java's
linked blocking queue;
https://github.com/thebusby/bagotricks/blob/master/src/bagotricks.clj#L204-L238

It uses fold instead of reduce to run in parallel, so has a slightly
different use case than above.


On Mon, Aug 26, 2013 at 10:14 AM, Mikera wrote:

> Nice idea Jozef!
>
> Hmmm this is another example of why nil-as-end-of-channel is a
> slightly problematic design decision for core.async: it makes this kind of
> code much more fiddly.
>
>
> On Monday, 26 August 2013 01:47:14 UTC+8, Jozef Wagner wrote:
>>
>> Hi,
>>
>> A distinctive feature of reducers is that "reducing is a one-shot thing".
>> The common understanding is that reducers are fast for cases where you want
>> to process whole collection at once, but for infinite and lazy seqs, you
>> have to use good old seqs.
>>
>> With core.async, it is now easy to create a transformation which produces
>> lazy seq from any reducible collection.
>>
>>   (defn lazy-seq*
>> [reducible]
>> (let [c (chan)
>>   NIL (Object.)
>>   encode-nil #(if (nil? %) NIL %)
>>   decode-nil #(if (identical? NIL %) nil %)]
>>   (thread
>>(reduce (fn [r v] (>!! c (encode-nil v))) nil reducible)
>>(close! c))
>>   (take-while (complement nil?) (repeatedly #(decode-nil (>
>>   (def s (lazy-seq* (clojure.core.reducers/map inc (range
>>
>>   (first s)
>>
>>   (take 100 s)
>>
>> This approach can be also extended to produce chunked seqs and chan
>> buffer can also be used to further tune the performance.
>>
>> JW
>>
>>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: lazy seq from reducers with core.async

[Mikera]

Nice idea Jozef!

Hmmm this is another example of why nil-as-end-of-channel is a slightly 
problematic design decision for core.async: it makes this kind of code much 
more fiddly.

On Monday, 26 August 2013 01:47:14 UTC+8, Jozef Wagner wrote:
>
> Hi,
>
> A distinctive feature of reducers is that "reducing is a one-shot thing". 
> The common understanding is that reducers are fast for cases where you want 
> to process whole collection at once, but for infinite and lazy seqs, you 
> have to use good old seqs.
>
> With core.async, it is now easy to create a transformation which produces 
> lazy seq from any reducible collection.
>
>   (defn lazy-seq*
> [reducible]
> (let [c (chan)
>   NIL (Object.)
>   encode-nil #(if (nil? %) NIL %)
>   decode-nil #(if (identical? NIL %) nil %)]
>   (thread
>(reduce (fn [r v] (>!! c (encode-nil v))) nil reducible)
>(close! c))
>   (take-while (complement nil?) (repeatedly #(decode-nil (  
>   (def s (lazy-seq* (clojure.core.reducers/map inc (range
>  
>   (first s)
>  
>   (take 100 s)
>
> This approach can be also extended to produce chunked seqs and chan buffer 
> can also be used to further tune the performance.
>
> JW
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

lazy seq from reducers with core.async

[Jozef Wagner]

Hi,

A distinctive feature of reducers is that "reducing is a one-shot thing". 
The common understanding is that reducers are fast for cases where you want 
to process whole collection at once, but for infinite and lazy seqs, you 
have to use good old seqs.

With core.async, it is now easy to create a transformation which produces 
lazy seq from any reducible collection.

  (defn lazy-seq*
[reducible]
(let [c (chan)
  NIL (Object.)
  encode-nil #(if (nil? %) NIL %)
  decode-nil #(if (identical? NIL %) nil %)]
  (thread
   (reduce (fn [r v] (>!! c (encode-nil v))) nil reducible)
   (close! c))
  (take-while (complement nil?) (repeatedly #(decode-nil (http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: butlast with reducers

[Jozef Wagner]

To continue on my previous post, here is the implementation of map
which returns both reducible and (lazy)seqable collection:

(defn my-map
[f c]
(let [coll (clojure.core.reducers/map f c)
  d (delay (clojure.core/map f c))]
  (reify
clojure.core.protocols/CollReduce
(coll-reduce [_ f1]
  (clojure.core.protocols/coll-reduce coll f1))
(coll-reduce [_ f1 init]
  (clojure.core.protocols/coll-reduce coll f1 init))
clojure.core.reducers/CollFold
(coll-fold [_ n combinef reducef]
  (clojure.core.reducers/coll-fold coll n combinef reducef))
clojure.lang.Seqable
(seq [_] @d

user>   (def x (my-map inc (range 100)))
#'user/x
user> x
#
user> (seq x)
(1 2 3 4 5 6 7 8 9 10 11 12 . . . 100)
user> (last x)
100
user> (reduce + x)
5050
user> (clojure.core.reducers/fold + x)
5050

Best,
JW

On Thursday, August 8, 2013 4:46:37 PM UTC+2, Jozef Wagner wrote:
>
> Wow, thank you very much! A perfect solution.
>
> At the end, wouldn't be good if the reducers would implement alongside 
> CollReduce also a Seqable interface, so that reducers could be used as a drop 
> in replacement for today's sequence functions (map, filter, ...)? CollReduce 
> implements 'eager' computations (when calling reduce and fold) while the 
> Seqable would implement a lazy ones (when calling seq).
>
> Functions working with seqs (e.g. first, rest) always call seq before working 
> with the collection, so this would be transparent for them. It may be a 
> breaking change and complects things a bit though...
>
> JW
>
> On Thursday, August 8, 2013 3:34:31 PM UTC+2, Christophe Grand wrote:
>>
>> ArrayDeque based versions:
>>
>> (defn drop-last [n coll]
>>   (reducer coll
>> (fn [f1]
>>   (let [buffer (java.util.ArrayDeque. (int n))]
>> (fn self
>>   ([] (f1))
>>   ([ret x]
>> (.add buffer x)
>> (if (<= (count buffer) n)
>>   ret
>>   (f1 ret (.pop buffer)
>>
>> (defn take-last [n coll]
>>   (reify clojure.core.protocols.CollReduce
>> (coll-reduce [this f1]
>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>> (coll-reduce [_ f1 init]
>>   (clojure.core.protocols/coll-reduce
>> (doto (clojure.core.protocols/coll-reduce
>>   coll
>>   (fn [^java.util.Deque q x]
>> (.add q x)
>> (when (> (count q) n)
>>   (.pop q))
>> q) (java.util.ArrayDeque. (int n))) prn)
>> f1 init
>>
>>
>>
>> On Thu, Aug 8, 2013 at 3:16 PM, Christophe Grand  wrote:
>>>
>>> You need to use a buffer to defer calls to the reduced function
>>>
>>> (defn drop-last [n coll]
>>>   (reducer coll
>>> (fn [f1]
>>>   (let [buffer
>>> (atom clojure.lang.PersistentQueue/EMPTY)]
>>> (fn self
>>>   ([] (f1))
>>>   ([ret x]
>>> (let [b (swap! buffer conj x)]
>>>   (if (<= (count @buffer) n)
>>> ret
>>> (do
>>>   (swap! buffer pop)
>>>   (f1 ret (peek b)))
>>>
>>> An array or a ring buffer should be used instead of  the atom and 
>>> persistent queue combo to reduce allocation.
>>>
>>> take-last is harder because you can't know when the reduction is over when 
>>> using #'reducer, so you have to implement CollReduce yourself:
>>>
>>> (defn take-last [n coll]
>>>   (reify clojure.core.protocols.CollReduce
>>> (coll-reduce [this f1]
>>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>>> (coll-reduce [_ f1 init]
>>>   (clojure.core.protocols/coll-reduce
>>> (clojure.core.protocols/coll-reduce
>>>   coll
>>>   (fn [q x]
>>> (let [q (conj q x)]
>>>   (if (<= (count q) n)
>>> q
>>> (pop q clojure.lang.PersistentQueue/EMPTY)
>>> f1 init
>>>
>>> again, use of a mutable array/buffer would be preferable.
>>>
>>> hth,
>>>
>>> Christophe
>>>
>>> On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner  wrote:
>>>>
>>>> Is it possible to implement efficient butlast (and drop-last, take-last) 
>>>> with reducers? The only solution

Re: butlast with reducers

[Jozef Wagner]

To continue on my previous email, here is the implementation of map which 
returns both reducible and (lazy) seqable object:

  (defn my-map
[f c]
(let [coll (clojure.core.reducers/map f c)
  d (delay (clojure.core/map f c))]
  (reify
clojure.core.protocols/CollReduce
(coll-reduce [_ f1]
  (clojure.core.protocols/coll-reduce coll f1))
(coll-reduce [_ f1 init]
  (clojure.core.protocols/coll-reduce coll f1 init))
clojure.core.reducers/CollFold
(coll-fold [_ n combinef reducef]
  (clojure.core.reducers/coll-fold coll n combinef reducef))
clojure.lang.Seqable
(seq [_] @d

user> (def x (my-map inc (range 100)))
#'user/x
user> x
#
user> (reduce + x)
5050
user> (clojure.core.reducers/fold + x)
5050
user> (seq x)
(1 2 3 4 5 6 7 8 9 10 11 12 . . . 100)
user> (last x)
100

JW

On Thursday, August 8, 2013 4:46:37 PM UTC+2, Jozef Wagner wrote:
>
> Wow, thank you very much! A perfect solution.
>
> At the end, wouldn't be good if the reducers would implement alongside 
> CollReduce also a Seqable interface, so that reducers could be used as a 
> drop in replacement for today's sequence functions (map, filter, ...)? 
> CollReduce implements 'eager' computations (when calling reduce and fold) 
> while the Seqable would implement a lazy ones (when calling seq).
>
> Functions working with seqs (e.g. first, rest) always call seq before 
> working with the collection, so this would be transparent for them. It 
> may be a breaking change and complects things a bit though...
>
> JW
>
> On Thursday, August 8, 2013 3:34:31 PM UTC+2, Christophe Grand wrote:
>>
>> ArrayDeque based versions:
>>
>> (defn drop-last [n coll]
>>   (reducer coll
>> (fn [f1]
>>   (let [buffer (java.util.ArrayDeque. (int n))]
>> (fn self
>>   ([] (f1))
>>   ([ret x]
>> (.add buffer x)
>> (if (<= (count buffer) n)
>>   ret
>>   (f1 ret (.pop buffer)
>>
>> (defn take-last [n coll]
>>   (reify clojure.core.protocols.CollReduce
>> (coll-reduce [this f1]
>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>> (coll-reduce [_ f1 init]
>>   (clojure.core.protocols/coll-reduce
>> (doto (clojure.core.protocols/coll-reduce 
>>   coll
>>   (fn [^java.util.Deque q x]
>> (.add q x)
>> (when (> (count q) n)
>>   (.pop q))
>> q) (java.util.ArrayDeque. (int n))) prn)
>> f1 init
>>
>>
>>
>> On Thu, Aug 8, 2013 at 3:16 PM, Christophe Grand wrote:
>>
>>> You need to use a buffer to defer calls to the reduced function
>>>
>>> (defn drop-last [n coll]
>>>   (reducer coll
>>> (fn [f1]
>>>   (let [buffer 
>>> (atom clojure.lang.PersistentQueue/EMPTY)]
>>> (fn self
>>>   ([] (f1))
>>>   ([ret x]
>>> (let [b (swap! buffer conj x)]
>>>   (if (<= (count @buffer) n)
>>> ret
>>> (do 
>>>   (swap! buffer pop)
>>>   (f1 ret (peek b)))
>>>
>>> An array or a ring buffer should be used instead of  the atom and 
>>> persistent queue combo to reduce allocation.
>>>
>>> take-last is harder because you can't know when the reduction is over 
>>> when using #'reducer, so you have to implement CollReduce yourself:
>>>
>>> (defn take-last [n coll]
>>>   (reify clojure.core.protocols.CollReduce
>>> (coll-reduce [this f1]
>>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>>> (coll-reduce [_ f1 init]
>>>   (clojure.core.protocols/coll-reduce
>>> (clojure.core.protocols/coll-reduce 
>>>   coll
>>>   (fn [q x]
>>> (let [q (conj q x)]
>>>   (if (<= (count q) n)
>>> q
>>> (pop q clojure.lang.PersistentQueue/EMPTY)
>>> f1 init
>>>
>>> again, use of a mutable array/buffer would be preferable. 
>>>
>>> hth,
>>>
>>> Christophe
>>>
>>> On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner wrote:
>>>
>>>> Is it possible to implement efficient butlast (and drop-last, 
>>>> take-last) with reducers? The only solution I can think of 

Re: butlast with reducers

[Jozef Wagner]

To continue my previous email, here is the implementation of map which is 
both reducible and (lazy) seqable:

 (defn my-map
[f c]
(let [coll (clojure.core.reducers/map f c)
  d (delay (clojure.core/map f c))]
  (reify
clojure.core.protocols/CollReduce
(coll-reduce [_ f1]
  (clojure.core.protocols/coll-reduce coll f1))
(coll-reduce [_ f1 init]
  (clojure.core.protocols/coll-reduce coll f1 init))
clojure.core.reducers/CollFold
(coll-fold [_ n combinef reducef]
  (clojure.core.reducers/coll-fold coll n combinef reducef))
clojure.lang.Seqable
(seq [_] @d



user> (def x (my-map inc (range 100)))
#'user/x
user> x
#
user> (seq x) ;; to lazy sequence
(1 2 3 4 5 6 7 8 9 10 11 12 . . . 100)
user> (last x) ;; using sequence
100
user> (reduce + x) ;; using reducer
5050
user> (clojure.core.reducers/fold + x) ;; using folder
5050

JW

On Thursday, August 8, 2013 4:46:37 PM UTC+2, Jozef Wagner wrote:
>
> Wow, thank you very much! A perfect solution.
>
> At the end, wouldn't be good if the reducers would implement alongside 
> CollReduce also a Seqable interface, so that reducers could be used as a 
> drop in replacement for today's sequence functions (map, filter, ...)? 
> CollReduce implements 'eager' computations (when calling reduce and fold) 
> while the Seqable would implement a lazy ones (when calling seq).
>
> Functions working with seqs (e.g. first, rest) always call seq before 
> working with the collection, so this would be transparent for them. It 
> may be a breaking change and complects things a bit though...
>
> JW
>
> On Thursday, August 8, 2013 3:34:31 PM UTC+2, Christophe Grand wrote:
>>
>> ArrayDeque based versions:
>>
>> (defn drop-last [n coll]
>>   (reducer coll
>> (fn [f1]
>>   (let [buffer (java.util.ArrayDeque. (int n))]
>> (fn self
>>   ([] (f1))
>>   ([ret x]
>> (.add buffer x)
>> (if (<= (count buffer) n)
>>   ret
>>   (f1 ret (.pop buffer)
>>
>> (defn take-last [n coll]
>>   (reify clojure.core.protocols.CollReduce
>> (coll-reduce [this f1]
>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>> (coll-reduce [_ f1 init]
>>   (clojure.core.protocols/coll-reduce
>> (doto (clojure.core.protocols/coll-reduce 
>>   coll
>>   (fn [^java.util.Deque q x]
>> (.add q x)
>> (when (> (count q) n)
>>   (.pop q))
>> q) (java.util.ArrayDeque. (int n))) prn)
>> f1 init
>>
>>
>>
>> On Thu, Aug 8, 2013 at 3:16 PM, Christophe Grand wrote:
>>
>>> You need to use a buffer to defer calls to the reduced function
>>>
>>> (defn drop-last [n coll]
>>>   (reducer coll
>>> (fn [f1]
>>>   (let [buffer 
>>> (atom clojure.lang.PersistentQueue/EMPTY)]
>>> (fn self
>>>   ([] (f1))
>>>   ([ret x]
>>> (let [b (swap! buffer conj x)]
>>>   (if (<= (count @buffer) n)
>>> ret
>>> (do 
>>>   (swap! buffer pop)
>>>   (f1 ret (peek b)))
>>>
>>> An array or a ring buffer should be used instead of  the atom and 
>>> persistent queue combo to reduce allocation.
>>>
>>> take-last is harder because you can't know when the reduction is over 
>>> when using #'reducer, so you have to implement CollReduce yourself:
>>>
>>> (defn take-last [n coll]
>>>   (reify clojure.core.protocols.CollReduce
>>> (coll-reduce [this f1]
>>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>>> (coll-reduce [_ f1 init]
>>>   (clojure.core.protocols/coll-reduce
>>> (clojure.core.protocols/coll-reduce 
>>>   coll
>>>   (fn [q x]
>>> (let [q (conj q x)]
>>>   (if (<= (count q) n)
>>> q
>>> (pop q clojure.lang.PersistentQueue/EMPTY)
>>> f1 init
>>>
>>> again, use of a mutable array/buffer would be preferable. 
>>>
>>> hth,
>>>
>>> Christophe
>>>
>>> On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner wrote:
>>>
>>>> Is it possible to implement efficient butlast (and drop-last, 
>>>> take-

Re: butlast with reducers

[Christophe Grand]

Gaving slept on it I realize that by adding before popping I allowed the
buffer to grow to n+1 thus causing a realloc.

(defn drop-last [n coll]
  (reducer coll
(fn [f1]
  (let [buffer (java.util.ArrayDeque. (int n))]
(fn self
  ([] (f1))
  ([ret x]
(let [ret (if (= (count buffer) n)
(f1 ret (.pop buffer))
ret)]
  (.add buffer x)
  ret)))


(defn take-last [n coll]
  (reify clojure.core.protocols.CollReduce
(coll-reduce [this f1]
  (clojure.core.protocols/coll-reduce this f1 (f1)))
(coll-reduce [_ f1 init]
  (clojure.core.protocols/coll-reduce
(clojure.core.protocols/coll-reduce
  coll
  (fn [^java.util.Deque q x]
(when (= (count q) n)
  (.pop q))
(.add q x)
q) (java.util.ArrayDeque. (int n)))
f1 init


On Thu, Aug 8, 2013 at 3:34 PM, Christophe Grand wrote:

> ArrayDeque based versions:
>
>
> (defn drop-last [n coll]
>   (reducer coll
> (fn [f1]
>   (let [buffer (java.util.ArrayDeque. (int n))]
>
> (fn self
>   ([] (f1))
>   ([ret x]
> (.add buffer x)
> (if (<= (count buffer) n)
>   ret
>   (f1 ret (.pop buffer)
>
>
> (defn take-last [n coll]
>   (reify clojure.core.protocols.CollReduce
> (coll-reduce [this f1]
>   (clojure.core.protocols/coll-reduce this f1 (f1)))
> (coll-reduce [_ f1 init]
>   (clojure.core.protocols/coll-reduce
> (doto (clojure.core.protocols/coll-reduce
>   coll
>   (fn [^java.util.Deque q x]
> (.add q x)
> (when (> (count q) n)
>   (.pop q))
> q) (java.util.ArrayDeque. (int n))) prn)
> f1 init
>
>
>
> On Thu, Aug 8, 2013 at 3:16 PM, Christophe Grand wrote:
>
>> You need to use a buffer to defer calls to the reduced function
>>
>> (defn drop-last [n coll]
>>   (reducer coll
>> (fn [f1]
>>   (let [buffer
>> (atom clojure.lang.PersistentQueue/EMPTY)]
>> (fn self
>>   ([] (f1))
>>   ([ret x]
>> (let [b (swap! buffer conj x)]
>>   (if (<= (count @buffer) n)
>> ret
>> (do
>>   (swap! buffer pop)
>>   (f1 ret (peek b)))
>>
>> An array or a ring buffer should be used instead of  the atom and
>> persistent queue combo to reduce allocation.
>>
>> take-last is harder because you can't know when the reduction is over
>> when using #'reducer, so you have to implement CollReduce yourself:
>>
>> (defn take-last [n coll]
>>   (reify clojure.core.protocols.CollReduce
>> (coll-reduce [this f1]
>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>> (coll-reduce [_ f1 init]
>>   (clojure.core.protocols/coll-reduce
>> (clojure.core.protocols/coll-reduce
>>   coll
>>   (fn [q x]
>> (let [q (conj q x)]
>>   (if (<= (count q) n)
>>     q
>> (pop q clojure.lang.PersistentQueue/EMPTY)
>> f1 init
>>
>> again, use of a mutable array/buffer would be preferable.
>>
>> hth,
>>
>> Christophe
>>
>> On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner wrote:
>>
>>> Is it possible to implement efficient butlast (and drop-last, take-last)
>>> with reducers? The only solution I can think of needs additional reduce to
>>> compute count, which may often be undesirable.
>>>
>>> Or is it OK to say that reducers are not designed for such cases?
>>>
>>> JW
>>>
>>> --
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> F

Re: butlast with reducers

[Jozef Wagner]

Wow, thank you very much! A perfect solution.

At the end, wouldn't be good if the reducers would implement alongside 
CollReduce also a Seqable interface, so that reducers could be used as a 
drop in replacement for today's sequence functions (map, filter, ...)? 
CollReduce implements 'eager' computations (when calling reduce and fold) 
while the Seqable would implement a lazy ones (when calling seq).

Functions working with seqs (e.g. first, rest) always call seq before 
working with the collection, so this would be transparent for them. It may 
be a breaking change and complects things a bit though...

JW

On Thursday, August 8, 2013 3:34:31 PM UTC+2, Christophe Grand wrote:
>
> ArrayDeque based versions:
>
> (defn drop-last [n coll]
>   (reducer coll
> (fn [f1]
>   (let [buffer (java.util.ArrayDeque. (int n))]
> (fn self
>   ([] (f1))
>   ([ret x]
> (.add buffer x)
> (if (<= (count buffer) n)
>   ret
>   (f1 ret (.pop buffer)
>
> (defn take-last [n coll]
>   (reify clojure.core.protocols.CollReduce
> (coll-reduce [this f1]
>   (clojure.core.protocols/coll-reduce this f1 (f1)))
> (coll-reduce [_ f1 init]
>   (clojure.core.protocols/coll-reduce
> (doto (clojure.core.protocols/coll-reduce 
>   coll
>   (fn [^java.util.Deque q x]
> (.add q x)
> (when (> (count q) n)
>   (.pop q))
> q) (java.util.ArrayDeque. (int n))) prn)
> f1 init
>
>
>
> On Thu, Aug 8, 2013 at 3:16 PM, Christophe Grand 
> 
> > wrote:
>
>> You need to use a buffer to defer calls to the reduced function
>>
>> (defn drop-last [n coll]
>>   (reducer coll
>> (fn [f1]
>>   (let [buffer 
>> (atom clojure.lang.PersistentQueue/EMPTY)]
>> (fn self
>>   ([] (f1))
>>   ([ret x]
>> (let [b (swap! buffer conj x)]
>>   (if (<= (count @buffer) n)
>> ret
>> (do 
>>   (swap! buffer pop)
>>   (f1 ret (peek b)))
>>
>> An array or a ring buffer should be used instead of  the atom and 
>> persistent queue combo to reduce allocation.
>>
>> take-last is harder because you can't know when the reduction is over 
>> when using #'reducer, so you have to implement CollReduce yourself:
>>
>> (defn take-last [n coll]
>>   (reify clojure.core.protocols.CollReduce
>> (coll-reduce [this f1]
>>   (clojure.core.protocols/coll-reduce this f1 (f1)))
>> (coll-reduce [_ f1 init]
>>   (clojure.core.protocols/coll-reduce
>> (clojure.core.protocols/coll-reduce 
>>   coll
>>   (fn [q x]
>> (let [q (conj q x)]
>>   (if (<= (count q) n)
>> q
>> (pop q clojure.lang.PersistentQueue/EMPTY)
>> f1 init
>>
>> again, use of a mutable array/buffer would be preferable. 
>>
>> hth,
>>
>> Christophe
>>
>> On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner 
>> 
>> > wrote:
>>
>>> Is it possible to implement efficient butlast (and drop-last, take-last) 
>>> with reducers? The only solution I can think of needs additional reduce to 
>>> compute count, which may often be undesirable. 
>>>
>>> Or is it OK to say that reducers are not designed for such cases?
>>>
>>> JW
>>>
>>> -- 
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com 
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to clojure+u...@googlegroups.com .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>  
>>>  
>>>
>>
>>
>>
>> -- 
>> On Clojure http://clj-me.cgrand.net/
>> Clojure Programming http://clojurebook.com
>

Re: butlast with reducers

[Christophe Grand]

ArrayDeque based versions:

(defn drop-last [n coll]
  (reducer coll
(fn [f1]
  (let [buffer (java.util.ArrayDeque. (int n))]
(fn self
  ([] (f1))
  ([ret x]
(.add buffer x)
(if (<= (count buffer) n)
  ret
  (f1 ret (.pop buffer)

(defn take-last [n coll]
  (reify clojure.core.protocols.CollReduce
(coll-reduce [this f1]
  (clojure.core.protocols/coll-reduce this f1 (f1)))
(coll-reduce [_ f1 init]
  (clojure.core.protocols/coll-reduce
(doto (clojure.core.protocols/coll-reduce
  coll
  (fn [^java.util.Deque q x]
(.add q x)
(when (> (count q) n)
  (.pop q))
q) (java.util.ArrayDeque. (int n))) prn)
f1 init



On Thu, Aug 8, 2013 at 3:16 PM, Christophe Grand wrote:

> You need to use a buffer to defer calls to the reduced function
>
> (defn drop-last [n coll]
>   (reducer coll
> (fn [f1]
>   (let [buffer
> (atom clojure.lang.PersistentQueue/EMPTY)]
> (fn self
>   ([] (f1))
>   ([ret x]
> (let [b (swap! buffer conj x)]
>   (if (<= (count @buffer) n)
> ret
> (do
>   (swap! buffer pop)
>   (f1 ret (peek b)))
>
> An array or a ring buffer should be used instead of  the atom and
> persistent queue combo to reduce allocation.
>
> take-last is harder because you can't know when the reduction is over when
> using #'reducer, so you have to implement CollReduce yourself:
>
> (defn take-last [n coll]
>   (reify clojure.core.protocols.CollReduce
> (coll-reduce [this f1]
>   (clojure.core.protocols/coll-reduce this f1 (f1)))
> (coll-reduce [_ f1 init]
>   (clojure.core.protocols/coll-reduce
> (clojure.core.protocols/coll-reduce
>   coll
>   (fn [q x]
> (let [q (conj q x)]
>   (if (<= (count q) n)
> q
> (pop q clojure.lang.PersistentQueue/EMPTY)
> f1 init
>
> again, use of a mutable array/buffer would be preferable.
>
> hth,
>
> Christophe
>
> On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner wrote:
>
>> Is it possible to implement efficient butlast (and drop-last, take-last)
>> with reducers? The only solution I can think of needs additional reduce to
>> compute count, which may often be undesirable.
>>
>> Or is it OK to say that reducers are not designed for such cases?
>>
>> JW
>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
>
> --
> On Clojure http://clj-me.cgrand.net/
> Clojure Programming http://clojurebook.com
> Training, Consulting & Contracting http://lambdanext.eu/
>



-- 
On Clojure http://clj-me.cgrand.net/
Clojure Programming http://clojurebook.com
Training, Consulting & Contracting http://lambdanext.eu/

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: butlast with reducers

[Christophe Grand]

You need to use a buffer to defer calls to the reduced function

(defn drop-last [n coll]
  (reducer coll
(fn [f1]
  (let [buffer
(atom clojure.lang.PersistentQueue/EMPTY)]
(fn self
  ([] (f1))
  ([ret x]
(let [b (swap! buffer conj x)]
  (if (<= (count @buffer) n)
ret
(do
  (swap! buffer pop)
  (f1 ret (peek b)))

An array or a ring buffer should be used instead of  the atom and
persistent queue combo to reduce allocation.

take-last is harder because you can't know when the reduction is over when
using #'reducer, so you have to implement CollReduce yourself:

(defn take-last [n coll]
  (reify clojure.core.protocols.CollReduce
(coll-reduce [this f1]
  (clojure.core.protocols/coll-reduce this f1 (f1)))
(coll-reduce [_ f1 init]
  (clojure.core.protocols/coll-reduce
(clojure.core.protocols/coll-reduce
  coll
  (fn [q x]
(let [q (conj q x)]
  (if (<= (count q) n)
q
(pop q clojure.lang.PersistentQueue/EMPTY)
f1 init

again, use of a mutable array/buffer would be preferable.

hth,

Christophe

On Thu, Aug 8, 2013 at 1:00 PM, Jozef Wagner  wrote:

> Is it possible to implement efficient butlast (and drop-last, take-last)
> with reducers? The only solution I can think of needs additional reduce to
> compute count, which may often be undesirable.
>
> Or is it OK to say that reducers are not designed for such cases?
>
> JW
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
On Clojure http://clj-me.cgrand.net/
Clojure Programming http://clojurebook.com
Training, Consulting & Contracting http://lambdanext.eu/

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

butlast with reducers

[Jozef Wagner]

Is it possible to implement efficient butlast (and drop-last, take-last) 
with reducers? The only solution I can think of needs additional reduce to 
compute count, which may often be undesirable. 

Or is it OK to say that reducers are not designed for such cases?

JW

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: cannot import namespace reducers

[Johannes Brauer]

if I remember correctly I solved the problem by reinstalling Java 7 from 
Oracle. Thereafter my $JAVA_HOME points to:
/Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home

Johannes
Am 24.07.2013 um 03:02 schrieb Keith Maynard 
mailto:kpmayn...@gmail.com>>
:

Please spam the list!!! I am sure anyone who receives that message is probably 
running Mac OS X 10.7.x or later and trying to unravel the mess between Java 6 
and Java 7.  Please post recommendations. I finally got Eclipse to see the 
jdl1.7.0_25.jdk now how do I get the OS to replace the Apple installed 1.6.x?

On Sunday, 16 June 2013 17:29:40 UTC-4, Las wrote:

Then you are having a path problem.
Your lein is still using java6
Check your $PATH and $JAVA_HOME env. variables.

As this not strictly clojure related, let's not spam the list, im happy to help 
off list

Sent from my phone

On Jun 16, 2013 11:22 PM, "Johannes Brauer"  wrote:
I am on clojure 1.5.1 and I use lein repl.

But after

(require '[clojure.core.reducers :as r])

I still get the same error message as with Java 6:
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21)

A second input of
(require '[clojure.core.reducers :as r])

generates the message:
Exception namespace 'clojure.core.reducers' not found  clojure.core/load-lib 
(core.clj:5380)

Johannes

Am 16.06.2013 um 23:16 schrieb László Török 
:

are you on clojure 1.5+ ?

If I launch the REPL using "lein repl"

and then

(require 'clojure.core.reducers)

it works ok for me.


2013/6/16 Johannes Brauer 
now, I've Java 7 installed and get another error message:
Exception namespace 'clojure.core.reducers' not found  clojure.core/load-lib 
(core.clj:5380)

any further hints?

Johannes
Am 16.06.2013 um 22:43 schrieb Johannes Brauer 
:

thank you, Las, for the quick tip. I will give Java 7 a try. I hope there are 
no problems on Mac OS 10.8.4

Johannes
Am 16.06.2013 um 22:15 schrieb László Török 
:

.. sorry, gmail's new annoying keyboard shortcut

b) include the dependency to the forkjoin library [1] that is not included in 
Java6

Las

[1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108


2013/6/16 László Török 
Hi,

there are two ways to deal with this:

a) use Java 7


2013/6/16 Johannes 
Hi,

trying
(require '[clojure.core.reducers :as r])
at the repl prompt the error message
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21)

What is going wrong?

Johannes



--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.





--
László Török



--
László Török

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.







Staatlich anerkannte private Fachhochschule
NORDAKADEMIE
Gemeinnützige Aktiengesellschaft
Köllner Chaussee 11
25337 Elmshorn

Vorstand:
Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. Vorstand)

Vorsitzender des Aufsichtsrats:
Dr. h.c. Hans-Heinrich Bruns

Sitz:
Elmshorn, Amtsgericht Pinneberg, HRB 1682


--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_ou

Re: cannot import namespace reducers

[Keith Maynard]

Please spam the list!!! I am sure anyone who receives that message is 
probably running Mac OS X 10.7.x or later and trying to unravel the mess 
between Java 6 and Java 7.  Please post recommendations. I finally got 
Eclipse to see the jdl1.7.0_25.jdk now how do I get the OS to replace the 
Apple installed 1.6.x?

On Sunday, 16 June 2013 17:29:40 UTC-4, Las wrote:
>
> Then you are having a path problem. 
> Your lein is still using java6
> Check your $PATH and $JAVA_HOME env. variables. 
>
> As this not strictly clojure related, let's not spam the list, im happy to 
> help off list
>
> Sent from my phone
> On Jun 16, 2013 11:22 PM, "Johannes Brauer" 
> > 
> wrote:
>
>>  I am on clojure 1.5.1 and I use lein repl. 
>>
>>  But after  
>>
>>  (require '[clojure.core.reducers :as r])
>>
>>  I still get the same error message as with Java 6:
>> CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
>> compiling:(clojure/core/reducers.clj:56:21)
>>
>>  A second input of
>> (require '[clojure.core.reducers :as r])
>>
>>  generates the message:
>> Exception namespace 'clojure.core.reducers' not found 
>>  clojure.core/load-lib (core.clj:5380)
>>
>>  Johannes
>>
>>  Am 16.06.2013 um 23:16 schrieb László Török 
>> >
>> :
>>
>>  are you on clojure 1.5+ ? 
>>
>>  If I launch the REPL using "lein repl"
>>
>>  and then
>>
>>  (require 'clojure.core.reducers)
>>  
>>  it works ok for me.
>>  
>>
>> 2013/6/16 Johannes Brauer >
>>
>>> now, I've Java 7 installed and get another error message: 
>>> Exception namespace 'clojure.core.reducers' not found 
>>>  clojure.core/load-lib (core.clj:5380)
>>>
>>>  any further hints?
>>>  
>>>  Johannes
>>>   Am 16.06.2013 um 22:43 schrieb Johannes Brauer 
>>> 
>>> >
>>> :
>>>  
>>>  thank you, Las, for the quick tip. I will give Java 7 a try. I hope 
>>> there are no problems on Mac OS 10.8.4 
>>>
>>>  Johannes
>>>  Am 16.06.2013 um 22:15 schrieb László Török 
>>> >
>>> :
>>>
>>>  .. sorry, gmail's new annoying keyboard shortcut 
>>>
>>>  b) include the dependency to the forkjoin library [1] that is not 
>>> included in Java6 
>>>
>>>  Las
>>>
>>>  [1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108
>>>  
>>>
>>> 2013/6/16 László Török >
>>>
 Hi, 

  there are two ways to deal with this:

  a) use Java 7
   

 2013/6/16 Johannes >

> Hi, 
>
>  trying
>  (require '[clojure.core.reducers :as r])
> at the repl prompt the error message
> CompilerException java.lang.ClassNotFoundException: 
> jsr166y.ForkJoinPool, compiling:(clojure/core/reducers.clj:56:21) 
>  
>  What is going wrong?
>
>  Johannes
>
>  
>  
>  -- 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient 
> with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google 
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to clojure+u...@googlegroups.com .
> For more options, visit https://groups.google.com/groups/opt_out.
>  
>  
>
  


   -- 
 László Török
  
>>>  
>>>
>>>
>>>  -- 
>>> László Török
>>>  
>>>  -- 
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with 
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com 
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to clojure+u...@googlegroups.com .
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>  
>>>  
>>>
>>>  
>>>  
>>> --
>>>
>>>
>>> Staatlich anerkannte private Fachhochschule
>>> NORDAKADEMIE
>>> Gemeinnützige Aktiengesellschaft
>>> Köllner Chaussee 11
>>> 25337 Elmshorn
>>>
>>> Vorstand:
>>> Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. 
>>> Vorstand)
>>>
>>> Vorsitzender des Aufsichtsrats:
>>> Dr. h.c. Hans-Heinrich Bruns
>>>
>>> Sitz:
>>> Elmshorn, Amtsgericht Pinneberg, HRB 1682
>>>
>>>  
>>>  -- 
>>> -- 
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo..

Re: cannot import namespace reducers

[László Török]

Then you are having a path problem.
Your lein is still using java6
Check your $PATH and $JAVA_HOME env. variables.

As this not strictly clojure related, let's not spam the list, im happy to
help off list

Sent from my phone
On Jun 16, 2013 11:22 PM, "Johannes Brauer"  wrote:

>  I am on clojure 1.5.1 and I use lein repl.
>
>  But after
>
>  (require '[clojure.core.reducers :as r])
>
>  I still get the same error message as with Java 6:
> CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool,
> compiling:(clojure/core/reducers.clj:56:21)
>
>  A second input of
> (require '[clojure.core.reducers :as r])
>
>  generates the message:
> Exception namespace 'clojure.core.reducers' not found
>  clojure.core/load-lib (core.clj:5380)
>
>  Johannes
>
>  Am 16.06.2013 um 23:16 schrieb László Török 
> :
>
>  are you on clojure 1.5+ ?
>
>  If I launch the REPL using "lein repl"
>
>  and then
>
>  (require 'clojure.core.reducers)
>
>  it works ok for me.
>
>
> 2013/6/16 Johannes Brauer 
>
>> now, I've Java 7 installed and get another error message:
>> Exception namespace 'clojure.core.reducers' not found
>>  clojure.core/load-lib (core.clj:5380)
>>
>>  any further hints?
>>
>>  Johannes
>>   Am 16.06.2013 um 22:43 schrieb Johannes Brauer 
>> :
>>
>>  thank you, Las, for the quick tip. I will give Java 7 a try. I hope
>> there are no problems on Mac OS 10.8.4
>>
>>  Johannes
>>  Am 16.06.2013 um 22:15 schrieb László Török 
>> :
>>
>>  .. sorry, gmail's new annoying keyboard shortcut
>>
>>  b) include the dependency to the forkjoin library [1] that is not
>> included in Java6
>>
>>  Las
>>
>>  [1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108
>>
>>
>> 2013/6/16 László Török 
>>
>>> Hi,
>>>
>>>  there are two ways to deal with this:
>>>
>>>  a) use Java 7
>>>
>>>
>>> 2013/6/16 Johannes 
>>>
 Hi,

  trying
  (require '[clojure.core.reducers :as r])
 at the repl prompt the error message
 CompilerException java.lang.ClassNotFoundException:
 jsr166y.ForkJoinPool, compiling:(clojure/core/reducers.clj:56:21)

  What is going wrong?

  Johannes



  --
 --
 You received this message because you are subscribed to the Google
 Groups "Clojure" group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google
 Groups "Clojure" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.



>>>
>>>
>>>
>>>   --
>>> László Török
>>>
>>
>>
>>
>>  --
>> László Török
>>
>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>>
>>
>> --
>>
>>
>> Staatlich anerkannte private Fachhochschule
>> NORDAKADEMIE
>> Gemeinnützige Aktiengesellschaft
>> Köllner Chaussee 11
>> 25337 Elmshorn
>>
>> Vorstand:
>> Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv.
>> Vorstand)
>>
>> Vorsitzender des Aufsichtsrats:
>> Dr. h.c. Hans-Heinrich Bruns
>>
>> Sitz:
>> Elmshorn, Amtsgericht Pinneberg, HRB 1682
>>
>>
>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>>
>>
>> -

Re: cannot import namespace reducers

[Johannes Brauer]

I am on clojure 1.5.1 and I use lein repl.

But after

(require '[clojure.core.reducers :as r])

I still get the same error message as with Java 6:
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21)

A second input of
(require '[clojure.core.reducers :as r])

generates the message:
Exception namespace 'clojure.core.reducers' not found  clojure.core/load-lib 
(core.clj:5380)

Johannes

Am 16.06.2013 um 23:16 schrieb László Török 
mailto:ltoro...@gmail.com>>
:

are you on clojure 1.5+ ?

If I launch the REPL using "lein repl"

and then

(require 'clojure.core.reducers)

it works ok for me.


2013/6/16 Johannes Brauer 
mailto:bra...@nordakademie.de>>
now, I've Java 7 installed and get another error message:
Exception namespace 'clojure.core.reducers' not found  clojure.core/load-lib 
(core.clj:5380)

any further hints?

Johannes
Am 16.06.2013 um 22:43 schrieb Johannes Brauer 
mailto:bra...@nordakademie.de>>
:

thank you, Las, for the quick tip. I will give Java 7 a try. I hope there are 
no problems on Mac OS 10.8.4

Johannes
Am 16.06.2013 um 22:15 schrieb László Török 
mailto:ltoro...@gmail.com>>
:

.. sorry, gmail's new annoying keyboard shortcut

b) include the dependency to the forkjoin library [1] that is not included in 
Java6

Las

[1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108


2013/6/16 László Török mailto:ltoro...@gmail.com>>
Hi,

there are two ways to deal with this:

a) use Java 7


2013/6/16 Johannes mailto:bra...@nordakademie.de>>
Hi,

trying
(require '[clojure.core.reducers :as r])
at the repl prompt the error message
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21)

What is going wrong?

Johannes



--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.





--
László Török



--
László Török

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.







Staatlich anerkannte private Fachhochschule
NORDAKADEMIE
Gemeinnützige Aktiengesellschaft
Köllner Chaussee 11
25337 Elmshorn

Vorstand:
Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. Vorstand)

Vorsitzender des Aufsichtsrats:
Dr. h.c. Hans-Heinrich Bruns

Sitz:
Elmshorn, Amtsgericht Pinneberg, HRB 1682


--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.







Staatlich anerkannte private Fachhochschule
NORDAKADEMIE
Gemeinnützige Aktiengesellschaft
Köllner Chaussee 11
25337 Elmshorn

Vorstand:
Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. Vorstand)

Vorsitzender des Aufsichtsrats:
Dr. h.c. Hans-Heinrich B

Re: cannot import namespace reducers

[László Török]

are you on clojure 1.5+ ?

If I launch the REPL using "lein repl"

and then

(require 'clojure.core.reducers)

it works ok for me.


2013/6/16 Johannes Brauer 

>  now, I've Java 7 installed and get another error message:
> Exception namespace 'clojure.core.reducers' not found
>  clojure.core/load-lib (core.clj:5380)
>
>  any further hints?
>
>  Johannes
>  Am 16.06.2013 um 22:43 schrieb Johannes Brauer 
> :
>
>  thank you, Las, for the quick tip. I will give Java 7 a try. I hope
> there are no problems on Mac OS 10.8.4
>
>  Johannes
>  Am 16.06.2013 um 22:15 schrieb László Török 
> :
>
>  .. sorry, gmail's new annoying keyboard shortcut
>
>  b) include the dependency to the forkjoin library [1] that is not
> included in Java6
>
>  Las
>
>  [1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108
>
>
> 2013/6/16 László Török 
>
>> Hi,
>>
>>  there are two ways to deal with this:
>>
>>  a) use Java 7
>>
>>
>> 2013/6/16 Johannes 
>>
>>> Hi,
>>>
>>>  trying
>>>  (require '[clojure.core.reducers :as r])
>>> at the repl prompt the error message
>>> CompilerException java.lang.ClassNotFoundException:
>>> jsr166y.ForkJoinPool, compiling:(clojure/core/reducers.clj:56:21)
>>>
>>>  What is going wrong?
>>>
>>>  Johannes
>>>
>>>
>>>
>>>  --
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>>
>>
>>   --
>> László Török
>>
>
>
>
>  --
> László Török
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>
> --
>
>
> Staatlich anerkannte private Fachhochschule
> NORDAKADEMIE
> Gemeinnützige Aktiengesellschaft
> Köllner Chaussee 11
> 25337 Elmshorn
>
> Vorstand:
> Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv.
> Vorstand)
>
> Vorsitzender des Aufsichtsrats:
> Dr. h.c. Hans-Heinrich Bruns
>
> Sitz:
> Elmshorn, Amtsgericht Pinneberg, HRB 1682
>
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>
> --
>
>
> Staatlich anerkannte private Fachhochschule
> NORDAKADEMIE
> Gemeinnützige Aktiengesellschaft
> Köllner Chaussee 11
> 25337 Elmshorn
>
> Vorstand:
> Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv.
> Vorstand)
>
> Vorsitzender des Aufsichtsrats:
> Dr. h.c. Hans-Heinrich Bruns
>
> Sitz:
> Elmshorn, Amtsgericht Pinneberg, HRB 1682
>
>   --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojur

Re: cannot import namespace reducers

[Johannes Brauer]

now, I've Java 7 installed and get another error message:
Exception namespace 'clojure.core.reducers' not found  clojure.core/load-lib 
(core.clj:5380)

any further hints?

Johannes
Am 16.06.2013 um 22:43 schrieb Johannes Brauer 
mailto:bra...@nordakademie.de>>
:

thank you, Las, for the quick tip. I will give Java 7 a try. I hope there are 
no problems on Mac OS 10.8.4

Johannes
Am 16.06.2013 um 22:15 schrieb László Török 
mailto:ltoro...@gmail.com>>
:

.. sorry, gmail's new annoying keyboard shortcut

b) include the dependency to the forkjoin library [1] that is not included in 
Java6

Las

[1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108


2013/6/16 László Török mailto:ltoro...@gmail.com>>
Hi,

there are two ways to deal with this:

a) use Java 7


2013/6/16 Johannes mailto:bra...@nordakademie.de>>
Hi,

trying
(require '[clojure.core.reducers :as r])
at the repl prompt the error message
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21)

What is going wrong?

Johannes



--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.





--
László Török



--
László Török

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.







Staatlich anerkannte private Fachhochschule
NORDAKADEMIE
Gemeinnützige Aktiengesellschaft
Köllner Chaussee 11
25337 Elmshorn

Vorstand:
Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. Vorstand)

Vorsitzender des Aufsichtsrats:
Dr. h.c. Hans-Heinrich Bruns

Sitz:
Elmshorn, Amtsgericht Pinneberg, HRB 1682


--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.







Staatlich anerkannte private Fachhochschule
NORDAKADEMIE
Gemeinnützige Aktiengesellschaft
Köllner Chaussee 11
25337 Elmshorn

Vorstand:
Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. Vorstand)

Vorsitzender des Aufsichtsrats:
Dr. h.c. Hans-Heinrich Bruns

Sitz:
Elmshorn, Amtsgericht Pinneberg, HRB 1682

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://g

Re: cannot import namespace reducers

[László Török]

I'm on Java7 and OS X 10.8.4, no problem over here. :)


2013/6/16 Johannes Brauer 

>  thank you, Las, for the quick tip. I will give Java 7 a try. I hope there
> are no problems on Mac OS 10.8.4
>
>  Johannes
>  Am 16.06.2013 um 22:15 schrieb László Török 
> :
>
>  .. sorry, gmail's new annoying keyboard shortcut
>
>  b) include the dependency to the forkjoin library [1] that is not
> included in Java6
>
>  Las
>
>  [1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108
>
>
> 2013/6/16 László Török 
>
>> Hi,
>>
>>  there are two ways to deal with this:
>>
>>  a) use Java 7
>>
>>
>> 2013/6/16 Johannes 
>>
>>> Hi,
>>>
>>>  trying
>>>  (require '[clojure.core.reducers :as r])
>>> at the repl prompt the error message
>>> CompilerException java.lang.ClassNotFoundException:
>>> jsr166y.ForkJoinPool, compiling:(clojure/core/reducers.clj:56:21)
>>>
>>>  What is going wrong?
>>>
>>>  Johannes
>>>
>>>
>>>
>>>  --
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>>
>>
>>   --
>> László Török
>>
>
>
>
>  --
> László Török
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
>
>
> --
>
>
> Staatlich anerkannte private Fachhochschule
> NORDAKADEMIE
> Gemeinnützige Aktiengesellschaft
> Köllner Chaussee 11
> 25337 Elmshorn
>
> Vorstand:
> Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv.
> Vorstand)
>
> Vorsitzender des Aufsichtsrats:
> Dr. h.c. Hans-Heinrich Bruns
>
> Sitz:
> Elmshorn, Amtsgericht Pinneberg, HRB 1682
>
>   --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
László Török

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: cannot import namespace reducers

[Johannes Brauer]

thank you, Las, for the quick tip. I will give Java 7 a try. I hope there are 
no problems on Mac OS 10.8.4

Johannes
Am 16.06.2013 um 22:15 schrieb László Török 
mailto:ltoro...@gmail.com>>
:

.. sorry, gmail's new annoying keyboard shortcut

b) include the dependency to the forkjoin library [1] that is not included in 
Java6

Las

[1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108


2013/6/16 László Török mailto:ltoro...@gmail.com>>
Hi,

there are two ways to deal with this:

a) use Java 7


2013/6/16 Johannes mailto:bra...@nordakademie.de>>
Hi,

trying
(require '[clojure.core.reducers :as r])
at the repl prompt the error message
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21)

What is going wrong?

Johannes



--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.





--
László Török



--
László Török

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to 
clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.







Staatlich anerkannte private Fachhochschule
NORDAKADEMIE
Gemeinnützige Aktiengesellschaft
Köllner Chaussee 11
25337 Elmshorn

Vorstand:
Prof. Dr. Georg Plate (Vorsitzender), Dipl.-Ing. Jörg Meier (stellv. Vorstand)

Vorsitzender des Aufsichtsrats:
Dr. h.c. Hans-Heinrich Bruns

Sitz:
Elmshorn, Amtsgericht Pinneberg, HRB 1682

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: cannot import namespace reducers

[László Török]

(require '[clojure.core.reducers :as r]) is correct.

(:require '[clojure.core.reducers :as r]) only works within the ns macro:
(ns 'yournamespace
  (:require '[clojure.core.reducers :as r]))

Las


2013/6/16 Mayank Jain 

> Try
>
> (:require '[clojure.core.reducers :as r])
>
>
> i.e. :require
>
>
> On Mon, Jun 17, 2013 at 1:38 AM, Johannes  wrote:
>
>> Hi,
>>
>> trying
>> (require '[clojure.core.reducers :as r])
>> at the repl prompt the error message
>> CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool,
>> compiling:(clojure/core/reducers.clj:56:21)
>>
>> What is going wrong?
>>
>> Johannes
>>
>>
>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
>
> --
> Regards,
> Mayank.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
László Török

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: cannot import namespace reducers

[László Török]

.. sorry, gmail's new annoying keyboard shortcut

b) include the dependency to the forkjoin library [1] that is not included
in Java6

Las

[1] http://mavenhub.com/mvn/central/org.coconut.forkjoin/jsr166y/070108


2013/6/16 László Török 

> Hi,
>
> there are two ways to deal with this:
>
> a) use Java 7
>
>
> 2013/6/16 Johannes 
>
>> Hi,
>>
>> trying
>> (require '[clojure.core.reducers :as r])
>> at the repl prompt the error message
>> CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool,
>> compiling:(clojure/core/reducers.clj:56:21)
>>
>> What is going wrong?
>>
>> Johannes
>>
>>
>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>
>
> --
> László Török
>



-- 
László Török

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: cannot import namespace reducers

[Mayank Jain]

Try

(:require '[clojure.core.reducers :as r])


i.e. :require


On Mon, Jun 17, 2013 at 1:38 AM, Johannes  wrote:

> Hi,
>
> trying
> (require '[clojure.core.reducers :as r])
> at the repl prompt the error message
> CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool,
> compiling:(clojure/core/reducers.clj:56:21)
>
> What is going wrong?
>
> Johannes
>
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
Regards,
Mayank.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: cannot import namespace reducers

[László Török]

Hi,

there are two ways to deal with this:

a) use Java 7


2013/6/16 Johannes 

> Hi,
>
> trying
> (require '[clojure.core.reducers :as r])
> at the repl prompt the error message
> CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool,
> compiling:(clojure/core/reducers.clj:56:21)
>
> What is going wrong?
>
> Johannes
>
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>



-- 
László Török

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

cannot import namespace reducers

[Johannes]

Hi,

trying
(require '[clojure.core.reducers :as r])
at the repl prompt the error message
CompilerException java.lang.ClassNotFoundException: jsr166y.ForkJoinPool, 
compiling:(clojure/core/reducers.clj:56:21) 

What is going wrong?

Johannes


-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Iota, reducers, word counting

[Catonano]

Well I reply to myself:

with pmap there´s no need to explicitly calculate the batches

The final version of the code is:

(defn parallel-count [iota-file]
  (reduce (partial merge-with +)
  (pmap (fn [a-subcollection]
  (frequencies (words a-subcollection)))
iota-file)))

which is not that dramatically far from the the vanilla version.

Right ?






2013/5/1 Catonano 

> Meikel,
>
> thank you for your reply
>
>
> 2013/4/30 Meikel Brandmeyer (kotarak) 
>
> Hello,
>>
>> your function does not follow the contract of fold. What you provided is
>> basically the reduce function, but it doesn't work together with fold. For
>> fold you need to merge the different maps you created in the subtasks. So
>> the combine function must look different.
>
>
> Admittedly I had missed the fact that fold expects a reducing function AND
> a combining function.
>
> But the worst thing is that I modified my-frequencies so that it could be
> called with no arguments, but who was going to call i like that ? Not my
> code and not anyone else !!
>
> What was I thinking ?
>
> I´m sorry for the silly question, I was just confused.
>
> Thanks for your help
>
> Additionally you cannot use transients at the moment with fold.
>>
>
> That´s a pity. I suppose I could explicitly split the input vector in
> parts and then call pvalues with a function implementing my word counting
> on each subpart. And then merging the results from each subpart
>
> But the reducers idiom is more beautiful because the modifications to the
> traditional map reduce version would be smaller, with that you haven´t got
> to explicitly split the input and call a specielized version of map.
>
> Oh well...
>
> It was just an exercise after all...
>
> Thanks again Meikel !
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Iota, reducers, word counting

[Catonano]

Meikel,

thank you for your reply


2013/4/30 Meikel Brandmeyer (kotarak) 

> Hello,
>
> your function does not follow the contract of fold. What you provided is
> basically the reduce function, but it doesn't work together with fold. For
> fold you need to merge the different maps you created in the subtasks. So
> the combine function must look different.


Admittedly I had missed the fact that fold expects a reducing function AND
a combining function.

But the worst thing is that I modified my-frequencies so that it could be
called with no arguments, but who was going to call i like that ? Not my
code and not anyone else !!

What was I thinking ?

I´m sorry for the silly question, I was just confused.

Thanks for your help

Additionally you cannot use transients at the moment with fold.
>

That´s a pity. I suppose I could explicitly split the input vector in parts
and then call pvalues with a function implementing my word counting on each
subpart. And then merging the results from each subpart

But the reducers idiom is more beautiful because the modifications to the
traditional map reduce version would be smaller, with that you haven´t got
to explicitly split the input and call a specielized version of map.

Oh well...

It was just an exercise after all...

Thanks again Meikel !

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Iota, reducers, word counting

[Meikel Brandmeyer (kotarak)]

Hello,

your function does not follow the contract of fold. What you provided is 
basically the reduce function, but it doesn't work together with fold. For 
fold you need to merge the different maps you created in the subtasks. So 
the combine function must look different.

(defn my-frequencies
  [coll]
  (r/fold
(fn
  ([] {})
  ([a b] (merge-with + a b)))
#(update-in %1 [%2] (fnil inc 0))
{}
coll))

Compare this example. The contract for the combine function requires that 
the combine function may be called with zero arguments to return the 
seeding value for the subtask. In our case an empty map. (This is were your 
example blows up.) To combine two maps we use merge-with. The update-in is 
basically the reduce function. The reducers library provides the monoid 
helper to ease the pain a little. (Although it doesn't buy us much in your 
example.)

(defn my-frequencies
  [coll]
  (r/fold
(r/monoid
  #(merge-with + %1 %2)
  (constantly {}))
#(update-in %1 [%2] (fnil inc 0))
{}
coll))

Additionally you cannot use transients at the moment with fold.

Kind regards
Meikel

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Iota, reducers, word counting

[Catonano]

Hello people,

I would like you to take a look at the question I posted here

http://meta.stackoverflow.com/questions/178326/why-is-this-question-not-good-enough

I´m trying to implement a word count of a large file using the machinery by
Iota ( https://github.com/thebusby/iota#readme ) just as an exercise

I´m stuck and I tried to post a question on stackoverflow, but my question
was not "up to the quality standards" of the site.

So I posted on meta and the guys there acknoledged tat my question was
decent.

Now I´d love to post it here but it would take too long and since it is
decent, I can safely ask you to look at it there.

Tanks for any hint

P.S. Of course I´d love you to comment on the Clojure related issues, not
about the meta issues about stacktrace of which I frankly care not so much

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Stanislav Yurin]

Indeed, that was my really bad example with 100.

On Saturday, April 27, 2013 4:19:07 PM UTC+3, Alan Busby wrote:
>
>
> On Sat, Apr 27, 2013 at 10:01 PM, Stanislav Yurin 
> 
> > wrote:
>
>> By the way, fold function has [n combinef reducef coll] implementation, 
>> where n is number of elements collection is folded by. 512 is just the 
>> default.
>>
>
> Yep I misspoke there, but it is still one of the tricks to be aware of.
>
> Lots of people do; 
> (->> (range 100)
>vec
>(r/filter even?)
>(r/fold +))
>
> And wonder why it's not running in parallel.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Alan Busby]

On Sat, Apr 27, 2013 at 10:01 PM, Stanislav Yurin  wrote:

> By the way, fold function has [n combinef reducef coll] implementation,
> where n is number of elements collection is folded by. 512 is just the
> default.
>

Yep I misspoke there, but it is still one of the tricks to be aware of.

Lots of people do;
(->> (range 100)
   vec
   (r/filter even?)
   (r/fold +))

And wonder why it's not running in parallel.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Stanislav Yurin]

Thanks Alan, looking into it.
By the way, fold function has [n combinef reducef coll] implementation, 
where n is number of elements collection is folded by. 512 is just the 
default.

On Saturday, April 27, 2013 3:51:39 PM UTC+3, Alan Busby wrote:
>
> On Sat, Apr 27, 2013 at 7:35 PM, Stanislav Yurin 
> 
> > wrote:
>
>> Actually, what I was trying to do, is to prototype multithreaded i/o 
>> operation via reducers. And then use fold to regulate number of concurrent 
>> operations.
>> But now something tells me I am doing not very clever thing.
>>
>
> I'm not entirely sure what you mean above, but I've been very happy using 
> reducers with I/O so far.
>
> Reducers just have a few tricks you need to be aware of first;
> 1. You wont get parallel processing unless the input is a vector.
> 2. Each thread gets ~512 elements each, so reducing a vector of 800 
> elements will only use two cores. 
> 3. How you aggregate your final result can greatly impact performance. 
> (Ex. (r/fold +) is fast, fold-into-vec is slower, etc)
>
> I wrote a library to use reducers over text files (input) and have found 
> it to be invaluable for working with giant TSV files.
> Link: https://github.com/thebusby/iota/
>
> I often then use fold-into-lazy-seq to write the output back to a file.
> Link: https://gist.github.com/thebusby/5472980
>
> Hope this helps! ;)
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Alan Busby]

On Sat, Apr 27, 2013 at 7:35 PM, Stanislav Yurin  wrote:

> Actually, what I was trying to do, is to prototype multithreaded i/o
> operation via reducers. And then use fold to regulate number of concurrent
> operations.
> But now something tells me I am doing not very clever thing.
>

I'm not entirely sure what you mean above, but I've been very happy using
reducers with I/O so far.

Reducers just have a few tricks you need to be aware of first;
1. You wont get parallel processing unless the input is a vector.
2. Each thread gets ~512 elements each, so reducing a vector of 800
elements will only use two cores.
3. How you aggregate your final result can greatly impact performance. (Ex.
(r/fold +) is fast, fold-into-vec is slower, etc)

I wrote a library to use reducers over text files (input) and have found it
to be invaluable for working with giant TSV files.
Link: https://github.com/thebusby/iota/

I often then use fold-into-lazy-seq to write the output back to a file.
Link: https://gist.github.com/thebusby/5472980

Hope this helps! ;)

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Stanislav Yurin]

Yep, thanks, my bad. I got the point.

Actually, what I was trying to do, is to prototype multithreaded i/o 
operation via reducers. And then use fold to regulate number of concurrent 
operations.
But now something tells me I am doing not very clever thing.

On Friday, April 26, 2013 5:27:46 PM UTC+3, Stanislav Yurin wrote:
>
> I was assuming that following code will fold in parallel, but it is 
> reduced sequentially
>
> (require '[clojure.core.reducers :as r])
> (defn test1 
> [x] 
> (Thread/sleep 1000) 
> (println (str "Finished:" x))
> x)
> (def xxx (r/map test1 (range 100)))
> (r/fold + xxx)
>
> What am I doing wrong?
> Thanks.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Jim - FooBar();]

+1 ! I use 'fold-into-vec' regularly :)

Jim

On 26/04/13 18:07, Alan Busby wrote:

Some additional pointers here (this is a little old though);
http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html


On Fri, Apr 26, 2013 at 11:51 PM, László Török > wrote:


Hi,

Not sure what you are trying to do, but xxx is a lazy seq, thus it
can only be consumed sequentially and fold falls back to reduce

You need a vector.

Las

Sent from my phone

On Apr 26, 2013 4:46 PM, "Stanislav Yurin" mailto:jusk...@gmail.com>> wrote:

I was assuming that following code will fold in parallel, but
it is reduced sequentially

(require '[clojure.core.reducers :as r])
(defn test1
[x]
(Thread/sleep 1000)
(println (str "Finished:" x))
x)
(def xxx (r/map test1 (range 100)))
(r/fold + xxx)

What am I doing wrong?
Thanks.
-- 
-- 
You received this message because you are subscribed to the Google

Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com

Note that posts from new members are moderated - please be
patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the
Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to clojure+unsubscr...@googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.


-- 
-- 
You received this message because you are subscribed to the Google

Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com

Note that posts from new members are moderated - please be patient
with your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to clojure+unsubscr...@googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.



--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient 
with your first post.

To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google 
Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to clojure+unsubscr...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.




--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[Alan Busby]

Some additional pointers here (this is a little old though);
http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html


On Fri, Apr 26, 2013 at 11:51 PM, László Török  wrote:

> Hi,
>
> Not sure what you are trying to do, but xxx is a lazy seq, thus it can
> only be consumed sequentially and fold falls back to reduce
>
> You need a vector.
>
> Las
>
> Sent from my phone
> On Apr 26, 2013 4:46 PM, "Stanislav Yurin"  wrote:
>
>> I was assuming that following code will fold in parallel, but it is
>> reduced sequentially
>>
>> (require '[clojure.core.reducers :as r])
>> (defn test1
>>  [x]
>> (Thread/sleep 1000)
>>  (println (str "Finished:" x))
>> x)
>> (def xxx (r/map test1 (range 100)))
>> (r/fold + xxx)
>>
>> What am I doing wrong?
>> Thanks.
>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Reducers newbie question

[László Török]

Hi,

Not sure what you are trying to do, but xxx is a lazy seq, thus it can only
be consumed sequentially and fold falls back to reduce

You need a vector.

Las

Sent from my phone
On Apr 26, 2013 4:46 PM, "Stanislav Yurin"  wrote:

> I was assuming that following code will fold in parallel, but it is
> reduced sequentially
>
> (require '[clojure.core.reducers :as r])
> (defn test1
> [x]
> (Thread/sleep 1000)
> (println (str "Finished:" x))
> x)
> (def xxx (r/map test1 (range 100)))
> (r/fold + xxx)
>
> What am I doing wrong?
> Thanks.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Reducers newbie question

[Stanislav Yurin]

I was assuming that following code will fold in parallel, but it is reduced 
sequentially

(require '[clojure.core.reducers :as r])
(defn test1 
[x] 
(Thread/sleep 1000) 
(println (str "Finished:" x))
x)
(def xxx (r/map test1 (range 100)))
(r/fold + xxx)

What am I doing wrong?
Thanks.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: would FixedThreadPool-backed reducers make sense?

[Gary Verhaegen]

The implementation of Clojure sequence library predates protocols.
This is, however, the way forward, at least in clojurescript :

https://github.com/clojure/clojurescript/blob/master/src/cljs/cljs/core.cljs#L191

On 28 March 2013 14:49, vemv  wrote:
> I recall from Rich's presentation on reducers (and it's intuitively true
> anyway) that FJ is not well suited to all workloads: uniform ones would do
> just fine with a fixed allocation of tasks to threads. I believe the
> tradeoff in that case is that one has to manage parallelism very explicitly.
>
> Now, is it feasible to provide a reducers implementation backed by a fixed
> thead pool? How well would it perform (provided the workload is suitable)?
>
> Similarly, I'd find tempting (just if a little crazy) to define clj's "seq
> library" (map, filter, reduce...) as a huge protocol, providing e.g.
> simple-lazy, chunked, FJ, and fixed-thread-pool implementations.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

would FixedThreadPool-backed reducers make sense?

[vemv]

I recall from Rich's presentation on reducers (and it's intuitively true 
anyway) that FJ is not well suited to all workloads: uniform ones would do 
just fine with a fixed allocation of tasks to threads. I believe the 
tradeoff in that case is that one has to manage parallelism very explicitly.

Now, is it feasible to provide a reducers implementation backed by a fixed 
thead pool? How well would it perform (provided the workload is suitable)?

Similarly, I'd find tempting (just if a little crazy) to define clj's "seq 
library" (map, filter, reduce...) as a huge protocol, providing e.g. 
simple-lazy, chunked, FJ, and fixed-thread-pool implementations.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[Alan Busby]

Hi Bernard,

I'm going to see if I can add some performance metrics in the test code to
try and determine where various bottlenecks may exist. If that goes well
I'll see if I can add a branch that returns Byte[] instead of String for
iota/vec.

I suspect this will just require modifying iota.FileVector.getChunk() and
iota.FileVector.cachedChunk .

I'm still not clear on a good way to handle ByteBuffer with offsets though,
because Java's mmap() doesn't support reading from files greater that
Integer/MAX_VALUE.

To work around this I've had to patch the following together;
https://github.com/thebusby/iota/blob/master/src/java/iota/Mmap.java

I do think I understand what you're asking for though.
Unfortunately for this level of tuning I still primarily fall back to C and
JNI, so my Java/Clojure is likely a little weak here.


Regarding why iota/vec returns nil on empty lines, it's to allow for
(->> (iota/vec "filename.tsv")
 (r/filter identity)
 ...


Hope this helps,
Alan


On Sat, Mar 9, 2013 at 9:41 AM, bernardH wrote:

> Hi Alan,
>
>
>
> On Friday, March 8, 2013 4:02:18 PM UTC+1, Alan Busby wrote:
>>
>> Hi Bernard,
>>
>> I'd certainly like to add support for binary files, but as I haven't had
>> a need for it myself I haven't had a good place to start.
>>
>> As Java NIO's mmap() doesn't support ranges over 2GB, I've had to paste
>> together multiple mmap's to cover files that are larger than 2GB.
>> So if a record ended up spanning two mmap()'s, you couldn't return the
>> raw data as a single object without copying it into a new buffer first.
>>
>> Also, if you provide a fixed record size in bytes for "doing the idx
>> offset maths", why do you need the end idx for the current line as well?
>> For example if you say file.bin is full of records each 100B in size, and
>> you ask for the 10th record; don't you already know that the length of the
>> record is 100B?
>>
>>
> Indeed, the correlation between txt/binary and char (i.e \n)
> delimited/fixed length record is very strong. However in my case I want to
> first handle a \n delimited (txt) file as binary for performance reasons.
> The context is that I have to consider all the lines of data, but might
> not have to do "heavy" processing on all of them, so I want to do as few
> work as possible on each line (i.e. not construct any java.lang.String).
> This is in no way Clojure specific, I have two implementations in Java of
> a small Minimum Spanning Tree program :
> - one is constructing Strings from all the lines:
> https://www.refheap.com/paste/12312
> - one is using offsets from a raw ByteBuffer :
> https://www.refheap.com/paste/12313
>
> As most of the lines are not really processed (just sorted according to
> the last field), being able to only peek at the relevant bytes instead of
> constructing full blown java.lang.Strings is a huge performance boost.
> FWIW, as far as performance i concerned, I draw the line not between
> Clojure and Java but between objects (constructed by copying some data
> somewhere on the heap) and arrays of primitive data types, because
> nowadays, cache locality trumps everything (once you got rid of reflection
> calls in Clojure, obviously).
>
> So ideally, maybe 2 x 2 combinations (String / offset in ByteArray) x
> (char delimited / fixed length) would be needed to cover all the needs.
>
> Thanks again for sharing your library !
>
> Cheers,
>
> Bernard
>
> PS: Is there a rationale for returning nil instead of empty String "" on
> empty lines with iota/vec?
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[bernardH]

Hi Alan,


On Friday, March 8, 2013 4:02:18 PM UTC+1, Alan Busby wrote:
>
> Hi Bernard, 
>
> I'd certainly like to add support for binary files, but as I haven't had a 
> need for it myself I haven't had a good place to start.
>
> As Java NIO's mmap() doesn't support ranges over 2GB, I've had to paste 
> together multiple mmap's to cover files that are larger than 2GB. 
> So if a record ended up spanning two mmap()'s, you couldn't return the raw 
> data as a single object without copying it into a new buffer first.
>
> Also, if you provide a fixed record size in bytes for "doing the idx 
> offset maths", why do you need the end idx for the current line as well?
> For example if you say file.bin is full of records each 100B in size, and 
> you ask for the 10th record; don't you already know that the length of the 
> record is 100B?
>
>
Indeed, the correlation between txt/binary and char (i.e \n) 
delimited/fixed length record is very strong. However in my case I want to 
first handle a \n delimited (txt) file as binary for performance reasons.
The context is that I have to consider all the lines of data, but might not 
have to do "heavy" processing on all of them, so I want to do as few work 
as possible on each line (i.e. not construct any java.lang.String).
This is in no way Clojure specific, I have two implementations in Java of a 
small Minimum Spanning Tree program :
- one is constructing Strings from all the lines: 
https://www.refheap.com/paste/12312
- one is using offsets from a raw ByteBuffer : 
https://www.refheap.com/paste/12313

As most of the lines are not really processed (just sorted according to the 
last field), being able to only peek at the relevant bytes instead of 
constructing full blown java.lang.Strings is a huge performance boost.
FWIW, as far as performance i concerned, I draw the line not between 
Clojure and Java but between objects (constructed by copying some data 
somewhere on the heap) and arrays of primitive data types, because 
nowadays, cache locality trumps everything (once you got rid of reflection 
calls in Clojure, obviously).

So ideally, maybe 2 x 2 combinations (String / offset in ByteArray) x (char 
delimited / fixed length) would be needed to cover all the needs.

Thanks again for sharing your library !

Cheers,

Bernard

PS: Is there a rationale for returning nil instead of empty String "" on 
empty lines with iota/vec?

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[Alan Busby]

Hi Bernard,

I'd certainly like to add support for binary files, but as I haven't had a
need for it myself I haven't had a good place to start.

As Java NIO's mmap() doesn't support ranges over 2GB, I've had to paste
together multiple mmap's to cover files that are larger than 2GB.
So if a record ended up spanning two mmap()'s, you couldn't return the raw
data as a single object without copying it into a new buffer first.

Also, if you provide a fixed record size in bytes for "doing the idx offset
maths", why do you need the end idx for the current line as well?
For example if you say file.bin is full of records each 100B in size, and
you ask for the 10th record; don't you already know that the length of the
record is 100B?

Hope I can help,
 Alan


On Fri, Mar 8, 2013 at 1:44 AM, bernardH wrote:

> Hi,
>
>
> On Wednesday, March 6, 2013 2:53:26 PM UTC+1, Alan Busby wrote:
>>
>>
>> With the release of Clojure 1.5 and it's new reducers, I figured this
>> would be a good time to release a library to help with file IO for
>> reducers. As reducers can only operate in parallel over specific
>> collections, like vec, it requires some work to use them against files.
>>
>> Iota wraps a text files using Java NIO's mmap() so you can treat files
>> larger than memory like a vector. Clojure's normal collection functions
>> (first, last, nth, etc) work, while enabling 1.5's reducer's to operate in
>> parallel for fold.
>>
>
> Thanks,  I already love your lib !
> However, in order to squeeze the last bits of performance, what I'd really
> need would to be directly access the raw data as byte arrays.
> The ideal API would provide my function with things looking and quaking
> like bytes arrays but doing the idx offset maths to directly access the
> maped data (read-only ), the next-to ideal would provide my function the
> ref to whole bytes array and give me the start and end idx for current line.
>
> (I'd like to port a Java code where the most significant performance boost
> was avoiding Strings creation when at all possible, which was possible by
> only peeking at most of the lines).
>
> Cheers,
>
> B.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[bernardH]

Hi,

On Wednesday, March 6, 2013 2:53:26 PM UTC+1, Alan Busby wrote:
>
>
> With the release of Clojure 1.5 and it's new reducers, I figured this 
> would be a good time to release a library to help with file IO for 
> reducers. As reducers can only operate in parallel over specific 
> collections, like vec, it requires some work to use them against files.
>
> Iota wraps a text files using Java NIO's mmap() so you can treat files 
> larger than memory like a vector. Clojure's normal collection functions 
> (first, last, nth, etc) work, while enabling 1.5's reducer's to operate in 
> parallel for fold.
>

Thanks,  I already love your lib !
However, in order to squeeze the last bits of performance, what I'd really 
need would to be directly access the raw data as byte arrays.
The ideal API would provide my function with things looking and quaking 
like bytes arrays but doing the idx offset maths to directly access the 
maped data (read-only ), the next-to ideal would provide my function the 
ref to whole bytes array and give me the start and end idx for current line.

(I'd like to port a Java code where the most significant performance boost 
was avoiding Strings creation when at all possible, which was possible by 
only peeking at most of the lines).

Cheers,

B.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[Paul deGrandis]

I agree, this is a really cool project.

Nice work!


-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[kovas boguta]

Very cool!

Been wanting to play with something like this.


On Wed, Mar 6, 2013 at 5:53 AM, Alan Busby  wrote:
> Hi all,
>
> With the release of Clojure 1.5 and it's new reducers, I figured this would
> be a good time to release a library to help with file IO for reducers. As
> reducers can only operate in parallel over specific collections, like vec,
> it requires some work to use them against files.
>
> Iota wraps a text files using Java NIO's mmap() so you can treat files
> larger than memory like a vector. Clojure's normal collection functions
> (first, last, nth, etc) work, while enabling 1.5's reducer's to operate in
> parallel for fold.
>
> Code is available on Github here: https://github.com/thebusby/iota
> Library is available on Clojars here: https://clojars.org/iota
>
>
> Hope it helps,
>Alan Busby
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: Iota 1.0.2 release (Reducers for text files)

[Michael Klishin]

2013/3/6 Alan Busby 

> Code is available on Github here: https://github.com/thebusby/iota
> Library is available on Clojars here: https://clojars.org/iota
>

Alan,

Please add dependency information to the README. Otherwise beginners
won't be able to use your project.

If you need an example: https://github.com/clojurewerkz/elastisch#artifacts
-- 
MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ANN: Iota 1.0.2 release (Reducers for text files)

[Alan Busby]

Hi all,

With the release of Clojure 1.5 and it's new reducers, I figured this would
be a good time to release a library to help with file IO for reducers. As
reducers can only operate in parallel over specific collections, like vec,
it requires some work to use them against files.

Iota wraps a text files using Java NIO's mmap() so you can treat files
larger than memory like a vector. Clojure's normal collection functions
(first, last, nth, etc) work, while enabling 1.5's reducer's to operate in
parallel for fold.

Code is available on Github here: https://github.com/thebusby/iota
Library is available on Clojars here: https://clojars.org/iota


Hope it helps,
   Alan Busby

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Alan Busby]

On Tue, Jan 8, 2013 at 12:18 AM, Wolodja Wentland  wrote:

> On Thu, Nov 01, 2012 at 22:34 +0900, Alan Busby wrote:
> > On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland 
> wrote:
>
> > > Oh, fold-into-map and fold-into-map-with would be wonderful and I
> tried to
> > > implement the former along the lines of fold-into-vec, but the
> performance was
> > > abysmal. I am now using fold-into-vec + r/map with zipmap which is
> better, but
> > > I wouldn't consider that optimal.
> >
> > The bottleneck with fold-into-map is that merge just doesn't scale up
> > well and you end up doing a significant amount of merging.
> > Currently a merge operation takes elements one at a time from one map,
> > and adds them one at a time to another.
> >
> > This is incredibly wasteful for large maps as you already have two
> > "pre-sorted" tries, and can pair each branch together recursively. If
> > each branch node stored the number of child nodes, then you can assign
> > different threads to work on different branches as well. This would be
> > perfect for reducers, but from a quick look it didn't appear that any
> > of the key internals were exposed to be taken advantage of.
> >
> > Unfortunately I haven't discovered a way to facilitate more efficient
> > merging without modifying the actual Clojure core source. I guess this
> > is my next step.
>
> Just stumbled over this old thread again and wondered if you found time to
> look into this again?
>

I've done a good bit of work here but haven't finished anything yet
unfortunately.

In the meantime I've been working on other reducer utility
functions/libraries like,
1. fold-into-lazy-seq ( https://gist.github.com/4479877 )
2. Iota, for treating large text files as input for a reducer (
https://github.com/thebusby/iota )

Unfortunately some changes in reducers between alpha2 and beta1 have slowed
performance down a bit, so I've been waiting for 1.5 to be released before
spending
too much time on it.

I guess all of the above is motivation for me to finally send in my Clojure
license stuff...

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Wolodja Wentland]

On Thu, Nov 01, 2012 at 22:34 +0900, Alan Busby wrote:
> On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland  wrote:

> > Oh, fold-into-map and fold-into-map-with would be wonderful and I tried to
> > implement the former along the lines of fold-into-vec, but the performance 
> > was
> > abysmal. I am now using fold-into-vec + r/map with zipmap which is better, 
> > but
> > I wouldn't consider that optimal.
> 
> The bottleneck with fold-into-map is that merge just doesn't scale up
> well and you end up doing a significant amount of merging.
> Currently a merge operation takes elements one at a time from one map,
> and adds them one at a time to another.
> 
> This is incredibly wasteful for large maps as you already have two
> "pre-sorted" tries, and can pair each branch together recursively. If
> each branch node stored the number of child nodes, then you can assign
> different threads to work on different branches as well. This would be
> perfect for reducers, but from a quick look it didn't appear that any
> of the key internals were exposed to be taken advantage of.
> 
> Unfortunately I haven't discovered a way to facilitate more efficient
> merging without modifying the actual Clojure core source. I guess this
> is my next step.

Just stumbled over this old thread again and wondered if you found time to
look into this again?
-- 
Wolodja 

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature

Re: group-by vs. reducers?

[Balint Erdi]

Thank you, Christophe, that makes total sense.

Interestingly, on my machine, the reducers version is still slower:

Run # 0
"Elapsed time: 1105.586 msecs"
"Elapsed time: 685.8 msecs"
"Elapsed time: 549.969 msecs"
Run # 1
"Elapsed time: 497.831 msecs"
"Elapsed time: 489.932 msecs"
"Elapsed time: 520.38 msecs"
Run # 2
"Elapsed time: 363.873 msecs"
"Elapsed time: 489.529 msecs"
"Elapsed time: 556.117 msecs"
Run # 3
"Elapsed time: 357.822 msecs"
"Elapsed time: 429.894 msecs"
"Elapsed time: 540.975 msecs"
Run # 4
"Elapsed time: 425.276 msecs"
"Elapsed time: 431.819 msecs"
"Elapsed time: 528.8 msecs"

I guess the JVM finds some serious optimizations during the runs for the 
non-reducer versions. I saw the same difference when using criterium for 
benchmarking.

Balint

On Friday, December 7, 2012 1:44:58 PM UTC+1, Christophe Grand wrote:
>
> Absolutely
>
>
> On Fri, Dec 7, 2012 at 1:29 PM, László Török 
> > wrote:
>
>> Hi,
>>
>> shouldn't 
>>
>> (fn [groups a]
>>   (assoc groups (f a) (conj (get groups a []) a)))
>>
>> be 
>>
>> (fn [groups a]
>>   (let [k (f a)]
>> (assoc groups k (conj (get groups k []) a
>>
>> ?
>>
>> Las
>>
>> 2012/12/7 Christophe Grand >
>>
>>> Hi,
>>>
>>> There are indeed too much allocationfoing on in your r/map.
>>> You don't need the rmap,
>>>
>>> Start from a plain old reduce like your reduce-by-naive, replace reduce 
>>> by r/fold, remove the seed and add the combine-fn (which now provides the 
>>> seed):
>>>
>>> (defn group-by-red [f coll]
>>>   (r/fold
>>> (partial merge-with concat)
>>> (fn [groups a]
>>>   (assoc groups (f a) (conj (get groups a []) a)))
>>> coll))
>>>
>>>
>>> Crude benchmark;
>>>  (let [v (vec (range 100))] 
>>>   (dotimes [n 5]
>>> (println "Run" n)
>>> (time (group-by #(mod % 29) v))
>>> (time (group-by-naive #(mod % 29) v))
>>> (time (group-by-red #(mod % 29) v
>>> Run 0
>>> "Elapsed time: 938.878 msecs"
>>> "Elapsed time: 778.161 msecs"
>>> "Elapsed time: 1035.997 msecs"
>>> Run 1
>>> "Elapsed time: 715.352 msecs"
>>> "Elapsed time: 771.913 msecs"
>>> "Elapsed time: 840.264 msecs"
>>> Run 2
>>> "Elapsed time: 656.588 msecs"
>>> "Elapsed time: 700.114 msecs"
>>> "Elapsed time: 777.323 msecs"
>>> Run 3
>>> "Elapsed time: 731.781 msecs"
>>> "Elapsed time: 713.547 msecs"
>>> "Elapsed time: 875.801 msecs"
>>> Run 4
>>> "Elapsed time: 689.711 msecs"
>>> "Elapsed time: 710.728 msecs"
>>> "Elapsed time: 1112.609 msecs"
>>> nil
>>>
>>> Let's play with the granularity (defaults to 512)
>>> (defn group-by-red [f coll]
>>>   (r/fold 2048
>>> (partial merge-with concat)
>>> (fn [groups a]
>>>   (assoc groups (f a) (conj (get groups a []) a)))
>>> coll))
>>>
>>> Re-benchmark:
>>> Run 0
>>> "Elapsed time: 810.676 msecs"
>>> "Elapsed time: 736.764 msecs"
>>> "Elapsed time: 547.651 msecs"
>>> Run 1
>>> "Elapsed time: 681.249 msecs"
>>> "Elapsed time: 850.046 msecs"
>>> "Elapsed time: 521.27 msecs"
>>> Run 2
>>> "Elapsed time: 669.385 msecs"
>>> "Elapsed time: 712.15 msecs"
>>> "Elapsed time: 518.502 msecs"
>>> Run 3
>>> "Elapsed time: 673.108 msecs"
>>> "Elapsed time: 745.688 msecs"
>>> "Elapsed time: 542.837 msecs"
>>> Run 4
>>> "Elapsed time: 654.196 msecs"
>>> "Elapsed time: 723.074 msecs"
>>> "Elapsed time: 506.861 msecs"
>>>
>>> Note that you are not exactly comparing apples to apples since 
>>> group-by-red is returning maps whose values are unrealized lazy sequence 
>>> (concats of concats of ocncats of vactors) instead of vectors
>>>
>>> (defn group-by-red [f coll]
>>>   (r/fold 2048
>>> (partial merge-with into)
>>> (fn [g

Re: group-by vs. reducers?

[Christophe Grand]

Absolutely


On Fri, Dec 7, 2012 at 1:29 PM, László Török  wrote:

> Hi,
>
> shouldn't
>
> (fn [groups a]
>   (assoc groups (f a) (conj (get groups a []) a)))
>
> be
>
> (fn [groups a]
>   (let [k (f a)]
> (assoc groups k (conj (get groups k []) a
>
> ?
>
> Las
>
> 2012/12/7 Christophe Grand 
>
>> Hi,
>>
>> There are indeed too much allocationfoing on in your r/map.
>> You don't need the rmap,
>>
>> Start from a plain old reduce like your reduce-by-naive, replace reduce
>> by r/fold, remove the seed and add the combine-fn (which now provides the
>> seed):
>>
>> (defn group-by-red [f coll]
>>   (r/fold
>> (partial merge-with concat)
>> (fn [groups a]
>>   (assoc groups (f a) (conj (get groups a []) a)))
>> coll))
>>
>>
>> Crude benchmark;
>>  (let [v (vec (range 100))]
>>   (dotimes [n 5]
>> (println "Run" n)
>> (time (group-by #(mod % 29) v))
>> (time (group-by-naive #(mod % 29) v))
>> (time (group-by-red #(mod % 29) v
>> Run 0
>> "Elapsed time: 938.878 msecs"
>> "Elapsed time: 778.161 msecs"
>> "Elapsed time: 1035.997 msecs"
>> Run 1
>> "Elapsed time: 715.352 msecs"
>> "Elapsed time: 771.913 msecs"
>> "Elapsed time: 840.264 msecs"
>> Run 2
>> "Elapsed time: 656.588 msecs"
>> "Elapsed time: 700.114 msecs"
>> "Elapsed time: 777.323 msecs"
>> Run 3
>> "Elapsed time: 731.781 msecs"
>> "Elapsed time: 713.547 msecs"
>> "Elapsed time: 875.801 msecs"
>> Run 4
>> "Elapsed time: 689.711 msecs"
>> "Elapsed time: 710.728 msecs"
>> "Elapsed time: 1112.609 msecs"
>> nil
>>
>> Let's play with the granularity (defaults to 512)
>> (defn group-by-red [f coll]
>>   (r/fold 2048
>> (partial merge-with concat)
>> (fn [groups a]
>>   (assoc groups (f a) (conj (get groups a []) a)))
>> coll))
>>
>> Re-benchmark:
>> Run 0
>> "Elapsed time: 810.676 msecs"
>> "Elapsed time: 736.764 msecs"
>> "Elapsed time: 547.651 msecs"
>> Run 1
>> "Elapsed time: 681.249 msecs"
>> "Elapsed time: 850.046 msecs"
>> "Elapsed time: 521.27 msecs"
>> Run 2
>> "Elapsed time: 669.385 msecs"
>> "Elapsed time: 712.15 msecs"
>> "Elapsed time: 518.502 msecs"
>> Run 3
>> "Elapsed time: 673.108 msecs"
>> "Elapsed time: 745.688 msecs"
>> "Elapsed time: 542.837 msecs"
>> Run 4
>> "Elapsed time: 654.196 msecs"
>> "Elapsed time: 723.074 msecs"
>> "Elapsed time: 506.861 msecs"
>>
>> Note that you are not exactly comparing apples to apples since
>> group-by-red is returning maps whose values are unrealized lazy sequence
>> (concats of concats of ocncats of vactors) instead of vectors
>>
>> (defn group-by-red [f coll]
>>   (r/fold 2048
>> (partial merge-with into)
>> (fn [groups a]
>>   (assoc groups (f a) (conj (get groups a []) a)))
>> coll))
>>
>> Run 0
>> "Elapsed time: 763.455 msecs"
>> "Elapsed time: 763.501 msecs"
>> "Elapsed time: 681.075 msecs"
>> Run 1
>> "Elapsed time: 645.52 msecs"
>> "Elapsed time: 731.545 msecs"
>> "Elapsed time: 476.381 msecs"
>> Run 2
>> "Elapsed time: 660.775 msecs"
>> "Elapsed time: 728.19 msecs"
>> "Elapsed time: 475.543 msecs"
>> Run 3
>> "Elapsed time: 657.255 msecs"
>> "Elapsed time: 725.995 msecs"
>> "Elapsed time: 494.038 msecs"
>> Run 4
>> "Elapsed time: 647.53 msecs"
>> "Elapsed time: 731.085 msecs"
>> "Elapsed time: 538.649 msecs"
>>
>> hth,
>>
>> Christophe
>>
>>
>> On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi wrote:
>>
>>> BTW I understood the most about reducers (still not quite there yet,
>>> though :) ) from Rich Hickey's talk at EuroClojure:
>>>
>>> https://vimeo.com/45561411
>>>
>>>
>>> On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
>>>>
>>>> Hey,
>>>>
>>>> Reducers is fascinating and q

Re: group-by vs. reducers?

[László Török]

Hi,

shouldn't

(fn [groups a]
  (assoc groups (f a) (conj (get groups a []) a)))

be

(fn [groups a]
  (let [k (f a)]
(assoc groups k (conj (get groups k []) a

?

Las

2012/12/7 Christophe Grand 

> Hi,
>
> There are indeed too much allocationfoing on in your r/map.
> You don't need the rmap,
>
> Start from a plain old reduce like your reduce-by-naive, replace reduce by
> r/fold, remove the seed and add the combine-fn (which now provides the
> seed):
>
> (defn group-by-red [f coll]
>   (r/fold
> (partial merge-with concat)
> (fn [groups a]
>   (assoc groups (f a) (conj (get groups a []) a)))
> coll))
>
>
> Crude benchmark;
>  (let [v (vec (range 100))]
>   (dotimes [n 5]
> (println "Run" n)
> (time (group-by #(mod % 29) v))
> (time (group-by-naive #(mod % 29) v))
> (time (group-by-red #(mod % 29) v
> Run 0
> "Elapsed time: 938.878 msecs"
> "Elapsed time: 778.161 msecs"
> "Elapsed time: 1035.997 msecs"
> Run 1
> "Elapsed time: 715.352 msecs"
> "Elapsed time: 771.913 msecs"
> "Elapsed time: 840.264 msecs"
> Run 2
> "Elapsed time: 656.588 msecs"
> "Elapsed time: 700.114 msecs"
> "Elapsed time: 777.323 msecs"
> Run 3
> "Elapsed time: 731.781 msecs"
> "Elapsed time: 713.547 msecs"
> "Elapsed time: 875.801 msecs"
> Run 4
> "Elapsed time: 689.711 msecs"
> "Elapsed time: 710.728 msecs"
> "Elapsed time: 1112.609 msecs"
> nil
>
> Let's play with the granularity (defaults to 512)
> (defn group-by-red [f coll]
>   (r/fold 2048
> (partial merge-with concat)
> (fn [groups a]
>   (assoc groups (f a) (conj (get groups a []) a)))
> coll))
>
> Re-benchmark:
> Run 0
> "Elapsed time: 810.676 msecs"
> "Elapsed time: 736.764 msecs"
> "Elapsed time: 547.651 msecs"
> Run 1
> "Elapsed time: 681.249 msecs"
> "Elapsed time: 850.046 msecs"
> "Elapsed time: 521.27 msecs"
> Run 2
> "Elapsed time: 669.385 msecs"
> "Elapsed time: 712.15 msecs"
> "Elapsed time: 518.502 msecs"
> Run 3
> "Elapsed time: 673.108 msecs"
> "Elapsed time: 745.688 msecs"
> "Elapsed time: 542.837 msecs"
> Run 4
> "Elapsed time: 654.196 msecs"
> "Elapsed time: 723.074 msecs"
> "Elapsed time: 506.861 msecs"
>
> Note that you are not exactly comparing apples to apples since
> group-by-red is returning maps whose values are unrealized lazy sequence
> (concats of concats of ocncats of vactors) instead of vectors
>
> (defn group-by-red [f coll]
>   (r/fold 2048
> (partial merge-with into)
> (fn [groups a]
>   (assoc groups (f a) (conj (get groups a []) a)))
> coll))
>
> Run 0
> "Elapsed time: 763.455 msecs"
> "Elapsed time: 763.501 msecs"
> "Elapsed time: 681.075 msecs"
> Run 1
> "Elapsed time: 645.52 msecs"
> "Elapsed time: 731.545 msecs"
> "Elapsed time: 476.381 msecs"
> Run 2
> "Elapsed time: 660.775 msecs"
> "Elapsed time: 728.19 msecs"
> "Elapsed time: 475.543 msecs"
> Run 3
> "Elapsed time: 657.255 msecs"
> "Elapsed time: 725.995 msecs"
> "Elapsed time: 494.038 msecs"
> Run 4
> "Elapsed time: 647.53 msecs"
> "Elapsed time: 731.085 msecs"
> "Elapsed time: 538.649 msecs"
>
> hth,
>
> Christophe
>
>
> On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi wrote:
>
>> BTW I understood the most about reducers (still not quite there yet,
>> though :) ) from Rich Hickey's talk at EuroClojure:
>>
>> https://vimeo.com/45561411
>>
>>
>> On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
>>>
>>> Hey,
>>>
>>> Reducers is fascinating and quite complex at the same time. I feel like
>>> "best practices" around it has not quite solidified yet.
>>>
>>> Here is how I made your example work:
>>>
>>> (ns group-by-reducers.core
>>>   (:require [clojure.core.reducers :as r :only [fold reduce map]])
>>>   (:require [criterium.core :as c]))
>>>
>>> (defn group-by-naive [f coll]
>>>   (reduce
>>> (fn [groups a]
>>>   (assoc groups (f a) (conj (get groups a []) a)))
>>> {}
>>> coll))
>>>
>>> (defn group-by-red [f coll]
>>>  

Re: group-by vs. reducers?

[Christophe Grand]

Hi,

There are indeed too much allocationfoing on in your r/map.
You don't need the rmap,

Start from a plain old reduce like your reduce-by-naive, replace reduce by
r/fold, remove the seed and add the combine-fn (which now provides the
seed):

(defn group-by-red [f coll]
  (r/fold
(partial merge-with concat)
(fn [groups a]
  (assoc groups (f a) (conj (get groups a []) a)))
coll))


Crude benchmark;
 (let [v (vec (range 100))]
  (dotimes [n 5]
(println "Run" n)
(time (group-by #(mod % 29) v))
(time (group-by-naive #(mod % 29) v))
(time (group-by-red #(mod % 29) v
Run 0
"Elapsed time: 938.878 msecs"
"Elapsed time: 778.161 msecs"
"Elapsed time: 1035.997 msecs"
Run 1
"Elapsed time: 715.352 msecs"
"Elapsed time: 771.913 msecs"
"Elapsed time: 840.264 msecs"
Run 2
"Elapsed time: 656.588 msecs"
"Elapsed time: 700.114 msecs"
"Elapsed time: 777.323 msecs"
Run 3
"Elapsed time: 731.781 msecs"
"Elapsed time: 713.547 msecs"
"Elapsed time: 875.801 msecs"
Run 4
"Elapsed time: 689.711 msecs"
"Elapsed time: 710.728 msecs"
"Elapsed time: 1112.609 msecs"
nil

Let's play with the granularity (defaults to 512)
(defn group-by-red [f coll]
  (r/fold 2048
(partial merge-with concat)
(fn [groups a]
  (assoc groups (f a) (conj (get groups a []) a)))
coll))

Re-benchmark:
Run 0
"Elapsed time: 810.676 msecs"
"Elapsed time: 736.764 msecs"
"Elapsed time: 547.651 msecs"
Run 1
"Elapsed time: 681.249 msecs"
"Elapsed time: 850.046 msecs"
"Elapsed time: 521.27 msecs"
Run 2
"Elapsed time: 669.385 msecs"
"Elapsed time: 712.15 msecs"
"Elapsed time: 518.502 msecs"
Run 3
"Elapsed time: 673.108 msecs"
"Elapsed time: 745.688 msecs"
"Elapsed time: 542.837 msecs"
Run 4
"Elapsed time: 654.196 msecs"
"Elapsed time: 723.074 msecs"
"Elapsed time: 506.861 msecs"

Note that you are not exactly comparing apples to apples since group-by-red
is returning maps whose values are unrealized lazy sequence (concats of
concats of ocncats of vactors) instead of vectors

(defn group-by-red [f coll]
  (r/fold 2048
(partial merge-with into)
(fn [groups a]
  (assoc groups (f a) (conj (get groups a []) a)))
coll))

Run 0
"Elapsed time: 763.455 msecs"
"Elapsed time: 763.501 msecs"
"Elapsed time: 681.075 msecs"
Run 1
"Elapsed time: 645.52 msecs"
"Elapsed time: 731.545 msecs"
"Elapsed time: 476.381 msecs"
Run 2
"Elapsed time: 660.775 msecs"
"Elapsed time: 728.19 msecs"
"Elapsed time: 475.543 msecs"
Run 3
"Elapsed time: 657.255 msecs"
"Elapsed time: 725.995 msecs"
"Elapsed time: 494.038 msecs"
Run 4
"Elapsed time: 647.53 msecs"
"Elapsed time: 731.085 msecs"
"Elapsed time: 538.649 msecs"

hth,

Christophe

On Fri, Dec 7, 2012 at 10:30 AM, Balint Erdi  wrote:

> BTW I understood the most about reducers (still not quite there yet,
> though :) ) from Rich Hickey's talk at EuroClojure:
>
> https://vimeo.com/45561411
>
>
> On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
>>
>> Hey,
>>
>> Reducers is fascinating and quite complex at the same time. I feel like
>> "best practices" around it has not quite solidified yet.
>>
>> Here is how I made your example work:
>>
>> (ns group-by-reducers.core
>>   (:require [clojure.core.reducers :as r :only [fold reduce map]])
>>   (:require [criterium.core :as c]))
>>
>> (defn group-by-naive [f coll]
>>   (reduce
>> (fn [groups a]
>>   (assoc groups (f a) (conj (get groups a []) a)))
>> {}
>> coll))
>>
>> (defn group-by-red [f coll]
>>   (letfn [(reduce-groups
>> ([] {})
>> ([g1 g2]
>>   (merge-with concat g1 g2)))]
>> (r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
>>
>> (defn -main
>>   [& args]
>>   (c/quick-bench (group-by #(mod % 29) (range 1)))
>>   (c/quick-bench (group-by-naive #(mod % 29) (range 1)))
>>   (c/quick-bench (group-by-red #(mod % 29) (range 1
>>
>> (available as a gist here: 
>> https://gist.github.com/**4232044<https://gist.github.com/4232044>
>> )
>>
>> The core and naive versions perform roughly equally (~4ms) while the
>> reducers version takes 10x as much time (~40ms) on my two-core laptop.
>>
>> I'm absolutely sure I'm doing something wrong (there is probably too much
>> memory allocated by

Re: group-by vs. reducers?

[Balint Erdi]

BTW I understood the most about reducers (still not quite there yet, though 
:) ) from Rich Hickey's talk at EuroClojure:

https://vimeo.com/45561411

On Friday, December 7, 2012 10:21:59 AM UTC+1, Balint Erdi wrote:
>
> Hey,
>
> Reducers is fascinating and quite complex at the same time. I feel like 
> "best practices" around it has not quite solidified yet.
>
> Here is how I made your example work:
>
> (ns group-by-reducers.core
>   (:require [clojure.core.reducers :as r :only [fold reduce map]])
>   (:require [criterium.core :as c]))
>
> (defn group-by-naive [f coll]
>   (reduce
> (fn [groups a]
>   (assoc groups (f a) (conj (get groups a []) a)))
> {}
> coll))
>
> (defn group-by-red [f coll]
>   (letfn [(reduce-groups
> ([] {})
> ([g1 g2]
>   (merge-with concat g1 g2)))]
> (r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll
>
> (defn -main
>   [& args]
>   (c/quick-bench (group-by #(mod % 29) (range 1)))
>   (c/quick-bench (group-by-naive #(mod % 29) (range 1)))
>   (c/quick-bench (group-by-red #(mod % 29) (range 1
>
> (available as a gist here: https://gist.github.com/4232044)
>
> The core and naive versions perform roughly equally (~4ms) while the 
> reducers version takes 10x as much time (~40ms) on my two-core laptop.
>
> I'm absolutely sure I'm doing something wrong (there is probably too much 
> memory allocated by the map function) and would like to hear the opinion of 
> someone more knowledgeable with reducers. I just did not want the subject 
> to be buried.
>
> Hope that still helps a bit,
> Balint
>
> ps. There are some tips and tricks in this post: 
> http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html
>
> On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
>>
>> Hi,
>>
>> As I was trying to wrap my head around the reducers library[1], I thought 
>> implementing group-by would be a good exercise to gain some insight.
>>
>> After spending a few hours with it, I'm still pretty much clueless, so 
>> hope to find someone here to help me out:
>>
>> So if I understood the reducer lingo introduced in [2],[3] and group-by 
>> correctly, it reduces the following reducing function on a collection
>>
>> (fn group-by-reducef [keyfn ret x]
>>  (let [k (keyfn x)]
>> (assoc ret k (conj (get ret k []) x
>>
>> where keyfn is provided by a partial function application.
>>
>> fold needs a combining function that takes two result maps that have 
>> already been grouped and merges them.
>> A naive implementation could look like
>>
>> (defn group-by-combinef
>>   ([] {})
>>   ([g1 g2]
>>  (persistent!
>>   (reduce (fn [res k v]
>> (assoc! res k (into (get res k []) v)))
>>   (transient g1) g2
>>
>> (defn group-by [f coll]
>>   (fold (partial gr-by-reducef f) gr-by-combinef coll))
>>
>> Now couple of questions:
>> 1) I expected fold to actually perform the operation, how can I force it 
>> to give me the result?
>> 2) Can somehow the actual reducing at the leaf nodes still take advantage 
>> of transient collections?
>> 3) I took a look at flatten as it seems the "closest" match. Again, if I 
>> call (flatten [[1 2] [2 4]]), I don't actually get the result. How do I get 
>> to the result?
>>
>>
>> Thanks!
>>
>> [1] 
>> https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj
>> [2] 
>> http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html
>> [3] http://clojure.com/blog/2012/05/15/anatomy-of-reducer.html
>> -- 
>> László Török
>>
>> 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: group-by vs. reducers?

[Balint Erdi]

Hey,

Reducers is fascinating and quite complex at the same time. I feel like 
"best practices" around it has not quite solidified yet.

Here is how I made your example work:

(ns group-by-reducers.core
  (:require [clojure.core.reducers :as r :only [fold reduce map]])
  (:require [criterium.core :as c]))

(defn group-by-naive [f coll]
  (reduce
(fn [groups a]
  (assoc groups (f a) (conj (get groups a []) a)))
{}
coll))

(defn group-by-red [f coll]
  (letfn [(reduce-groups
([] {})
([g1 g2]
  (merge-with concat g1 g2)))]
(r/fold reduce-groups (r/map #(hash-map (f %) [%]) coll

(defn -main
  [& args]
  (c/quick-bench (group-by #(mod % 29) (range 1)))
  (c/quick-bench (group-by-naive #(mod % 29) (range 1)))
  (c/quick-bench (group-by-red #(mod % 29) (range 1

(available as a gist here: https://gist.github.com/4232044)

The core and naive versions perform roughly equally (~4ms) while the 
reducers version takes 10x as much time (~40ms) on my two-core laptop.

I'm absolutely sure I'm doing something wrong (there is probably too much 
memory allocated by the map function) and would like to hear the opinion of 
someone more knowledgeable with reducers. I just did not want the subject 
to be buried.

Hope that still helps a bit,
Balint

ps. There are some tips and tricks in this 
post: http://www.thebusby.com/2012/07/tips-tricks-with-clojure-reducers.html

On Monday, December 3, 2012 5:04:17 PM UTC+1, Las wrote:
>
> Hi,
>
> As I was trying to wrap my head around the reducers library[1], I thought 
> implementing group-by would be a good exercise to gain some insight.
>
> After spending a few hours with it, I'm still pretty much clueless, so 
> hope to find someone here to help me out:
>
> So if I understood the reducer lingo introduced in [2],[3] and group-by 
> correctly, it reduces the following reducing function on a collection
>
> (fn group-by-reducef [keyfn ret x]
>  (let [k (keyfn x)]
> (assoc ret k (conj (get ret k []) x
>
> where keyfn is provided by a partial function application.
>
> fold needs a combining function that takes two result maps that have 
> already been grouped and merges them.
> A naive implementation could look like
>
> (defn group-by-combinef
>   ([] {})
>   ([g1 g2]
>  (persistent!
>   (reduce (fn [res k v]
> (assoc! res k (into (get res k []) v)))
>   (transient g1) g2
>
> (defn group-by [f coll]
>   (fold (partial gr-by-reducef f) gr-by-combinef coll))
>
> Now couple of questions:
> 1) I expected fold to actually perform the operation, how can I force it 
> to give me the result?
> 2) Can somehow the actual reducing at the leaf nodes still take advantage 
> of transient collections?
> 3) I took a look at flatten as it seems the "closest" match. Again, if I 
> call (flatten [[1 2] [2 4]]), I don't actually get the result. How do I get 
> to the result?
>
>
> Thanks!
>
> [1] 
> https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj
> [2] 
> http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html
> [3] http://clojure.com/blog/2012/05/15/anatomy-of-reducer.html
> -- 
> László Török
>
> 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

group-by vs. reducers?

[László Török]

Hi,

As I was trying to wrap my head around the reducers library[1], I thought
implementing group-by would be a good exercise to gain some insight.

After spending a few hours with it, I'm still pretty much clueless, so hope
to find someone here to help me out:

So if I understood the reducer lingo introduced in [2],[3] and group-by
correctly, it reduces the following reducing function on a collection

(fn group-by-reducef [keyfn ret x]
 (let [k (keyfn x)]
(assoc ret k (conj (get ret k []) x

where keyfn is provided by a partial function application.

fold needs a combining function that takes two result maps that have
already been grouped and merges them.
A naive implementation could look like

(defn group-by-combinef
  ([] {})
  ([g1 g2]
 (persistent!
  (reduce (fn [res k v]
(assoc! res k (into (get res k []) v)))
  (transient g1) g2

(defn group-by [f coll]
  (fold (partial gr-by-reducef f) gr-by-combinef coll))

Now couple of questions:
1) I expected fold to actually perform the operation, how can I force it to
give me the result?
2) Can somehow the actual reducing at the leaf nodes still take advantage
of transient collections?
3) I took a look at flatten as it seems the "closest" match. Again, if I
call (flatten [[1 2] [2 4]]), I don't actually get the result. How do I get
to the result?


Thanks!

[1]
https://github.com/clojure/clojure/blob/master/src/clj/clojure/core/reducers.clj
[2]
http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html
[3] http://clojure.com/blog/2012/05/15/anatomy-of-reducer.html
-- 
László Török

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Alan Busby]

On Thu, Nov 1, 2012 at 8:27 PM, Wolodja Wentland  wrote:
> It seems to me as if we are currently figuring out which (boilerplate?)
> functions are missing in reducers.clj and that we will have a nice and
> well-integrated library at the end.

To be fair, it's in beta and it's open source; so if anyone thinks it
needs something they can always volunteer. :)



>> > P.S. Would it be possible to have something like fold-into-vec in 
>> > clojure.reducers?
>>
>> Don't forget fold-into-map and fold-into-map-with, but both of those
>> will likely require a better merge/merge-with function for maps. :(
>
> Oh, fold-into-map and fold-into-map-with would be wonderful and I tried to
> implement the former along the lines of fold-into-vec, but the performance was
> abysmal. I am now using fold-into-vec + r/map with zipmap which is better, but
> I wouldn't consider that optimal.

The bottleneck with fold-into-map is that merge just doesn't scale up
well and you end up doing a significant amount of merging.
Currently a merge operation takes elements one at a time from one map,
and adds them one at a time to another.

This is incredibly wasteful for large maps as you already have two
"pre-sorted" tries, and can pair each branch together recursively. If
each branch node stored the number of child nodes, then you can assign
different threads to work on different branches as well. This would be
perfect for reducers, but from a quick look it didn't appear that any
of the key internals were exposed to be taken advantage of.

Unfortunately I haven't discovered a way to facilitate more efficient
merging without modifying the actual Clojure core source. I guess this
is my next step.


If you're interesting in using reducers over text files, I've uploaded
a library to facilitate that here;
http://github.com/thebusby/iota

I'll be cleaning up the code with more documentation and examples at
some point in the near future, and will send out an announcement then.
In the mean time though feel free to use it.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Wolodja Wentland]

On Thu, Nov 01, 2012 at 15:11 +0900, Alan Busby wrote:
> On Mon, Oct 29, 2012 at 10:00 PM, Wolodja Wentland  wrote:
> > I find this behaviour quite unfortunate because I now have to explicitly 
> > test
> > for nil? and ensure consistent behaviour. This inconsistency violates the
> > principle of least-surprise and I am not sure if the current behaviour is
> > intentional or merely an unfortunate implementation detail.
> 
> fold wont work in parallel on list/seqs/etc, so if you're trying to
> get the improved threading performance out of fold/fold-into-vec
> you'll have to supply a vector.

I am aware of that, but just tried to highlight the behaviour with nil. Your
ensure-vec function will actually do the right thing™ here, so I might just
start using that.

It seems to me as if we are currently figuring out which (boilerplate?)
functions are missing in reducers.clj and that we will have a nice and
well-integrated library at the end.

> > P.S. Would it be possible to have something like fold-into-vec in 
> > clojure.reducers?
> 
> Don't forget fold-into-map and fold-into-map-with, but both of those
> will likely require a better merge/merge-with function for maps. :(

Oh, fold-into-map and fold-into-map-with would be wonderful and I tried to
implement the former along the lines of fold-into-vec, but the performance was
abysmal. I am now using fold-into-vec + r/map with zipmap which is better, but
I wouldn't consider that optimal.
-- 
Wolodja 

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Alan Busby]

On Mon, Oct 29, 2012 at 10:00 PM, Wolodja Wentland  wrote:
> I find this behaviour quite unfortunate because I now have to explicitly test
> for nil? and ensure consistent behaviour. This inconsistency violates the
> principle of least-surprise and I am not sure if the current behaviour is
> intentional or merely an unfortunate implementation detail.

fold wont work in parallel on list/seqs/etc, so if you're trying to
get the improved threading performance out of fold/fold-into-vec
you'll have to supply a vector.

(vec nil) => []

Then;
(->> data
vec
(r/map inc)
fold-into-vec)



Unfortunately if you "vec" a vector, it'll actually do the work so you may want;

(defn ensure-vec [xs]
"Ensure's that the value provided is a vector"
(if (= (type xs) clojure.lang.PersistentVector)
  xs
  (vec xs)))



> P.S. Would it be possible to have something like fold-into-vec in 
> clojure.reducers?

Don't forget fold-into-map and fold-into-map-with, but both of those
will likely require a better merge/merge-with function for maps. :(



P.S.
As soon as I can find a moment, I can provide a JVM friendly library
for reducing over mmap'ed text files as well.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Herwig Hochleitner]

Created an issue: http://dev.clojure.org/jira/browse/CLJ-1098

Don't know if patch is welcome, but fix should be trivial, so not much is
lost if declined.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Wolodja Wentland]

On Tue, Oct 30, 2012 at 15:17 +0100, Herwig Hochleitner wrote:
>I've also run into this. Maybe this is just an oversight, since clojure
>handles nils gracefully almost everywhere else.
>Should CollFold and IKVReduce be extended to nil, or is there some
>rationale against it?

I would much rather prefer consistent behaviour than the current mixture and
can't really think of a good reason why we see the current behaviour. Maybe
someone more knowledgeable with the actual implementation of those function
can shed some light on the issue.

Guess it wouldn't be too far fetched to consider this to be a bug, but I
thought I ask here first.
-- 
Wolodja 

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[Herwig Hochleitner]

I've also run into this. Maybe this is just an oversight, since clojure
handles nils gracefully almost everywhere else.

Should CollFold and IKVReduce be extended to nil, or is there some
rationale against it?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: reduce, reduce-kv, map, mapv, reducers/map and nil

[abp]

>
> I find this behaviour quite unfortunate because I now have to explicitly 
> test 
> for nil? and ensure consistent behaviour.


Yes, especially unfortunate considering that Rich said the reducers lib 
could be used as a drop in replacement for core to speed up programs, or 
something along the lines.

On Monday, October 29, 2012 2:00:35 PM UTC+1, Wolodja Wentland wrote:
>
> Hi all, 
>
> I am currently testing performance of different reduce and map 
> implementations 
> in my programs and have problems because their treatment of nil is 
> different. 
> The "normal" clojure.core implementations of reduce and map work well when 
> called on nil, but reduce-kv and functions in clojure.reducers throw 
> exceptions. 
>
> --- snip --- 
> user=> (defn fold-into-vec 
>   "Provided a reducer, concatenate into a vector. 
>
>   Note: same as (into [] coll), but parallel." 
>   ([coll] 
>(r/fold (r/monoid into vector) conj coll)) 
>   ([n coll] 
>(r/fold n (r/monoid into vector) conj coll))) 
>
> user=> (map (fn [el] (* 2 el)) nil) 
> () 
> user=> (mapv (fn [el] (* 2 el)) nil) 
> [] 
> user=> (fold-into-vec (r/map (fn [el] (* 2 el)) nil)) 
> # implementation of method: :coll-fold of protocol: 
> #'clojure.core.reducers/CollFold found for class: nil> 
> user=> (reduce (fn [ret el] (+ el el)) {} nil) 
> {} 
> user=> (reduce (fn [ret [k v]] (+ k v)) {} nil) 
> {} 
> user=> (reduce-kv (fn [ret k v] (+ k v)) {} nil) 
> # implementation of method: :kv-reduce of protocol: 
> #'clojure.core.protocols/IKVReduce found for class: nil> 
> --- snip --- 
>
> I find this behaviour quite unfortunate because I now have to explicitly 
> test 
> for nil? and ensure consistent behaviour. This inconsistency violates the 
> principle of least-surprise and I am not sure if the current behaviour is 
> intentional or merely an unfortunate implementation detail. 
>
> Wouldn't it make sense if reduce-kv and r/map mirror the behaviour of 
> reduce, 
> map and mapv in core? 
>
> P.S. Would it be possible to have something like fold-into-vec in 
> clojure.reducers? 
> -- 
> Wolodja > 
>
> 4096R/CAF14EFC 
> 081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC 
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

reduce, reduce-kv, map, mapv, reducers/map and nil

[Wolodja Wentland]

Hi all,

I am currently testing performance of different reduce and map implementations
in my programs and have problems because their treatment of nil is different.
The "normal" clojure.core implementations of reduce and map work well when
called on nil, but reduce-kv and functions in clojure.reducers throw
exceptions.

--- snip ---
user=> (defn fold-into-vec
  "Provided a reducer, concatenate into a vector.

  Note: same as (into [] coll), but parallel."
  ([coll]
   (r/fold (r/monoid into vector) conj coll))
  ([n coll]
   (r/fold n (r/monoid into vector) conj coll)))

user=> (map (fn [el] (* 2 el)) nil)
()
user=> (mapv (fn [el] (* 2 el)) nil)
[]
user=> (fold-into-vec (r/map (fn [el] (* 2 el)) nil))
#
user=> (reduce (fn [ret el] (+ el el)) {} nil)
{}
user=> (reduce (fn [ret [k v]] (+ k v)) {} nil)
{}
user=> (reduce-kv (fn [ret k v] (+ k v)) {} nil)
#
--- snip ---

I find this behaviour quite unfortunate because I now have to explicitly test
for nil? and ensure consistent behaviour. This inconsistency violates the
principle of least-surprise and I am not sure if the current behaviour is
intentional or merely an unfortunate implementation detail.

Wouldn't it make sense if reduce-kv and r/map mirror the behaviour of reduce,
map and mapv in core?

P.S. Would it be possible to have something like fold-into-vec in 
clojure.reducers?
-- 
Wolodja 

4096R/CAF14EFC
081C B7CD FF04 2BA9 94EA  36B2 8B7F 7D30 CAF1 4EFC


signature.asc
Description: Digital signature

Re: trampoline not compatible with reducers?

[Jim - FooBar();]

On 16/10/12 21:45, Alan Malloy wrote:

But really I don't think trampoline is a very compelling problem-
solver here: the primary (only?) gain from trampoline is in reducing
stack depth, and I rather doubt that you can calculate minimax for
even a few hundred ply. So you're in no danger of a stackoverflow, but
are sacrificing readability and a smidge of performance to keep your
stack "really small" instead of "not big enough to be a problem".


I see what you mean...thanks for the detailed answer.

Jim

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: trampoline not compatible with reducers?

[Alan Malloy]

Reducers don't enter into the picture at all: this code is just as
broken with clojure.core/reduce and clojure.core/map. The immediate
problem that Kevin is trying to draw your attention to is that you are
calling (reduce max (map my-fn coll)), where my-fn sometimes returns a
number, and sometimes a thunk. Obviously max can't be expected to
compare thunks for you, so that's no good. His suggestion is to make
sure to trampoline the results of (map my-fn coll) until you get out a
list of numbers, and then find the max.

But really I don't think trampoline is a very compelling problem-
solver here: the primary (only?) gain from trampoline is in reducing
stack depth, and I rather doubt that you can calculate minimax for
even a few hundred ply. So you're in no danger of a stackoverflow, but
are sacrificing readability and a smidge of performance to keep your
stack "really small" instead of "not big enough to be a problem".

On Oct 16, 12:45 pm, "Jim - FooBar();"  wrote:
> 'my-min' and 'my-max' simply wrap core/min core/max. You mean i have to
> trampoline these calls as well? return something like this? :
>
> #(r/reduce (trampoline my-max) ;;was my-max
>                                     (r/map (fn [child] (minimize (:tree
> child) (dec d))) (:children tree)))
>
> Jim
>
> On 16/10/12 20:01, Kevin Downey wrote:
>
>
>
>
>
>
>
> > if you look further down the stacktrace (where it refers to your code
> > instead of clojure.lang.Numbers.lt) it will give you line numbers in
> > your code to look at.
>
> > you are calling these trampolined functions without trampoline.
>
> > On Tue, Oct 16, 2012 at 11:24 AM, Jim - FooBar();  
> > wrote:
> >> On 16/10/12 19:15, Kevin Downey wrote:
> >>> you are declaring the functions return doubles, but in fact returning
> >>> functions or doubles
> >> yes you're right (my bad) but the same thing happens without the
> >> type-hinting - albeit in a different place and different originating
> >> function:
>
> >> ClassCastException Clondie24.lib.search$search$maximize__3081$fn__3082
> >> cannot be cast to java.lang.Number  clojure.lang.Numbers.lt
> >> (Numbers.java:219)
>
> >> Jim
>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Clojure" group.
> >> To post to this group, send email to clojure@googlegroups.com
> >> Note that posts from new members are moderated - please be patient with 
> >> your
> >> first post.
> >> To unsubscribe from this group, send email to
> >> clojure+unsubscr...@googlegroups.com
> >> For more options, visit this group at
> >>http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: trampoline not compatible with reducers?

[Jim - FooBar();]

'my-min' and 'my-max' simply wrap core/min core/max. You mean i have to 
trampoline these calls as well? return something like this? :


#(r/reduce (trampoline my-max) ;;was my-max
   (r/map (fn [child] (minimize (:tree 
child) (dec d))) (:children tree)))


Jim


On 16/10/12 20:01, Kevin Downey wrote:

if you look further down the stacktrace (where it refers to your code
instead of clojure.lang.Numbers.lt) it will give you line numbers in
your code to look at.

you are calling these trampolined functions without trampoline.

On Tue, Oct 16, 2012 at 11:24 AM, Jim - FooBar();  wrote:

On 16/10/12 19:15, Kevin Downey wrote:

you are declaring the functions return doubles, but in fact returning
functions or doubles

yes you're right (my bad) but the same thing happens without the
type-hinting - albeit in a different place and different originating
function:

ClassCastException Clondie24.lib.search$search$maximize__3081$fn__3082
cannot be cast to java.lang.Number  clojure.lang.Numbers.lt
(Numbers.java:219)

Jim




--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en





--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: trampoline not compatible with reducers?

[Kevin Downey]

if you look further down the stacktrace (where it refers to your code
instead of clojure.lang.Numbers.lt) it will give you line numbers in
your code to look at.

you are calling these trampolined functions without trampoline.

On Tue, Oct 16, 2012 at 11:24 AM, Jim - FooBar();  wrote:
> On 16/10/12 19:15, Kevin Downey wrote:
>>
>> you are declaring the functions return doubles, but in fact returning
>> functions or doubles
>
> yes you're right (my bad) but the same thing happens without the
> type-hinting - albeit in a different place and different originating
> function:
>
> ClassCastException Clondie24.lib.search$search$maximize__3081$fn__3082
> cannot be cast to java.lang.Number  clojure.lang.Numbers.lt
> (Numbers.java:219)
>
> Jim
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en



-- 
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: trampoline not compatible with reducers?

[Jim - FooBar();]

On 16/10/12 19:15, Kevin Downey wrote:

you are declaring the functions return doubles, but in fact returning
functions or doubles
yes you're right (my bad) but the same thing happens without the 
type-hinting - albeit in a different place and different originating 
function:


ClassCastException Clondie24.lib.search$search$maximize__3081$fn__3082 
cannot be cast to java.lang.Number  clojure.lang.Numbers.lt 
(Numbers.java:219)


Jim



--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: trampoline not compatible with reducers?

[Kevin Downey]

you are declaring the functions return doubles, but in fact returning
functions or doubles

On Tue, Oct 16, 2012 at 11:06 AM, Jim - FooBar();  wrote:
> Hi everyone,
>
> I'm pretty sure i'm using trampoline the right way but still I get this
> exception:
>
> ClassCastException Clondie24.lib.search$search$minimize__3075$fn__3076
> cannot be cast to java.lang.Number
> Clondie24.lib.search/search/minimize--3075 (search.clj:47)
>
> here is the code:
>
> (defn search "The recursive bit of the min-max algorithm."
> [eval-fn tree depth]
> (letfn [(minimize ^double [tree d] (if (zero? d) (eval-fn (:root tree)
> (:direction tree))
> #(r/reduce my-min
>   (r/map (fn [child] (maximize (:tree child)
> (dec d))) (:children tree)
> (maximize ^double [tree d] (if (zero? d) (eval-fn (:root tree)
> (:direction tree))
> #(r/reduce my-max
>(r/map (fn [child] (minimize (:tree
> child) (dec d))) (:children tree)]
> (trampoline minimize tree (int depth
>
> Can anyone shine some light? Is it something that has to do with reducers?
>
> Jim
>
> ps: it works fine without trampoline and '#' behind 'r/reduce'.
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en



-- 
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

trampoline not compatible with reducers?

[Jim - FooBar();]

Hi everyone,

I'm pretty sure i'm using trampoline the right way but still I get this 
exception:


ClassCastException Clondie24.lib.search$search$minimize__3075$fn__3076 
cannot be cast to java.lang.Number  
Clondie24.lib.search/search/minimize--3075 (search.clj:47)


here is the code:

(defn search "The recursive bit of the min-max algorithm."
[eval-fn tree depth]
(letfn [(minimize ^double [tree d] (if (zero? d) (eval-fn (:root tree) 
(:direction tree))

#(r/reduce my-min
  (r/map (fn [child] (maximize (:tree 
child) (dec d))) (:children tree)
(maximize ^double [tree d] (if (zero? d) (eval-fn (:root tree) 
(:direction tree))

#(r/reduce my-max
   (r/map (fn [child] (minimize (:tree 
child) (dec d))) (:children tree)]

(trampoline minimize tree (int depth

Can anyone shine some light? Is it something that has to do with reducers?

Jim

ps: it works fine without trampoline and '#' behind 'r/reduce'.




--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Reducers reduce my performance

[Tassilo Horn]

Kevin Downey  writes:

Hi Kevin,

>> This is the new version using reducers (:as r).  The problem here is
>> that to stop the iteration, one has to reduce the intermediate result
>> in every step and check if new reachable vertices (n) could be found.
>> If so, we need to do another iteration.
>
> this is almost certainly the problem, if you are using reducers you
> should not make intermediate results concrete.  given the structure of
> your computation it may not be a good fit for reducers.

I guessed that, but wanted to be extra-sure.

> you might be able to restructure in such a way as return something
> with a custom impl of CollReduce that performs this computation when
> reduced, which would get you out of returning concrete results.

Hm, yes, sounds good.  I think, I need to have another look at the
reducers implementation...

Thanks for the hint,
Tassilo

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Reducers reduce my performance

[Kevin Downey]

On Fri, Sep 14, 2012 at 4:22 AM, Tassilo Horn  wrote:
> Hi all,
>
> I have some code which uses a lot of map/mapcat/filter stuff and is
> totally eager (result is always an ordered set).  That looked to me like
> a good candidate for reducers.
>
> Basically, my code enables me to write something like this:
>
> --8<---cut here---start->8---
> (reachables p [p-seq
> [--> 'ContainsType]
> [p-+ [p-seq
>[<-- 'Defines]
>[--> 'Imports]
>[p-opt [<-- 'ContainsType])
> --8<---cut here---end--->8---
>
> which calculates the set of reachable vertices in a graph starting at a
> vertex p (or a set of vertices p) which can be reached by traversing a
> path matching the structure given as vector.  So from p, we navigate a
> sequence (p-seq) of one forward ContainsType-edge, then one-or-many
> times (p-+) traversing the sequence of an incoming Defines-edge followed
> by an outgoing Imports-edge followed by optionally (p-opt) an incoming
> ContainsType-edge.
>
> p-seq, p-+, p-opt, <--, --> are all functions receiving a set of
> vertices and the nested rest of the vector or edge type symbol.  The
> current implementation simply recurses over the vector and applies the
> functions, combining their results.
>
> Ok, that works very good and performs well, but I've though reducers
> might give me some extra-performance.  So now I rewrote all those
> functions to use reducers and return reducibles instead of ordered sets.
> The shape has changed to this, so instead of recursively applying the
> functions in a vector we create one big reducer function that does the
> job, and reachables simply applies that to p.
>
> --8<---cut here---start->8---
> (reachables p (p-seq
>(--> 'ContainsType)
>(p-+ (p-seq
>  (<-- 'Defines)
>  (--> 'Imports)
>  (p-opt (<-- 'ContainsType))
> --8<---cut here---end--->8---
>
> The good thing is that it calculates exactly the same set.  The bad
> thing is that it's about a factor of 2 or 3 slower.
>
> I tried to track down the cause, and the main bottleneck is the p-+
> function which got much slower than before.
>
> This is the new version using reducers (:as r).  The problem here is
> that to stop the iteration, one has to reduce the intermediate result in
> every step and check if new reachable vertices (n) could be found.  If
> so, we need to do another iteration.

this is almost certainly the problem, if you are using reducers you
should not make intermediate results concrete. given the structure of
your computation it may not be a good fit for reducers.

you might be able to restructure in such a way as return something
with a custom impl of CollReduce that performs this computation when
reduced, which would get you out of returning concrete results.

> --8<---cut here---start->8---
> (defn ^:private p-*-or-+
>   [p include-coll]
>   (fn [coll] ;; coll is a reducible
> (loop [ret (if include-coll
>  (into (ordered.set/ordered-set) coll)
>  (ordered.set/ordered-set))
>n (into (ordered.set/ordered-set)
>(if include-coll
>  (r/remove ret (p coll))
>  (p coll)))]
>   (if (seq n)
> (recur (into ret n)
>(into (ordered.set/ordered-set)
>  (r/remove ret (p n
> ret
>
> (defn p-+ [p]
>   (p-*-or-+ p false))
> --8<---cut here---end--->8---
>
> This is the original version, where the ordered set of vertices to start
> with is already given as first parameter, and the fn does the job itself
> rather than returning a fn that can do it.  (As you can see, the old
> version uses reducers internally, too, but the functions themselves all
> receive and return sets).
>
> --8<---cut here---start->8---
> (defn ^:private p-*-or-+
>   [v p ret]
>   (let [n (into (ordered.set/ordered-set)
> ;; oset is like set for ordered-sets
> (r/remove ret (oset (*p-apply* v p]
> (if (seq n)
>   (recur n p (into-oset ret n))
>   ret)))
>
> (defn p-* [v p]
>   (p-*-or-+ v p (oset v)))
> --8<---cut here---end--->8---
>
> S

Reducers reduce my performance

[Tassilo Horn]

Hi all,

I have some code which uses a lot of map/mapcat/filter stuff and is
totally eager (result is always an ordered set).  That looked to me like
a good candidate for reducers.

Basically, my code enables me to write something like this:

--8<---cut here---start->8---
(reachables p [p-seq
[--> 'ContainsType]
[p-+ [p-seq
   [<-- 'Defines]
   [--> 'Imports]
   [p-opt [<-- 'ContainsType])
--8<---cut here---end--->8---

which calculates the set of reachable vertices in a graph starting at a
vertex p (or a set of vertices p) which can be reached by traversing a
path matching the structure given as vector.  So from p, we navigate a
sequence (p-seq) of one forward ContainsType-edge, then one-or-many
times (p-+) traversing the sequence of an incoming Defines-edge followed
by an outgoing Imports-edge followed by optionally (p-opt) an incoming
ContainsType-edge.

p-seq, p-+, p-opt, <--, --> are all functions receiving a set of
vertices and the nested rest of the vector or edge type symbol.  The
current implementation simply recurses over the vector and applies the
functions, combining their results.

Ok, that works very good and performs well, but I've though reducers
might give me some extra-performance.  So now I rewrote all those
functions to use reducers and return reducibles instead of ordered sets.
The shape has changed to this, so instead of recursively applying the
functions in a vector we create one big reducer function that does the
job, and reachables simply applies that to p.

--8<---cut here---start->8---
(reachables p (p-seq
   (--> 'ContainsType)
   (p-+ (p-seq
 (<-- 'Defines)
 (--> 'Imports)
 (p-opt (<-- 'ContainsType))
--8<---cut here---end--->8---

The good thing is that it calculates exactly the same set.  The bad
thing is that it's about a factor of 2 or 3 slower.

I tried to track down the cause, and the main bottleneck is the p-+
function which got much slower than before.

This is the new version using reducers (:as r).  The problem here is
that to stop the iteration, one has to reduce the intermediate result in
every step and check if new reachable vertices (n) could be found.  If
so, we need to do another iteration.

--8<---cut here---start->8---
(defn ^:private p-*-or-+
  [p include-coll]
  (fn [coll] ;; coll is a reducible
(loop [ret (if include-coll
 (into (ordered.set/ordered-set) coll)
 (ordered.set/ordered-set))
   n (into (ordered.set/ordered-set)
   (if include-coll
 (r/remove ret (p coll))
 (p coll)))]
  (if (seq n)
(recur (into ret n)
   (into (ordered.set/ordered-set)
 (r/remove ret (p n
ret

(defn p-+ [p]
  (p-*-or-+ p false))
--8<---cut here---end--->8---

This is the original version, where the ordered set of vertices to start
with is already given as first parameter, and the fn does the job itself
rather than returning a fn that can do it.  (As you can see, the old
version uses reducers internally, too, but the functions themselves all
receive and return sets).

--8<---cut here---start->8---
(defn ^:private p-*-or-+
  [v p ret]
  (let [n (into (ordered.set/ordered-set)
;; oset is like set for ordered-sets
(r/remove ret (oset (*p-apply* v p]
(if (seq n)
  (recur n p (into-oset ret n))
  ret)))

(defn p-* [v p]
  (p-*-or-+ v p (oset v)))
--8<---cut here---end--->8---

So is that simply a use-case where reducers don't fit in very well, or
am I doing something wrong?

Thanks for any hints,
Tassilo

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en