Re: Issue when moving to Clojure 1.10

[Didier]

Okay, so after reading through the linked issue here: 
https://dev.clojure.org/jira/browse/CLJ-322 I'm not sure, as a tool 
builder, what is the ideal path forward.

This is what I seem to understand would be ideal, let me know if I'm wrong:

1) AOT compile everything. So basically, always AOT every lib. This is to 
make sure that we don't face this issue: 
https://dev.clojure.org/jira/browse/CLJ-1544

2) Figure out some mechanism within the built tooling to only include the 
classes for which the source exist in the current project being built into 
the resulting Jar. Meaning, only namespaces found in the source folder 
should have their compiled classes be included into the Jar.

This setup should minimize conflicts, from what I understand. The only 
thing I'm not sure about is, are classes compiled with an older version of 
Clojure and JDK always going to work with a newer version of Clojure and 
JDK?

On Friday, 25 January 2019 22:11:46 UTC-8, Didier wrote:
>
> So I got to the bottom bottom of the problem here.
>
> This is a scenario:
>
> 1) Library A depends on library B and Clojure 1.10
> 2) Library B must be AOTed due to a gen-class, but depends on Clojure 1.9
> 3) Library A does not work, because it is now using the Clojure core spec 
> 1.9 compiled transitively through the Library B AOT, and found in its Jar, 
> since .class get precedence over source files.
>
> So, this means that a library with a dependency on another one that is AOT 
> can cause spec conflicts.
>
> This is new now that spec is out. And so I first caught this when some of 
> our libs were using 1.9, and I started moving others to 1.10.
>
> The way I solved this is by hacking our build, so that the clojure 
> compiled classes from the AOT don't get included in the Jar after they are 
> compiled.
>
> In my opinion, this is a bit of a problem. What would be nice is either to 
> have a way to compile only a gen-class, and not the namespace that contains 
> it. Or not compile things transitively. Or maybe just a way for compile to 
> exclude clojure core namespaces.
>
> Now, if you depend on libraries that don't need AOT, it is not an issue. 
> But if you do, it forces you to re-compile all your dependencies and do a 
> full big bang upgrade to the new Clojure version. You can't just use libs 
> compiled with older versions of spec. While before, you were able to use 
> libs compiled with older version of Clojure from packages that use newer 
> version of Clojure.
>
> On Wednesday, 16 January 2019 20:47:43 UTC-8, Mark Derricutt wrote:
>>
>> On 16 Jan 2019, at 18:17, Alex Miller wrote:
>>
>> Yes, it's one of the downsides of AOT. 
>>
>> I'd say it's not so much a downside of AOT - but of having a single 
>> output path for classes. I've long wanted Chas's patch to be applied, or 
>> something like it.
>>
>> Having everyone reinvent the mechanism whenever they happen to need AOT 
>> is kind of annoying - rare, but still annoying.
>> --
>>
>> "The ease with which a change can be implemented has no relevance at all 
>> to whether it is the right change for the (Java) Platform for all time." — 
>> Mark Reinhold.
>>
>> Mark Derricutt
>> http://www.theoryinpractice.net
>> http://www.chaliceofblood.net
>> http://plus.google.com/+MarkDerricutt
>> http://twitter.com/talios
>> http://facebook.com/mderricutt
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue when moving to Clojure 1.10

[Didier]

So I got to the bottom bottom of the problem here.

This is a scenario:

1) Library A depends on library B and Clojure 1.10
2) Library B must be AOTed due to a gen-class, but depends on Clojure 1.9
3) Library A does not work, because it is now using the Clojure core spec 
1.9 compiled transitively through the Library B AOT, and found in its Jar, 
since .class get precedence over source files.

So, this means that a library with a dependency on another one that is AOT 
can cause spec conflicts.

This is new now that spec is out. And so I first caught this when some of 
our libs were using 1.9, and I started moving others to 1.10.

The way I solved this is by hacking our build, so that the clojure compiled 
classes from the AOT don't get included in the Jar after they are compiled.

In my opinion, this is a bit of a problem. What would be nice is either to 
have a way to compile only a gen-class, and not the namespace that contains 
it. Or not compile things transitively. Or maybe just a way for compile to 
exclude clojure core namespaces.

Now, if you depend on libraries that don't need AOT, it is not an issue. 
But if you do, it forces you to re-compile all your dependencies and do a 
full big bang upgrade to the new Clojure version. You can't just use libs 
compiled with older versions of spec. While before, you were able to use 
libs compiled with older version of Clojure from packages that use newer 
version of Clojure.

On Wednesday, 16 January 2019 20:47:43 UTC-8, Mark Derricutt wrote:
>
> On 16 Jan 2019, at 18:17, Alex Miller wrote:
>
> Yes, it's one of the downsides of AOT. 
>
> I'd say it's not so much a downside of AOT - but of having a single output 
> path for classes. I've long wanted Chas's patch to be applied, or something 
> like it.
>
> Having everyone reinvent the mechanism whenever they happen to need AOT is 
> kind of annoying - rare, but still annoying.
> --
>
> "The ease with which a change can be implemented has no relevance at all 
> to whether it is the right change for the (Java) Platform for all time." — 
> Mark Reinhold.
>
> Mark Derricutt
> http://www.theoryinpractice.net
> http://www.chaliceofblood.net
> http://plus.google.com/+MarkDerricutt
> http://twitter.com/talios
> http://facebook.com/mderricutt
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: r/fold combinef and reducef init values

[Sean Corfield]

Which docs are you reading? The docstring for r/fold says this – with no 
indication of calling (reducef) with no arguments (well, unless you do not 
supply combinef – in which case reducef will be used for that, so (reducef) 
would be called to seed the reductions):

"Reduces a collection using a (potentially parallel) reduce-combine
  strategy. The collection is partitioned into groups of approximately
  n (default 512), each of which is reduced with reducef (with a seed
  value obtained by calling (combinef) with no arguments). The results
  of these reductions are then reduced with combinef (default
  reducef). combinef must be associative, and, when called with no
  arguments, (combinef) must produce its identity element. These
  operations may be performed in parallel, but the results will
  preserve order."

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

From: Brian Craft
Sent: Friday, January 25, 2019 3:36 PM
Subject: r/fold combinef and reducef init values

>From the docs:

r/fold takes a reducible collection and partitions it into groups of 
approximately n (default 512) elements. Each group is reduced using the reducef 
function. The reducef function will be called with no arguments to produce an 
identity value in each partition. The results of those reductions are then 
reduced with the combinef (defaults to reducef) function. When called with no 
arguments, (combinef) must produce its identity element - this will be called 
multiple times. Operations may be performed in parallel. Results will preserve 
order.

So, this seems to say r/fold will partition the collection and reduce each 
partition using the (reducef) as the init value.

Then, all these intermediate results will be reduced with combinef, using 
(combinef) as the init value.

However, in test it seems (reducef) is never called, and (combinef) is used as 
the init value for calls to reducef.

  (defn combinef
([] {:combine :f})
([acc v] acc))

  (defn reducef
([] {:reduce :f})
([acc v]
 (println "acc" acc "v" v)
 v))

  (clojure.core.reducers/fold combinef reducef ["foo" "bar"])

; outputs:
acc {:combine :f} v foo
acc foo v bar
"bar"

The accumulator in reducef is the init value from combinef, not the init value 
from reducef.

What's going on?
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: r/fold combinef and reducef init values

[Brian Craft]

Looks like it's something that's changed over different clojure releases.

On Friday, January 25, 2019 at 3:35:58 PM UTC-8, Brian Craft wrote:
>
> From the docs:
>
> r/fold takes a reducible collection and partitions it into groups of 
> approximately n (default 512) elements. Each group is reduced using the 
> reducef function. The reducef function will be called with no arguments to 
> produce an identity value in each partition. The results of those 
> reductions are then reduced with the combinef (defaults to reducef) 
> function. When called with no arguments, (combinef) must produce its 
> identity element - this will be called multiple times. Operations may be 
> performed in parallel. Results will preserve order.
>
> So, this seems to say r/fold will partition the collection and reduce each 
> partition using the (reducef) as the init value.
>
> Then, all these intermediate results will be reduced with combinef, using 
> (combinef) as the init value.
>
> However, in test it seems (reducef) is never called, and (combinef) is 
> used as the init value for calls to reducef.
>
>   (defn combinef
> ([] {:combine :f})
> ([acc v] acc))
>
>   (defn reducef
> ([] {:reduce :f})
> ([acc v]
>  (println "acc" acc "v" v)
>  v))
>
>   (clojure.core.reducers/fold combinef reducef ["foo" "bar"])
>
> ; outputs:
> acc {:combine :f} v foo
> acc foo v bar
> "bar"
>
> The accumulator in reducef is the init value from combinef, not the init 
> value from reducef.
>
> What's going on?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

r/fold combinef and reducef init values

[Brian Craft]

>From the docs:

r/fold takes a reducible collection and partitions it into groups of 
approximately n (default 512) elements. Each group is reduced using the 
reducef function. The reducef function will be called with no arguments to 
produce an identity value in each partition. The results of those 
reductions are then reduced with the combinef (defaults to reducef) 
function. When called with no arguments, (combinef) must produce its 
identity element - this will be called multiple times. Operations may be 
performed in parallel. Results will preserve order.

So, this seems to say r/fold will partition the collection and reduce each 
partition using the (reducef) as the init value.

Then, all these intermediate results will be reduced with combinef, using 
(combinef) as the init value.

However, in test it seems (reducef) is never called, and (combinef) is used 
as the init value for calls to reducef.

  (defn combinef
([] {:combine :f})
([acc v] acc))

  (defn reducef
([] {:reduce :f})
([acc v]
 (println "acc" acc "v" v)
 v))

  (clojure.core.reducers/fold combinef reducef ["foo" "bar"])

; outputs:
acc {:combine :f} v foo
acc foo v bar
"bar"

The accumulator in reducef is the init value from combinef, not the init 
value from reducef.

What's going on?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: undocumented one-argument call of reducer

[Alex Miller]

The ‘completing’ function can be used to add the missing arity to a reducing 
function that is missing it.

http://clojure.github.io/clojure/clojure.core-api.html#clojure.core/completing

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: undocumented one-argument call of reducer

[James Reeves]

The 1-argument form of a reducing function is called at the end of the
reduce. So for example:

(defn mean
  "Transducer that returns the mean average."
  ([] [0 0])
  ([[sum count]]
   (/ sum count))
  ([[sum count] value]
   [(+ sum value) (inc count))


On Fri, 25 Jan 2019 at 22:19, Brian Craft  wrote:

> The transducers doc suggests transduce works with standard reducing
> functions, but then transduce makes a one-argument call to the function.
>
> The code docs for transduce say the reducing function must support a
> one-argument call, but don't give any information about what that call
> should do.
>
> It could be helpful to document this requirement, especially since the
> examples are all written with "+", which is rather misleading since it's
> not a standard reducing function with respect to its arity, and there is
> this unusual requirement on the arity of the function.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
James Reeves
booleanknot.com

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transducer parallelism

[Laurens Van Houtven]

Hi Brian,

It's not quite what you asked but https://github.com/aphyr/tesser will make
a locally and remotely running parallel TF/IDF easy and fun :)

lvh

On Fri, Jan 25, 2019 at 4:19 PM Brian Craft  wrote:

> Are there any docs on transducer parallelism? I had the impression, from
> various sources, that they could operate in parallel, but in doing some
> benchmarks over a largish collection (counting character frequencies in
> 1.3M strings), transduce never uses more than one thread. Is this expected?
> If not, how would one debug it?
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transducer parallelism

[James Reeves]

A transducer produces a reducing function, which processes in serial. You
can manually divide up the work into sections if you have some way of
combining the results at the end, however. I believe the
clojure.core.reducers namespace will work with any reducing function that
doesn't need a finalizer (e.g. 1-arity form), though you'll have to supply
your own combining function.

On Fri, 25 Jan 2019 at 22:19, Brian Craft  wrote:

> Are there any docs on transducer parallelism? I had the impression, from
> various sources, that they could operate in parallel, but in doing some
> benchmarks over a largish collection (counting character frequencies in
> 1.3M strings), transduce never uses more than one thread. Is this expected?
> If not, how would one debug it?
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


-- 
James Reeves
booleanknot.com

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: transducer parallelism

[alpeware]

If you want parallelism, you'll want to use fold [0] or async's pipeline 
[1].

Personally, I've found Rich's talk the best way to help me understand how 
transducers were added to the language. [2]

You may also want to look at cgrand's xforms library [3]. For example, it 
allows you to express frequencies as -

(into {} (x/by-key identity x/count) coll)

Hope this helps.

[0] https://clojuredocs.org/clojure.core.reducers/fold
[1] https://clojuredocs.org/clojure.core.async/pipeline
[2] https://www.youtube.com/watch?v=6mTbuzafcII
[3] https://github.com/cgrand/xforms

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

transducer parallelism

[Brian Craft]

Are there any docs on transducer parallelism? I had the impression, from 
various sources, that they could operate in parallel, but in doing some 
benchmarks over a largish collection (counting character frequencies in 
1.3M strings), transduce never uses more than one thread. Is this expected? 
If not, how would one debug it?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

undocumented one-argument call of reducer

[Brian Craft]

The transducers doc suggests transduce works with standard reducing 
functions, but then transduce makes a one-argument call to the function.

The code docs for transduce say the reducing function must support a 
one-argument call, but don't give any information about what that call 
should do.

It could be helpful to document this requirement, especially since the 
examples are all written with "+", which is rather misleading since it's 
not a standard reducing function with respect to its arity, and there is 
this unusual requirement on the arity of the function.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.