Re: shapeless in spark 2.1.0

2016-12-29 Thread Koert Kuipers 
we also use spray for webservices that execute on spark, and spray depends
on even older (and incompatible) shapeless 1.x
to get rid of the old shapeless i would have to upgrade from spray to
akka-http, which means going to java 8

this might also affect spark-job-server, which it seems uses spray.

On Thu, Dec 29, 2016 at 1:17 PM, Maciej Szymkiewicz 
wrote:

> Breeze 0.13 (RC-1 right now) bumps shapeless to 2.2.0 and 2.2.5 for Scala
> 2.10 and 2.11 respectively:
>
> https://github.com/scalanlp/breeze/pull/509
>
>
> On 12/29/2016 07:13 PM, Ryan Williams wrote:
>
> Other option would presumably be for someone to make a release of breeze
> with old-shapeless shaded... unless shapeless classes are exposed in
> breeze's public API, in which case you'd have to copy the relevant
> shapeless classes into breeze and then publish that?
>
> On Thu, Dec 29, 2016, 1:05 PM Sean Owen  wrote:
>
>> It is breeze, but, what's the option? It can't be excluded. I think this
>> falls in the category of things an app would need to shade in this
>> situation.
>>
>> On Thu, Dec 29, 2016, 16:49 Koert Kuipers  wrote:
>>
>> i just noticed that spark 2.1.0 bring in a new transitive dependency on
>> shapeless 2.0.0
>>
>> shapeless is a popular library for scala users, and shapeless 2.0.0 is
>> old (2014) and not compatible with more current versions.
>>
>> so this means a spark user that uses shapeless in his own development
>> cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>>
>> wish i had noticed this sooner
>>
>>
> --
> Maciej Szymkiewicz
>
>

Re: shapeless in spark 2.1.0

2016-12-29 Thread Maciej Szymkiewicz 
Breeze 0.13 (RC-1 right now) bumps shapeless to 2.2.0 and 2.2.5 for
Scala 2.10 and 2.11 respectively:

https://github.com/scalanlp/breeze/pull/509

On 12/29/2016 07:13 PM, Ryan Williams wrote:
>
> Other option would presumably be for someone to make a release of
> breeze with old-shapeless shaded... unless shapeless classes are
> exposed in breeze's public API, in which case you'd have to copy the
> relevant shapeless classes into breeze and then publish that?
>
>
> On Thu, Dec 29, 2016, 1:05 PM Sean Owen  > wrote:
>
> It is breeze, but, what's the option? It can't be excluded. I
> think this falls in the category of things an app would need to
> shade in this situation. 
>
> On Thu, Dec 29, 2016, 16:49 Koert Kuipers  > wrote:
>
> i just noticed that spark 2.1.0 bring in a new transitive
> dependency on shapeless 2.0.0
>
> shapeless is a popular library for scala users, and shapeless
> 2.0.0 is old (2014) and not compatible with more current versions.
>
> so this means a spark user that uses shapeless in his own
> development cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>
> wish i had noticed this sooner
>

-- 
Maciej Szymkiewicz

Re: shapeless in spark 2.1.0

2016-12-29 Thread Ryan Williams 
Other option would presumably be for someone to make a release of breeze
with old-shapeless shaded... unless shapeless classes are exposed in
breeze's public API, in which case you'd have to copy the relevant
shapeless classes into breeze and then publish that?

On Thu, Dec 29, 2016, 1:05 PM Sean Owen  wrote:

> It is breeze, but, what's the option? It can't be excluded. I think this
> falls in the category of things an app would need to shade in this
> situation.
>
> On Thu, Dec 29, 2016, 16:49 Koert Kuipers  wrote:
>
> i just noticed that spark 2.1.0 bring in a new transitive dependency on
> shapeless 2.0.0
>
> shapeless is a popular library for scala users, and shapeless 2.0.0 is old
> (2014) and not compatible with more current versions.
>
> so this means a spark user that uses shapeless in his own development
> cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>
> wish i had noticed this sooner
>
>

Re: shapeless in spark 2.1.0

2016-12-29 Thread Sean Owen 
It is breeze, but, what's the option? It can't be excluded. I think this
falls in the category of things an app would need to shade in this
situation.

On Thu, Dec 29, 2016, 16:49 Koert Kuipers  wrote:

> i just noticed that spark 2.1.0 bring in a new transitive dependency on
> shapeless 2.0.0
>
> shapeless is a popular library for scala users, and shapeless 2.0.0 is old
> (2014) and not compatible with more current versions.
>
> so this means a spark user that uses shapeless in his own development
> cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>
> wish i had noticed this sooner
>

Re: shapeless in spark 2.1.0

2016-12-29 Thread Ryan Williams 
`mvn dependency:tree -Dverbose -Dincludes=:shapeless_2.11` shows:

[INFO] \- org.apache.spark:spark-mllib_2.11:jar:2.1.0:provided
[INFO]\- org.scalanlp:breeze_2.11:jar:0.12:provided
[INFO]   \- com.chuusai:shapeless_2.11:jar:2.0.0:provided

On Thu, Dec 29, 2016 at 12:11 PM Herman van Hövell tot Westerflier <
hvanhov...@databricks.com> wrote:

> Which dependency pulls in shapeless?
>
> On Thu, Dec 29, 2016 at 5:49 PM, Koert Kuipers  wrote:
>
> i just noticed that spark 2.1.0 bring in a new transitive dependency on
> shapeless 2.0.0
>
> shapeless is a popular library for scala users, and shapeless 2.0.0 is old
> (2014) and not compatible with more current versions.
>
> so this means a spark user that uses shapeless in his own development
> cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>
> wish i had noticed this sooner
>
>
>
>
> --
>
> Herman van Hövell
>
> Software Engineer
>
> Databricks Inc.
>
> hvanhov...@databricks.com
>
> +31 6 420 590 27
>
> databricks.com
>
> [image: http://databricks.com] 
>

Re: shapeless in spark 2.1.0

2016-12-29 Thread Herman van Hövell tot Westerflier 
Which dependency pulls in shapeless?

On Thu, Dec 29, 2016 at 5:49 PM, Koert Kuipers  wrote:

> i just noticed that spark 2.1.0 bring in a new transitive dependency on
> shapeless 2.0.0
>
> shapeless is a popular library for scala users, and shapeless 2.0.0 is old
> (2014) and not compatible with more current versions.
>
> so this means a spark user that uses shapeless in his own development
> cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>
> wish i had noticed this sooner
>



-- 

Herman van Hövell

Software Engineer

Databricks Inc.

hvanhov...@databricks.com

+31 6 420 590 27

databricks.com

[image: http://databricks.com]

shapeless in spark 2.1.0

2016-12-29 Thread Koert Kuipers 
i just noticed that spark 2.1.0 bring in a new transitive dependency on
shapeless 2.0.0

shapeless is a popular library for scala users, and shapeless 2.0.0 is old
(2014) and not compatible with more current versions.

so this means a spark user that uses shapeless in his own development
cannot upgrade safely from 2.0.0 to 2.1.0, i think.

wish i had noticed this sooner