Add spark dependency on on org.opencypher:okapi-shade.okapi

[Weichen Xu]

Hi everyone,

I'd like to call a new vote on the issue: should we add dependency
"org.opencypher:okapi-shade.okapi" into spark ? The issue background is:

Spark is going to add a big feature "Spark Graph", the prototypical
implementation is here
https://github.com/apache/spark/pull/24297
which will introduce dependency org.opencypher:okapi-shade.okapi


Xiangrui already mentioned 2 concerns on this dependency change:

> On the technical side, my main concern is the runtime dependency on
> org.opencypher:okapi-shade.okapi depends on several Scala libraries. We
> came out with the solution to shade a few Scala libraries to avoid
> pollution. However, I'm not super confident that the approach is
> sustainable for two reasons: a) there exists no proper shading libraries
> for Scala, 2) We will have to wait for upgrades from those Scala libraries
> before we can upgrade Spark to use a newer Scala version. So it would be
> great if some Scala experts can help review the current implementation and
> help assess the risk.


So let's discuss and vote whether this is a good choice.
Before this spark graph feature to get into spark ASAP, this issue should
be resolved first.

This vote is open until next Tuseday (Oct. 22).

[ ] +1: Accept the proposal
[ ] +0
[ ] -1: I don't think this is a good idea because ...

Thank you!

Weichen

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

[Mats Rydberg]

Hello Weichen, community

I'm sorry, I'm feeling a little bit confused about this vote. Is this about
the PR (https://github.com/apache/spark/pull/24490) that was merged in
early June and introduced the spark-graph module including the okapi-shade
dependency?

Regarding the okapi-shade dependency which was developed as part of the
above PR work, some advice was offered by Scala experts at TripleQuote
which helped find a satisfactory solution. The shading mechanism used is
standard and very comparable to a Java library shading solution.

The PR you link (https://github.com/apache/spark/pull/24297) is not meant
for merging. It is just a proof-of-concept branch containing a full
implementation of the system, which is kept up-to-date with the API
discussion on the currently proposed PR:
https://github.com/apache/spark/pull/24851.

Thank you
Mats


On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu 
wrote:

> Hi everyone,
>
> I'd like to call a new vote on the issue: should we add dependency
> "org.opencypher:okapi-shade.okapi" into spark ? The issue background is:
>
> Spark is going to add a big feature "Spark Graph", the prototypical
> implementation is here
> https://github.com/apache/spark/pull/24297
> which will introduce dependency org.opencypher:okapi-shade.okapi
> 
>
> Xiangrui already mentioned 2 concerns on this dependency change:
>
>> On the technical side, my main concern is the runtime dependency on
>> org.opencypher:okapi-shade.okapi depends on several Scala libraries. We
>> came out with the solution to shade a few Scala libraries to avoid
>> pollution. However, I'm not super confident that the approach is
>> sustainable for two reasons: a) there exists no proper shading libraries
>> for Scala, 2) We will have to wait for upgrades from those Scala libraries
>> before we can upgrade Spark to use a newer Scala version. So it would be
>> great if some Scala experts can help review the current implementation and
>> help assess the risk.
>
>
> So let's discuss and vote whether this is a good choice.
> Before this spark graph feature to get into spark ASAP, this issue should
> be resolved first.
>
> This vote is open until next Tuseday (Oct. 22).
>
> [ ] +1: Accept the proposal
> [ ] +0
> [ ] -1: I don't think this is a good idea because ...
>
> Thank you!
>
> Weichen
>
>

Cancel DSv2 sync this week

[Ryan Blue]

Hi everyone,

I can't make it to the DSv2 sync tomorrow, so let's skip it. If anyone
would prefer to have one and is willing to take notes, I can send out the
invite. Just let me know, otherwise let's consider it cancelled.

Thanks,

rb

-- 
Ryan Blue
Software Engineer
Netflix

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

[Weichen Xu]

Hi Mats Rydberg,

Although this dependency "org.opencypher:okapi-shade.okapi" was added into
spark, but Xiangrui raised two concerns (see above mail) about it, so we'd
better rethink on this and consider whether this is a good choice, so I
call this vote.

Thanks!

On Tue, Oct 15, 2019 at 10:56 PM Mats Rydberg 
wrote:

> Hello Weichen, community
>
> I'm sorry, I'm feeling a little bit confused about this vote. Is this
> about the PR (https://github.com/apache/spark/pull/24490) that was merged
> in early June and introduced the spark-graph module including the
> okapi-shade dependency?
>
> Regarding the okapi-shade dependency which was developed as part of the
> above PR work, some advice was offered by Scala experts at TripleQuote
> which helped find a satisfactory solution. The shading mechanism used is
> standard and very comparable to a Java library shading solution.
>
> The PR you link (https://github.com/apache/spark/pull/24297) is not meant
> for merging. It is just a proof-of-concept branch containing a full
> implementation of the system, which is kept up-to-date with the API
> discussion on the currently proposed PR:
> https://github.com/apache/spark/pull/24851.
>
> Thank you
> Mats
>
>
> On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu 
> wrote:
>
>> Hi everyone,
>>
>> I'd like to call a new vote on the issue: should we add dependency
>> "org.opencypher:okapi-shade.okapi" into spark ? The issue background is:
>>
>> Spark is going to add a big feature "Spark Graph", the prototypical
>> implementation is here
>> https://github.com/apache/spark/pull/24297
>> which will introduce dependency org.opencypher:okapi-shade.okapi
>> 
>>
>> Xiangrui already mentioned 2 concerns on this dependency change:
>>
>>> On the technical side, my main concern is the runtime dependency on
>>> org.opencypher:okapi-shade.okapi depends on several Scala libraries. We
>>> came out with the solution to shade a few Scala libraries to avoid
>>> pollution. However, I'm not super confident that the approach is
>>> sustainable for two reasons: a) there exists no proper shading libraries
>>> for Scala, 2) We will have to wait for upgrades from those Scala libraries
>>> before we can upgrade Spark to use a newer Scala version. So it would be
>>> great if some Scala experts can help review the current implementation and
>>> help assess the risk.
>>
>>
>> So let's discuss and vote whether this is a good choice.
>> Before this spark graph feature to get into spark ASAP, this issue should
>> be resolved first.
>>
>> This vote is open until next Tuseday (Oct. 22).
>>
>> [ ] +1: Accept the proposal
>> [ ] +0
>> [ ] -1: I don't think this is a good idea because ...
>>
>> Thank you!
>>
>> Weichen
>>
>>

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

[Sean Owen]

I do not have a very informed opinion here, so take this with a grain of salt.

I'd say that we need to either commit a coherent version of this for
Spark 3, or not at all. If it doesn't have support, I'd back out the
existing changes.
I was initially skeptical about how much this needs to be in Spark vs
a third-party package, and that still stands.

The addition of another dependency isn't that big a deal IMHO, but,
yes, it does add something to the maintenance overhead. But that's all
the more true of a new module.

I don't feel strongly about it, but if this isn't obviously getting
support from any committers, can we keep it as a third party library
for now?


On Tue, Oct 15, 2019 at 8:53 PM Weichen Xu  wrote:
>
> Hi Mats Rydberg,
>
> Although this dependency "org.opencypher:okapi-shade.okapi" was added into 
> spark, but Xiangrui raised two concerns (see above mail) about it, so we'd 
> better rethink on this and consider whether this is a good choice, so I call 
> this vote.
>
> Thanks!
>
> On Tue, Oct 15, 2019 at 10:56 PM Mats Rydberg  wrote:
>>
>> Hello Weichen, community
>>
>> I'm sorry, I'm feeling a little bit confused about this vote. Is this about 
>> the PR (https://github.com/apache/spark/pull/24490) that was merged in early 
>> June and introduced the spark-graph module including the okapi-shade 
>> dependency?
>>
>> Regarding the okapi-shade dependency which was developed as part of the 
>> above PR work, some advice was offered by Scala experts at TripleQuote which 
>> helped find a satisfactory solution. The shading mechanism used is standard 
>> and very comparable to a Java library shading solution.
>>
>> The PR you link (https://github.com/apache/spark/pull/24297) is not meant 
>> for merging. It is just a proof-of-concept branch containing a full 
>> implementation of the system, which is kept up-to-date with the API 
>> discussion on the currently proposed PR: 
>> https://github.com/apache/spark/pull/24851.
>>
>> Thank you
>> Mats
>>
>>
>> On Tue, Oct 15, 2019 at 10:38 AM Weichen Xu  
>> wrote:
>>>
>>> Hi everyone,
>>>
>>> I'd like to call a new vote on the issue: should we add dependency 
>>> "org.opencypher:okapi-shade.okapi" into spark ? The issue background is:
>>>
>>> Spark is going to add a big feature "Spark Graph", the prototypical 
>>> implementation is here
>>> https://github.com/apache/spark/pull/24297
>>> which will introduce dependency org.opencypher:okapi-shade.okapi
>>>
>>> Xiangrui already mentioned 2 concerns on this dependency change:

 On the technical side, my main concern is the runtime dependency on 
 org.opencypher:okapi-shade.okapi depends on several Scala libraries. We 
 came out with the solution to shade a few Scala libraries to avoid 
 pollution. However, I'm not super confident that the approach is 
 sustainable for two reasons: a) there exists no proper shading libraries 
 for Scala, 2) We will have to wait for upgrades from those Scala libraries 
 before we can upgrade Spark to use a newer Scala version. So it would be 
 great if some Scala experts can help review the current implementation and 
 help assess the risk.
>>>
>>>
>>> So let's discuss and vote whether this is a good choice.
>>> Before this spark graph feature to get into spark ASAP, this issue should 
>>> be resolved first.
>>>
>>> This vote is open until next Tuseday (Oct. 22).
>>>
>>> [ ] +1: Accept the proposal
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>> Thank you!
>>>
>>> Weichen
>>>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

branch-3.0 vs branch-3.0-preview (?)

[Dongjoon Hyun]

Hi,

It seems that we have `branch-3.0-preview` branch.

https://github.com/apache/spark/commits/branch-3.0-preview

Can we have `branch-3.0` instead of `branch-3.0-preview`?

We can tag `v3.0.0-preview` on `branch-3.0` and continue to use for
`v3.0.0` later.

Bests,
Dongjoon.