Re: Apache Spark 3.0 timeline

2019-10-16 Thread Wenchen Fan
> I figure we are probably moving to code freeze late in the year, release early next year? Sounds good! On Thu, Oct 17, 2019 at 7:51 AM Dongjoon Hyun wrote: > Thanks! That sounds reasonable. I'm +1. :) > > Historically, 2.0-preview was on May 2016 and 2.0 was on July, 2016. 3.0 > seems to be

Unsubscribe

2019-10-16 Thread Kiran B
-- Thank you, Kiran, 9959311399.

unsubscribe

2019-10-16 Thread Jason Man, CLSA
unsubscribe The content of this communication is intended for the recipient and is subject to CLSA Legal and Regulatory Notices. These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon request. Please consider before printing. CLSA is ISO14001 certified and committed to

Re: Apache Spark 3.0 timeline

2019-10-16 Thread Dongjoon Hyun
Thanks! That sounds reasonable. I'm +1. :) Historically, 2.0-preview was on May 2016 and 2.0 was on July, 2016. 3.0 seems to be different. Bests, Dongjoon On Wed, Oct 16, 2019 at 16:38 Sean Owen wrote: > I think the branch question is orthogonal but yeah we can probably make an > updated

Re: Apache Spark 3.0 timeline

2019-10-16 Thread Sean Owen
I think the branch question is orthogonal but yeah we can probably make an updated statement about 3.0 release. Clearly a preview is imminent. I figure we are probably moving to code freeze late in the year, release early next year? Any better ideas about estimates to publish? They aren't binding.

Apache Spark 3.0 timeline

2019-10-16 Thread Dongjoon Hyun
Hi, All. I saw the following comment from Wenchen in the previous email thread. > Personally I'd like to avoid cutting branch-3.0 right now, otherwise we need to merge PRs into two branches in the following several months. Since 3.0.0-preview seems to be already here for RC, can we update our

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

2019-10-16 Thread Reynold Xin
Just curious - did we discuss why this shouldn't be another Apache sister project? On Wed, Oct 16, 2019 at 10:21 AM, Sean Owen < sro...@gmail.com > wrote: > > > > We don't all have to agree on whether to add this -- there are like 10 > people with an opinion -- and I certainly would not veto

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

2019-10-16 Thread Sean Owen
We don't all have to agree on whether to add this -- there are like 10 people with an opinion -- and I certainly would not veto it. In practice a medium-sized changes needs someone to review/merge it all the way through, and nobody strongly objecting. I too don't know what to make of the

Re: branch-3.0 vs branch-3.0-preview (?)

2019-10-16 Thread Sean Owen
We do not have to do anything to branch-3.0-preview; it's just for the convenience of the RM. Just continue to merge to master for 3.0. If it happens that some state of the master branch works as a preview release, sure, just tag and release. We might get away with it. But if for example we have

Re: branch-3.0 vs branch-3.0-preview (?)

2019-10-16 Thread Dongjoon Hyun
Technically, `branch-3.0-preview` has many issues. First of all, are we going to delete `branch-3.0-preview` after releasing `3.0-preview`? I guess we didn't delete old branches (including feature branches like jdbc, yarn branches) Second, our merge script already starts to show

Re: [k8s] Spark operator (the Java one)

2019-10-16 Thread Erik Erlandson
Folks have (correctly) pointed out that an operator does not need to be coupled to the Apache Spark project. However, I believe there are some strategic community benefits to supporting a Spark operator that should be weighed against the costs of maintaining one. *) The Kubernetes ecosystem is

Re: [k8s] Spark operator (the Java one)

2019-10-16 Thread Yinan Li
Hi Erik, I agree with what you said about the community benefits of supporting operators for Spark. However, that doesn't necessarily mean the operators need and should be part of Spark core. > *) The Kubernetes ecosystem is evolving toward adopting operators as the de facto standard for

Re: Add spark dependency on on org.opencypher:okapi-shade.okapi

2019-10-16 Thread Martin Junghanns
I'm slightly confused about this discussion. I worked on all of the aforementioned PRs: the module PR that has been merged, the current PR that introduces the Graph API and the PoC PR that contains the full implementation. The issues around shading were addressed and the module PR eventually got

Re: branch-3.0 vs branch-3.0-preview (?)

2019-10-16 Thread Sean Owen
I don't think we would want to cut 'branch-3.0' right now, which would imply that master is 3.1. We don't want to merge every new change into two branches. It may still be useful to have `branch-3.0-preview` as a short-lived branch just used to manage the preview release, as we will need to let

Re: branch-3.0 vs branch-3.0-preview (?)

2019-10-16 Thread Reynold Xin
Can we just tag master? On Wed, Oct 16, 2019 at 12:34 AM, Wenchen Fan < cloud0...@gmail.com > wrote: > > Does anybody remember what we did for 2.0 preview? Personally I'd like to > avoid cutting branch-3.0 right now, otherwise we need to merge PRs into > two branches in the following several

Re: branch-3.0 vs branch-3.0-preview (?)

2019-10-16 Thread Wenchen Fan
Does anybody remember what we did for 2.0 preview? Personally I'd like to avoid cutting branch-3.0 right now, otherwise we need to merge PRs into two branches in the following several months. Thanks, Wenchen On Wed, Oct 16, 2019 at 3:01 PM Xingbo Jiang wrote: > Hi Dongjoon, > > I'm not sure

Re: branch-3.0 vs branch-3.0-preview (?)

2019-10-16 Thread Xingbo Jiang
Hi Dongjoon, I'm not sure about the best practice of maintaining a preview release branch, since new features might still go into Spark 3.0 after preview release, I guess it might make more sense to have separated branches for 3.0.0 and 3.0-preview. However, I'm open to both solutions, if we