Re: Thoughts on Spark 3 release, or a preview release

uesday, September 17, 2019 at 12:00 AM > *To: *Erik Erlandson > *Cc: *Sean Owen , dev > *Subject: *Re: Thoughts on Spark 3 release, or a preview release > > > > https://issues.apache.org/jira/browse/SPARK-28264 [issues.apache.org] > <h

Re: Thoughts on Spark 3 release, or a preview release

the Spark 3 preview release specifically on SPARK-25299. -Matt Cheah From: Xiao Li Date: Tuesday, September 17, 2019 at 12:00 AM To: Erik Erlandson Cc: Sean Owen , dev Subject: Re: Thoughts on Spark 3 release, or a preview release https://issues.apache.org/jira/browse/SPARK-28264

Re: Thoughts on Spark 3 release, or a preview release

https://issues.apache.org/jira/browse/SPARK-28264 SPARK-28264 Revisiting Python / pandas UDF sounds critical for 3.0 preview Xiao On Mon, Sep 16, 2019 at 12:22 PM Erik Erlandson wrote: > > I'm in favor of adding SPARK-25299 > - Use remote

Re: Thoughts on Spark 3 release, or a preview release

I'm in favor of adding SPARK-25299 - Use remote storage for persisting shuffle data https://issues.apache.org/jira/browse/SPARK-25299 If that is far enough along to get onto the roadmap. On Wed, Sep 11, 2019 at 11:37 AM Sean Owen wrote: >

Re: Thoughts on Spark 3 release, or a preview release

Thank you, Fokko. Probably best to discuss further off-list. I'm almost embarrassed to describe our current workaround — it involves among other things a custom Shader implementation for the Maven Shade plugin. michael > On Sep 13, 2019, at 3:07 AM, Driesprong, Fokko wrote: > > Michael

Re: Thoughts on Spark 3 release, or a preview release

I don't expect to see a large DS V2 API change from now on. But we may update the API a little bit if we find problems during the preview. On Sat, Sep 14, 2019 at 10:16 PM Sean Owen wrote: > I don't think this suggests anything is finalized, including APIs. I > would not guess there will be

Re: Thoughts on Spark 3 release, or a preview release

I don't think this suggests anything is finalized, including APIs. I would not guess there will be major changes from here though. On Fri, Sep 13, 2019 at 4:27 PM Andrew Melo wrote: > > Hi Spark Aficionados- > > On Fri, Sep 13, 2019 at 15:08 Ryan Blue wrote: >> >> +1 for a preview release. >>

Re: Thoughts on Spark 3 release, or a preview release

Hi Spark Aficionados- On Fri, Sep 13, 2019 at 15:08 Ryan Blue wrote: > +1 for a preview release. > > DSv2 is quite close to being ready. I can only think of a couple issues > that we need to merge, like getting a fix for stats estimation done. I'll > have a better idea once I've caught up from

Re: Thoughts on Spark 3 release, or a preview release

+1 for a preview release. DSv2 is quite close to being ready. I can only think of a couple issues that we need to merge, like getting a fix for stats estimation done. I'll have a better idea once I've caught up from being away for ApacheCon and I'll add this to the agenda for our next DSv2 sync

Re: Thoughts on Spark 3 release, or a preview release

Ur, Sean. I prefer a full release like 2.0.0-preview. https://archive.apache.org/dist/spark/spark-2.0.0-preview/ And, thank you, Xingbo! Could you take a look at website generation? It seems to be broken on `master`. Bests, Dongjoon. On Fri, Sep 13, 2019 at 11:30 AM Xingbo Jiang wrote: >

Re: Thoughts on Spark 3 release, or a preview release

Hi all, I would like to volunteer to be the release manager of Spark 3 preview, thanks! Sean Owen 于2019年9月13日周五 上午11:21写道： > Well, great to hear the unanimous support for a Spark 3 preview > release. Now, I don't know how to make releases myself :) I would > first open it up to our revered

Re: Thoughts on Spark 3 release, or a preview release

Well, great to hear the unanimous support for a Spark 3 preview release. Now, I don't know how to make releases myself :) I would first open it up to our revered release managers: would anyone be interested in trying to make one? sounds like it's not too soon to get what's in master out for

Re: Thoughts on Spark 3 release, or a preview release

+1 for preview release On Fri, Sep 13, 2019 at 9:58 AM Thomas Graves wrote: > +1, I think having preview release would be great. > > Tom > > On Fri, Sep 13, 2019 at 4:55 AM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> +1 as a contributor and as a user. Given the amount

Re: Thoughts on Spark 3 release, or a preview release

+1, I think having preview release would be great. Tom On Fri, Sep 13, 2019 at 4:55 AM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > +1 as a contributor and as a user. Given the amount of testing required > for all the new cool stuff like java 11 support, major >

Re: Thoughts on Spark 3 release, or a preview release

+1 as a contributor and as a user. Given the amount of testing required for all the new cool stuff like java 11 support, major refactorings/deprecations etc, a preview version would help a lot the community making adoption smoother long term. I would also add to the list of issues, Scala 2.13

Re: Thoughts on Spark 3 release, or a preview release

Michael Heuer, that's an interesting issue. 1.8.2 to 1.9.0 is almost binary compatible (94%): http://people.apache.org/~busbey/avro/1.9.0-RC4/1.8.2_to_1.9.0RC4_compat_report.html. Most of the stuff is removing the Jackson and Netty API from Avro's public API and deprecating the Joda library. I

Re: Thoughts on Spark 3 release, or a preview release

+1! Long due for a preview release. On Thu, Sep 12, 2019 at 5:26 PM, Holden Karau < hol...@pigscanfly.ca > wrote: > > I like the idea from the PoV of giving folks something to start testing > against and exploring so they can raise issues with us earlier in the > process and we have more time

Re: Thoughts on Spark 3 release, or a preview release

I like the idea from the PoV of giving folks something to start testing against and exploring so they can raise issues with us earlier in the process and we have more time to make calls around this. On Thu, Sep 12, 2019 at 4:15 PM John Zhuge wrote: > +1 Like the idea as a user and a DSv2

Re: Thoughts on Spark 3 release, or a preview release

+1 as both a contributor and a user. From: John Zhuge Date: Thursday, September 12, 2019 at 4:15 PM To: Jungtaek Lim Cc: Jean Georges Perrin , Hyukjin Kwon , Dongjoon Hyun , dev Subject: Re: Thoughts on Spark 3 release, or a preview release +1 Like the idea as a user and a DSv2

Re: Thoughts on Spark 3 release, or a preview release

+1 Like the idea as a user and a DSv2 contributor. On Thu, Sep 12, 2019 at 4:10 PM Jungtaek Lim wrote: > +1 (as a contributor) from me to have preview release on Spark 3 as it > would help to test the feature. When to cut preview release is > questionable, as major works are ideally to be done

Re: Thoughts on Spark 3 release, or a preview release

+1 (as a contributor) from me to have preview release on Spark 3 as it would help to test the feature. When to cut preview release is questionable, as major works are ideally to be done before that - if we are intended to introduce new features before official release, that should work regardless

Re: Thoughts on Spark 3 release, or a preview release

As a user/non committer, +1 I love the idea of an early 3.0.0 so we can test current dev against it, I know the final 3.x will probably need another round of testing when it gets out, but less for sure... I know I could checkout and compile, but having a “packaged” preversion is great if it

Re: Thoughts on Spark 3 release, or a preview release

I would love to see Spark + Hadoop + Parquet + Avro compatibility problems resolved, e.g. https://issues.apache.org/jira/browse/SPARK-25588 https://issues.apache.org/jira/browse/SPARK-27781

Thoughts on Spark 3 release, or a preview release

I'm curious what current feelings are about ramping down towards a Spark 3 release. It feels close to ready. There is no fixed date, though in the past we had informally tossed around "back end of 2019". For reference, Spark 1 was May 2014, Spark 2 was July 2016. I'd expect Spark 2 to last longer,