[VOTE] SPARK 2.4.0 (RC2)

2018-09-27 Thread Wenchen Fan
Please vote on releasing the following candidate as Apache Spark version 2.4.0. The vote is open until October 1 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.0 [ ] -1 Do not release this package because ... To l

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-27 Thread Wenchen Fan
I'm adding my own +1, since all the problems mentioned in the RC1 voting email are all resolved. And there is no blocker issue for 2.4.0 AFAIK. On Fri, Sep 28, 2018 at 10:59 AM Wenchen Fan wrote: > Please vote on releasing the following candidate as Apache Spark version > 2.4.0. > > The vote is

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-27 Thread Saisai Shao
Only "without-hadoop" profile has 2.12 binary, is it expected? Thanks Saisai Wenchen Fan 于2018年9月28日周五 上午11:08写道： > I'm adding my own +1, since all the problems mentioned in the RC1 voting > email are all resolved. And there is no blocker issue for 2.4.0 AFAIK. > > On Fri, Sep 28, 2018 at 10:59

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-27 Thread Wenchen Fan
Yes, that's proposed by Sean. This time we should publish a Scala 2.12 build, both in maven and the download page. On Fri, Sep 28, 2018 at 11:34 AM Saisai Shao wrote: > Only "without-hadoop" profile has 2.12 binary, is it expected? > > Thanks > Saisai > > Wenchen Fan 于2018年9月28日周五 上午11:08写道： >

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-28 Thread Wenchen Fan
I don't think this bug is so serious to fail an RC, it's only about metrics IIUC, and it's not a regression in 2.4. I agree we should backport this fix to 2.3 and 2.4, and we should update our scala 2.12 jenkins build to use scala 2.12.7. cc Shane do you know how to change it? BTW end users can s

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-28 Thread Sean Owen
I don't even know how or if this manifests as a bug. The code is indeed incorrect and the 2.12 compiler flags it. We fixed a number of these in SPARK-25398. While I want to get this into 2.4 if we have another RC, I don't see evidence this is a blocker. It is not specific to Scala 2.12. Using Scal

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-28 Thread Dongjoon Hyun
Hi, Wenchen. The current issue link seems to be out of order for me. The list of bug fixes going into 2.4.0 can be found at the following URL: https://issues.apache.org/jira/projects/SPARK/versions/2.4.0 Could you send out with the following issue link for next RCs? https://issues.apache.org/ji

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-28 Thread Darcy Shen
I agree it is a non-important Spark bug. I mean the Option and String comparison. The bug is easy to fix and obvious to confirm. If the desc of PR may not be accurate，feel free to edit the title or content. I am on a vocation from 9.29   ：）But the scala bug of Wra

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-28 Thread Sean Owen
Go ahead and file a JIRA to update to 2.12.7 with these details. We'll assess whether it is a blocker. On Fri, Sep 28, 2018 at 12:09 PM Darcy Shen wrote: > > I agree it is a non-important Spark bug. I mean the Option and String > comparison. The bug is easy to fix and obvious to confirm. If the

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-28 Thread Sean Owen
+1, with comments: There are 5 critical issues for 2.4, and no blockers: SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4 SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs SPARK-25319 Spark MLlib, GraphX 2.4 QA umbrella SPARK-25326 ML, Graph 2.4 QA: Programm

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-29 Thread Stavros Kontopoulos
+1 Stavros On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen wrote: > +1, with comments: > > There are 5 critical issues for 2.4, and no blockers: > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4 > SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs > SPARK-2531

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-09-30 Thread Denny Lee
+1 (non-binding) On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > +1 > > Stavros > > On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen wrote: > >> +1, with comments: >> >> There are 5 critical issues for 2.4, and no blockers: >> SPARK-25378 ArrayData.t

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Ted Yu
+1 Original message From: Denny Lee Date: 9/30/18 10:30 PM (GMT-08:00) To: Stavros Kontopoulos Cc: Sean Owen , Wenchen Fan , dev Subject: Re: [VOTE] SPARK 2.4.0 (RC2) +1 (non-binding) On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos wrote: +1 Stavros On Sat, Sep

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Marco Gaido
hen Fan , dev < > dev@spark.apache.org> > Subject: Re: [VOTE] SPARK 2.4.0 (RC2) > > +1 (non-binding) > > > On Sat, Sep 29, 2018 at 10:24 AM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> +1 >> >> Stavros >

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Holden Karau
Original message >> From: Denny Lee >> Date: 9/30/18 10:30 PM (GMT-08:00) >> To: Stavros Kontopoulos >> Cc: Sean Owen , Wenchen Fan , dev >> >> Subject: Re: [VOTE] SPARK 2.4.0 (RC2) >> >> +1 (non-binding) >> >> >>

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Xiangrui Meng
tt 2018 alle ore 09:11 Ted Yu ha >> scritto: >> >>> +1 >>> >>> Original message >>> From: Denny Lee >>> Date: 9/30/18 10:30 PM (GMT-08:00) >>> To: Stavros Kontopoulos >>> Cc: Sean Owen , Wenchen Fan , >&g

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Sean Owen
Given that this release is probably still 2 weeks from landing, I don't think that waiting on a spark-tensorflow-connector release with TF 1.12 in mid-October is a big deal. Users can use the library with Spark 2.3.x for a week or two before upgrading, if that's the case. I think this kind of bug f

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Wenchen Fan
This RC fails because of the correctness bug: SPARK-25538 I'll start a new RC once the fix(https://github.com/apache/spark/pull/22602) is merged. Thanks, Wenchen On Tue, Oct 2, 2018 at 1:21 AM Sean Owen wrote: > Given that this release is probably still 2 weeks from landing, I don't > think th

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-01 Thread Michael Heuer
FYI I’ve open two new issues against 2.4.0 rc2 https://issues.apache.org/jira/browse/SPARK-25587 https://issues.apache.org/jira/browse/SPARK-25588 that are regressions against 2.3.1, and may

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-04 Thread Shixiong(Ryan) Zhu
-1. Found an issue in a new 2.4 Java API: https://issues.apache.org/jira/browse/SPARK-25644 We should fix it in 2.4.0 to avoid future breaking changes. Best Regards, Ryan On Mon, Oct 1, 2018 at 7:22 PM Michael Heuer wrote: > FYI I’ve open two new issues against 2.4.0 rc2 > > https://issues.apa

Re: [VOTE] SPARK 2.4.0 (RC2)

2018-10-04 Thread shane knapp
> > Using Scala 2.12.7 is a not an infra change, but change to the build, > but again it's not even specific to 2.12.7. We should use the latest > if we can though. > > yep, exactly. we don't even have scala install on any of our jenkins nodes. this is all taken care of via build/mvn at runtime.