Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

Hyun > *Subject:* Re: [VOTE] Release Apache Spark 2.4.4 (RC3) > > +1 (from the last blocker PR) > > 2019년 8월 29일 (목) 오전 8:20, Takeshi Yamamuro 님이 작성: > >> I checked the tests passed again on the same env. >> It looks ok. >> >> >> On Thu, Aug 29, 2019 at 6:15

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 Run tests, R tests, r-hub Debian, Ubuntu, mac, Windows From: Hyukjin Kwon Sent: Wednesday, August 28, 2019 9:14 PM To: Takeshi Yamamuro Cc: dev; Dongjoon Hyun Subject: Re: [VOTE] Release Apache Spark 2.4.4 (RC3) +1 (from the last blocker PR) 2019년 8월 29일 (목

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 (from the last blocker PR) 2019년 8월 29일 (목) 오전 8:20, Takeshi Yamamuro 님이 작성: > I checked the tests passed again on the same env. > It looks ok. > > > On Thu, Aug 29, 2019 at 6:15 AM Marcelo Vanzin > wrote: > >> +1 >> >> On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun >> wrote: >> > >> >

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

I checked the tests passed again on the same env. It looks ok. On Thu, Aug 29, 2019 at 6:15 AM Marcelo Vanzin wrote: > +1 > > On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun > wrote: > > > > Please vote on releasing the following candidate as Apache Spark version > 2.4.4. > > > > The vote is

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 On Tue, Aug 27, 2019 at 4:06 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.4. > > The vote is open until August 30th 5PM PST and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. > > [ ] +1 Release this

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 Installed PySpark in a py3 virtual env & checked the PKG-INFO which we've had difficulty with previously and looked good. On Wed, Aug 28, 2019 at 10:17 AM DB Tsai wrote: > +1 > > Thanks! > > On Wed, Aug 28, 2019 at 7:14 AM Wenchen Fan wrote: > >> +1, no more blocking issues that I'm aware

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 Thanks! On Wed, Aug 28, 2019 at 7:14 AM Wenchen Fan wrote: > +1, no more blocking issues that I'm aware of. > > On Wed, Aug 28, 2019 at 8:33 PM Sean Owen wrote: > >> +1 from me again. >> >> On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun >> wrote: >> > >> > Please vote on releasing the

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1, no more blocking issues that I'm aware of. On Wed, Aug 28, 2019 at 8:33 PM Sean Owen wrote: > +1 from me again. > > On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun > wrote: > > > > Please vote on releasing the following candidate as Apache Spark version > 2.4.4. > > > > The vote is open

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1 from me again. On Tue, Aug 27, 2019 at 6:06 PM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.4. > > The vote is open until August 30th 5PM PST and passes if a majority +1 PMC > votes are cast, with a minimum of 3 +1 votes. > > [ ] +1

RE: [VOTE] Release Apache Spark 2.4.4 (RC3)

Apache Spark 2.4.4 (RC3) +1. - Checked checksums and signatures of artifacts. - Checked to have all binaries and maven repo. - Checked document generation (including a new change after RC2) - Build with `-Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-2.6` on AdoptOpenJDK8_202

Re: [VOTE] Release Apache Spark 2.4.4 (RC3)

+1. - Checked checksums and signatures of artifacts. - Checked to have all binaries and maven repo. - Checked document generation (including a new change after RC2) - Build with `-Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Phadoop-2.6` on AdoptOpenJDK8_202. - Tested with both

[VOTE] Release Apache Spark 2.4.4 (RC3)

Please vote on releasing the following candidate as Apache Spark version 2.4.4. The vote is open until August 30th 5PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.4 [ ] -1 Do not release this package because

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

Hi, All. Thank you for testing. 2.4.4 RC2 vote fails due to the PySpark correctness issue. Since the blocker is merged, I'll start RC3 soon. Bests, Dongjoon. On Mon, Aug 26, 2019 at 10:55 PM Hyukjin Kwon wrote: > -1 > > Seems there's one critical correctness issue specifically in branch-2.4

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

-1 Seems there's one critical correctness issue specifically in branch-2.4 ... Please take a look for https://github.com/apache/spark/pull/25593 2019년 8월 27일 (화) 오후 2:38, Takeshi Yamamuro 님이 작성: > Hi, Dongjoon > > I checked that all the test passed on my Mac/x86_64 env with: > -Pyarn

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

Hi, Dongjoon I checked that all the test passed on my Mac/x86_64 env with: -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes -Pkubernetes-integration-tests -Psparkr maropu@~/spark-2.4.4-rc2:$java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build

Re: [VOTE] Release Apache Spark 2.4.4 (RC2)

+1 as per response to RC1. The existing issues identified there seem to have been fixed. On Mon, Aug 26, 2019 at 2:45 AM Dongjoon Hyun wrote: > > Please vote on releasing the following candidate as Apache Spark version > 2.4.4. > > The vote is open until August 29th 1AM PST and passes if a

[VOTE] Release Apache Spark 2.4.4 (RC2)

Please vote on releasing the following candidate as Apache Spark version 2.4.4. The vote is open until August 29th 1AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.4 [ ] -1 Do not release this package because

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

FYI, three more patches landed to `branch-2.4` until yesterday. [SPARK-28642][SQL][2.4] Hide credentials in show create table [SPARK-27330][SS][2.4] support task abort in foreach writer [SPARK-28025][SS][2.4] Fix FileContextBasedCheckpointFileManager leaking crc files For some non-blocker

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

Hi, All. This 2.4.4 RC1 vote fails and the reported PRs are merged to `branch-2.4`. The following is the commit list since `v2.4.4-rc1` tag. Preparing development version 2.4.5-SNAPSHOT [SPARK-28749][TEST][BRANCH-2.4] Fix PySpark tests not to require kafka-0-8 [SPARK-28775][CORE][TESTS] Skip

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

Sounds fine, we probably needed SPARK-28775 anyway. I merged that and SPARK-28749. It looks like it's just the one you're talking about right now, SPARK-28699. The rest of the tests seemed to pass OK, release looks good, but bears more testing by everyone out there before a next RC. On Tue, Aug

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

Unfortunately, I need to -1. Recently we found that the repartition correctness bug can still be reproduced. The root cause has been identified and there are 2 PRs to fix 2 related issues: https://github.com/apache/spark/pull/25491 https://github.com/apache/spark/pull/25498 I think we should

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

Thank you for testing, Sean and Herman. There are three reporting until now. 1. SPARK-28775 is for JDK 8u221+ testing at Apache Spark 3.0/2.4/2.3. 2. SPARK-28749 is for Scala 2.12 Python testing at Apache Spark 2.4 only. 3. SPARK-28699 is for disabling radix sort for ShuffleExchangeExec at

Re: [VOTE] Release Apache Spark 2.4.4 (RC1)

Things are looking pretty good so far, but a few notes: I thought we might need this PR to make the 2.12 build of 2.4.x not try to build Kafka 0.8 support, but, I'm not seeing that 2.4.x + 2.12 builds or tests it? https://github.com/apache/spark/pull/25482 I can merge this to 2.4 shortly anyway,

[VOTE] Release Apache Spark 2.4.4 (RC1)

Please vote on releasing the following candidate as Apache Spark version 2.4.4. The vote is open until August 22nd 10AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.4 [ ] -1 Do not release this package because

Re: Release Apache Spark 2.4.4

en , Kazuaki Ishizaki > Cc:dev > Date:2019/08/16 05:10 > Subject: [EXTERNAL] Re: Release Apache Spark 2.4.4 > -- > > > > +1 for that. > > Kazuaki volunteered for 2.3.4 release last month. AFAIK, he has been > prepa

Re: Release Apache Spark 2.4.4

Sure, I will launch a separate e-mail thread for discussing 2.3.4 later. Regards, Kazuaki Ishizaki, Ph.D. From: Dongjoon Hyun To: Sean Owen , Kazuaki Ishizaki Cc: dev Date: 2019/08/16 05:10 Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4 +1 for that. Kazuaki

Re: Release Apache Spark 2.4.4

+1 for that. Kazuaki volunteered for 2.3.4 release last month. AFAIK, he has been preparing that. - https://lists.apache.org/thread.html/6fafeefb7715e8764ccfe5d30c90d7444378b5f4f383ec95e2f1d7de@%3Cdev.spark.apache.org%3E I believe we can handle them after 2.4.4 RC1 (or concurrently.) Hi,

Re: Release Apache Spark 2.4.4

While we're on the topic: In theory, branch 2.3 is meant to be unsupported as of right about now. There are 69 fixes in branch 2.3 since 2.3.3 was released in Februrary: https://issues.apache.org/jira/projects/SPARK/versions/12344844 Some look moderately important. Should we also, or first,

Re: Release Apache Spark 2.4.4

Thank you, DB, Takeshi, Hyukjin, Sean, Kazuaki, Holden, Wenchen! I'll create tag for 2.4.4-rc1 next Monday. For SPARK-27234, it looks like that to me, too. Thanks, Dongjoon. On Wed, Aug 14, 2019 at 9:13 AM Holden Karau wrote: > That looks like more of a feature than a bug fix unless I’m

Re: Release Apache Spark 2.4.4

That looks like more of a feature than a bug fix unless I’m missing something? On Tue, Aug 13, 2019 at 11:58 PM Hyukjin Kwon wrote: > Adding Shixiong > > WDYT? > > 2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성: > >> Can the following be included? >> >> [SPARK-27234][SS][PYTHON] Use

Re: Release Apache Spark 2.4.4

Adding Shixiong WDYT? 2019년 8월 14일 (수) 오후 2:30, Terry Kim 님이 작성: > Can the following be included? > > [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in > EpochTracker (to support Python UDFs) > > > Thanks, > Terry > > On Tue,

Re: Release Apache Spark 2.4.4

Can the following be included? [SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs) Thanks, Terry On Tue, Aug 13, 2019 at 10:24 PM Wenchen Fan wrote: > +1 > > On Wed, Aug 14, 2019 at 12:52

Re: Release Apache Spark 2.4.4

+1 On Wed, Aug 14, 2019 at 12:52 PM Holden Karau wrote: > +1 > Does anyone have any critical fixes they’d like to see in 2.4.4? > > On Tue, Aug 13, 2019 at 5:22 PM Sean Owen wrote: > >> Seems fine to me if there are enough valuable fixes to justify another >> release. If there are any other

Re: Release Apache Spark 2.4.4

+1 Does anyone have any critical fixes they’d like to see in 2.4.4? On Tue, Aug 13, 2019 at 5:22 PM Sean Owen wrote: > Seems fine to me if there are enough valuable fixes to justify another > release. If there are any other important fixes imminent, it's fine to > wait for those. > > > On Tue,

RE: Release Apache Spark 2.4.4

Thanks, Dongjoon! +1 Kazuaki Ishizaki, From: Hyukjin Kwon To: Takeshi Yamamuro Cc: Dongjoon Hyun , dev , User Date: 2019/08/14 09:21 Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4 +1 2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님 이 작성: Hi, Thanks for your

Re: Release Apache Spark 2.4.4

Seems fine to me if there are enough valuable fixes to justify another release. If there are any other important fixes imminent, it's fine to wait for those. On Tue, Aug 13, 2019 at 6:16 PM Dongjoon Hyun wrote: > > Hi, All. > > Spark 2.4.3 was released three months ago (8th May). > As of today

Re: Release Apache Spark 2.4.4

+1 2019년 8월 14일 (수) 오전 9:13, Takeshi Yamamuro 님이 작성: > Hi, > > Thanks for your notification, Dongjoon! > I put some links for the other committers/PMCs to access the info easily: > > A commit list in github from the last release: >

Re: Release Apache Spark 2.4.4

Hi, Thanks for your notification, Dongjoon! I put some links for the other committers/PMCs to access the info easily: A commit list in github from the last release: https://github.com/apache/spark/compare/5ac2014e6c118fbeb1fe8e5c8064c4a8ee9d182a...branch-2.4 A issue list in jira:

Re: Release Apache Spark 2.4.4

+1 On Tue, Aug 13, 2019 at 4:16 PM Dongjoon Hyun wrote: > > Hi, All. > > Spark 2.4.3 was released three months ago (8th May). > As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` > since 2.4.3. > > It would be great if we can have Spark 2.4.4. > Shall we start `2.4.4

Release Apache Spark 2.4.4

Hi, All. Spark 2.4.3 was released three months ago (8th May). As of today (13th August), there are 112 commits (75 JIRAs) in `branch-24` since 2.4.3. It would be great if we can have Spark 2.4.4. Shall we start `2.4.4 RC1` next Monday (19th August)? Last time, there was a request for K8s issue

Re: Re: Release Apache Spark 2.4.4 before 3.0.0

org>, Apache Spark PMC > Date: 2019/07/13 07:18 > Subject:[EXTERNAL] Re: Release Apache Spark 2.4.4 before 3.0.0 > -- > > > > Thank you, Jacek. > > BTW, I added `@private` since we need PMC's help to make an Apache Spark >

Re: Re: Release Apache Spark 2.4.4 before 3.0.0

TERNAL] Re: Release Apache Spark 2.4.4 before 3.0.0 Thank you, Jacek. BTW, I added `@private` since we need PMC's help to make an Apache Spark release. Can I get more feedbacks from the other PMC members? Please me know if you have any concerns (e.g. Release date or Release manager

Re: Release Apache Spark 2.4.4 before 3.0.0

Hi Dongjoon, Should we also consider fixing https://issues.apache.org/jira/browse/SPARK-27812 before the cut? Best, Stavros On Mon, Jul 15, 2019 at 7:04 PM Dongjoon Hyun wrote: > Hi, Apache Spark PMC members. > > Can we cut Apache Spark 2.4.4 next Monday (22nd July)? > > Bests, > Dongjoon. >

Re: Release Apache Spark 2.4.4 before 3.0.0

Hi, Apache Spark PMC members. Can we cut Apache Spark 2.4.4 next Monday (22nd July)? Bests, Dongjoon. On Fri, Jul 12, 2019 at 3:18 PM Dongjoon Hyun wrote: > Thank you, Jacek. > > BTW, I added `@private` since we need PMC's help to make an Apache Spark > release. > > Can I get more feedbacks

Re: Release Apache Spark 2.4.4 before 3.0.0

Thank you, Jacek. BTW, I added `@private` since we need PMC's help to make an Apache Spark release. Can I get more feedbacks from the other PMC members? Please me know if you have any concerns (e.g. Release date or Release manager?) As one of the community members, I assumed the followings (if

Re: Release Apache Spark 2.4.4 before 3.0.0

Hi, Thanks Dongjoon Hyun for stepping up as a release manager! Much appreciated. If there's a volunteer to cut a release, I'm always to support it. In addition, the more frequent releases the better for end users so they have a choice to upgrade and have all the latest fixes or wait. It's their

Re: Release Apache Spark 2.4.4 before 3.0.0

Additionally, one more correctness patch landed yesterday. - SPARK-28015 Check stringToDate() consumes entire input for the and -[m]m formats Bests, Dongjoon. On Tue, Jul 9, 2019 at 10:11 AM Dongjoon Hyun wrote: > Thank you for the reply, Sean. Sure. 2.4.x should be a LTS

Re: Release Apache Spark 2.4.4 before 3.0.0

Thank you for the reply, Sean. Sure. 2.4.x should be a LTS version. The main reason of 2.4.4 release (before 3.0.0) is to have a better basis for comparison to 3.0.0. For example, SPARK-27798 had an old bug, but its correctness issue is only exposed at Spark 2.4.3. It would be great if we can

Re: Release Apache Spark 2.4.4 before 3.0.0

We will certainly want a 2.4.4 release eventually. In fact I'd expect 2.4.x gets maintained for longer than the usual 18 months, as it's the last 2.x branch. It doesn't need to happen before 3.0, but could. Usually maintenance releases happen 3-4 months apart and the last one was 2 months ago. If

Release Apache Spark 2.4.4 before 3.0.0

Hi, All. Spark 2.4.3 was released two months ago (8th May). As of today (9th July), there exist 45 fixes in `branch-2.4` including the following correctness or blocker issues. - SPARK-26038 Decimal toScalaBigInt/toJavaBigInteger not work for decimals not fitting in long - SPARK-26045