Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-25 Thread Hyukjin Kwon
Thanks, Xiao. I will close this vote within a couple of hours. 2021년 2월 26일 (금) 오후 4:30, Xiao Li 님이 작성: > I confirmed that Q17 and Q39a/b have matching results between Spark 3.0 > and 3.1 after enabling spark.sql.legacy.statisticalAggregate. The result > changes are expected. For more details, yo

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-25 Thread Xiao Li
I confirmed that Q17 and Q39a/b have matching results between Spark 3.0 and 3.1 after enabling spark.sql.legacy.statisticalAggregate. The result changes are expected. For more details, you can read the PR https://github.com/apache/spark/pull/29983/ Also, the result of Q18 is affected by the overflo

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-25 Thread Ismaël Mejía
Since the TPC-DS performance tests are one of the main validation sources for regressions on Spark releases maybe it is time to automate the query outputs validation to find correctness issues eagerly (it would be also nice to validate the performance regressions but correctness >>> performance).

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Reynold Xin
+1 Correctness issues are serious! On Wed, Feb 24, 2021 at 11:08 AM, Mridul Muralidharan < mri...@gmail.com > wrote: > > That is indeed cause for concern. > +1 on extending the voting deadline until we finish investigation of this. > > > > > Regards, > Mridul > > > > On Wed, Feb 24, 2021

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Mridul Muralidharan
That is indeed cause for concern. +1 on extending the voting deadline until we finish investigation of this. Regards, Mridul On Wed, Feb 24, 2021 at 12:55 PM Xiao Li wrote: > -1 Could we extend the voting deadline? > > A few TPC-DS queries (q17, q18, q39a, q39b) are returning different > resul

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Xiao Li
-1 Could we extend the voting deadline? A few TPC-DS queries (q17, q18, q39a, q39b) are returning different results between Spark 3.0 and Spark 3.1. We need a few more days to understand whether these changes are expected. Xiao Mridul Muralidharan 于2021年2月24日周三 上午10:41写道: > > Sounds good, tha

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Mridul Muralidharan
Sounds good, thanks for clarifying Hyukjin ! +1 on release. Regards, Mridul On Wed, Feb 24, 2021 at 2:46 AM Hyukjin Kwon wrote: > I remember HiveExternalCatalogVersionsSuite was flaky for a while which > is fixed in > https://github.com/apache/spark/commit/0d5d248bdc4cdc71627162a3d20c42ad19f24

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Terry Kim
+1 (non-binding) Tested against .NET for Apache Spark. Thanks, Terry On Wed, Feb 24, 2021 at 8:05 AM Dongjoon Hyun wrote: > +1 > > Bests, > Dongjoon > > On Wed, Feb 24, 2021 at 5:46 AM Gabor Somogyi > wrote: > >> +1 (non-binding) >> >> Tested my added security related featues, found an issue

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Dongjoon Hyun
+1 Bests, Dongjoon On Wed, Feb 24, 2021 at 5:46 AM Gabor Somogyi wrote: > +1 (non-binding) > > Tested my added security related featues, found an issue but not a blocker. > > On Wed, 24 Feb 2021, 09:47 Hyukjin Kwon, wrote: > >> I remember HiveExternalCatalogVersionsSuite was flaky for a while

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Gabor Somogyi
+1 (non-binding) Tested my added security related featues, found an issue but not a blocker. On Wed, 24 Feb 2021, 09:47 Hyukjin Kwon, wrote: > I remember HiveExternalCatalogVersionsSuite was flaky for a while which > is fixed in > https://github.com/apache/spark/commit/0d5d248bdc4cdc71627162a3d

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Hyukjin Kwon
I remember HiveExternalCatalogVersionsSuite was flaky for a while which is fixed in https://github.com/apache/spark/commit/0d5d248bdc4cdc71627162a3d20c42ad19f24ef4 and .. KafkaDelegationTokenSuite is flaky ( https://issues.apache.org/jira/browse/SPARK-31250). 2021년 2월 24일 (수) 오후 5:19, Mridul Mural

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-24 Thread Mridul Muralidharan
Signatures, digests, etc check out fine. Checked out tag and build/tested with -Pyarn -Phadoop-2.7 -Phive -Phive-thriftserver -Pmesos -Pkubernetes I keep getting test failures with * org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite * org.apache.spark.sql.kafka010.KafkaDelegationTokenSuit

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-23 Thread Maxim Gekk
+1 (non-binding) On Wed, Feb 24, 2021 at 2:42 AM Cheng Su wrote: > +1 (non-binding) > > > > *From: *Takeshi Yamamuro > *Date: *Tuesday, February 23, 2021 at 3:30 PM > *To: *Hyukjin Kwon , dev > *Subject: *Re: [VOTE] Release Spark 3.1.1 (RC3) > > > > +1 >

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-23 Thread Cheng Su
+1 (non-binding) From: Takeshi Yamamuro Date: Tuesday, February 23, 2021 at 3:30 PM To: Hyukjin Kwon , dev Subject: Re: [VOTE] Release Spark 3.1.1 (RC3) +1 On Wed, Feb 24, 2021 at 2:07 AM John Zhuge mailto:jzh...@apache.org>> wrote: +1 (non-binding) On Mon, Feb 22, 2021 at 10

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-23 Thread Takeshi Yamamuro
+1 On Wed, Feb 24, 2021 at 2:07 AM John Zhuge wrote: > +1 (non-binding) > > On Mon, Feb 22, 2021 at 10:19 PM Gengliang Wang wrote: > >> +1 (non-binding) >> >> On Tue, Feb 23, 2021 at 10:56 AM Yuming Wang wrote: >> >>> +1 @Sean Owen I do not have this issue: >>> >>> [info] SparkSQLEnvSuite: >

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-23 Thread John Zhuge
+1 (non-binding) On Mon, Feb 22, 2021 at 10:19 PM Gengliang Wang wrote: > +1 (non-binding) > > On Tue, Feb 23, 2021 at 10:56 AM Yuming Wang wrote: > >> +1 @Sean Owen I do not have this issue: >> >> [info] SparkSQLEnvSuite: >> 19:45:15.430 WARN org.apache.hadoop.util.NativeCodeLoader: Unable t

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-22 Thread Gengliang Wang
+1 (non-binding) On Tue, Feb 23, 2021 at 10:56 AM Yuming Wang wrote: > +1 @Sean Owen I do not have this issue: > > [info] SparkSQLEnvSuite: > 19:45:15.430 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-22 Thread Yuming Wang
+1 @Sean Owen I do not have this issue: [info] SparkSQLEnvSuite: 19:45:15.430 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19:45:56.366 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of na

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-22 Thread Sean Owen
+1 LGTM, same results as last time. Does anyone see the error below? It is probably env-specific as the Jenkins jobs don't hit this. Just checking. SPARK-29604 external listeners should be initialized with Spark classloader *** FAILED *** java.lang.RuntimeException: [download failed: tomcat#jas

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-22 Thread Herman van Hovell
+1 On Mon, Feb 22, 2021 at 12:59 PM Jungtaek Lim wrote: > +1 (non-binding) > > Verified signatures. Only a few commits added after RC2 which don't seem > to change the SS behavior, so I'd carry over my +1 from RC2. > > On Mon, Feb 22, 2021 at 3:57 PM Hyukjin Kwon wrote: > >> Starting with my +1

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-22 Thread Jungtaek Lim
+1 (non-binding) Verified signatures. Only a few commits added after RC2 which don't seem to change the SS behavior, so I'd carry over my +1 from RC2. On Mon, Feb 22, 2021 at 3:57 PM Hyukjin Kwon wrote: > Starting with my +1 (binding). > > 2021년 2월 22일 (월) 오후 3:56, Hyukjin Kwon 님이 작성: > >> Plea

Re: [VOTE] Release Spark 3.1.1 (RC3)

2021-02-21 Thread Hyukjin Kwon
Starting with my +1 (binding). 2021년 2월 22일 (월) 오후 3:56, Hyukjin Kwon 님이 작성: > Please vote on releasing the following candidate as Apache Spark version > 3.1.1. > > The vote is open until February 24th 11PM PST and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. > > [

[VOTE] Release Spark 3.1.1 (RC3)

2021-02-21 Thread Hyukjin Kwon
Please vote on releasing the following candidate as Apache Spark version 3.1.1. The vote is open until February 24th 11PM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.1.1 [ ] -1 Do not release this package because