Re: [Discuss][SPIP] DataSource V2 SQL push down

2021-04-04 Thread huaxin gao
Hello Chang, Thanks for proposing the SPIP and initiating the discussion. However, I think the problem with your proposal is that you haven’t taken into consideration file-based data sources such as parquet, ORC, etc. As far as I know, most of the Spark users have file-based data sources. As a

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-04-04 Thread Liang-Chi Hsieh
Thanks Hyukjin and Dongjoon! :) Then I will start RC. Dongjoon Hyun-2 wrote > Given that Maven passed already with that profile and you tested locally, > I'm +1 for staring RC. > > Thanks, > Dongjoon. > > On Sun, Apr 4, 2021 at 2:24 AM Hyukjin Kwon > gurwls223@ > wrote: > >> I would

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-04-04 Thread Dongjoon Hyun
Given that Maven passed already with that profile and you tested locally, I'm +1 for staring RC. Thanks, Dongjoon. On Sun, Apr 4, 2021 at 2:24 AM Hyukjin Kwon wrote: > I would +1for just going ahead. That looks flaky to me too. > > Thanks Langchi for driving this! > > > On Sun, 4 Apr 2021,

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-04-04 Thread Hyukjin Kwon
I would +1for just going ahead. That looks flaky to me too. Thanks Langchi for driving this! On Sun, 4 Apr 2021, 18:17 Liang-Chi Hsieh, wrote: > Hi devs, > > Currently no open issues or ongoing issues targeting 2.4. > > On QA test dashboard, only spark-branch-2.4-test-sbt-hadoop-2.6 is in red

Re: Apache Spark 2.4.8 (and EOL of 2.4)

2021-04-04 Thread Liang-Chi Hsieh
Hi devs, Currently no open issues or ongoing issues targeting 2.4. On QA test dashboard, only spark-branch-2.4-test-sbt-hadoop-2.6 is in red status. The failed test is org.apache.spark.sql.streaming.StreamingQueryManagerSuite.awaitAnyTermination with timeout and resetTerminated. It looks a flaky