Re: [DISCUSS] Spark 4.0.0 release

+1 发件人: Jungtaek Lim 日期: 2024年5月2日 星期四 10:21 收件人: Holden Karau 抄送: Chao Sun , Xiao Li , Tathagata Das , Wenchen Fan , Cheng Pan , Nicholas Chammas , Dongjoon Hyun , Cheng Pan , Spark dev list , Anish Shrigondekar 主题: Re: [DISCUSS] Spark 4.0.0 release +1 love to see it! On Thu, May 2,

Re: [FYI] SPARK-47993: Drop Python 3.8

+1 发件人: Ruifeng Zheng 日期: 2024年4月26日 星期五 15:05 收件人: Xinrong Meng 抄送: Dongjoon Hyun , "dev@spark.apache.org" 主题: Re: [FYI] SPARK-47993: Drop Python 3.8 +1 On Fri, Apr 26, 2024 at 10:26 AM Xinrong Meng mailto:xinr...@apache.org>> wrote: +1 On Thu, Apr 25, 2024 at 2:08 PM Holden Karau

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 for me Jie Yang 发件人: Mich Talebzadeh 日期: 2024年4月14日 星期日 15:41 收件人: Dongjoon Hyun , Spark dev list 主题: Re: [VOTE] SPARK-4: Use ANSI SQL mode by default + 1 for me It makes it more compatible with the other ANSI SQL compliant products. Mich Talebzadeh, Technologist | Solutions

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 Jie Yang 发件人: Haejoon Lee 日期: 2024年3月11日 星期一 17:09 收件人: Gengliang Wang 抄送: dev 主题: Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark +1 On Mon, Mar 11, 2024 at 10:36 AM Gengliang Wang mailto:ltn...@gmail.com>> wrote: Hi all, I'd like to start the vote for SPIP: Structured

Re: [ANNOUNCE] Apache Spark 3.5.1 released

That sounds like a great suggestion. 发件人: Jungtaek Lim 日期: 2024年3月5日 星期二 10:46 收件人: Hyukjin Kwon 抄送: yangjie01 , Dongjoon Hyun , dev , user 主题: Re: [ANNOUNCE] Apache Spark 3.5.1 released Yes, it's relevant to that PR. I wonder, if we want to expose version switcher, it should

Re: [VOTE] Release Apache Spark 3.5.1 (RC2)

Very sorry. When I was fixing `SPARK-45242 (https://github.com/apache/spark/pull/43594)`, I noticed that its `Affects Version` and `Fix Version` of SPARK-45242 were both 4.0, and I didn't realize that it had also been merged into branch-3.5, so I didn't advocate for SPARK-45357 to be

Re: [DISCUSS] Release Spark 3.5.1?

+1 在 2024/2/4 13:13，“Kent Yao”mailto:y...@apache.org>> 写入: +1 Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> 于2024年2月3日周六 21:14写道： > > Hi dev, > > looks like there are a huge number of commits being pushed to branch-3.5 > after 3.5.0 was released, 200+ commits. > > $ git log --oneline

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 发件人: Reynold Xin 日期: 2023年11月25日 星期六 14:35 收件人: Dongjoon Hyun 抄送: Ye Zhou , Mridul Muralidharan , Kent Yao , dev 主题: Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files +1 On Fri, Nov 24, 2023 at 10:19 PM, Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: +1 Thanks,

Re: Apache Spark 3.4.2 (?)

+1 发件人: Yuming Wang 日期: 2023年11月7日 星期二 07:00 收件人: Santosh Pingale 抄送: Dongjoon Hyun , dev 主题: Re: Apache Spark 3.4.2 (?) +1 On Tue, Nov 7, 2023 at 3:55 AM Santosh Pingale wrote: Makes sense given the nature of those commits. On Mon, Nov 6, 2023, 7:52 PM Dongjoon Hyun

Re: Welcome to Our New Apache Spark Committer and PMCs

Congratulations! Jie Yang 发件人: Dongjoon Hyun 日期: 2023年10月4日 星期三 13:04 收件人: Hyukjin Kwon 抄送: Hussein Awala , Rui Wang , Gengliang Wang , Xiao Li , "dev@spark.apache.org" 主题: Re: Welcome to Our New Apache Spark Committer and PMCs Congratulations! Dongjoon. On Tue, Oct 3, 2023 at 5:25 PM

Re: [VOTE] Updating documentation hosted for EOL and maintenance releases

+1 发件人: Yikun Jiang 日期: 2023年9月26日 星期二 18:06 收件人: dev 抄送: Hyukjin Kwon , Ruifeng Zheng 主题: Re: [VOTE] Updating documentation hosted for EOL and maintenance releases +1, I believe it is a wise choice to update the EOL policy of the document based on the real demands of community users.

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

+1 发件人: Jia Fan 日期: 2023年9月12日 星期二 10:08 收件人: Ruifeng Zheng 抄送: Hyukjin Kwon , Xiao Li , Mridul Muralidharan , Peter Toth , Spark dev list , Yuanjian Li 主题: Re: [VOTE] Release Apache Spark 3.5.0 (RC5) +1 Ruifeng Zheng mailto:ruife...@apache.org>> 于2023年9月12日周二 08:46写道： +1 On Tue, Sep 12,

Re: [VOTE] Release Apache Spark 3.5.0 (RC4)

+1 发件人: Gengliang Wang 日期: 2023年9月7日 星期四 12:53 收件人: Yuanjian Li 抄送: Xiao Li , "her...@databricks.com.invalid" , Spark dev list 主题: Re: [VOTE] Release Apache Spark 3.5.0 (RC4) +1 On Wed, Sep 6, 2023 at 9:46 PM Yuanjian Li mailto:xyliyuanj...@gmail.com>> wrote: +1 (non-binding) Xiao Li

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

Hi, Sean I have performed testing with Java 17 and Scala 2.13 using maven (`mvn clean install` and `mvn package test`), and have not encountered the issue you mentioned. The test for the connect module depends on the `spark-protobuf` module to complete the `package,` was it successful? Or

Re: [VOTE] Release Apache Spark 3.5.0 (RC2)

-1, due to SPARK-43646 and SPARK-44784 not yet being fixed. Jie Yang 发件人: Sean Owen 日期: 2023年8月20日 星期日 04:43 收件人: Yuanjian Li 抄送: Spark dev list 主题: Re: [VOTE] Release Apache Spark 3.5.0

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

84/job/15819181762 I think we should address this issue before the release of Apache Spark 3.5.0. Jie Yang 发件人: Yuanjian Li 日期: 2023年8月12日 星期六 15:20 收件人: Yuming Wang 抄送: yangjie01 , Sean Owen , Spark dev list 主题: Re: [VOTE] Release Apache Spark 3.5.0 (RC1) Thanks for all updates! The vote has

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

3 at 9:30 AM yangjie01 mailto:yangji...@baidu.com>> wrote: HI，Dongjoon and Yuming I submitted a PR a few days ago to try to fix this issue: https://github.com/apache/spark/pull/42167<https://mailshield.baidu.com/check?q=zJC5kBC6NRCGy3lXApap3GX6%2bKB9Gi%2b%2fTr0LBfwtxiuVHIiRznzQ7iofG2KJFsJ

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

HI，Dongjoon and Yuming I submitted a PR a few days ago to try to fix this issue: https://github.com/apache/spark/pull/42167. The reason for the failure is that the branch daily test and the master use the same yml file. Jie Yang 发件人: Dongjoon Hyun 日期: 2023年8月8日 星期二 00:18 收件人: Yuming Wang

Re: [VOTE] Release Apache Spark 3.5.0 (RC1)

I submitted a PR last week to try and solve this issue: https://github.com/apache/spark/pull/42236. 发件人: Sean Owen 日期: 2023年8月7日 星期一 11:05 收件人: Yuanjian Li 抄送: Spark dev list 主题: Re: [VOTE] Release Apache Spark 3.5.0 (RC1) 【外部邮件】信息安全要牢记，账号密码不传递！

Re: Welcome two new Apache Spark committers

Congratulations, Peter and Xiduo ~ 发件人: Hyukjin Kwon 日期: 2023年8月7日 星期一 10:30 收件人: Ruifeng Zheng 抄送: Xiao Li , Debasish Das , Wenchen Fan , Spark dev list 主题: Re: Welcome two new Apache Spark committers Woohoo! On Mon, 7 Aug 2023 at 11:28, Ruifeng Zheng mailto:ruife...@apache.org>> wrote:

Re: Time for Spark v3.5.0 release

+1 发件人: Maxim Gekk 日期: 2023年7月4日 星期二 17:24 收件人: Kent Yao 抄送: "dev@spark.apache.org" 主题: Re: Time for Spark v3.5.0 release +1 On Tue, Jul 4, 2023 at 11:55 AM Kent Yao mailto:y...@apache.org>> wrote: +1， thank you Kent On 2023/07/04 05:32:52 Dongjoon Hyun wrote: > +1 > > Thank you, Yuanjian

Re: [ANNOUNCE] Apache Spark 3.4.1 released

Thanks Dongjoon ~ 在 2023/6/24 10:29，“L. C. Hsieh”mailto:vii...@gmail.com>> 写入: Thanks Dongjoon! On Fri, Jun 23, 2023 at 7:10 PM Hyukjin Kwon mailto:gurwls...@apache.org>> wrote: > > Thanks! > > On Sat, Jun 24, 2023 at 11:01 AM Mridul Muralidharan > wrote: >> >> >>

Re: [VOTE] Release Spark 3.4.1 (RC1)

+1 发件人: Dongjoon Hyun 日期: 2023年6月22日 星期四 23:35 收件人: Chao Sun 抄送: Yuming Wang , Jacek Laskowski , dev 主题: Re: [VOTE] Release Spark 3.4.1 (RC1) Thank you everyone for your participation. The vote is open until June 23rd 1AM (PST) and I'll conclude this vote after that. Dongjoon. On Thu,

Re: [VOTE] Release Spark 3.4.1 (RC1)

+1 在 2023/6/21 13:20，“L. C. Hsieh”mailto:vii...@gmail.com>> 写入: +1 On Tue, Jun 20, 2023 at 8:48 PM Dongjoon Hyun mailto:dongj...@apache.org>> wrote: > > +1 > > Dongjoon > > On 2023/06/20 02:51:32 Jia Fan wrote: > > +1 > > > > Dongjoon Hyun mailto:dongj...@apache.org>> > > 于2023年6月20日周二

Re: ASF policy violation and Scala version issues

Yes, you're right. 发件人: Jungtaek Lim 日期: 2023年6月12日 星期一 11:37 收件人: Dongjoon Hyun 抄送: yangjie01 , Grisha Weintraub , Nan Zhu , Sean Owen , "dev@spark.apache.org" 主题: Re: ASF policy violation and Scala version issues Are we concerned that a library does not release a new version w

Re: ASF policy violation and Scala version issues

Perhaps we should reconsider our reliance on and use of Ammonite? There are still no new available versions of Ammonite one week after the release of Scala 2.12.18 and 2.13.11. The question related to version release in the Ammonite community also did not receive a response, which makes me feel

Re: Apache Spark 3.4.1 Release?

+1 Thank you Dongjoon ~ 发件人: Ruifeng Zheng 日期: 2023年6月10日 星期六 09:39 收件人: Xiao Li 抄送: Wenchen Fan , Xinrong Meng , dev 主题: Re: Apache Spark 3.4.1 Release? +1 Thank you Dongjoon! On Fri, Jun 9, 2023 at 11:54 PM Xiao Li wrote: +1 On Fri, Jun 9, 2023 at 08:30 Wenchen Fan

Re: JDK version support policy?

+1 on dropping Java 8 in Spark 4.0, and I even hope Spark 4.0 can only support Java 17 and the upcoming Java 21. 发件人: Denny Lee 日期: 2023年6月7日 星期三 07:10 收件人: Sean Owen 抄送: David Li , "dev@spark.apache.org" 主题: Re: JDK version support policy? +1 on dropping Java 8 in Spark 4.0, saying this as

Re: Apache Spark 4.0 Timeframe?

+1，Agree to start to prepare Apache Spark 4.0 after creating branch-3.5 on July 16th. As I am not yet familiar with Scala 3, I am unable to make good suggestions for choosing the Scala version. But I want to know if Spark 4.0 chooses to use the Scala 2.13.x, is it impossible to switch Scala

Re: [CONNECT] New Clients for Go and Rust

+1 on start this with a separate repo. Which new clients can be placed in the main repo should be discussed after they are mature enough, Yang Jie 发件人: Denny Lee 日期: 2023年5月24日 星期三 21:31 收件人: Hyukjin Kwon 抄送: Maciej , "dev@spark.apache.org" 主题: Re: [CONNECT] New Clients for Go and Rust +1

Re: hadoop-2 profile to be removed in 3.5.0

Thanks Chao ~ Yang Jie 发件人: Dongjoon Hyun 日期: 2023年4月16日 星期日 00:08 收件人: Chao Sun 抄送: dev 主题: Re: hadoop-2 profile to be removed in 3.5.0 Thank you so much for head-ups, Chao! Dongjoon. On Fri, Apr 14, 2023 at 6:33 PM Chao Sun mailto:sunc...@apache.org>> wrote: Hi all, Just a heads up

Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 (non-binding) 发件人: Sean Owen 日期: 2023年4月10日 星期一 21:19 收件人: Dongjoon Hyun 抄送: "dev@spark.apache.org" 主题: Re: [VOTE] Release Apache Spark 3.2.4 (RC1) +1 from me On Sun, Apr 9, 2023 at 7:19 PM Dongjoon Hyun mailto:dongj...@apache.org>> wrote: I'll start with my +1. I verified the checksum,

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 发件人: Sean Owen 日期: 2023年4月8日 星期六 20:27 收件人: Xinrong Meng 抄送: dev 主题: Re: [VOTE] Release Apache Spark 3.4.0 (RC7) +1 form me, same result as last time. On Fri, Apr 7, 2023 at 6:30 PM Xinrong Meng mailto:xinrong.apa...@gmail.com>> wrote: Please vote on releasing the following

Re: [VOTE] Release Apache Spark 3.4.0 (RC6)

-1 for me due to this RC not include the fix of SPARK-39696, SPARK-39696 will fix a data race issue in access to TaskMetrics.externalAccums when using Scala 2.13.8 and this issue will cause high-frequency Executor crash when use Scala 2.13 distribution according to the user's

Re: Apache Spark 3.2.4 EOL Release?

Hi, Dongjoon Hyun Maybe we need include the fix of SPARK-39696 in Apache Spark 3.2.4 EOL Release, this will fix a data race issue in access to TaskMetrics.externalAccums when using Scala 2.13.8 and the corresponding Scala 2.13 release of Spark 3.2.x also uses Scala 2.13.8. 1.

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

023 at 3:47 AM L. C. Hsieh > >>> > >> mailto:vii...@gmail.com>> > >>> wrote: > >>> > >> > >>> > >>> +1 > >>> > >>> > >>> > >>> Thanks Xinrong. > >>> > >>

Re: Apache Spark 3.2.4 EOL Release?

+1 发件人: Yuming Wang 日期: 2023年4月5日 星期三 14:39 收件人: Xinrong Meng 抄送: Hyukjin Kwon , Chao Sun , Holden Karau , "L. C. Hsieh" , Mridul Muralidharan , "dev@spark.apache.org" , huaxin gao 主题: Re: Apache Spark 3.2.4 EOL Release? +1 On Wed, Apr 5, 2023 at 9:09 AM Xinrong Meng

Re: [VOTE] Release Apache Spark 3.4.0 (RC5)

+1, checked Java 17 + Scala 2.13 + Python 3.10.10. 发件人: Herman van Hovell 日期: 2023年3月31日 星期五 12:12 收件人: Sean Owen 抄送: Xinrong Meng , dev 主题: Re: [VOTE] Release Apache Spark 3.4.0 (RC5) +1 On Thu, Mar 30, 2023 at 11:05 PM Sean Owen mailto:sro...@apache.org>> wrote: +1 same result from me as

Re: please help the problem of big parquet file can not be splitted to read

e Davidson 抄送: yangjie01 , Spark Dev List 主题: Re:Re: please help the problem of big parquet file can not be splitted to read @Yangjie. the meta file is attached. I use "hadoop jar parquet-tools-1.11.2.jar meta hdfs://horton/user/yazou/VenInv/shifu_norm_emb_bert/emb_valid_sel_train_

Re: please help the problem of big parquet file can not be splitted to read

Is there only one RowGroup for this file? You can check this by printing the file's metadata using the `meta` command of `parquet-cli`. Yang Jie 发件人: zhangliyun 日期: 2023年3月23日 星期四 15:16 收件人: Spark Dev List 主题: please help the problem of big parquet file can not be splitted to read hi all i

Re: [VOTE] Release Apache Spark 3.4.0 (RC4)

Can you test `./build/mvn clean package -Phive` ? Thanks 发件人: Bjørn Jørgensen 日期: 2023年3月11日 星期六 20:33 收件人: Xinrong Meng 抄送: beliefer , dev 主题: Re: Re: [VOTE] Release Apache Spark 3.4.0 (RC4) Ubuntu 23.04 java --version openjdk 17.0.6 2023-01-17 OpenJDK Runtime Environment (build

Re: [Question] Can't start Spark Connect

Yeah, after executing `./build/mvn -DskipTests clean package ` on the command line, you may need to manually reload maven projects in intellij, otherwise intellij will not immediately respond to the command line behavior. Yang Jie 发件人: Lucifer Tyrant 日期: 2023年3月9日 星期四 00:11 收件人: Jia Fan 抄送:

Re: [VOTE] Release Spark 3.3.2 (RC1)

on, Feb 13, 2023 at 10:25 AM yangjie01 mailto:yangji...@baidu.com>> wrote: Which Python version do you use for testing? When I use the latest Python 3.11, I can reproduce similar test failures (43 tests of sql module fail), but when I use python 3.10, they will succeed YangJie

Re: [VOTE] Release Spark 3.3.2 (RC1)

Which Python version do you use for testing? When I use the latest Python 3.11, I can reproduce similar test failures (43 tests of sql module fail), but when I use python 3.10, they will succeed YangJie 发件人: Bjørn Jørgensen 日期: 2023年2月13日 星期一 05:09 收件人: Sean Owen 抄送: "L. C. Hsieh" , Spark

Re: Time for release v3.3.2

+1 Thanks Liang-Chi! YangJie 发件人: huaxin gao 日期: 2023年1月31日 星期二 10:03 收件人: Dongjoon Hyun 抄送: Hyukjin Kwon , Chao Sun , "L. C. Hsieh" , Spark dev list 主题: Re: Time for release v3.3.2 +1 Thanks Liang-Chi! On Mon, Jan 30, 2023 at 6:01 PM Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote:

Re: Time for Spark 3.4.0 release?

Thanks Xinrong, 发件人: Dongjoon Hyun 日期: 2023年1月25日 星期三 15:49 收件人: Hyukjin Kwon 抄送: Xinrong Meng , "dev@spark.apache.org" 主题: Re: Time for Spark 3.4.0 release? Great! Thank you so much, Xinrong! Dongjoon On Tue, Jan 24, 2023 at 7:17 PM Hyukjin Kwon mailto:gurwls...@gmail.com>> wrote: Thanks