Re: [ANNOUNCE] Apache Spark 3.5.1 released

2024-03-04 Thread Yang Jie
hmm... I guess this is meant to cc @Bingkun Pan ? On 2024/03/05 02:16:12 Hyukjin Kwon wrote: > Is this related to https://github.com/apache/spark/pull/42428? > > cc @Yang,Jie(INF) > > On Mon, 4 Mar 2024 at 22:21, Jungtaek Lim > wrote: > > > Shall we revisit this

Re: [VOTE] Release Spark 3.3.4 (RC1)

2023-12-11 Thread Yang Jie
+1 On 2023/12/11 03:03:39 "L. C. Hsieh" wrote: > +1 > > On Sun, Dec 10, 2023 at 6:15 PM Kent Yao wrote: > > > > +1(non-binding > > > > Kent Yao > > > > Yuming Wang 于2023年12月11日周一 09:33写道: > > > > > > +1 > > > > > > On Mon, Dec 11, 2023 at 5:55 AM Dongjoon Hyun wrote: > > >> > > >> +1 > > >> >

Re: Apache Spark 3.3.4 EOL Release?

2023-12-04 Thread Yang Jie
+1 for a 3.3.4 EOL Release. Thanks Dongjoon. Jie Yang On 2023/12/04 15:08:25 Tom Graves wrote: > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > Tom > On Friday, December 1, 2023 at 02:48:22 PM CST, Dongjoon Hyun > wrote: > > Hi, All. > > Since the Apache Spark 3.3.0 RC6 vote passed

Re: Remove HiveContext from Apache Spark 4.0

2023-11-29 Thread Yang Jie
Thank you very much for the feedback from Dongjoon and Xiao Li. After carefully reading https://lists.apache.org/thread/mrx0y078cf3ozs7czykvv864y6dr55xq, I have decided to abandon the deletion of HiveContext. As Xiao Li said, its maintenance cost is not high, but it will increase the cost of

Re: [VOTE] Release Spark 3.4.2 (RC1)

2023-11-29 Thread Yang Jie
+1(non-binding) Jie Yang On 2023/11/29 02:08:04 Kent Yao wrote: > +1(non-binding) > > Kent Yao > > On 2023/11/27 01:12:53 Dongjoon Hyun wrote: > > Hi, Marc. > > > > Given that it exists in 3.4.0 and 3.4.1, I don't think it's a release > > blocker for Apache Spark 3.4.2. > > > > When the

Re: What else could be removed in Spark 4?

2023-08-16 Thread Yang Jie
I would like to know how we should handle the two Kinesis-related modules in Spark 4.0. They have a very low frequency of code updates, and because the corresponding tests are not continuously executed in any GitHub Actions pipeline, so I think they significantly lack quality assurance. On top

Re: Time for Spark 3.4.0 release?

2023-01-03 Thread Yang,Jie(INF)
+1 for me YangJie 发件人: Hyukjin Kwon 日期: 2023年1月4日 星期三 13:16 收件人: Xinrong Meng 抄送: dev 主题: Re: Time for Spark 3.4.0 release? SGTM +1 On Wed, Jan 4, 2023 at 2:13 PM Xinrong Meng mailto:xinrong.apa...@gmail.com>> wrote: Hi All, Shall we cut branch-3.4 on January 16th, 2023? We proposed

Re: maven build failing in spark sql w/BouncyCastleProvider CNFE

2022-12-07 Thread Yang,Jie(INF)
bcpkix-jdk15on test ``` Yang Jie 发件人: "Yang,Jie(INF)" 日期: 2022年12月6日 星期二 18:27 收件人: Steve Loughran 抄送: Hyukjin Kwon , Apache Spark Dev 主题: Re: maven build failing in spark sql w/BouncyCastleProvider CNFE I think we can try scala-maven-plugin 4.8.0 发件人: Steve Loughran

Re: maven build failing in spark sql w/BouncyCastleProvider CNFE

2022-12-06 Thread Yang,Jie(INF)
I think we can try scala-maven-plugin 4.8.0 发件人: Steve Loughran 日期: 2022年12月6日 星期二 18:19 收件人: "Yang,Jie(INF)" 抄送: Hyukjin Kwon , Apache Spark Dev 主题: Re: maven build failing in spark sql w/BouncyCastleProvider CNFE On Tue, 6 Dec 2022 at 04:10, Yang,Jie(INF) mailto:yangji...

Re: maven build failing in spark sql w/BouncyCastleProvider CNFE

2022-12-05 Thread Yang,Jie(INF)
Steve, did compile failed happen when mvn build Spark master with hadoop 3.4.0-SNAPSHOT? 发件人: Hyukjin Kwon 日期: 2022年12月6日 星期二 10:27 抄送: Apache Spark Dev 主题: Re: maven build failing in spark sql w/BouncyCastleProvider CNFE Steve, does the lower version of scala plugin work for you? If that

Re: [ANNOUNCE] Apache Spark 3.2.3 released

2022-11-30 Thread Yang,Jie(INF)
Thanks, Chao! 发件人: Maxim Gekk 日期: 2022年11月30日 星期三 19:40 收件人: Jungtaek Lim 抄送: Wenchen Fan , Chao Sun , dev , user 主题: Re: [ANNOUNCE] Apache Spark 3.2.3 released Thank you, Chao! On Wed, Nov 30, 2022 at 12:42 PM Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> wrote: Thanks Chao for

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-16 Thread Yang,Jie(INF)
+1,non-binding The test combination of Java 11 + Scala 2.12 and Java 11 + Scala 2.13 has passed. Yang Jie 发件人: Chris Nauroth 日期: 2022年11月17日 星期四 04:27 收件人: Yuming Wang 抄送: "Yang,Jie(INF)" , Dongjoon Hyun , huaxin gao , "L. C. Hsieh" , Chao Sun , dev 主题: Re: [VOTE]

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

2022-11-16 Thread Yang,Jie(INF)
+1, non-binding Yang Jie 发件人: Mridul Muralidharan 日期: 2022年11月16日 星期三 17:35 收件人: Kent Yao 抄送: Gengliang Wang , dev 主题: Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications +1 Would be great to see history server performance improvements and lower

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Yang,Jie(INF)
I switched Scala 2.13 to Scala 2.12 today. The test is still in progress and it has not been hung. Yang Jie 发件人: Dongjoon Hyun 日期: 2022年11月16日 星期三 01:17 收件人: "Yang,Jie(INF)" 抄送: huaxin gao , "L. C. Hsieh" , Chao Sun , dev 主题: Re: [VOTE] Release Spark 3.2.3 (RC1) Did y

Re: [VOTE] Release Spark 3.2.3 (RC1)

2022-11-15 Thread Yang,Jie(INF)
ase being executed is `SPARK-28323: PythonUDF should be able to use in join condition`, does anyone have the same problem? Yang Jie 发件人: huaxin gao 日期: 2022年11月15日 星期二 13:59 收件人: "L. C. Hsieh" 抄送: Dongjoon Hyun , Chao Sun , dev 主题: Re: [VOTE] Release Spark 3.2.3 (RC1) +1 Thanks Chao!

Re: Upgrade guava to 31.1-jre and remove hadoop2

2022-11-06 Thread Yang,Jie(INF)
Hi, Bjørn Currently, I don't think we can consider upgrading Guava for the following reasons: 1. Although Spark 3.4.0 will no longer release the hadoop-2 distribution, but the build and testing process is still running. We need to keep it and will not consider upgrading Guava until it is

Re: [ANNOUNCE] Apache Spark 3.3.1 released

2022-10-26 Thread Yang,Jie(INF)
Thanks Yuming and all developers ~ Yang Jie 发件人: Maxim Gekk 日期: 2022年10月26日 星期三 15:19 收件人: Hyukjin Kwon 抄送: "L. C. Hsieh" , Dongjoon Hyun , Yuming Wang , dev , User 主题: Re: [ANNOUNCE] Apache Spark 3.3.1 released Congratulations everyone with the new release, and thanks to Yumi

Re: Apache Spark 3.2.3 Release?

2022-10-18 Thread Yang,Jie(INF)
+1 发件人: vaquar khan 日期: 2022年10月19日 星期三 10:08 收件人: "416161...@qq.com" 抄送: Yuming Wang , kazuyuki tanimura , Gengliang Wang , huaxin gao , Dongjoon Hyun , Sean Owen , Chao Sun , dev 主题: Re: Apache Spark 3.2.3 Release? +1 On Tue, Oct 18, 2022, 8:58 PM

Re: [VOTE] Release Spark 3.3.1 (RC4)

2022-10-18 Thread Yang,Jie(INF)
Use maven to test Java 17 + Scala 2.13 and test passed, +1 for me 发件人: Sean Owen 日期: 2022年10月17日 星期一 21:34 收件人: Yuming Wang 抄送: dev 主题: Re: [VOTE] Release Spark 3.3.1 (RC4) +1 from me, same as last time On Sun, Oct 16, 2022 at 9:14 PM Yuming Wang mailto:wgy...@gmail.com>> wrote: Please

答复: Welcome Yikun Jiang as a Spark committer

2022-10-08 Thread Yang,Jie(INF)
Congratulations Yikun! Regards, Yang Jie 发件人: Mridul Muralidharan 发送时间: 2022年10月8日 14:16:02 收件人: Yuming Wang 抄送: Hyukjin Kwon; dev; Yikun Jiang 主题: Re: Welcome Yikun Jiang as a Spark committer Congratulations ! Regards, Mridul On Sat, Oct 8, 2022 at 12:19 AM

Re: Dropping Apache Spark Hadoop2 Binary Distribution?

2022-10-03 Thread Yang,Jie(INF)
Hi, Dongjoon Our company(Baidu) is still using the combination of Spark 3.3 + Hadoop 2.7.4 in the production environment. Hadoop 2.7.4 is an internally maintained version compiled by Java 8. Although we are using Hadoop 2, I still support this proposal because it is positive and exciting.

Re: [VOTE] SPIP: Support Docker Official Image for Spark

2022-09-21 Thread Yang,Jie(INF)
+1 (non-binding) Regards, Yang Jie 发件人: Gengliang Wang 日期: 2022年9月22日 星期四 12:22 收件人: Xiangrui Meng 抄送: Kent Yao , Hyukjin Kwon , dev 主题: Re: [VOTE] SPIP: Support Docker Official Image for Spark +1 On Wed, Sep 21, 2022 at 7:26 PM Xiangrui Meng mailto:men...@gmail.com>> wrote: +1

答复: [DISCUSS] SPIP: Support Docker Official Image for Spark

2022-09-19 Thread Yang,Jie(INF)
+1 (non-binding) Yang Jie 发件人: Yikun Jiang 发送时间: 2022年9月19日 14:23:14 收件人: Denny Lee 抄送: bo zhaobo; Yuming Wang; Kent Yao; Gengliang Wang; Hyukjin Kwon; dev; zrf 主题: Re: [DISCUSS] SPIP: Support Docker Official Image for Spark Thanks for your support! @all

Re: Time for Spark 3.3.1 release?

2022-09-12 Thread Yang,Jie(INF)
+1 Thanks Yuming ~ 发件人: Hyukjin Kwon 日期: 2022年9月13日 星期二 08:19 收件人: Gengliang Wang 抄送: "L. C. Hsieh" , Dongjoon Hyun , Yuming Wang , dev 主题: Re: Time for Spark 3.3.1 release? +1 On Tue, 13 Sept 2022 at 06:45, Gengliang Wang mailto:ltn...@gmail.com>> wrote: +1. Thank you, Yuming! On Mon,

Re: memory module of yarn container

2022-08-25 Thread Yang,Jie(INF)
Hi, vtygoss In my memory, the memoryOverhead in Spark 2.3 includes all the memories that are not executor onHeap memory, including the memory used by Spark offheapMemoryPool(executorOffHeapMemory, this concept also exists in Spark 2.3), PySparkWorker, PipeRDD used, netty memory pool, JVM

Re: Welcoming three new PMC members

2022-08-10 Thread Yang,Jie(INF)
Congratulations! Regards, Yang Jie 发件人: Yuming Wang 日期: 2022年8月10日 星期三 16:42 收件人: Yikun Jiang 抄送: dev 主题: Re: Welcoming three new PMC members Congratulations! On Wed, Aug 10, 2022 at 4:35 PM Yikun Jiang mailto:yikunk...@gmail.com>> wrote: Congratulations! Regards, Yikun On Wed,

Re: Welcome Xinrong Meng as a Spark committer

2022-08-09 Thread Yang,Jie(INF)
Congratulations! Regards, Yang Jie 发件人: Hyukjin Kwon 日期: 2022年8月9日 星期二 16:12 收件人: dev 主题: Welcome Xinrong Meng as a Spark committer Hi all, The Spark PMC recently added Xinrong Meng as a committer on the project. Xinrong is the major contributor of PySpark especially Pandas API on Spark

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-12 Thread Yang,Jie(INF)
+1 (non-binding) Yang Jie 发件人: Dongjoon Hyun 日期: 2022年7月12日 星期二 16:03 收件人: dev 抄送: Cheng Su , "Yang,Jie(INF)" , Sean Owen 主题: Re: [VOTE] Release Spark 3.2.2 (RC1) +1 Dongjoon. On Mon, Jul 11, 2022 at 11:34 PM Cheng Su mailto:scnj...@gmail.com>> wrote: +1 (non-binding). B

Re: [VOTE] Release Spark 3.2.2 (RC1)

2022-07-11 Thread Yang,Jie(INF)
Does this happen when running all UTs? I ran this suite several times alone using OpenJDK(zulu) 8u322-b06 on my Mac, but no similar error occurred 发件人: Sean Owen 日期: 2022年7月12日 星期二 10:45 收件人: Dongjoon Hyun 抄送: dev 主题: Re: [VOTE] Release Spark 3.2.2 (RC1) Is anyone seeing this error? I'm on

Re: Apache Spark 3.2.2 Release?

2022-07-07 Thread Yang,Jie(INF)
+1 (non-binding) Thank you Dongjoon ~ 发件人: Ruifeng Zheng 日期: 2022年7月7日 星期四 16:28 收件人: dev 主题: Re: Apache Spark 3.2.2 Release? +1 thank you Dongjoon! [图像已被发件人删除。] Ruifeng Zheng ruife...@foxmail.com -- Original -- From:

Contributor data in github-page no longer updated after May 1

2022-05-11 Thread Yang,Jie(INF)
Hi, teams The contributors data in the following page seems no longer updated after May 1, Can anyone fix it? https://github.com/apache/spark/graphs/contributors?from=2022-05-01=2022-05-11=c Warm regards, YangJie

Re: [VOTE] Release Spark 3.1.1 (RC1)

2021-01-19 Thread Yang,Jie(INF)
+1 发件人: Gengliang Wang 日期: 2021年1月19日 星期二 下午3:04 收件人: Jungtaek Lim 抄送: Yuming Wang , Hyukjin Kwon , dev 主题: Re: [VOTE] Release Spark 3.1.1 (RC1) +1 (non-binding) On Tue, Jan 19, 2021 at 2:05 PM Jungtaek Lim mailto:kabhwan.opensou...@gmail.com>> wrote: +1 (non-binding) * verified