Re: [VOTE] Apache Spark PMC asks Databricks to differentiate its Spark version string

statement vote is only about Apache Spark PMC's stance ("Ask or not Ask"). If the vote decides not to ask, that's it. Dongjoon. On Fri, Jun 16, 2023 at 2:23 PM Sean Owen wrote: > On Fri, Jun 16, 2023 at 3:58 PM Dongjoon Hyun > wrote: > >> I started the thread about a

Re: [VOTE][RESULT] Release Spark 3.4.1 (RC1)

Thank you, Mridul. :) On Fri, Jun 23, 2023 at 7:26 AM Mridul Muralidharan wrote: > A late +1 from me too … forgot to send this yesterday :-) > > Regards, > Mridul > > On Fri, Jun 23, 2023 at 3:20 AM Dongjoon Hyun wrote: > >> The vote passes with 15 +1s (10 binding

Re: [VOTE][SPIP] PySpark Test Framework

+1 Dongjoon On Wed, Jun 21, 2023 at 8:56 PM Hyukjin Kwon wrote: > +1 > > On Thu, 22 Jun 2023 at 02:20, Jacek Laskowski wrote: > >> +0 >> >> Pozdrawiam, >> Jacek Laskowski >> >> "The Internals Of" Online Books >> Follow me on https://twitter.com/jaceklaskowski

Re: [DISCUSS] Unified Apache Spark Docker image tag?

. > > [1] > https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o/edit?disco=AAAAf2TyFr0 > > Regards, > Yikun > > > On Tue, May 9, 2023 at 5:03 AM Dongjoon Hyun wrote: > >> Thank you for initiating the discussion in the community. Yes,

Re: Remove protobuf 2.5.0 from Spark dependencies

Thank you for sharing, Steve. Dongjoon On Tue, May 16, 2023 at 11:44 AM Steve Loughran wrote: > I have some bad news here which is even though hadoop cut protobuf 2.5 > support, hbase team put it back in (HADOOP-17046). I don't know if the > shaded hadoop client has removed that dependency on

Re: [UPDATE] Apache Spark 3.5.0 Release Window

Thank you! Dongjoon On Thu, May 11, 2023 at 5:48 PM Xinrong Meng wrote: > Hi All, > > Apache Spark 3.5.0 Release Window is adjusted by #461 > . > > Please check the latest information on the official website >

Re: [DISCUSS] Unified Apache Spark Docker image tag?

Thank you for initiating the discussion in the community. Yes, we need to give more context in the dev mailing list. This root cause is not about SPARK-40941 or SPARK-40513. Technically, this situation started 16 days ago due to SPARK-43148 because it made some breaking changes.

Re: [DISCUSS] Unified Apache Spark Docker image tag?

ark). > Anyway, I am very sorry if there is any misleading, really many thanks for > your feedback and review. > > On Tue, May 9, 2023 at 12:37 PM Dongjoon Hyun wrote: > >> To Yikun, >> >> It seems that your reply (the following) didn't reach out to the mailing >>

Re: [DISCUSS] Unified Apache Spark Docker image tag?

it?disco=f2TyFr0 > > Regards, > Yikun > > > On Tue, May 9, 2023 at 5:03 AM Dongjoon Hyun wrote: > >> Thank you for initiating the discussion in the community. Yes, we need to >> give more context in the dev mailing list. >> >> This root cause is n

Re: Heads-up: Update on Spark 3.5.1 RC

Thank you for the update, Jungtaek. Dongjoon. On Tue, Feb 13, 2024 at 7:29 AM Jungtaek Lim wrote: > Hi, > > Just a head-up since I didn't give an update for a week after the last > update from the discussion thread. > > I've been following the automated release process and encountered several

Re: ASF board report draft for February

+1, it looks good to me. Thank you, Matei. Dongjoon On Sat, Feb 17, 2024 at 11:21 AM Matei Zaharia wrote: > Hi all, > > I missed some reminder emails about our board report this month, but here > is my draft. I’ll submit it tomorrow if that’s ok. > > == > > Issues for the board: >

Re: [VOTE] Release Spark 3.3.4 (RC1)

> > > > On Sun, Dec 10, 2023 at 6:15 PM Kent Yao wrote: > > > > > > > > +1(non-binding > > > > > > > > Kent Yao > > > > > > > > Yuming Wang 于2023年12月11日周一 09:33写道： > > > > > > > > > >

[VOTE][RESULT] Release Spark 3.3.4 (RC1)

The vote passes with 6 +1s (3 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Yuming Wang * - Kent Yao - Liang-Chi Hsieh * - Yang Jie - Malcolm Decuire +0: None -1: None

[ANNOUNCE] Apache Spark 3.3.4 released

would not have been possible without you. Dongjoon Hyun

Re: [VOTE] Release Spark 3.3.4 (RC1)

+1 Dongjoon On 2023/12/08 21:41:00 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.3.4. > > The vote is open until December 15th 1AM (PST) and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. >

Re: Spark on Yarn with Java 17

Hi, Jason. Apache Spark 4.0.0 depends on only Apache Hadoop client library. You can track all `Apache Spark 4` activities including Hadoop dependency here. https://issues.apache.org/jira/browse/SPARK-44111 (Prepare Apache Spark 4.0.0) According to the release history, the original suggested

Re: [VOTE] Release Spark 3.3.4 (RC1)

the above exception, another exception occurred: > > Traceback (most recent call last): > File "", line 1, in > File > "/home/mridul/work/apache/vote/spark/python/pyspark/serializers.py", line > 468, in dumps > raise pickle.PicklingError(msg) >

Re: Spark on Yarn with Java 17

d > Java 8 runtime? > > On Fri, Dec 8, 2023 at 4:33 PM Dongjoon Hyun wrote: > >> Hi, Jason. >> >> Apache Spark 4.0.0 depends on only Apache Hadoop client library. >> >> You can track all `Apache Spark 4` activities including Hadoop dependency >

Re: [VOTE] Release Spark 3.4.2 (RC1)

+1 Dongjoon. On 2023/11/25 10:48:41 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.4.2. > > The vote is open until November 30th 1AM (PST) and passes if a majority +1 > PMC votes are cast, with a minimum of 3 +1 votes. >

[VOTE] Release Spark 3.4.2 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.4.2. The vote is open until November 30th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.2 [ ] -1 Do not release this package

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 Thanks, Dongjoon. On Fri, Nov 24, 2023 at 7:14 PM Ye Zhou wrote: > +1(non-binding) > > On Fri, Nov 24, 2023 at 11:16 Mridul Muralidharan > wrote: > >> >> +1 >> >> Regards, >> Mridul >> >> On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: >> >>> Hi Spark Dev, >>> >>> Following the discussion

Re: Remove HiveContext from Apache Spark 4.0

Thank you for the heads-up. I agree with your intention and the fact that it's not useful in Apache Spark 4.0.0. However, as you know, historically, it was removed once and explicitly added back to the Apache Spark 3.0 via the vote. SPARK-31088 Add back HiveContext and createExternalTable (As a

Re: [VOTE] Release Spark 3.4.2 (RC1)

On Wed, Nov 29, 2023 at 5:08 AM Yang Jie wrote: > > >> > > >> +1(non-binding) > > >> > > >> Jie Yang > > >> > > >> On 2023/11/29 02:08:04 Kent Yao wrote: > > >> > +1(non-binding) > > >> >

[FYI] SPARK-45981: Improve Python language test coverage

Hi, All. As a part of Apache Spark 4.0.0 (SPARK-44111), the Apache Spark community starts to have test coverage for all supported Python versions from Today. - https://github.com/apache/spark/actions/runs/7061665420 Here is a summary. 1. Main CI: All PRs and commits on `master` branch are

Apache Spark 3.3.4 EOL Release?

Hi, All. Since the Apache Spark 3.3.0 RC6 vote passed on Jun 14, 2022, branch-3.3 has been maintained and served well until now. - https://github.com/apache/spark/releases/tag/v3.3.0 (tagged on Jun 9th, 2022) - https://lists.apache.org/thread/zg6k1spw6k1c7brgo6t7qldvsqbmfytm (vote result on June

`orc-format` 1.0 (ORC-1531) for Apache ORC 2.0

Hi, All. As one of the key parts of Apache ORC 2.0, we've been discussing a new repository and module, `orc-format`, in the following. https://github.com/apache/orc/issues/1543 Now, we are ready to create a new repo. Please take a look at the POC repo and code and let us know your thoughts.

Re: Apache Spark 3.3.4 EOL Release?

04 15:08:25 Tom Graves wrote: > > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. > > > Tom > > > On Friday, December 1, 2023 at 02:48:22 PM CST, Dongjoon Hyun < > dongjoon.h...@gmail.com> wrote: > > > > > > Hi, All. > > > > >

Re: Apache Spark 3.3.4 EOL Release?

; > > >> > > Thanks Dongjoon! >> > > >> > > On Mon, Dec 4, 2023 at 9:26 AM Yang Jie wrote: >> > > > >> > > > +1 for a 3.3.4 EOL Release. Thanks Dongjoon. >> > > > >> > > > Jie

[VOTE] Release Spark 3.3.4 (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.3.4. The vote is open until December 15th 1AM (PST) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.3.4 [ ] -1 Do not release this package

Re: [VOTE] Release Spark 3.4.2 (RC1)

coder.scala:62) > at org.apache.spark.sql.Encoders$.bean(Encoders.scala:179) > at org.apache.spark.sql.Encoders.bean(Encoders.scala) > > > https://issues.apache.org/jira/browse/SPARK-45311 > > Thanks ! > > Marc Le Bihan > > > On 25/11/2023 11:48, Dongjoon Hyun wrote: &g

[VOTE][RESULT] Release Spark 3.4.2 (RC1)

The vote passes with 6 +1s (3 binding +1s) and one non-binding -1. Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Kent Yao - Yang Jie - Mridul Muralidharan * - Liang-Chi Hsieh * - Jia Fan +0: None -1: - Marc Le Bihan

[ANNOUNCE] Apache Spark 3.4.2 released

not have been possible without you. Dongjoon Hyun

Re: [DISCUSS] Release Spark 3.5.1?

+1 On Sat, Feb 3, 2024 at 9:18 PM yangjie01 wrote: > +1 > > 在 2024/2/4 13:13，“Kent Yao”mailto:y...@apache.org>> 写入: > > > +1 > > > Jungtaek Lim kabhwan.opensou...@gmail.com>> 于2024年2月3日周六 21:14写道： > > > > Hi dev, > > > > looks like there are a huge number of commits being pushed to branch-3.5

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Apache Spark 3.4.2 (?)

t; On Wed, Nov 8, 2023 at 5:29 AM kazuyuki tanimura > > wrote: > >> > >> +1 > >> > >> Kazu > >> > >> On Nov 7, 2023, at 5:23 PM, L. C. Hsieh wrote: > >> > >> +1 > >> > >> On Tue, Nov 7, 2023 at 4:56 PM D

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

Thank you for proposing a new UI test framework for Apache Spark 4.0. It looks very useful. Thanks, Dongjoon. On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > Hi Spark Dev, > > This is a call to discuss a new SPIP: Testing Framework for > Spark UI Javascript files [1]. The SPIP aims to

Re: Versioning of Spark Operator

ctor Go Client? For example, > Spark Operator 3.5.x supports Spark 3.5 and above. > > Best, > Bo > > > On Tue, Apr 9, 2024 at 10:14 AM Dongjoon Hyun wrote: > > > Ya, that's simple and possible. > > > > However, it may cause many confusions because it implie

Re: Versioning of Spark Operator

Ya, that would work. Inevitably, I looked at Apache Flink K8s Operator's JIRA and GitHub repo. It looks reasonable to me. Although they share the same JIRA, they choose different patterns per place. 1. In POM file and Maven Artifact, independent version number. 1.8.0 2. Tag is also based on

[DISCUSS] SPARK-44444: Use ANSI SQL mode by default

Hi, All. Thanks to you, we've been achieving many things and have on-going SPIPs. I believe it's time to scope Apache Spark 4.0.0 (SPARK-44111) more narrowly by asking your opinions about Apache Spark's ANSI SQL mode. https://issues.apache.org/jira/browse/SPARK-44111 Prepare Apache Spark

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

t; Thanks Dongjoon to drive this! > >> > >> > >> -Rui > >> > >> On Mon, Apr 15, 2024 at 10:10 AM Xinrong Meng wrote: > >> > >>> +1 > >>> > >>> Thank you @Dongjoon Hyun ! > >>> > >>> On

[VOTE][RESULT] SPARK-44444: Use ANSI SQL mode by default

The vote passes with 24 +1s (13 binding +1s). Thanks to all who helped with the vote! (* = binding) +1: - Dongjoon Hyun * - Gengliang Wang * - Chao Sun * - Hyukjin Kwon * - Liang-Chi Hsieh * - Holden Karau * - Huaxin Gao * - Denny Lee - Xiao Li * - Mich Talebzadeh - Christiano Anderson - Yang Jie

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

ror Attribution Framework > > <https://issues.apache.org/jira/browse/SPARK-38615> will also be beneficial > > in migrating to ANSI SQL mode. > > > > > > Gengliang > > > > > > On Thu, Apr 11, 2024 at 7:56 PM Dongjoon Hyun > <mailto:dongjoon.h

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

I'll start from my +1. Dongjoon. On 2024/04/13 22:22:05 Dongjoon Hyun wrote: > Please vote on SPARK-4 to use ANSI SQL mode by default. > The technical scope is defined in the following PR which is > one line of code change and one line of migration guide. > > - DISC

[VOTE] SPARK-44444: Use ANSI SQL mode by default

Please vote on SPARK-4 to use ANSI SQL mode by default. The technical scope is defined in the following PR which is one line of code change and one line of migration guide. - DISCUSSION: https://lists.apache.org/thread/ztlwoz1v1sn81ssks12tb19x37zozxlz - JIRA:

Re: [DISCUSS] Un-deprecate Trigger.Once

For that case, I believe it's enough for us to revise the deprecation message only by making sure that Apache Spark will keep it without removal for backward-compatibility purposes only. That's what the users asked, isn't that? > deprecation of Trigger.Once confuses users that the trigger won't

[ANNOUNCE] Apache Spark 3.4.3 released

not have been possible without you. Dongjoon Hyun

[FYI] SPARK-47046: Apache Spark 4.0.0 Dependency Audit and Cleanup

on the above reports or have new ones for Apache Spark 4.0.0. Dongjoon Hyun

Re: [DISCUSS] Spark 4.0.0 release

Thank you for volunteering, Wenchen. Dongjoon. On 2024/04/12 15:11:04 Wenchen Fan wrote: > Hi all, > > It's close to the previously proposed 4.0.0 release date (June 2024), and I > think it's time to prepare for it and discuss the ongoing projects: > >- ANSI by default >- Spark Connect

Re: [VOTE] Add new `Versions` in Apache Spark JIRA for Versioning of Spark Operator

+1 Thank you! I hope we can customize `dev/merge_spark_pr.py` script per repository after this PR. Dongjoon. On 2024/04/12 03:28:36 "L. C. Hsieh" wrote: > Hi all, > > Thanks for all discussions in the thread of "Versioning of Spark > Operator":

[VOTE] Release Spark 3.4.3 (RC2)

Please vote on releasing the following candidate as Apache Spark version 3.4.3. The vote is open until April 18th 1AM (PDT) and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.3 [ ] -1 Do not release this package because

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

ten v1.1.1 support Spark3.2 and > 3.3. > > We are planning to support Spark3.4 and 3.5 in Gluten v1.2.0. > > Spark4.0 support for Gluten is depending on the release schedule in > Spark community. > > > > On 2024/04/09 07:14:13 Dongjoon Hyun wrote: > > > Thank you for s

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

;>>> complex >>>>>>>>>> SQL queries or existing SQL-based workflows, using Hive may be >>>>>>>>>> advantageous. >>>>>>>>>> 3) If you are looking for performance, spark's native catalog >>>>

[VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

Please vote on SPARK-46122 to set spark.sql.legacy.createHiveTableByDefault to `false` by default. The technical scope is defined in the following PR. - DISCUSSION: https://lists.apache.org/thread/ylk96fg4lvn6klxhj6t6yh42lyqb8wmd - JIRA: https://issues.apache.org/jira/browse/SPARK-46122 - PR:

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

I'll start with my +1. Dongjoon. On 2024/04/26 16:45:51 Dongjoon Hyun wrote: > Please vote on SPARK-46122 to set spark.sql.legacy.createHiveTableByDefault > to `false` by default. The technical scope is defined in the following PR. > > - DISCUSSION: > https://lists.ap

[DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

Hi, All. It's great to see community activities to polish 4.0.0 more and more. Thank you all. I'd like to bring SPARK-46122 (another SQL topic) to you from the subtasks of SPARK-4 (Prepare Apache Spark 4.0.0), - https://issues.apache.org/jira/browse/SPARK-46122 Set

[FYI] SPARK-47993: Drop Python 3.8

FYI, there is a proposal to drop Python 3.8 because its EOL is October 2024. https://github.com/apache/spark/pull/46228 [SPARK-47993][PYTHON] Drop Python 3.8 Since it's still alive and there will be an overlap between the lifecycle of Python 3.8 and Apache Spark 4.0.0, please give us your

Re: [VOTE] Release Spark 3.4.3 (RC2)

I'll start with my +1. - Checked checksum and signature - Checked Scala/Java/R/Python/SQL Document's Spark version - Checked published Maven artifacts - All CIs passed. Thanks, Dongjoon. On 2024/04/15 04:22:26 Dongjoon Hyun wrote: > Please vote on releasing the following candidate as Apa

Re: [VOTE] Release Spark 3.4.3 (RC2)

> > > > > > +1 > > > > > > On Tue, Apr 16, 2024 at 1:38 PM Hyukjin Kwon > > wrote: > > >> > > >> +1 > > >> > > >> On Wed, Apr 17, 2024 at 3:57 AM L. C. Hsieh wrote: > > >>> > > >>&

[VOTE][RESULT] Release Spark 3.4.3 (RC2)

The vote passes with 10 +1s (8 binding +1s). Thanks to all who helped with the release! (* = binding) +1: - Dongjoon Hyun * - Mridul Muralidharan * - Wenchen Fan * - Liang-Chi Hsieh * - Gengliang Wang * - Hyukjin Kwon * - Bo Yang - DB Tsai * - Kent Yao - Huaxin Gao * +0: None -1: None

[VOTE][RESULT] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

The vote passes with 11 +1s (6 binding +1s) and one -1. Thanks to all who helped with the vote! (* = binding) +1: - Dongjoon Hyun * - Gengliang Wang * - Liang-Chi Hsieh * - Holden Karau * - Zhou Jiang - Cheng Pan - Hyukjin Kwon * - DB Tsai * - Ye Xianjin - XiDuo You - Nimrod Ofek +0: None -1

Re: [DISCUSS] Spark 4.0.0 release

, 2024 at 7:57 AM Dongjoon Hyun wrote: > Could you file an INFRA JIRA issue with the error message and context > first, Wenchen? > > As you know, if we see something, we had better file a JIRA issue because > it could be not only an Apache Spark project issue but also all ASF p

Re: [DISCUSS] Spark 4.0.0 release

gt; YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>>>> >>>>>>> >>>>>>> On Tue, May 7, 2024 at 10:55 AM Nimrod Ofek >>>>>>> wrote: >>>>>>> >>&g

Re: [DISCUSS] Spark 4.0.0 release

> On Thu, May 9, 2024 at 11:06 PM Dongjoon Hyun > wrote: > >> In addition, FYI, I was the latest release manager with Apache Spark >> 3.4.3 (2024-04-15 Vote) >> >> According to my work log, I uploaded the following binaries to SVN from >> EC2 (us-west-2)

Re: [DISCUSS] Spark 4.0.0 release

es targeting the Delta 4.0 release are still incomplete. >>> >>> Thanks! >>> >>> >>> On Wed, Apr 17, 2024 at 8:35 AM Wenchen Fan wrote: >>> >>>> Thank you all for the replies! >>>> >>>> To @Nicholas Chammas : Th

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

+1 On Sun, May 12, 2024 at 3:50 PM huaxin gao wrote: > +1 > > On Sat, May 11, 2024 at 4:35 PM L. C. Hsieh wrote: > >> +1 >> >> On Sat, May 11, 2024 at 3:11 PM Chao Sun wrote: >> > >> > +1 >> > >> > On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: >> >> >> >> Hi all, >> >> >> >> I’d like to

Re: [DISCUSS] Spark 4.0.0 release

olden Karau >> *抄送**: *Chao Sun , Xiao Li , >> Tathagata Das , Wenchen Fan < >> cloud0...@gmail.com>, Cheng Pan , Nicholas Chammas < >> nicholas.cham...@gmail.com>, Dongjoon Hyun , >> Cheng Pan , Spark dev list , >> Anish Shrigondekar >> *主题**: *Re

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

t; 1) Hive provides a more mature and widely adopted catalog >>>>>>>>>>> solution that integrates well with other components in the Hadoop >>>>>>>>>>> ecosystem, such as HDFS, HBase, and YARN. IIf you are Hadoop >&g

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

sand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Mon, 29 Apr 2024 at 17:32, Dongjoon Hyun wrote: > > > It's a surprise to me to see t

Re: ASF board report draft for May

+1 for Holden's comment. Yes, it would be great to mention `it` as "soon". (If Wenchen release it on Monday, we can simply mention the release) In addition, Apache Spark PMC received an official notice from ASF Infra team. https://lists.apache.org/thread/rgy1cg17tkd3yox7qfq87ht12sqclkbg >

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

Ya, I also have a similar opinion with Mridul. +1 Thank you, Gengliang. Dongjoon. On Mon, Mar 11, 2024 at 1:34 PM Mridul Muralidharan wrote: > > I am supportive of the proposal - this is a step in the right direction ! > Additional metadata (explicit and inferred) for log records, and

Re: [DISCUSS] MySQL version support policy

Hi, Cheng. Thank you for the suggestion. Your suggestion seems to have at least two themes. A. Adding a new Apache Spark community policy (contract) to guarantee MySQL LTS Versions Support. B. Dropping the support of non-LTS version support (MySQL 8.3/8.2/8.1) And, it brings me three questions.

Re: The dedicated repository for Kubernetes Operator for Apache Spark

Thank you, Liang-Chi! Dongjoon. On Wed, Mar 27, 2024 at 10:56 PM L. C. Hsieh wrote: > Hi all, > > For the passed SPIP: An Official Kubernetes Operator for Apache Spark, > the developers have been working on code cleaning and refactoring for > open source in the last few months. They are ready

Re: SPIP: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

Thank you for sharing, Jia. I have the same questions like the previous Weiting's thread. Do you think you can share the future milestone of Apache Gluten? I'm wondering when the first stable release will come and how we can coordinate across the ASF communities. > This project is still under

Re: Introducing Apache Gluten(incubating), a middle layer to offload Spark to native engine

Thank you for sharing, Weiting. Do you think you can share the future milestone of Apache Gluten? I'm wondering when the first stable release will come and how we can coordinate across the ASF communities. > This project is still under active development now, and doesn't have a stable release. >

Re: Versioning of Spark Operator

Hi, Liang-Chi. Thank you for leading Apache Spark K8s operator as a shepherd. I took a look at `Apache Spark Connect Go` repo mentioned in the thread. Sadly, there is no release at all and no activity since last 6 months. It seems to be the first time for Apache Spark community to consider

Re: Versioning of Spark Operator

elopers and > > intuitive for users. > > > > Regards, > > Mridul > > > > > > On Tue, Apr 9, 2024 at 10:09 AM Dongjoon Hyun > <mailto:dongj...@apache.org>> wrote: > >> Hi, Liang-Chi. > >> > >> Thank you for leading Ap

Apache Spark 3.4.3 (?)

Hi, All. Apache Spark 3.4.2 tag was created on Nov 24th and `branch-3.4` has 85 commits including important security and correctness patches like SPARK-45580, SPARK-46092, SPARK-46466, SPARK-46794, and SPARK-46862. https://github.com/apache/spark/releases/tag/v3.4.2 $ git log --oneline

Re: Apache Spark 3.4.3 (?)

Thank you, Holden, Mridul, Kent, Liang-Chi, Mich, Jungtaek. I added `Target Version: 3.4.3` to SPARK-47318 and am going to continue to prepare for RC1 (April 15th). Dongjoon. - To unsubscribe e-mail:

Re: [VOTE] SPIP: Pure Python Package in PyPI (Spark Connect)

+1 Thank you, Hyukjin. Dongjoon On Sun, Mar 31, 2024 at 19:07 Haejoon Lee wrote: > +1 > > On Mon, Apr 1, 2024 at 10:15 AM Hyukjin Kwon wrote: > >> Hi all, >> >> I'd like to start the vote for SPIP: Pure Python Package in PyPI (Spark >> Connect) >> >> JIRA

Re: [VOTE] Release Apache Spark 3.5.1 (RC2)

Hi, All. Unfortunately, the Apache Spark `3.5.1 RC2` document artifact seems to be generated from unknown source code instead of the correct source code of the tag, `3.5.1`. https://spark.apache.org/docs/3.5.1/ [image: Screenshot 2024-02-23 at 14.13.07.png] Dongjoon. On Wed, Feb 21, 2024 at

Re: When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

gt; Could you please share a list of fixes as the link provided by you is > not working. > > On Thu, Feb 29, 2024 at 11:27 AM Dongjoon Hyun > wrote: > >> Hi, >> >> If you are observing correctness issues, you may hit some old (and fixed) >> correctness is

Re: [ANNOUNCE] Apache Spark 3.5.1 released

master [image: Screenshot 2024-02-29 at 21.12.24.png] Could you do the follow-up, please? Thank you in advance. Dongjoon. On Thu, Feb 29, 2024 at 2:48 PM John Zhuge wrote: > Excellent work, congratulations! > > On Wed, Feb 28, 2024 at 10:12 PM Dongjoon Hyun > wrote: > >> C

Re: When Spark job shows FetchFailedException it creates few duplicate data and we see few data also missing , please explain why

Hi, If you are observing correctness issues, you may hit some old (and fixed) correctness issues. For example, from Apache Spark 3.2.1 to 3.2.4, we fixed 31 correctness issues.

Re: [ANNOUNCE] Apache Spark 3.5.1 released

Congratulations! Bests, Dongjoon. On Wed, Feb 28, 2024 at 11:43 AM beliefer wrote: > Congratulations! > > > > At 2024-02-28 17:43:25, "Jungtaek Lim" > wrote: > > Hi everyone, > > We are happy to announce the availability of Spark 3.5.1! > > Spark 3.5.1 is a maintenance release containing

<    3   4   5   6   7   8