Re: [VOTE] SPARK 4.0.0-preview1 (RC1)

762) ~[?:?] at org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749) ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1] ... 38 more Thanks, Cheng Pan > On May 11, 2024, at 13:55, Wenchen Fan wrote: > > Please vote on releasing the followi

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 (non-binding) Thanks, Cheng Pan On Sat, Apr 27, 2024 at 9:29 AM Holden Karau wrote: > > +1 > > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 > YouTube Live Streams: https://www.youtube.co

Re: [DISCUSS] Spark 4.0.0 release

will we have preview release for 4.0.0 like we did for 2.0.0 and 3.0.0? Thanks, Cheng Pan > On Apr 15, 2024, at 09:58, Jungtaek Lim wrote: > > W.r.t. state data source - reader (SPARK-45511), there are several follow-up > tickets, but we don't plan to address them soon.

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1, non-binding Thanks, Cheng Pan > On Apr 15, 2024, at 14:14, John Zhuge wrote: > > +1 (non-binding) > > On Sun, Apr 14, 2024 at 7:18 PM Jungtaek Lim > wrote: > +1 (non-binding), thanks Dongjoon. > > On Sun, Apr 14, 2024 at 7:22 AM Dongjoon Hyun wrote: &

Re: External Spark shuffle service for k8s

-samples/emr-remote-shuffle-service [4] https://github.com/apache/celeborn/issues/2140 Thanks, Cheng Pan > On Apr 6, 2024, at 21:41, Mich Talebzadeh wrote: > > I have seen some older references for shuffle service for k8s, > although it is not clear they are talking about a generic shuff

Re: [DISCUSS] MySQL version support policy

L-compatible product, in the same position as Amazon RDS for MySQL, neither official support declaration nor CI verification is required, but considering the adoption rate of those products, reasonable patches should be considered too. Thanks, Cheng Pan On 2024/03/25 06:47:10 Dongjoon Hyun wrote: > H

[DISCUSS] MySQL version support policy

-innovation-and-long-term-support-lts-versions/ [3] https://github.com/apache/spark/pull/45581 [4] https://aws.amazon.com/rds/mysql/ [5] https://learn.microsoft.com/en-us/azure/mysql/concepts-version-policy Thanks, Cheng Pan

Re: [VOTE] Release Apache Spark 3.5.1 (RC2)

+1 (non-binding) - Build successfully from source code. - Pass integration tests with Spark ClickHouse Connector[1] [1] https://github.com/housepower/spark-clickhouse-connector/pull/299 Thanks, Cheng Pan > On Feb 20, 2024, at 10:56, Jungtaek Lim wrote: > > Thanks Sean, let's

Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 (non-binding) Thanks, Cheng Pan > On Nov 15, 2023, at 01:41, L. C. Hsieh wrote: > > Hi all, > > I’d like to start a vote for SPIP: An Official Kubernetes Operator for > Apache Spark. > > The proposal is to develop an official Java-based Kubernetes operator > f

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

> Not really - this is not designed to be a replacement for the current > approach. That's what I assumed too. But my question is, as a user, how to write a spark-submit command to submit a Spark app to leverage this operator? Thanks, Cheng Pan > On Nov 11, 2023, at 03:21, Zhou Jia

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

Thanks for this impressive proposal, I have a basic question, how does spark-submit work with this operator? Or it enforces that we must use `kubectl apply -f spark-job.yaml`(or K8s client in programming way) to submit Spark app? Thanks, Cheng Pan > On Nov 10, 2023, at 04:05, Zhou Ji

Re: [VOTE] Release Apache Spark 3.3.3 (RC1)

+1 (non-binding) Passed integration test with Apache Kyuubi. Thanks for driving this release. Thanks, Cheng Pan > On Aug 11, 2023, at 06:36, L. C. Hsieh wrote: > > +1 > > Thanks Yuming. > > On Thu, Aug 10, 2023 at 3:24 PM Dongjoon Hyun wrote: >> >> +1

Re: What else could be removed in Spark 4?

-hive Thanks, Cheng Pan > On Aug 8, 2023, at 10:09, Wenchen Fan wrote: > > I think the principle is we should remove things that block us from > supporting new things like Java 21, or come with a significant maintenance > cost. If there is no benefit to removing deprecated API

Re: What else could be removed in Spark 4?

ink it's impossible. [1] sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala [2] https://issues.apache.org/jira/browse/SPARK-42539 [3] https://issues.apache.org/jira/browse/HIVE-27560 [4] https://github.com/apache/spark/pull/33989#issuecomment-926277286 Thanks, Ch

Re: Welcome two new Apache Spark committers

Congratulations! Peter and Xiduo! Thanks, Cheng Pan > On Aug 7, 2023, at 10:58, Gengliang Wang wrote: > > Congratulations! Peter and Xiduo! - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS] SPIP: Python Data Source API

API. Thanks, Cheng Pan > On Jun 16, 2023, at 12:14, Allison Wang > wrote: > > Hi everyone, > > I would like to start a discussion on “Python Data Source API”. > > This proposal aims to introduce a simple API in Python for Data Sources. The > idea is to enable

Re: Apache Spark 3.5.0 Expectations (?)

/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala#L215 Thanks, Cheng Pan > On May 31, 2023, at 03:49, Bjørn Jørgensen wrote: > > @Dongjoon Hyun Thank you. > > I have two points to discuss. > First, we are currently conducting tests wi

Re: Remove protobuf 2.5.0 from Spark dependencies

+CC dev@hbase Thanks, Cheng Pan On Fri, May 19, 2023 at 4:08 AM Steve Loughran wrote: > > > > On Thu, 18 May 2023 at 03:45, Cheng Pan wrote: >> >> Steve, thanks for the information, I think HADOOP-17046 should be fine for >> the Spark case. >> >> Had

Re: Remove protobuf 2.5.0 from Spark dependencies

+CC dev@hbase From: Steve Loughran Date: Friday, May 19, 2023 at 04:08To: Cc: dev Subject: Re: Remove protobuf 2.5.0 from Spark dependencies  On Thu, 18 May 2023 at 03:45, Cheng Pan <cheng...@apache.org> wrote:Steve, thanks for the information, I think HADOOP-17046 should be fine for the

Re: Remove protobuf 2.5.0 from Spark dependencies

classes from hadoop-client-runtime. Thanks, Cheng Pan On May 17, 2023 at 04:10:43, Dongjoon Hyun wrote: > Thank you for sharing, Steve. > > Dongjoon > > On Tue, May 16, 2023 at 11:44 AM Steve Loughran > wrote: > >> I have some bad news here which is even though hadoop

Remove protobuf 2.5.0 from Spark dependencies

by the kinesis client, into the kinesis assembly jar - Spark itself's core/connect/protobuf modules use protobuf 3, also shaded and relocated all protobuf 3 deps. Feel free to comment if you still have any concerns. [1] https://github.com/apache/spark/pull/41153 Thanks, Cheng Pan

Re: Spark Multiple Hive Metastore Catalog Support

] https://github.com/apache/kyuubi/tree/master/extensions/spark/kyuubi-spark-connector-hive Thanks, Cheng Pan On Apr 18, 2023 at 00:38:23, Elliot West wrote: > Hi Ankit, > > While not a part of Spark, there is a project called 'WaggleDance' that > can federate multiple Hive m

Re: [Question] LimitedInputStream license issue in Spark source.

add the ASF License > header to this file[1]? [1] > ./common/src/main/java/org/apache/celeborn/common/network/util/LimitedInputStream.java Willem Jiang Thanks, Cheng Pan On Mar 1, 2023 at 15:04:52, Dongjoon Hyun wrote: > Since both license headers are Apache License 2.0, w

Re: Allow Spark on K8s to integrate w/ External Log Service

The key point here is, how do you jump to the log service from Spark UI to explore or download logs of each Pod like Spark on Yarn? Thanks, Cheng Pan On Nov 2, 2022 at 18:32:26, Martin Andersson wrote: > Hello Cheng. > > I don't quite understand, why can't you configure Log4

Allow Spark on K8s to integrate w/ External Log Service

/pull/32456 Thanks, Cheng Pan

Re: [VOTE] Release Spark 3.3.1 (RC4)

+1 (non-binding) - Passed Apache Kyuubi (Incubating) integration tests[1] - Run some jobs on our internal K8s cluster [1] https://github.com/apache/incubator-kyuubi/pull/3507 Thanks, Cheng Pan On Wed, Oct 19, 2022 at 9:13 AM Yikun Jiang wrote: > > +1, also test passed with spark-

Re: [VOTE] Release Spark 3.3.0 (RC5)

+1 (non-binding) * Verified SPARK-39313 has been address[1] * Passed integration test w/ Apache Kyuubi (Incubating)[2] [1] https://github.com/housepower/spark-clickhouse-connector/pull/123 [2] https://github.com/apache/incubator-kyuubi/pull/2817 Thanks, Cheng Pan On Wed, Jun 8, 2022 at 7:04 AM

Re: [VOTE] Release Spark 3.2.0 (RC7)

+1 (non-binding) Integration test passed[1] with my project[2]. [1] https://github.com/housepower/spark-clickhouse-connector/runs/3834335017 [2] https://github.com/housepower/spark-clickhouse-connector Thanks, Cheng Pan On Sat, Oct 9, 2021 at 2:01 PM Ye Zhou wrote: > +1 (non-bind