Re: [VOTE] Release Spark 3.5.2 (RC4)

2024-07-29 Thread Peter Toth
+1 XiDuo You ezt írta (időpont: 2024. júl. 30., K, 7:56): > +1 > > Zhou Jiang 于2024年7月30日周二 02:08写道: > > > > +1 (non-binding) > > > > Zhou JIANG > > > > > > > > On Mon, Jul 29, 2024 at 11:06 L. C. Hsieh wrote: > >> > >> +1 > >> > >> On Mon, Jul 29, 2024 at 7:33 AM Wenchen Fan > wrote: > >> >

Re: [VOTE] Release Spark 3.5.2 (RC4)

2024-07-29 Thread XiDuo You
+1 Zhou Jiang 于2024年7月30日周二 02:08写道: > > +1 (non-binding) > > Zhou JIANG > > > > On Mon, Jul 29, 2024 at 11:06 L. C. Hsieh wrote: >> >> +1 >> >> On Mon, Jul 29, 2024 at 7:33 AM Wenchen Fan wrote: >> > >> > +1 >> > >> > On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun >> > wrote: >> >> >> >> +1

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

2024-07-29 Thread Hyukjin Kwon
Let me clarify them. This is TL;DR: - The PR (https://github.com/apache/spark/pull/47328) mentioned Databricks in the PR description (it is edited now), which had to be avoided. - I also updated committer guidelines in spark-website to prevent such cases in the future. - Otherwise, the change was

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

2024-07-29 Thread Russell Spitzer
The original text (viewable in github) was Why are the changes needed? In *databricks runtime*certain storage configuration cases, RDD read / write API has some issue for certain storage types that requires the account key, but Dataframe read / write API works. On Mon, Jul 29, 2024 at 2:11 PM

Re: [VOTE] Release Spark 3.5.2 (RC4)

2024-07-29 Thread Zhou Jiang
+1 (non-binding) *Zhou JIANG* On Mon, Jul 29, 2024 at 11:06 L. C. Hsieh wrote: > +1 > > On Mon, Jul 29, 2024 at 7:33 AM Wenchen Fan wrote: > > > > +1 > > > > On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun > wrote: > >> > >> +1 > >> > >> Thank you, Kent. > >> > >> Dongjoon. > >> > >> On

Re: [VOTE] Release Spark 3.5.2 (RC4)

2024-07-29 Thread L. C. Hsieh
+1 On Mon, Jul 29, 2024 at 7:33 AM Wenchen Fan wrote: > > +1 > > On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun > wrote: >> >> +1 >> >> Thank you, Kent. >> >> Dongjoon. >> >> On Fri, Jul 26, 2024 at 6:37 AM Kent Yao wrote: >>> >>> Hi dev, >>> >>> Please vote on releasing the following

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

2024-07-29 Thread Sean Owen
Also from ASF community perspective - I think all are agreed this was merged too fast. But, I'm missing where this is somehow due to the needs of a single vendor. Where is this related to file systems or keys? did I miss it from another discussion or PR, or is this actually about a different

Re: [DISCUSS] Why do we remove RDD usage and RDD-backed code?

2024-07-29 Thread Steve Loughran
I'm going to join in from an ASF community perspective. Nobody should be making fundamental changes to an ASF code base with a PR up and then merged two hours later because of the needs of a single vendor of a downstream product. This doesn't even give people in different time zones the chance to

Please review (ValidateExternalType should return child in error)

2024-07-29 Thread Mark Andreev
Hi Spark Devs, Please review my PR [ https://github.com/apache/spark/pull/47522 ] that relates to ticket [ https://issues.apache.org/jira/browse/SPARK-49044 ]. Context: When we have mixed schema rows, the error message "{actual} is not a valid external type for schema of {expected}" doesn't help

Re: [VOTE] Release Spark 3.5.2 (RC4)

2024-07-29 Thread Wenchen Fan
+1 On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun wrote: > +1 > > Thank you, Kent. > > Dongjoon. > > On Fri, Jul 26, 2024 at 6:37 AM Kent Yao wrote: > >> Hi dev, >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.5.2. >> >> The vote is open until Jul 29,

Re: Spark decommission

2024-07-29 Thread Steve Loughran
On Fri, 5 Jul 2024 at 01:44, Arun Ravi wrote: > Hi Rajesh, > > We use it production at scale. We run spark on kubernetes on aws cloud and > here are the key things that we do > 1) we run driver on on-demand node > 2) we have configured decommission along with fallback option on to S3, > try the