Re: [VOTE] Release Apache Spark 3.2.4 (RC1)

+1 On Tue, Apr 11, 2023 at 10:09 AM Hyukjin Kwon wrote: > +1 > > On Tue, 11 Apr 2023 at 11:04, Ruifeng Zheng wrote: > >> +1 (non-binding) >> >> Thank you for driving this release! >> >> -- >> Ruifeng Zheng >> ruife...@foxmail.com >> >>

Re: [VOTE] Release Apache Spark 3.4.0 (RC7)

+1 On Tue, Apr 11, 2023 at 9:57 AM Yuming Wang wrote: > +1. > > On Tue, Apr 11, 2023 at 9:14 AM Yikun Jiang wrote: > >> +1 (non-binding) >> >> Also ran the docker image related test (signatures/standalone/k8s) with >> rc7: https://github.com/apache/spark-docker/pull/32 >> >> Regards, >> Yikun

Re: Time for release v3.3.2

+1, thanks! On Tue, Jan 31, 2023 at 3:17 PM Maxim Gekk wrote: > +1 > > On Tue, Jan 31, 2023 at 10:12 AM John Zhuge wrote: > >> +1 Thanks Liang-Chi for driving the release! >> >> On Mon, Jan 30, 2023 at 10:26 PM Yuming Wang wrote: >> >>> +1 >>> >>> On Tue, Jan 31, 2023 at 12:18 PM yangjie01

Re: Apache Spark 3.4.1 Release?

+1 On Fri, Jun 9, 2023 at 8:52 PM Xinrong Meng wrote: > +1. Thank you Doonjoon! > > Thanks, > > Xinrong Meng > > Mridul Muralidharan 于2023年6月9日 周五上午5:22写道： > >> >> +1, thanks Dongjoon ! >> >> Regards, >> Mridul >> >> On Thu, Jun 8, 2023 at 7:16 PM Jia Fan >> wrote: >> >>> +1 >>> >>>

Re: [Feature Request] create *permanent* Spark View from DataFrame via PySpark

DataFrame view stores the logical plan, while SQL view stores SQL text. I don't think we can support this feature until we have a reliable way to materialize logical plans. On Sun, Jun 4, 2023 at 10:31 PM Mich Talebzadeh wrote: > Try sending it to dev@spark.apache.org (and join that group) > >

Re: What else could be removed in Spark 4?

I think the principle is we should remove things that block us from supporting new things like Java 21, or come with a significant maintenance cost. If there is no benefit to removing deprecated APIs (just to keep the codebase clean?), I'd prefer to leave them there and not bother. On Tue, Aug 8,

Welcome two new Apache Spark committers

Hi all, The Spark PMC recently voted to add two new committers. Please join me in welcoming them to their new role! - Peter Toth (Spark SQL) - Xiduo You (Spark SQL) They consistently make contributions to the project and clearly showed their expertise. We are very excited to have them join as

Re: [DISCUSS] SPIP: Python Data Source API

In an ideal world, every data source you want to connect to already has a Spark data source implementation (either v1 or v2), then this Python API is useless. But I feel it's common that people want to do quick data exploration, and the target data system is not popular enough to have an existing

Re: [VOTE] Release Apache Spark 3.5.1 (RC2)

+1, thanks for making the release! On Sat, Feb 17, 2024 at 3:54 AM Sean Owen wrote: > Yeah let's get that fix in, but it seems to be a minor test only issue so > should not block release. > > On Fri, Feb 16, 2024, 9:30 AM yangjie01 wrote: > >> Very sorry. When I was fixing `SPARK-45242 ( >>

Re: [VOTE] SPIP: Structured Streaming - Arbitrary State API v2

+1 On Thu, Jan 11, 2024 at 9:32 AM L. C. Hsieh wrote: > +1 > > On Wed, Jan 10, 2024 at 9:06 AM Bhuwan Sahni > wrote: > >> +1. This is a good addition. >> >> >> *Bhuwan Sahni* >> Staff Software Engineer >> >> bhuwan.sa...@databricks.com >> 500 108th Ave. NE >>

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

+1, very useful! On Wed, Nov 22, 2023 at 10:29 AM Dongjoon Hyun wrote: > Thank you for proposing a new UI test framework for Apache Spark 4.0. > > It looks very useful. > > Thanks, > Dongjoon. > > > On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > >> Hi Spark Dev, >> >> This is a call to

Re: SPIP: Enhancing the Flexibility of Spark's Physical Plan to Enable Execution on Various Native Engines

It's good to reduce duplication between different native accelerators of Spark, and AFAIK there is already a project trying to solve it: https://substrait.io/ I'm not sure why we need to do this inside Spark, instead of doing the unification for a wider scope (for all engines, not only Spark).

[DISCUSS] Spark 4.0.0 release

collation support - Spark k8s operator versioning Please help to add more items to this list that are missed here. I would like to volunteer as the release manager for Apache Spark 4.0.0 if there is no objection. Thank you all for the great work that fills Spark 4.0! Wenchen Fan

Re: [DISCUSS] SPARK-44444: Use ANSI SQL mode by default

+1, the existing "NULL on error" behavior is terrible for data quality. I have one concern about error reporting with DataFrame APIs. Query execution is lazy and where the error happens can be far away from where the dataframe/column was created. We are improving it (PR

Re: [DISCUSS] Spark 4.0.0 release

add more context on the plan for transformWithState) > > > > On Sat, Apr 13, 2024 at 3:15 AM Wenchen Fan wrote: > > Hi all, > > > > It's close to the previously proposed 4.0.0 release date (June 2024), > and I think it's time to prepare for it and discuss

Re: [VOTE] Release Spark 3.4.3 (RC2)

+1 On Mon, Apr 15, 2024 at 12:31 PM Dongjoon Hyun wrote: > I'll start with my +1. > > - Checked checksum and signature > - Checked Scala/Java/R/Python/SQL Document's Spark version > - Checked published Maven artifacts > - All CIs passed. > > Thanks, > Dongjoon. > > On 2024/04/15 04:22:26

Re: [VOTE] SPARK-44444: Use ANSI SQL mode by default

+1 On Sun, Apr 14, 2024 at 6:28 AM Dongjoon Hyun wrote: > I'll start from my +1. > > Dongjoon. > > On 2024/04/13 22:22:05 Dongjoon Hyun wrote: > > Please vote on SPARK-4 to use ANSI SQL mode by default. > > The technical scope is defined in the following PR which is > > one line of code

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

+1 On Thu, Apr 25, 2024 at 2:46 PM Kent Yao wrote: > +1 > > Nit: the umbrella ticket is SPARK-44111, not SPARK-4. > > Thanks, > Kent Yao > > Dongjoon Hyun 于2024年4月25日周四 14:39写道： > > > > Hi, All. > > > > It's great to see community activities to polish 4.0.0 more and more. > > Thank you

Re: [DISCUSS] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Thu, 25 Apr 2024 at 11:17, Wenchen Fan wrote: > >

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

ave a more reasonable default behavior: creating Parquet tables (or whatever is specified by `spark.sql.sources.default`). On Tue, Apr 30, 2024 at 10:45 AM Wenchen Fan wrote: > @Mich Talebzadeh there seems to be a > misunderstanding here. The Spark native data source table is still stor

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

024 at 2:08 AM Mich Talebzadeh > wrote: > >> >> Hi @Wenchen Fan >> >> Thanks for your response. I believe we have not had enough time to >> "DISCUSS" this matter. >> >> Currently in order to make Spark take advantage of Hive, I create a soft >

Re: [VOTE] SPARK-46122: Set spark.sql.legacy.createHiveTableByDefault to false

@Mich Talebzadeh thanks for sharing your concern! Note: creating Spark native data source tables is usually Hive compatible as well, unless we use features that Hive does not support (TIMESTAMP NTZ, ANSI INTERVAL, etc.). I think it's a better default to create Spark native table in this case,

Re: [DISCUSS] Spark 4.0.0 release

>>>>> >>>>> בתאריך יום ד׳, 8 במאי 2024, 00:54, מאת Holden Karau ‏< >>>>> holden.ka...@gmail.com>: >>>>> >>>>>> Indeed. We could conceivably build the release in CI/CD but the final >>>>>&

Re: ASF board report draft for May

The preview release also needs a vote. I'll try my best to cut the RC on Monday, but the actual release may take some time. Hopefully, we can get it out this week but if the vote fails, it will take longer as we need more RCs. On Mon, May 6, 2024 at 7:22 AM Dongjoon Hyun wrote: > +1 for

Re: [DISCUSS] Spark - How to improve our release processes

Thanks for starting the discussion! To add a bit more color, we should at least add a test job to make sure the release script can produce the packages correctly. Today it's kind of being manually tested by the release manager each time, which slows down the release process. It's better if we can

Re: [DISCUSS] Spark 4.0.0 release

;> Could you file an INFRA JIRA issue with the error message and context >> first, Wenchen? >> >> As you know, if we see something, we had better file a JIRA issue because >> it could be not only an Apache Spark project issue but also all ASF project >> issues. >>

Re: [DISCUSS] Spark 4.0.0 release

MB > > Dongjoon. > > > > On Thu, May 9, 2024 at 8:12 AM Wenchen Fan wrote: > >> I've created a ticket: https://issues.apache.org/jira/browse/INFRA-25776 >> >> On Thu, May 9, 2024 at 11:06 PM Dongjoon Hyun >> wrote: >> >>> In addition, FYI, I

Re: [DISCUSS] SPIP: Stored Procedures API for Catalogs

Thanks for leading this project! Let's move forward. On Fri, May 10, 2024 at 10:31 AM L. C. Hsieh wrote: > Thanks Anton. Thank you, Wenchen, Dongjoon, Ryan, Serge, Allison and > others if I miss those who are participating in the discussion. > > I suppose we have reached a consensus or close to

Re: [DISCUSS] clarify the definition of behavior changes

modate the second group of users. > > On 1 May 2024, at 06:08, Wenchen Fan wrote: > > Hi all, > > It's exciting to see innovations keep happening in the Spark community and > Spark keeps evolving itself. To make these innovations available to more > users, it's important to help

Re: [DISCUSS] Spark 4.0.0 release

a Preview release, > the faster we can start getting feedback for fixing things for a great > Spark 4.0 final release. > > So I urge the community to produce a Spark 4.0 Preview soon even if > certain features targeting the Delta 4.0 release are still incomplete. > > Thanks! > >

Re: [DISCUSS] clarify the definition of behavior changes

park as a service from a >> provider >> 2. Providers/Operators: There are some users that provide spark as a >> service for their internal(on-prem setup with yarn/k8s)/external(Something >> like EMR) customers >> 3. ? >> >> Perhaps we need to consider

[VOTE] SPARK 4.0.0-preview1 (RC1)

Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 16 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package

Re: [DISCUSS] Spark - How to improve our release processes

ng again to fix this problem, but it needs to be in > collaboration with a committer since I cannot fully test the release > scripts. (This testing gap is what doomed my last attempt at fixing this > problem.) > > Nick > > > On May 13, 2024, at 12:18 AM, Wenchen Fan wrote: &

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

+1 On Mon, May 13, 2024 at 10:30 AM Kent Yao wrote: > +1 > > Dongjoon Hyun 于2024年5月13日周一 08:39写道： > > > > +1 > > > > On Sun, May 12, 2024 at 3:50 PM huaxin gao > wrote: > >> > >> +1 > >> > >> On Sat, May 11, 2024 at 4:35 PM L. C. Hsieh wrote: > >>> > >>> +1 > >>> > >>> On Sat, May 11, 2024

Re: [DISCUSS] Spark - How to improve our release processes

After finishing the 4.0.0-preview1 RC1, I have more experience with this topic now. In fact, the main job of the release process: building packages and documents, is tested in Github Action jobs. However, the way we test them is different from what we do in the release scripts. 1. the execution

Re: [DISCUSS] Spark 4.0.0 release

wrote: > +1 > > > > *发件人**: *Jungtaek Lim > *日期**: *2024年5月2日 星期四 10:21 > *收件人**: *Holden Karau > *抄送**: *Chao Sun , Xiao Li , > Tathagata Das , Wenchen Fan < > cloud0...@gmail.com>, Cheng Pan , Nicholas Chammas < > nicholas.cham...@gmail.com>,

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

+1 On Tue, May 14, 2024 at 8:19 AM Zhou Jiang wrote: > +1 (non-binding) > > On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: > >> Hi all, >> >> I’d like to start a vote for SPIP: Stored Procedures API for Catalogs. >> >> Please also refer to: >> >>- Discussion thread: >>

Re: [VOTE] SPARK 4.0.0-preview1 (RC1)

kproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749) > ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1] > ... 38 more > > Thanks, > Cheng Pan > > > > On May 11, 2024, at 13:55, Wenchen Fan wrote: > > > > Please vote on releasing t

Re: [DISCUSS] clarify the definition of behavior changes

migration guide would have helped. > > > > To summarize: the migration guide was invaluable, we appreciated every > entry, and we'd appreciate Wenchen's stricter definition of "behavior > changes" (especially for silent ones). > > > > *From: *Nimrod Ofek >

Re: Potential Impact of Hive Upgrades on Spark Tables

Yes, Spark has a shim layer to support all Hive versions. It shouldn't be an issue as many users create native Spark data source tables already today, by explicitly putting the `USING` clause in the CREATE TABLE statement. On Wed, May 1, 2024 at 12:56 AM Mich Talebzadeh wrote: > @Wenchen

[DISCUSS] clarify the definition of behavior changes

Hi all, It's exciting to see innovations keep happening in the Spark community and Spark keeps evolving itself. To make these innovations available to more users, it's important to help users upgrade to newer Spark versions easily. We've done a good job on it: the PR template requires the author

Re: [VOTE] SPIP: Structured Logging Framework for Apache Spark

+1 On Mon, Mar 11, 2024 at 5:26 PM Hyukjin Kwon wrote: > +1 > > On Mon, 11 Mar 2024 at 18:11, yangjie01 > wrote: > >> +1 >> >> >> >> Jie Yang >> >> >> >> *发件人**: *Haejoon Lee >> *日期**: *2024年3月11日 星期一 17:09 >> *收件人**: *Gengliang Wang >> *抄送**: *dev >> *主题**: *Re: [VOTE] SPIP: Structured

[VOTE] SPARK 4.0.0-preview1 (RC2)

Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 31 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package

[VOTE] SPARK 4.0.0-preview1 (RC3)

Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 31 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

I agree with the idea of a versionless programming guide. But one thing we need to make sure of is we give clear messages for things that are only available in a new version. My proposal is: 1. keep the old versions' programming guide unchanged. For example, people can still access

[ANNOUNCE] Announcing Apache Spark 4.0.0-preview1

Hi all, To enable wide-scale community testing of the upcoming Spark 4.0 release, the Apache Spark community has posted a preview release of Spark 4.0. This preview is not a stable release in terms of either API or functionality, but it is meant to give the community early access to try the code

Re: [VOTE] SPARK 4.0.0-preview1 (RC2)

ug fix > > > https://github.com/apache/spark/commit/6cd1ccc56321dfa52672cd25f4cfdf2bbc86b3ea > . > > The bug can lead to the unrecoverable job failure. > > > > Thanks, > > Yi > > > > On Tue, May 28, 2024 at 3:45 PM Wenchen Fan wrote: > > > > > Please vote on releasing

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

one correction: "The tag to be voted on is v4.0.0-preview1-rc2 (commit 7cfe5a6e44e8d7079ae29ad3e2cee7231cd3dc66)" should be "The tag to be voted on is v4.0.0-preview1-rc3 (commit 7a7a8bc4bab591ac8b98b2630b38c57adf619b82):" On Tue, May 28, 2024 at 11:48 AM Wenchen Fan w

Re: [DISCUSS] clarify the definition of behavior changes

Hi all, I've created a PR to put the behavior change guideline on the Spark website: https://github.com/apache/spark-website/pull/518 . Please leave comments if you have any, thanks! On Wed, May 15, 2024 at 1:41 AM Wenchen Fan wrote: > Thanks all for the feedback here! Let me put up a

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

se 2) we create an automation to allow > us to backport docs changes to old branches 3) once merged in Spark, the > automation rebuilds all the static sites and creates PRs in spark-website. > The downside is that backport merge conflicts *will *force developers to > backport changes t

Re: [DISCUSS] Versionless Spark Programming Guide Proposal

o work together and find a good path forward here. >>>> >>>> >>>> On Wed, Jun 5, 2024 at 3:22 PM Neil Ramaswamy >>>> wrote: >>>> >>>>> Thanks all for the responses. Let me try to address everything. >>>>> &

Re: 4.0.0-preview1 test report: running on Yarn

Thanks for sharing! Yea Spark 4.0 is built using Java 17. On Tue, Jun 18, 2024 at 5:07 AM George Magiros wrote: > I successfully submitted and ran org.apache.spark.examples.SparkPi on Yarn > using 4.0.0-preview1. However I got it to work only after fixing an issue > with the Yarn nodemanagers

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

The vote passes with 6+1s (4 binding +1s). (* = binding) +1: Wenchen Fan (*) Kent Yao Cheng Pan Xiao Li (*) Gengliang Wang (*) Tathagata Das (*) Thanks all! On Fri, May 31, 2024 at 6:07 PM Tathagata Das wrote: > +1 > - Tested RC3 with Delta Lake. All our Scala and Python test

Re: [VOTE] Move Spark Connect server to builtin package (Client API layer stays external)

+1 On Thu, Jul 4, 2024 at 10:41 AM Gengliang Wang wrote: > +1 > > On Wed, Jul 3, 2024 at 4:48 PM Reynold Xin > wrote: > >> +1 >> >> On Wed, Jul 3, 2024 at 4:45 PM L. C. Hsieh wrote: >> >>> +1 >>> >>> On Wed, Jul 3, 2024 at 3:54 PM Dongjoon Hyun >>> wrote: >>> > >>> > +1 >>> > >>> > Dongjoon

Re: [VOTE] Allow GitHub Actions runs for contributors' PRs without approvals in apache/spark-connect-go

+1 On Tue, Jul 9, 2024 at 10:47 AM Reynold Xin wrote: > +1 > > On Mon, Jul 8, 2024 at 7:44 PM haydn wrote: > >> +1 >> >> On Mon, Jul 8, 2024 at 7:41 PM haydn wrote: >> >>> +1 >>> >>> On Mon, Jul 8, 2024 at 19:41 Takuya UESHIN >>> wrote: >>> +1 On Mon, Jul 8, 2024 at 6:05 PM

Re: [VOTE] Release Spark 3.5.2 (RC1)

> The vote is open until Jul 18 Is it a typo? It's July 18 today. On Thu, Jul 18, 2024 at 6:30 PM Kent Yao wrote: > Hi dev, > > Please vote on releasing the following candidate as Apache Spark version > 3.5.2. > > The vote is open until Jul 18, 11 AM UTC, and passes if a majority +1 > PMC

Re: [DISCUSS] Differentiate Spark without Spark Connect from Spark Connect

Classic SGTM. On Mon, Jul 22, 2024 at 1:12 PM Jungtaek Lim wrote: > I'd propose not to change the name of "Spark Connect" - the name > represents the characteristic of the mode (separation of layer for client > and server). Trying to remove the part of "Connect" would just make > confusion. > >

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

+1 On Wed, Jul 24, 2024 at 10:51 AM Kent Yao wrote: > +1(non-binding), I have checked: > > - Download links are OK > - Signatures, Checksums, and the KEYS file are OK > - LICENSE and NOTICE are present > - No unexpected binary files in source releases > - Successfully built from source > >

Re: [VOTE] Differentiate Spark without Spark Connect from Spark Connect

+1 On Tue, Jul 23, 2024 at 8:40 AM Xinrong Meng wrote: > +1 > > Thank you @Hyukjin Kwon ! > > On Mon, Jul 22, 2024 at 5:20 PM Gengliang Wang wrote: > >> +1 >> >> On Mon, Jul 22, 2024 at 5:19 PM Hyukjin Kwon >> wrote: >> >>> Starting with my own +1. >>> >>> On Tue, 23 Jul 2024 at 09:12,

Re: [VOTE] Release Spark 3.5.2 (RC4)

+1 On Sat, Jul 27, 2024 at 10:03 AM Dongjoon Hyun wrote: > +1 > > Thank you, Kent. > > Dongjoon. > > On Fri, Jul 26, 2024 at 6:37 AM Kent Yao wrote: > >> Hi dev, >> >> Please vote on releasing the following candidate as Apache Spark version >> 3.5.2. >> >> The vote is open until Jul 29,

Re: [外部邮件] [VOTE] Release Spark 3.5.2 (RC2)

I'm changing my vote to -1 as we found a regression that breaks Delta Lake's generated column feature. The fix was merged just now: https://github.com/apache/spark/pull/47483 Can we cut a new RC? On Thu, Jul 25, 2024 at 3:13 PM Mridul Muralidharan wrote: > > +1 > > Signatures, digests, etc

<    1   2   3   4   5   6