(spark) branch master updated (651f68782ab7 -> c7caac9b10ca)

2024-06-04 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 651f68782ab7 [SPARK-48531][INFRA] Fix `Black` target version to Python 3.9 add c7caac9b10ca [SPARK-47972][SQL

[jira] [Assigned] (SPARK-48318) Hash join support for strings with collation (complex types)

2024-06-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48318: --- Assignee: Uroš Bojanić > Hash join support for strings with collation (complex ty

(spark) branch master updated: [SPARK-48318][SQL] Enable hash join support for all collations (complex types)

2024-06-04 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c852c4f72acb [SPARK-48318][SQL] Enable hash

[jira] [Resolved] (SPARK-48318) Hash join support for strings with collation (complex types)

2024-06-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48318. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46722 [https

(spark) branch master updated: [SPARK-47972][SQL] Restrict CAST expression for collations

2024-06-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e4e8bb5936d3 [SPARK-47972][SQL] Restrict CAST

[jira] [Assigned] (SPARK-47972) Restrict CAST expression for collations

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47972: --- Assignee: Mihailo Milosevic > Restrict CAST expression for collati

[jira] [Resolved] (SPARK-47972) Restrict CAST expression for collations

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47972. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46474 [https

[jira] [Resolved] (SPARK-48413) ALTER COLUMN with collation

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48413. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46734 [https

(spark) branch master updated: [SPARK-48413][SQL] ALTER COLUMN with collation

2024-06-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f9542d008402 [SPARK-48413][SQL] ALTER COLUMN

[ANNOUNCE] Announcing Apache Spark 4.0.0-preview1

2024-06-03 Thread Wenchen Fan
Spark 4.0.0-preview1, head over to the download page: https://archive.apache.org/dist/spark/spark-4.0.0-preview1 . It's also available in PyPI, with version name "4.0.0.dev1". Thanks, Wenchen

[ANNOUNCE] Announcing Apache Spark 4.0.0-preview1

2024-06-03 Thread Wenchen Fan
Spark 4.0.0-preview1, head over to the download page: https://archive.apache.org/dist/spark/spark-4.0.0-preview1 . It's also available in PyPI, with version name "4.0.0.dev1". Thanks, Wenchen

[jira] [Assigned] (SPARK-48503) Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48503: --- Assignee: Jack Chen > Scalar subquery with group-by and non-equality predicate incorrec

[jira] [Resolved] (SPARK-48503) Scalar subquery with group-by and non-equality predicate incorrectly allowed, wrong results

2024-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48503. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46839 [https

(spark) branch master updated: [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed

2024-06-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 5d71ef0716f7 [SPARK-48503][SQL] Fix invalid

svn commit: r69509 - in /dev/spark: v4.0.0-preview1-rc1-bin/ v4.0.0-preview1-rc1-docs/ v4.0.0-preview1-rc2-bin/ v4.0.0-preview1-rc2-docs/

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 01:12:24 2024 New Revision: 69509 Log: Removing RC artifacts. Removed: dev/spark/v4.0.0-preview1-rc1-bin/ dev/spark/v4.0.0-preview1-rc1-docs/ dev/spark/v4.0.0-preview1-rc2-bin/ dev/spark/v4.0.0-preview1-rc2-docs

svn commit: r69508 - /release/spark/KEYS

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 00:59:52 2024 New Revision: 69508 Log: Update KEYS Modified: release/spark/KEYS Modified: release/spark/KEYS == --- release/spark/KEYS (original) +++ release/spark/KEYS Mon Jun 3 00

svn commit: r69507 - /dev/spark/v4.0.0-preview1-rc3-bin/ /release/spark/spark-4.0.0-preview1/

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 00:59:50 2024 New Revision: 69507 Log: Apache Spark 4.0.0-preview1 Added: release/spark/spark-4.0.0-preview1/ - copied from r69506, dev/spark/v4.0.0-preview1-rc3-bin/ Removed: dev/spark/v4.0.0-preview1-rc3-bin

svn commit: r69506 - /dev/spark/v4.0.0-preview1-rc3-docs/

2024-06-02 Thread wenchen
Author: wenchen Date: Mon Jun 3 00:59:48 2024 New Revision: 69506 Log: Remove RC artifacts Removed: dev/spark/v4.0.0-preview1-rc3-docs/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional

(spark) tag v4.0.0-preview1 created (now 7a7a8bc4bab5)

2024-06-02 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview1 in repository https://gitbox.apache.org/repos/asf/spark.git at 7a7a8bc4bab5 (commit) No new revisions were added by this update

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

2024-06-02 Thread Wenchen Fan
The vote passes with 6+1s (4 binding +1s). (* = binding) +1: Wenchen Fan (*) Kent Yao Cheng Pan Xiao Li (*) Gengliang Wang (*) Tathagata Das (*) Thanks all! On Fri, May 31, 2024 at 6:07 PM Tathagata Das wrote: > +1 > - Tested RC3 with Delta Lake. All our Scala and Python test

[jira] [Assigned] (SPARK-48391) use addAll instead of add function in TaskMetrics to accelerate

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48391: --- Assignee: jiahong.li > use addAll instead of add function in TaskMetrics to acceler

(spark) branch master updated (96365c86962b -> 3cd35f8cb646)

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 96365c86962b [SPARK-48465][SQL] Avoid no-op empty relation propagation add 3cd35f8cb646 [SPARK-48391][CORE

[jira] [Resolved] (SPARK-48391) use addAll instead of add function in TaskMetrics to accelerate

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48391. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48391][CORE] Using addAll instead of add function in fromAccumulatorInfos method of TaskMetrics Class

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 744b070fa964 [SPARK-48391][CORE] Using

[jira] [Resolved] (SPARK-48465) Avoid no-op empty relation propagation in AQE

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48465. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46814 [https

(spark) branch master updated (844821c82da5 -> 96365c86962b)

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 844821c82da5 [SPARK-47578][R] Migrate RPackageUtils with variables to structured logging framework add

[jira] [Resolved] (SPARK-48430) Fix map value extraction when map contains collated strings

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48430. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46758 [https

(spark) branch master updated (747437c80aa8 -> f083e61925e9)

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 747437c80aa8 [SPARK-48476][SQL] fix NPE error message for null delmiter csv add f083e61925e9 [SPARK-48430][SQL

[jira] [Resolved] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48476. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46810 [https

[jira] [Assigned] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48476: --- Assignee: Milan Stefanovic > NPE thrown when delimiter set to null in

(spark) branch master updated: [SPARK-48476][SQL] fix NPE error message for null delmiter csv

2024-05-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 747437c80aa8 [SPARK-48476][SQL] fix NPE error

[jira] [Assigned] (SPARK-48419) Foldable propagation replace foldable column should use origin column

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48419: --- Assignee: KnightChess > Foldable propagation replace foldable column should use ori

[jira] [Resolved] (SPARK-48419) Foldable propagation replace foldable column should use origin column

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48419. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46742 [https

(spark) branch master updated: [SPARK-48419][SQL] Foldable propagation replace foldable column shoul…

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3e27543128c8 [SPARK-48419][SQL] Foldable

[jira] [Resolved] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48468. - Fix Version/s: 4.0.0 Resolution: Fixed > Add LogicalQueryStage interface in catal

[jira] [Assigned] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48468: --- Assignee: Ziqi Liu > Add LogicalQueryStage interface in catal

(spark) branch master updated: [SPARK-48468] Add LogicalQueryStage interface in catalyst

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6b4f97e1411c [SPARK-48468] Add

[jira] [Resolved] (SPARK-48477) Refactor CollationSuite, CoalesceShufflePartitionsSuite, SQLExecutionSuite

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48477. - Fix Version/s: 4.0.0 Resolution: Fixed > Refactor CollationSu

(spark) branch master updated: [SPARK-48477][SQL][TESTS] Use withSQLConf in tests: Refactor CollationSuite, CoalesceShufflePartitionsSuite, SQLExecutionSuite

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e1f5a7c856ab [SPARK-48477][SQL][TESTS] Use

[jira] [Resolved] (SPARK-48292) Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48292. - Fix Version/s: 4.0.0 Assignee: angerszhu (was: L. C. Hsieh) Resolution: Fixed

[jira] [Assigned] (SPARK-48292) Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48292: --- Assignee: L. C. Hsieh > Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator sho

(spark) branch master updated (69afd4be9c93 -> f68d761c9b21)

2024-05-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 69afd4be9c93 [SPARK-47361][SQL] Derby: Calculate suitable precision and scale for DECIMAL type add f68d761c9b21

(spark) branch branch-3.5 updated: [SPARK-41049][SQL][FOLLOW-UP][3.5] stateful expressions test uses different pretty name

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new c87b6483a3e0 [SPARK-41049][SQL][FOLLOW

[jira] [Resolved] (SPARK-48431) Do not forward predicates on collated columns to file readers

2024-05-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48431. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46760 [https

(spark) branch master updated: [SPARK-48431][SQL] Do not forward predicates on collated columns to file readers

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a3b8420e5eec [SPARK-48431][SQL] Do not forward

(spark) branch branch-3.5 updated: [SPARK-48273][SQL][FOLLOWUP] Explicitly create non-Hive table in identifier-clause.sql

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 043944e1b549 [SPARK-48273][SQL

(spark) branch master updated: [SPARK-48273][SQL][FOLLOWUP] Explicitly create non-Hive table in identifier-clause.sql

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new cf47293b5fc7 [SPARK-48273][SQL][FOLLOWUP

[jira] [Resolved] (SPARK-48462) Refactor HiveQuerySuite.scala and HiveTableScanSuite

2024-05-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48462. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46792 [https

(spark) branch master updated (0461745f1616 -> dc6b493dd1f4)

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 0461745f1616 [SPARK-48281][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr

[jira] [Assigned] (SPARK-48281) Alter string search logic for: instr, substring_index (UTF8_BINARY_LCASE)

2024-05-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48281: --- Assignee: Uroš Bojanić > Alter string search logic for: instr, substring_in

[jira] [Resolved] (SPARK-48281) Alter string search logic for: instr, substring_index (UTF8_BINARY_LCASE)

2024-05-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48281. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46589 [https

(spark) branch master updated: [SPARK-48281][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (StringInStr, SubstringIndex)

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0461745f1616 [SPARK-48281][SQL] Alter string

[jira] [Resolved] (SPARK-48444) Refactor SQLQuerySuite

2024-05-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48444. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46778 [https

(spark) branch master updated: [SPARK-48444][SQL][TESTS] Use withSQLConf in tests: Refactor SQLQuerySuite

2024-05-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 49204b10af58 [SPARK-48444][SQL][TESTS] Use

[jira] [Resolved] (SPARK-48000) Hash join support for strings with collation (StringType only)

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48000. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46599 [https

[jira] [Assigned] (SPARK-48000) Hash join support for strings with collation (StringType only)

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48000: --- Assignee: Uroš Bojanić > Hash join support for strings with collation (StringType o

(spark) branch master updated (a86bca131028 -> e6236af3d08c)

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from a86bca131028 [SPARK-48445][SQL] Don't inline UDFs with expensive children add e6236af3d08c [SPARK-48000][SQL

Re: [DISCUSS] clarify the definition of behavior changes

2024-05-28 Thread Wenchen Fan
Hi all, I've created a PR to put the behavior change guideline on the Spark website: https://github.com/apache/spark-website/pull/518 . Please leave comments if you have any, thanks! On Wed, May 15, 2024 at 1:41 AM Wenchen Fan wrote: > Thanks all for the feedback here! Let me put up a

Re: [VOTE] SPARK 4.0.0-preview1 (RC3)

2024-05-28 Thread Wenchen Fan
one correction: "The tag to be voted on is v4.0.0-preview1-rc2 (commit 7cfe5a6e44e8d7079ae29ad3e2cee7231cd3dc66)" should be "The tag to be voted on is v4.0.0-preview1-rc3 (commit 7a7a8bc4bab591ac8b98b2630b38c57adf619b82):" On Tue, May 28, 2024 at 11:48 AM Wenchen Fan w

[VOTE] SPARK 4.0.0-preview1 (RC3)

2024-05-28 Thread Wenchen Fan
Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 31 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package

svn commit: r69427 - in /dev/spark/v4.0.0-preview1-rc3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/R/articles/ _site/api/R/articles/sparkr-vignettes_files/ _site/api/R/articles/sparkr-vignettes_

2024-05-28 Thread wenchen
Author: wenchen Date: Tue May 28 17:45:42 2024 New Revision: 69427 Log: Apache Spark v4.0.0-preview1-rc3 docs [This commit notification would consist of 4816 parts, which exceeds the limit of 50 ones, so it was shortened to the summary

[jira] [Assigned] (SPARK-48221) Alter string search logic for: startsWith, endsWith, contains, locate (UTF8_BINARY_LCASE)

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48221: --- Assignee: Uroš Bojanić > Alter string search logic for: startsWith, endsWith, conta

[jira] [Resolved] (SPARK-48221) Alter string search logic for: startsWith, endsWith, contains, locate (UTF8_BINARY_LCASE)

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48221. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46511 [https

(spark) branch master updated: [SPARK-48221][SQL] Alter string search logic for UTF8_BINARY_LCASE collation (Contains, StartsWith, EndsWith, StringLocate)

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 249390017ef4 [SPARK-48221][SQL] Alter string

[jira] [Assigned] (SPARK-48273) Late rewrite of PlanWithUnresolvedIdentifier

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48273: --- Assignee: Nikola Mandic > Late rewrite of PlanWithUnresolvedIdentif

[jira] [Resolved] (SPARK-48159) Datetime expressions (all collations)

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48159. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46618 [https

(spark) branch master updated (731a2cfcffae -> e9a3ed857954)

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 731a2cfcffae [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier add e9a3ed857954 [SPARK-48159

[jira] [Resolved] (SPARK-48273) Late rewrite of PlanWithUnresolvedIdentifier

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48273. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 7313d71438e4 [SPARK-48273][SQL] Fix

(spark) branch master updated (7fe1b93884aa -> 731a2cfcffae)

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 7fe1b93884aa [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers add 731a2cfcffae

[jira] [Resolved] (SPARK-46841) Language support for collations

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46841. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46180 [https

[jira] [Assigned] (SPARK-46841) Language support for collations

2024-05-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-46841: --- Assignee: Nikola Mandic > Language support for collati

(spark) branch master updated: [SPARK-46841][SQL] Add collation support for ICU locales and collation specifiers

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7fe1b93884aa [SPARK-46841][SQL] Add collation

svn commit: r69426 - /dev/spark/v4.0.0-preview1-rc3-bin/

2024-05-28 Thread wenchen
Author: wenchen Date: Tue May 28 16:50:57 2024 New Revision: 69426 Log: Apache Spark v4.0.0-preview1-rc3 Added: dev/spark/v4.0.0-preview1-rc3-bin/ dev/spark/v4.0.0-preview1-rc3-bin/SparkR_4.0.0-preview1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc3-bin/SparkR_4.0.0-preview1

(spark) 01/01: Preparing Spark release v4.0.0-preview1-rc3

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to tag v4.0.0-preview1-rc3 in repository https://gitbox.apache.org/repos/asf/spark.git commit 7a7a8bc4bab591ac8b98b2630b38c57adf619b82 Author: Wenchen Fan AuthorDate: Tue May 28 16:23:00 2024 +

(spark) tag v4.0.0-preview1-rc3 created (now 7a7a8bc4bab5)

2024-05-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview1-rc3 in repository https://gitbox.apache.org/repos/asf/spark.git at 7a7a8bc4bab5 (commit) This tag includes the following new commits: new 7a7a8bc4bab5 Preparing Spark

Re: [VOTE] SPARK 4.0.0-preview1 (RC2)

2024-05-28 Thread Wenchen Fan
ug fix > > > https://github.com/apache/spark/commit/6cd1ccc56321dfa52672cd25f4cfdf2bbc86b3ea > . > > The bug can lead to the unrecoverable job failure. > > > > Thanks, > > Yi > > > > On Tue, May 28, 2024 at 3:45 PM Wenchen Fan wrote: > > > > > Please vote on releasing

svn commit: r69425 - /dev/spark/KEYS

2024-05-28 Thread wenchen
Author: wenchen Date: Tue May 28 16:09:45 2024 New Revision: 69425 Log: Update KEYS Modified: dev/spark/KEYS Modified: dev/spark/KEYS == --- dev/spark/KEYS (original) +++ dev/spark/KEYS Tue May 28 16:09:45 2024

[VOTE] SPARK 4.0.0-preview1 (RC2)

2024-05-28 Thread Wenchen Fan
Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 31 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package

svn commit: r69418 - /dev/spark/v4.0.0-preview1-rc2-bin/

2024-05-28 Thread wenchen
Author: wenchen Date: Tue May 28 07:41:08 2024 New Revision: 69418 Log: Apache Spark v4.0.0-preview1-rc2 Added: dev/spark/v4.0.0-preview1-rc2-bin/ dev/spark/v4.0.0-preview1-rc2-bin/SparkR_4.0.0-preview1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc2-bin/SparkR_4.0.0-preview1

svn commit: r69417 - /dev/spark/v4.0.0-preview1-rc2-bin/

2024-05-28 Thread wenchen
Author: wenchen Date: Tue May 28 06:35:58 2024 New Revision: 69417 Log: Deleting Removed: dev/spark/v4.0.0-preview1-rc2-bin/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail

(spark) branch branch-3.5 updated: [SPARK-41049][SQL][FOLLOW-UP] Mark map related expressions as stateful expressions

2024-05-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new f42c029fac5c [SPARK-41049][SQL][FOLLOW

(spark) branch master updated: [SPARK-41049][SQL][FOLLOW-UP] Mark map related expressions as stateful expressions

2024-05-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new af1ac1edc2a9 [SPARK-41049][SQL][FOLLOW-UP] Mark

svn commit: r69416 - in /dev/spark/v4.0.0-preview1-rc2-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/R/articles/ _site/api/R/articles/sparkr-vignettes_files/ _site/api/R/articles/sparkr-vignettes_

2024-05-27 Thread wenchen
Author: wenchen Date: Tue May 28 05:29:59 2024 New Revision: 69416 Log: Apache Spark v4.0.0-preview1-rc2 docs [This commit notification would consist of 4816 parts, which exceeds the limit of 50 ones, so it was shortened to the summary

svn commit: r69415 - /dev/spark/v4.0.0-preview1-rc2-bin/

2024-05-27 Thread wenchen
Author: wenchen Date: Tue May 28 04:31:46 2024 New Revision: 69415 Log: Apache Spark v4.0.0-preview1-rc2 Added: dev/spark/v4.0.0-preview1-rc2-bin/ dev/spark/v4.0.0-preview1-rc2-bin/SparkR_4.0.0-preview1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc2-bin/SparkR_4.0.0-preview1

(spark) branch master updated: [SPARK-48239][INFRA][FOLLOWUP] install the missing `jq` library

2024-05-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 416d7f24fc35 [SPARK-48239][INFRA][FOLLOWUP

(spark) tag v4.0.0-preview1-rc2 created (now 7cfe5a6e44e8)

2024-05-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview1-rc2 in repository https://gitbox.apache.org/repos/asf/spark.git at 7cfe5a6e44e8 (commit) This tag includes the following new commits: new 7cfe5a6e44e8 Preparing Spark

(spark) 01/01: Preparing Spark release v4.0.0-preview1-rc2

2024-05-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to tag v4.0.0-preview1-rc2 in repository https://gitbox.apache.org/repos/asf/spark.git commit 7cfe5a6e44e8d7079ae29ad3e2cee7231cd3dc66 Author: Wenchen Fan AuthorDate: Fri May 24 18:53:15 2024 +

(spark) tag v4.0.0-preview-rc1 deleted (was 9fec87d16a04)

2024-05-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git *** WARNING: tag v4.0.0-preview-rc1 was deleted! *** was 9fec87d16a04 Preparing Spark release v4.0.0-preview

[jira] [Assigned] (SPARK-48364) Type casting for AbstractMapType

2024-05-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48364: --- Assignee: Uroš Bojanić > Type casting for AbstractMapT

[jira] [Resolved] (SPARK-48364) Type casting for AbstractMapType

2024-05-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48364. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46661 [https

(spark) branch master updated: [SPARK-48364][SQL] Add AbstractMapType type casting and fix RaiseError parameter map to work with collated strings

2024-05-22 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6be3560f3c89 [SPARK-48364][SQL] Add

[jira] [Resolved] (SPARK-48215) DateFormatClass (all collations)

2024-05-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48215. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46561 [https

[jira] [Assigned] (SPARK-48215) DateFormatClass (all collations)

2024-05-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48215: --- Assignee: Nebojsa Savic > DateFormatClass (all collati

(spark) branch master updated: [SPARK-48215][SQL] Extending support for collated strings on date_format expression

2024-05-22 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e04d3d7c430a [SPARK-48215][SQL] Extending

(spark) branch master updated: [SPARK-48031] Decompose viewSchemaMode config, add SHOW CREATE TABLE support

2024-05-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 617ac1aec748 [SPARK-48031] Decompose

[jira] [Resolved] (SPARK-48305) CurrentLike - Database/Schema, Catalog, User (all collations)

2024-05-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48305. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46613 [https

(spark) branch master updated: [SPARK-48305][SQL] Add collation support for CurrentLike expressions

2024-05-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6a17b794338b [SPARK-48305][SQL] Add collation

[jira] [Assigned] (SPARK-48175) Store collation information in metadata and not in type for SER/DE

2024-05-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48175: --- Assignee: Stefan Kandic > Store collation information in metadata and not in type for

[jira] [Resolved] (SPARK-48175) Store collation information in metadata and not in type for SER/DE

2024-05-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48175. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46280 [https

  1   2   3   4   5   6   7   8   9   10   >