Re: [DISCUSS] clarify the definition of behavior changes

2024-05-15 Thread Wenchen Fan
t a note in the migration guide would have helped. > > > > To summarize: the migration guide was invaluable, we appreciated every > entry, and we'd appreciate Wenchen's stricter definition of "behavior > changes" (especially for silent ones). > > > >

Re: [VOTE] SPARK 4.0.0-preview1 (RC1)

2024-05-15 Thread Wenchen Fan
.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749) > ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1] > ... 38 more > > Thanks, > Cheng Pan > > > > On May 11, 2024, at 13:55, Wenchen Fan wrote: > > > > Please vote on releas

[jira] [Resolved] (SPARK-48277) Improve error message for ErrorClassesJsonReader.getErrorMessage

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48277. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46584 [https

(spark) branch master updated (8c0a7ba82c98 -> 5e87e9fbd6e6)

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 8c0a7ba82c98 [SPARK-48160][SQL] Add collation support for XPATH expressions add 5e87e9fbd6e6 [SPARK-48277

[jira] [Assigned] (SPARK-48160) XPath expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48160: --- Assignee: Uroš Bojanić > XPath expressions (all collati

[jira] [Resolved] (SPARK-48160) XPath expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48160. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46508 [https

(spark) branch master updated: [SPARK-48160][SQL] Add collation support for XPATH expressions

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8c0a7ba82c98 [SPARK-48160][SQL] Add collation

[jira] [Assigned] (SPARK-48162) Miscellaneous expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48162: --- Assignee: Uroš Bojanić > Miscellaneous expressions (all collati

[jira] [Resolved] (SPARK-48162) Miscellaneous expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48162. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46461 [https

(spark) branch master updated: [SPARK-48162][SQL] Add collation support for MISC expressions

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 723354039f1d [SPARK-48162][SQL] Add collation

[jira] [Updated] (SPARK-48271) Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-48271: Summary: Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER (was: support

[jira] [Resolved] (SPARK-48263) Collate function support for non UTF8_BINARY strings

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48263. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46574 [https

(spark) branch master updated: [SPARK-48263] Collate function support for non UTF8_BINARY strings

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 91da2caa409c [SPARK-48263] Collate function

[jira] [Assigned] (SPARK-48263) Collate function support for non UTF8_BINARY strings

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48263: --- Assignee: Nebojsa Savic > Collate function support for non UTF8_BINARY stri

(spark) branch master updated: [SPARK-47301][SQL][TESTS][FOLLOWUP] Remove workaround for ParquetIOSuite

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 97bf1ee9f6f7 [SPARK-47301][SQL][TESTS][FOLLOWUP

[jira] [Resolved] (SPARK-48172) Fix escaping issues in JDBCDialects

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48172. - Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution: Fixed

(spark) branch branch-3.4 updated: [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new a848e2790cba [SPARK-48172][SQL] Fix

(spark) branch branch-3.5 updated: [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new f37fa436cd4e [SPARK-48172][SQL] Fix

(spark) branch master updated: [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 47006a493f98 [SPARK-48172][SQL] Fix escaping

[jira] [Resolved] (SPARK-48155) PropagateEmpty relation cause LogicalQueryStage only with broadcast without join then execute failed

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48155. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46523 [https

[jira] [Assigned] (SPARK-48155) PropagateEmpty relation cause LogicalQueryStage only with broadcast without join then execute failed

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48155: --- Assignee: angerszhu > PropagateEmpty relation cause LogicalQueryStage only with broadc

(spark) branch master updated: [SPARK-48155][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e5ad5e94a8c8 [SPARK-48155][SQL

[jira] [Created] (SPARK-48271) support char/varchar in RowEncoder

2024-05-14 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48271: --- Summary: support char/varchar in RowEncoder Key: SPARK-48271 URL: https://issues.apache.org/jira/browse/SPARK-48271 Project: Spark Issue Type: Improvement

(spark) branch master updated: [SPARK-46707][SQL][FOLLOWUP] Push down throwable predicate through aggregates

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6766c39b458a [SPARK-46707][SQL][FOLLOWUP] Push

[jira] [Resolved] (SPARK-48157) CSV expressions (all collations)

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48157. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46504 [https

[jira] [Assigned] (SPARK-48157) CSV expressions (all collations)

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48157: --- Assignee: Uroš Bojanić > CSV expressions (all collati

(spark) branch master updated: [SPARK-48157][SQL] Add collation support for CSV expressions

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e6c914f63079 [SPARK-48157][SQL] Add collation

[jira] [Resolved] (SPARK-48229) inputFile expressions (all collations)

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48229. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46503 [https

(spark) branch master updated: [SPARK-48229][SQL] Add collation support for inputFile expressions

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9241b8e8c0df [SPARK-48229][SQL] Add collation

[jira] [Assigned] (SPARK-48265) Infer window group limit batch should do constant folding

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48265: --- Assignee: angerszhu > Infer window group limit batch should do constant fold

[jira] [Resolved] (SPARK-48265) Infer window group limit batch should do constant folding

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48265. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48265][SQL] Infer window group limit batch should do constant folding

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 34588a82239a [SPARK-48265][SQL] Infer

(spark) branch master updated: [SPARK-48265][SQL] Infer window group limit batch should do constant folding

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7974811218c9 [SPARK-48265][SQL] Infer window

(spark) branch master updated: [SPARK-48027][SQL][FOLLOWUP] Add comments for the other code branch

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0ea808880e22 [SPARK-48027][SQL][FOLLOWUP] Add

[jira] [Resolved] (SPARK-48241) CSV parsing failure with char/varchar type columns

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48241. - Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46565 [https

[jira] [Assigned] (SPARK-48241) CSV parsing failure with char/varchar type columns

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48241: --- Assignee: Jiayi Liu > CSV parsing failure with char/varchar type colu

(spark) branch branch-3.5 updated: [SPARK-48241][SQL][3.5] CSV parsing failure with char/varchar type columns

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 19d12b249f0f [SPARK-48241][SQL][3.5

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

2024-05-13 Thread Wenchen Fan
+1 On Tue, May 14, 2024 at 8:19 AM Zhou Jiang wrote: > +1 (non-binding) > > On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: > >> Hi all, >> >> I’d like to start a vote for SPIP: Stored Procedures API for Catalogs. >> >> Please also refer to: >> >>- Discussion thread: >> https://lists.apa

(spark) branch master updated: [SPARK-48241][SQL] CSV parsing failure with char/varchar type columns

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b14abb3a2ed0 [SPARK-48241][SQL] CSV parsing

(spark) branch master updated (42f2132d1fc9 -> 3456d4f69a86)

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 42f2132d1fc9 [SPARK-48206][SQL][TESTS] Add tests for window rewrites with RewriteWithExpression add

[jira] [Resolved] (SPARK-48206) Add tests for window expression rewrites in RewriteWithExpression

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48206. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46492 [https

[jira] [Assigned] (SPARK-48206) Add tests for window expression rewrites in RewriteWithExpression

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48206: --- Assignee: Kelvin Jiang > Add tests for window expression rewrites in RewriteWithExpress

(spark) branch master updated: [SPARK-48206][SQL][TESTS] Add tests for window rewrites with RewriteWithExpression

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 42f2132d1fc9 [SPARK-48206][SQL][TESTS] Add

[jira] [Assigned] (SPARK-48031) Add schema evolution options to views

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48031: --- Assignee: Serge Rielau > Add schema evolution options to vi

[jira] [Resolved] (SPARK-48031) Add schema evolution options to views

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48031. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46267 [https

Re: [DISCUSS] Spark - How to improve our release processes

2024-05-13 Thread Wenchen Fan
Hi Nicholas, Thanks for your help! I'm definitely interested in participating in this unification work. Let me know how I can help. Wenchen On Mon, May 13, 2024 at 1:41 PM Nicholas Chammas wrote: > Re: unification > > We also have a long-standing problem with how we

[jira] [Created] (SPARK-48260) disable output committer coordination in one test of ParquetIOSuite

2024-05-13 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48260: --- Summary: disable output committer coordination in one test of ParquetIOSuite Key: SPARK-48260 URL: https://issues.apache.org/jira/browse/SPARK-48260 Project: Spark

[jira] [Created] (SPARK-48252) Update CommonExpressionRef when necessary

2024-05-13 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48252: --- Summary: Update CommonExpressionRef when necessary Key: SPARK-48252 URL: https://issues.apache.org/jira/browse/SPARK-48252 Project: Spark Issue Type: Bug

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

2024-05-12 Thread Wenchen Fan
+1 On Mon, May 13, 2024 at 10:30 AM Kent Yao wrote: > +1 > > Dongjoon Hyun 于2024年5月13日周一 08:39写道: > > > > +1 > > > > On Sun, May 12, 2024 at 3:50 PM huaxin gao > wrote: > >> > >> +1 > >> > >> On Sat, May 11, 2024 at 4:35 PM L. C. Hsieh wrote: > >>> > >>> +1 > >>> > >>> On Sat, May 11, 2024 at

Re: [DISCUSS] Spark - How to improve our release processes

2024-05-12 Thread Wenchen Fan
se scripts as Github Action jobs, but I think it's more important to do the unification now. Thanks, Wenchen On Fri, May 10, 2024 at 12:34 AM Hussein Awala wrote: > Hello, > > I can answer some of your common questions with other Apache projects. > > > Who currently has p

[VOTE] SPARK 4.0.0-preview1 (RC1)

2024-05-10 Thread Wenchen Fan
Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 16 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package be

svn commit: r69098 - /dev/spark/v4.0.0-preview1-rc1-bin/

2024-05-10 Thread wenchen
Author: wenchen Date: Sat May 11 04:28:26 2024 New Revision: 69098 Log: Apache Spark v4.0.0-preview1-rc1 Added: dev/spark/v4.0.0-preview1-rc1-bin/ dev/spark/v4.0.0-preview1-rc1-bin/SparkR_4.0.0-preview1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc1-bin/SparkR_4.0.0-preview1

svn commit: r69097 - /dev/spark/v4.0.0-preview1-rc1-bin/

2024-05-10 Thread wenchen
Author: wenchen Date: Sat May 11 03:59:33 2024 New Revision: 69097 Log: prepare for re-uploading Removed: dev/spark/v4.0.0-preview1-rc1-bin/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional

svn commit: r69092 - in /dev/spark/v4.0.0-preview1-rc1-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/R/articles/ _site/api/R/articles/sparkr-vignettes_files/ _site/api/R/articles/sparkr-vignettes_

2024-05-10 Thread wenchen
Author: wenchen Date: Fri May 10 16:44:08 2024 New Revision: 69092 Log: Apache Spark v4.0.0-preview1-rc1 docs [This commit notification would consist of 4810 parts, which exceeds the limit of 50 ones, so it was shortened to the summary

(spark) branch master updated: [SPARK-48143][SQL] Use lightweight exceptions for control-flow between UnivocityParser and FailureSafeParser

2024-05-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a6632ffa16f6 [SPARK-48143][SQL] Use lightweight

[jira] [Assigned] (SPARK-48146) Fix error with aggregate function in With child

2024-05-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48146: --- Assignee: Kelvin Jiang > Fix error with aggregate function in With ch

[jira] [Resolved] (SPARK-48146) Fix error with aggregate function in With child

2024-05-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48146. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46443 [https

(spark) branch master updated: [SPARK-48146][SQL] Fix aggregate function in With expression child assertion

2024-05-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7ef0440ef221 [SPARK-48146][SQL] Fix aggregate

[jira] [Resolved] (SPARK-48158) XML expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48158. - Fix Version/s: 4.0.0 Assignee: Uroš Bojanić Resolution: Fixed > XML expressi

(spark) branch master updated (33cac4436e59 -> 2df494fd4e4e)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 33cac4436e59 [SPARK-47847][CORE] Deprecate `spark.network.remoteReadNioBufferConversion` add 2df494fd4e4e

Re: [DISCUSS] SPIP: Stored Procedures API for Catalogs

2024-05-09 Thread Wenchen Fan
Thanks for leading this project! Let's move forward. On Fri, May 10, 2024 at 10:31 AM L. C. Hsieh wrote: > Thanks Anton. Thank you, Wenchen, Dongjoon, Ryan, Serge, Allison and > others if I miss those who are participating in the discussion. > > I suppose we have reached a cons

[jira] [Assigned] (SPARK-48222) Sync Ruby Bundler to 2.4.22 and refresh Gem lock file

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48222: --- Assignee: Nicholas Chammas > Sync Ruby Bundler to 2.4.22 and refresh Gem lock f

[jira] [Resolved] (SPARK-48222) Sync Ruby Bundler to 2.4.22 and refresh Gem lock file

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48222. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46512 [https

(spark) branch master updated: [SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9a2818820f11 [SPARK-48222][INFRA][DOCS] Sync

Re: [DISCUSS] Spark 4.0.0 release

2024-05-09 Thread Wenchen Fan
spark/pull/44628#discussion_r1595718574 I'll continue after the issue is fixed. On Fri, May 10, 2024 at 12:29 AM Dongjoon Hyun wrote: > Please re-try to upload, Wenchen. ASF Infra team bumped up our upload > limit based on our request. > > > Your upload limit has been i

svn commit: r69065 - /dev/spark/v4.0.0-preview1-rc1-bin/

2024-05-09 Thread wenchen
Author: wenchen Date: Thu May 9 16:31:11 2024 New Revision: 69065 Log: Apache Spark v4.0.0-preview1-rc1 Added: dev/spark/v4.0.0-preview1-rc1-bin/ dev/spark/v4.0.0-preview1-rc1-bin/pyspark-4.0.0.dev1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc1-bin/pyspark-4.0.0.dev1

Re: [DISCUSS] Spark 4.0.0 release

2024-05-09 Thread Wenchen Fan
: > >> Could you file an INFRA JIRA issue with the error message and context >> first, Wenchen? >> >> As you know, if we see something, we had better file a JIRA issue because >> it could be not only an Apache Spark project issue but also all ASF project >> issue

[jira] [Assigned] (SPARK-47409) StringTrim & StringTrimLeft/Right/Both (binary & lowercase collation only)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47409: --- Assignee: David Milicevic > StringTrim & StringTrimLeft/Right/Both (binary &

[jira] [Resolved] (SPARK-47409) StringTrim & StringTrimLeft/Right/Both (binary & lowercase collation only)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47409. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46206 [https

(spark) branch master updated: [SPARK-47409][SQL] Add support for collation for StringTrim type of functions/expressions (for UTF8_BINARY & LCASE)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 21333f8c1fc0 [SPARK-47409][SQL] Add support for

Re: [DISCUSS] Spark - How to improve our release processes

2024-05-09 Thread Wenchen Fan
Thanks for starting the discussion! To add a bit more color, we should at least add a test job to make sure the release script can produce the packages correctly. Today it's kind of being manually tested by the release manager each time, which slows down the release process. It's better if we can a

(spark) branch master updated: [SPARK-47803][FOLLOWUP] Check nulls when casting nested type to variant

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3fd38d4c07f6 [SPARK-47803][FOLLOWUP] Check

[jira] [Resolved] (SPARK-47421) URL expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47421. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46460 [https

[jira] [Assigned] (SPARK-47421) URL expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47421: --- Assignee: Uroš Bojanić > URL expressions (all collati

(spark) branch master updated (045ec6a166c8 -> 34ee0d8414b2)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 045ec6a166c8 [SPARK-48208][SS] Skip providing memory usage metrics from RocksDB if bounded memory usage is enabled

[jira] [Resolved] (SPARK-47354) Variant expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47354. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46424 [https

[jira] [Assigned] (SPARK-47354) Variant expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47354: --- Assignee: Uroš Bojanić > Variant expressions (all collati

(spark) branch master updated (a4ab82b8f340 -> 91da4ac25148)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from a4ab82b8f340 [SPARK-48186][SQL] Add support for AbstractMapType add 91da4ac25148 [SPARK-47354][SQL] Add

[jira] [Resolved] (SPARK-48186) Add support for AbstractMapType

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48186. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46458 [https

[jira] [Assigned] (SPARK-48186) Add support for AbstractMapType

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48186: --- Assignee: Uroš Bojanić > Add support for AbstractMapT

(spark) branch master updated (6cc3dc2ef4d2 -> a4ab82b8f340)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 6cc3dc2ef4d2 [SPARK-48169][SPARK-48143][SQL] Revert BadRecordException optimizations add a4ab82b8f340 [SPARK

(spark) branch master updated: [SPARK-48169][SPARK-48143][SQL] Revert BadRecordException optimizations

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6cc3dc2ef4d2 [SPARK-48169][SPARK-48143][SQL

Re: [DISCUSS] Spark 4.0.0 release

2024-05-09 Thread Wenchen Fan
gt;> Nimrod >>>>> >>>>> >>>>> בתאריך יום ד׳, 8 במאי 2024, 00:54, מאת Holden Karau ‏< >>>>> holden.ka...@gmail.com>: >>>>> >>>>>> Indeed. We could conceivably build the release in CI/CD

[jira] [Resolved] (SPARK-48197) avoid assert error for invalid lambda function

2024-05-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48197. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48197][SQL] Avoid assert error for invalid lambda function

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 541e1c4da131 [SPARK-48197][SQL] Avoid

(spark) branch master updated (337f980f0073 -> 7e79e91dc8c5)

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 337f980f0073 [SPARK-48204][INFRA] Fix release script for Spark 4.0+ add 7e79e91dc8c5 [SPARK-48197][SQL] Avoid

(spark) tag v4.0.0-preview1-rc1 created (now 7dcf77c739c3)

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview1-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git at 7dcf77c739c3 (commit) This tag includes the following new commits: new 7dcf77c739c3 Preparing Spark

(spark) 01/01: Preparing Spark release v4.0.0-preview1-rc1

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to tag v4.0.0-preview1-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git commit 7dcf77c739c3854260464d732dbfb9a0f54706e7 Author: Wenchen Fan AuthorDate: Thu May 9 02:32:06 2024 +

[jira] [Created] (SPARK-48204) fix release script for Spark 4.0+

2024-05-08 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48204: --- Summary: fix release script for Spark 4.0+ Key: SPARK-48204 URL: https://issues.apache.org/jira/browse/SPARK-48204 Project: Spark Issue Type: Bug

(spark) tag v4.0.0-preview-rc1 created (now 9fec87d16a04)

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to tag v4.0.0-preview-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git at 9fec87d16a04 (commit) This tag includes the following new commits: new 9fec87d16a04 Preparing Spark

(spark) 01/01: Preparing Spark release v4.0.0-preview-rc1

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to tag v4.0.0-preview-rc1 in repository https://gitbox.apache.org/repos/asf/spark.git commit 9fec87d16a0418759d835541557ad22f20940e9e Author: Wenchen Fan AuthorDate: Wed May 8 14:16:23 2024 +

[jira] [Resolved] (SPARK-48161) JSON expressions (all collations)

2024-05-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48161. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46462 [https

(spark) branch master updated (8950add773e6 -> 8d7081639ab4)

2024-05-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 8950add773e6 [SPARK-48188][SQL] Consistently use normalized plan for cache add 8d7081639ab4 [SPARK-48161][SQL

[jira] [Created] (SPARK-48188) Consistently use normalized plan for cache

2024-05-07 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48188: --- Summary: Consistently use normalized plan for cache Key: SPARK-48188 URL: https://issues.apache.org/jira/browse/SPARK-48188 Project: Spark Issue Type: Bug

Re: [DISCUSS] Spark 4.0.0 release

2024-05-07 Thread Wenchen Fan
UPDATE: Unfortunately, it took me quite some time to set up my laptop and get it ready for the release process (docker desktop doesn't work anymore, my pgp key is lost, etc.). I'll start the RC process at my tomorrow. Thanks for your patience! Wenchen On Fri, May 3, 2024 at 7:47 AM

svn commit: r69013 - /dev/spark/KEYS

2024-05-07 Thread wenchen
Author: wenchen Date: Tue May 7 17:07:43 2024 New Revision: 69013 Log: Update KEYS Modified: dev/spark/KEYS Modified: dev/spark/KEYS == --- dev/spark/KEYS (original) +++ dev/spark/KEYS Tue May 7 17:07:43 2024

[jira] [Assigned] (SPARK-47297) Format expressions (all collations)

2024-05-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47297: --- Assignee: Uroš Bojanić > Format expressions (all collati

[jira] [Resolved] (SPARK-47297) Format expressions (all collations)

2024-05-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47297. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46423 [https

(spark) branch master updated: [SPARK-47297][SQL] Add collation support for format expressions

2024-05-07 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 148f5335427c [SPARK-47297][SQL] Add collation

[jira] [Created] (SPARK-48173) CheckAnalsis should see the entire query plan

2024-05-07 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48173: --- Summary: CheckAnalsis should see the entire query plan Key: SPARK-48173 URL: https://issues.apache.org/jira/browse/SPARK-48173 Project: Spark Issue Type: Bug

<    3   4   5   6   7   8   9   10   11   12   >