[jira] [Assigned] (SPARK-48175) Store collation information in metadata and not in type for SER/DE

2024-05-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48175: --- Assignee: Stefan Kandic > Store collation information in metadata and not in type for

[jira] [Resolved] (SPARK-48175) Store collation information in metadata and not in type for SER/DE

2024-05-18 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48175. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46280 [https

(spark) branch master updated: [SPARK-48175][SQL][PYTHON] Store collation information in metadata and not in type for SER/DE

2024-05-18 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6f6b4860268d [SPARK-48175][SQL][PYTHON] Store

(spark) branch master updated (15fb4787354a -> 3edd6c7e1d50)

2024-05-17 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 15fb4787354a [SPARK-48321][CONNECT][TESTS] Avoid using deprecated methods in dsl add 3edd6c7e1d50 [SPARK-48312

[jira] [Resolved] (SPARK-48308) Unify getting data schema without partition columns in FileSourceStrategy

2024-05-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48308. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46619 [https

[jira] [Assigned] (SPARK-48308) Unify getting data schema without partition columns in FileSourceStrategy

2024-05-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48308: --- Assignee: Johan Lasperas > Unify getting data schema without partition colu

(spark) branch master updated: [SPARK-48308][CORE] Unify getting data schema without partition columns in FileSourceStrategy

2024-05-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 57948c865e06 [SPARK-48308][CORE] Unify getting

[jira] [Resolved] (SPARK-48288) Add source data type to connector.Cast expression

2024-05-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48288. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46596 [https

(spark) branch master updated (fa83d0f8fce7 -> 4be0828e6e6a)

2024-05-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from fa83d0f8fce7 [SPARK-48296][SQL] Codegen Support for `to_xml` add 4be0828e6e6a [SPARK-48288] Add source data type

[jira] [Resolved] (SPARK-48252) Update CommonExpressionRef when necessary

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48252. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46552 [https

[jira] [Assigned] (SPARK-48252) Update CommonExpressionRef when necessary

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48252: --- Assignee: Wenchen Fan > Update CommonExpressionRef when necess

(spark) branch master updated: [SPARK-48252][SQL] Update CommonExpressionRef when necessary

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ca3593288d57 [SPARK-48252][SQL] Update

(spark) branch branch-3.4 updated: [SPARK-48172][SQL] Fix escaping issues in JDBCDialects

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 0e7156d2d801 [SPARK-48172][SQL] Fix

[jira] [Resolved] (SPARK-48172) Fix escaping issues in JDBCDialects

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48172. - Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution: Fixed

(spark) branch branch-3.5 updated: [SPARK-48172][SQL] Fix escaping issues in JDBCDialects

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 210ed2521d3d [SPARK-48172][SQL] Fix

(spark) branch master updated: [SPARK-48172][SQL] Fix escaping issues in JDBCDialects

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9e386b472981 [SPARK-48172][SQL] Fix escaping

Re: [DISCUSS] clarify the definition of behavior changes

2024-05-15 Thread Wenchen Fan
migration guide would have helped. > > > > To summarize: the migration guide was invaluable, we appreciated every > entry, and we'd appreciate Wenchen's stricter definition of "behavior > changes" (especially for silent ones). > > > > *From: *Nimrod Ofek >

Re: [VOTE] SPARK 4.0.0-preview1 (RC1)

2024-05-15 Thread Wenchen Fan
kproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:749) > ~[spark-core_2.13-4.0.0-preview1.jar:4.0.0-preview1] > ... 38 more > > Thanks, > Cheng Pan > > > > On May 11, 2024, at 13:55, Wenchen Fan wrote: > > > > Please vote on releasing t

[jira] [Resolved] (SPARK-48277) Improve error message for ErrorClassesJsonReader.getErrorMessage

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48277. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46584 [https

(spark) branch master updated (8c0a7ba82c98 -> 5e87e9fbd6e6)

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 8c0a7ba82c98 [SPARK-48160][SQL] Add collation support for XPATH expressions add 5e87e9fbd6e6 [SPARK-48277

[jira] [Assigned] (SPARK-48160) XPath expressions (all collations)

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48160: --- Assignee: Uroš Bojanić > XPath expressions (all collati

[jira] [Resolved] (SPARK-48160) XPath expressions (all collations)

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48160. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46508 [https

(spark) branch master updated: [SPARK-48160][SQL] Add collation support for XPATH expressions

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8c0a7ba82c98 [SPARK-48160][SQL] Add collation

[jira] [Assigned] (SPARK-48162) Miscellaneous expressions (all collations)

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48162: --- Assignee: Uroš Bojanić > Miscellaneous expressions (all collati

[jira] [Resolved] (SPARK-48162) Miscellaneous expressions (all collations)

2024-05-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48162. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46461 [https

(spark) branch master updated: [SPARK-48162][SQL] Add collation support for MISC expressions

2024-05-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 723354039f1d [SPARK-48162][SQL] Add collation

[jira] [Updated] (SPARK-48271) Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-48271: Summary: Turn match error in RowEncoder into UNSUPPORTED_DATA_TYPE_FOR_ENCODER (was: support

[jira] [Resolved] (SPARK-48263) Collate function support for non UTF8_BINARY strings

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48263. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46574 [https

(spark) branch master updated: [SPARK-48263] Collate function support for non UTF8_BINARY strings

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 91da2caa409c [SPARK-48263] Collate function

[jira] [Assigned] (SPARK-48263) Collate function support for non UTF8_BINARY strings

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48263: --- Assignee: Nebojsa Savic > Collate function support for non UTF8_BINARY stri

(spark) branch master updated: [SPARK-47301][SQL][TESTS][FOLLOWUP] Remove workaround for ParquetIOSuite

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 97bf1ee9f6f7 [SPARK-47301][SQL][TESTS][FOLLOWUP

[jira] [Resolved] (SPARK-48172) Fix escaping issues in JDBCDialects

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48172. - Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution: Fixed

(spark) branch branch-3.4 updated: [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new a848e2790cba [SPARK-48172][SQL] Fix

(spark) branch branch-3.5 updated: [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new f37fa436cd4e [SPARK-48172][SQL] Fix

(spark) branch master updated: [SPARK-48172][SQL] Fix escaping issues in JDBC Dialects

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 47006a493f98 [SPARK-48172][SQL] Fix escaping

[jira] [Resolved] (SPARK-48155) PropagateEmpty relation cause LogicalQueryStage only with broadcast without join then execute failed

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48155. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46523 [https

[jira] [Assigned] (SPARK-48155) PropagateEmpty relation cause LogicalQueryStage only with broadcast without join then execute failed

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48155: --- Assignee: angerszhu > PropagateEmpty relation cause LogicalQueryStage only with broadc

(spark) branch master updated: [SPARK-48155][SQL] AQEPropagateEmptyRelation for join should check if remain child is just BroadcastQueryStageExec

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e5ad5e94a8c8 [SPARK-48155][SQL

[jira] [Created] (SPARK-48271) support char/varchar in RowEncoder

2024-05-14 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48271: --- Summary: support char/varchar in RowEncoder Key: SPARK-48271 URL: https://issues.apache.org/jira/browse/SPARK-48271 Project: Spark Issue Type: Improvement

(spark) branch master updated: [SPARK-46707][SQL][FOLLOWUP] Push down throwable predicate through aggregates

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6766c39b458a [SPARK-46707][SQL][FOLLOWUP] Push

[jira] [Resolved] (SPARK-48157) CSV expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48157. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46504 [https

[jira] [Assigned] (SPARK-48157) CSV expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48157: --- Assignee: Uroš Bojanić > CSV expressions (all collati

(spark) branch master updated: [SPARK-48157][SQL] Add collation support for CSV expressions

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e6c914f63079 [SPARK-48157][SQL] Add collation

[jira] [Resolved] (SPARK-48229) inputFile expressions (all collations)

2024-05-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48229. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46503 [https

(spark) branch master updated: [SPARK-48229][SQL] Add collation support for inputFile expressions

2024-05-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9241b8e8c0df [SPARK-48229][SQL] Add collation

[jira] [Assigned] (SPARK-48265) Infer window group limit batch should do constant folding

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48265: --- Assignee: angerszhu > Infer window group limit batch should do constant fold

[jira] [Resolved] (SPARK-48265) Infer window group limit batch should do constant folding

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48265. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-48265][SQL] Infer window group limit batch should do constant folding

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 34588a82239a [SPARK-48265][SQL] Infer

(spark) branch master updated: [SPARK-48265][SQL] Infer window group limit batch should do constant folding

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7974811218c9 [SPARK-48265][SQL] Infer window

(spark) branch master updated: [SPARK-48027][SQL][FOLLOWUP] Add comments for the other code branch

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0ea808880e22 [SPARK-48027][SQL][FOLLOWUP] Add

[jira] [Resolved] (SPARK-48241) CSV parsing failure with char/varchar type columns

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48241. - Fix Version/s: 3.5.2 Resolution: Fixed Issue resolved by pull request 46565 [https

[jira] [Assigned] (SPARK-48241) CSV parsing failure with char/varchar type columns

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48241: --- Assignee: Jiayi Liu > CSV parsing failure with char/varchar type colu

(spark) branch branch-3.5 updated: [SPARK-48241][SQL][3.5] CSV parsing failure with char/varchar type columns

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 19d12b249f0f [SPARK-48241][SQL][3.5

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

2024-05-13 Thread Wenchen Fan
+1 On Tue, May 14, 2024 at 8:19 AM Zhou Jiang wrote: > +1 (non-binding) > > On Sat, May 11, 2024 at 2:10 PM L. C. Hsieh wrote: > >> Hi all, >> >> I’d like to start a vote for SPIP: Stored Procedures API for Catalogs. >> >> Please also refer to: >> >>- Discussion thread: >>

(spark) branch master updated: [SPARK-48241][SQL] CSV parsing failure with char/varchar type columns

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b14abb3a2ed0 [SPARK-48241][SQL] CSV parsing

(spark) branch master updated (42f2132d1fc9 -> 3456d4f69a86)

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 42f2132d1fc9 [SPARK-48206][SQL][TESTS] Add tests for window rewrites with RewriteWithExpression add

[jira] [Resolved] (SPARK-48206) Add tests for window expression rewrites in RewriteWithExpression

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48206. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46492 [https

[jira] [Assigned] (SPARK-48206) Add tests for window expression rewrites in RewriteWithExpression

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48206: --- Assignee: Kelvin Jiang > Add tests for window expression rewrites in RewriteWithExpress

(spark) branch master updated: [SPARK-48206][SQL][TESTS] Add tests for window rewrites with RewriteWithExpression

2024-05-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 42f2132d1fc9 [SPARK-48206][SQL][TESTS] Add

[jira] [Assigned] (SPARK-48031) Add schema evolution options to views

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48031: --- Assignee: Serge Rielau > Add schema evolution options to vi

[jira] [Resolved] (SPARK-48031) Add schema evolution options to views

2024-05-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48031. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46267 [https

Re: [DISCUSS] Spark - How to improve our release processes

2024-05-13 Thread Wenchen Fan
Hi Nicholas, Thanks for your help! I'm definitely interested in participating in this unification work. Let me know how I can help. Wenchen On Mon, May 13, 2024 at 1:41 PM Nicholas Chammas wrote: > Re: unification > > We also have a long-standing problem with how we mana

[jira] [Created] (SPARK-48260) disable output committer coordination in one test of ParquetIOSuite

2024-05-13 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48260: --- Summary: disable output committer coordination in one test of ParquetIOSuite Key: SPARK-48260 URL: https://issues.apache.org/jira/browse/SPARK-48260 Project: Spark

[jira] [Created] (SPARK-48252) Update CommonExpressionRef when necessary

2024-05-13 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-48252: --- Summary: Update CommonExpressionRef when necessary Key: SPARK-48252 URL: https://issues.apache.org/jira/browse/SPARK-48252 Project: Spark Issue Type: Bug

Re: [VOTE] SPIP: Stored Procedures API for Catalogs

2024-05-12 Thread Wenchen Fan
+1 On Mon, May 13, 2024 at 10:30 AM Kent Yao wrote: > +1 > > Dongjoon Hyun 于2024年5月13日周一 08:39写道: > > > > +1 > > > > On Sun, May 12, 2024 at 3:50 PM huaxin gao > wrote: > >> > >> +1 > >> > >> On Sat, May 11, 2024 at 4:35 PM L. C. Hsieh wrote: > >>> > >>> +1 > >>> > >>> On Sat, May 11, 2024

Re: [DISCUSS] Spark - How to improve our release processes

2024-05-12 Thread Wenchen Fan
as Github Action jobs, but I think it's more important to do the unification now. Thanks, Wenchen On Fri, May 10, 2024 at 12:34 AM Hussein Awala wrote: > Hello, > > I can answer some of your common questions with other Apache projects. > > > Who currently has permissions

[VOTE] SPARK 4.0.0-preview1 (RC1)

2024-05-10 Thread Wenchen Fan
Please vote on releasing the following candidate as Apache Spark version 4.0.0-preview1. The vote is open until May 16 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 4.0.0-preview1 [ ] -1 Do not release this package

svn commit: r69098 - /dev/spark/v4.0.0-preview1-rc1-bin/

2024-05-10 Thread wenchen
Author: wenchen Date: Sat May 11 04:28:26 2024 New Revision: 69098 Log: Apache Spark v4.0.0-preview1-rc1 Added: dev/spark/v4.0.0-preview1-rc1-bin/ dev/spark/v4.0.0-preview1-rc1-bin/SparkR_4.0.0-preview1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc1-bin/SparkR_4.0.0-preview1

svn commit: r69097 - /dev/spark/v4.0.0-preview1-rc1-bin/

2024-05-10 Thread wenchen
Author: wenchen Date: Sat May 11 03:59:33 2024 New Revision: 69097 Log: prepare for re-uploading Removed: dev/spark/v4.0.0-preview1-rc1-bin/ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional

svn commit: r69092 - in /dev/spark/v4.0.0-preview1-rc1-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/R/articles/ _site/api/R/articles/sparkr-vignettes_files/ _site/api/R/articles/sparkr-vignettes_

2024-05-10 Thread wenchen
Author: wenchen Date: Fri May 10 16:44:08 2024 New Revision: 69092 Log: Apache Spark v4.0.0-preview1-rc1 docs [This commit notification would consist of 4810 parts, which exceeds the limit of 50 ones, so it was shortened to the summary

(spark) branch master updated: [SPARK-48143][SQL] Use lightweight exceptions for control-flow between UnivocityParser and FailureSafeParser

2024-05-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a6632ffa16f6 [SPARK-48143][SQL] Use lightweight

[jira] [Assigned] (SPARK-48146) Fix error with aggregate function in With child

2024-05-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48146: --- Assignee: Kelvin Jiang > Fix error with aggregate function in With ch

[jira] [Resolved] (SPARK-48146) Fix error with aggregate function in With child

2024-05-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48146. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46443 [https

(spark) branch master updated: [SPARK-48146][SQL] Fix aggregate function in With expression child assertion

2024-05-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7ef0440ef221 [SPARK-48146][SQL] Fix aggregate

[jira] [Resolved] (SPARK-48158) XML expressions (all collations)

2024-05-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48158. - Fix Version/s: 4.0.0 Assignee: Uroš Bojanić Resolution: Fixed > XML expressi

(spark) branch master updated (33cac4436e59 -> 2df494fd4e4e)

2024-05-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 33cac4436e59 [SPARK-47847][CORE] Deprecate `spark.network.remoteReadNioBufferConversion` add 2df494fd4e4e

Re: [DISCUSS] SPIP: Stored Procedures API for Catalogs

2024-05-09 Thread Wenchen Fan
Thanks for leading this project! Let's move forward. On Fri, May 10, 2024 at 10:31 AM L. C. Hsieh wrote: > Thanks Anton. Thank you, Wenchen, Dongjoon, Ryan, Serge, Allison and > others if I miss those who are participating in the discussion. > > I suppose we have reached a consen

[jira] [Assigned] (SPARK-48222) Sync Ruby Bundler to 2.4.22 and refresh Gem lock file

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48222: --- Assignee: Nicholas Chammas > Sync Ruby Bundler to 2.4.22 and refresh Gem lock f

[jira] [Resolved] (SPARK-48222) Sync Ruby Bundler to 2.4.22 and refresh Gem lock file

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48222. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46512 [https

(spark) branch master updated: [SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9a2818820f11 [SPARK-48222][INFRA][DOCS] Sync

Re: [DISCUSS] Spark 4.0.0 release

2024-05-09 Thread Wenchen Fan
/44628#discussion_r1595718574 I'll continue after the issue is fixed. On Fri, May 10, 2024 at 12:29 AM Dongjoon Hyun wrote: > Please re-try to upload, Wenchen. ASF Infra team bumped up our upload > limit based on our request. > > > Your upload limit has been increased to 650

svn commit: r69065 - /dev/spark/v4.0.0-preview1-rc1-bin/

2024-05-09 Thread wenchen
Author: wenchen Date: Thu May 9 16:31:11 2024 New Revision: 69065 Log: Apache Spark v4.0.0-preview1-rc1 Added: dev/spark/v4.0.0-preview1-rc1-bin/ dev/spark/v4.0.0-preview1-rc1-bin/pyspark-4.0.0.dev1.tar.gz (with props) dev/spark/v4.0.0-preview1-rc1-bin/pyspark-4.0.0.dev1

Re: [DISCUSS] Spark 4.0.0 release

2024-05-09 Thread Wenchen Fan
;> Could you file an INFRA JIRA issue with the error message and context >> first, Wenchen? >> >> As you know, if we see something, we had better file a JIRA issue because >> it could be not only an Apache Spark project issue but also all ASF project >> issues. >>

[jira] [Assigned] (SPARK-47409) StringTrim & StringTrimLeft/Right/Both (binary & lowercase collation only)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47409: --- Assignee: David Milicevic > StringTrim & StringTrimLeft/Right/Both (binary &

[jira] [Resolved] (SPARK-47409) StringTrim & StringTrimLeft/Right/Both (binary & lowercase collation only)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47409. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46206 [https

(spark) branch master updated: [SPARK-47409][SQL] Add support for collation for StringTrim type of functions/expressions (for UTF8_BINARY & LCASE)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 21333f8c1fc0 [SPARK-47409][SQL] Add support

Re: [DISCUSS] Spark - How to improve our release processes

2024-05-09 Thread Wenchen Fan
Thanks for starting the discussion! To add a bit more color, we should at least add a test job to make sure the release script can produce the packages correctly. Today it's kind of being manually tested by the release manager each time, which slows down the release process. It's better if we can

(spark) branch master updated: [SPARK-47803][FOLLOWUP] Check nulls when casting nested type to variant

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3fd38d4c07f6 [SPARK-47803][FOLLOWUP] Check

[jira] [Resolved] (SPARK-47421) URL expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47421. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46460 [https

[jira] [Assigned] (SPARK-47421) URL expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47421: --- Assignee: Uroš Bojanić > URL expressions (all collati

(spark) branch master updated (045ec6a166c8 -> 34ee0d8414b2)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 045ec6a166c8 [SPARK-48208][SS] Skip providing memory usage metrics from RocksDB if bounded memory usage is enabled

[jira] [Resolved] (SPARK-47354) Variant expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-47354. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46424 [https

[jira] [Assigned] (SPARK-47354) Variant expressions (all collations)

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-47354: --- Assignee: Uroš Bojanić > Variant expressions (all collati

(spark) branch master updated (a4ab82b8f340 -> 91da4ac25148)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from a4ab82b8f340 [SPARK-48186][SQL] Add support for AbstractMapType add 91da4ac25148 [SPARK-47354][SQL] Add

[jira] [Resolved] (SPARK-48186) Add support for AbstractMapType

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48186. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46458 [https

[jira] [Assigned] (SPARK-48186) Add support for AbstractMapType

2024-05-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48186: --- Assignee: Uroš Bojanić > Add support for AbstractMapT

(spark) branch master updated (6cc3dc2ef4d2 -> a4ab82b8f340)

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 6cc3dc2ef4d2 [SPARK-48169][SPARK-48143][SQL] Revert BadRecordException optimizations add a4ab82b8f340 [SPARK

(spark) branch master updated: [SPARK-48169][SPARK-48143][SQL] Revert BadRecordException optimizations

2024-05-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6cc3dc2ef4d2 [SPARK-48169][SPARK-48143][SQL

Re: [DISCUSS] Spark 4.0.0 release

2024-05-09 Thread Wenchen Fan
>>>>> >>>>> בתאריך יום ד׳, 8 במאי 2024, 00:54, מאת Holden Karau ‏< >>>>> holden.ka...@gmail.com>: >>>>> >>>>>> Indeed. We could conceivably build the release in CI/CD but the final >>>>>&

[jira] [Resolved] (SPARK-48197) avoid assert error for invalid lambda function

2024-05-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48197. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

  1   2   3   4   5   6   7   8   9   10   >