Re: [VOTE] Officialy Deprecate GraphX in Spark 4

2024-10-03 Thread Wenchen Fan
+1 On Tue, Oct 1, 2024 at 4:20 PM beliefer wrote: > +1. > > I didn't hear users need it. > > > At 2024-10-01 02:01:17, "Holden Karau" wrote: > > I think it has been de-facto deprecated, we haven’t updated it > meaningfully in several years. I think removing the API would be excessive > but depr

Re: [VOTE] Single-pass Analyzer for Catalyst

2024-10-03 Thread Wenchen Fan
+1 On Wed, Oct 2, 2024 at 7:50 AM Peter Toth wrote: > +1 > > On Tue, Oct 1, 2024, 08:33 Yang Jie wrote: > >> +1, Thanks >> >> Jie Yang >> >> On 2024/10/01 03:26:40 John Zhuge wrote: >> > +1 (non-binding) >> > >> > On Mon, Sep 30, 2024 at 7:42 PM Gengliang Wang >> > wrote: >> > >> > > +1 >> > >

[jira] [Assigned] (SPARK-49743) OptimizeCsvJsonExpr should not change the schema of underlying StructType in GetArrayStructFields

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49743: --- Assignee: Nikhil Sheoran > OptimizeCsvJsonExpr should not change the schema of underly

[jira] [Resolved] (SPARK-49743) OptimizeCsvJsonExpr should not change the schema of underlying StructType in GetArrayStructFields

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49743. - Fix Version/s: 3.5.4 Resolution: Fixed Issue resolved by pull request 48308 [https

(spark) branch branch-3.5 updated: [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 75860a3cc50f [SPARK-49743][SQL

[jira] [Resolved] (SPARK-48196) Make lazy val plans in QueryExecution Trys to not re-execute on error

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48196. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48211 [https

[jira] [Assigned] (SPARK-48196) Make lazy val plans in QueryExecution Trys to not re-execute on error

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48196: --- Assignee: Juliusz Sompolski > Make lazy val plans in QueryExecution Trys to not re-exec

(spark) branch master updated: [SPARK-48196][SQL] Turn QueryExecution lazy val plans into LazyTry

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a7fa2700e0f0 [SPARK-48196][SQL] Turn

(spark) branch master updated (c54c017e9309 -> 97ae372634b1)

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from c54c017e9309 [SPARK-49666][SQL] Add feature flag for trim collation feature add 97ae372634b1 [SPARK-49819

(spark) branch master updated (dbfa909422ad -> 3065dd92ab8f)

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from dbfa909422ad [SPARK-49816][SQL] Should only update out-going-ref-count for referenced outer CTE relation add

(spark) branch master updated (885c3fac7246 -> d85e7bc0beb4)

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 885c3fac7246 [SPARK-49823][SS] Avoid flush during shutdown in rocksdb close path add d85e7bc0beb4 [SPARK-49811

(spark) branch branch-3.5 updated (a49d6f42f598 -> 16b8153063b2)

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git from a49d6f42f598 [SPARK-47949][SQL][DOCKER][TESTS] MsSQLServer: Bump up mssql docker image version to 2022-CU12-GDR1

(spark) branch master updated (97ae372634b1 -> dbfa909422ad)

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 97ae372634b1 [SPARK-49819] Disable CollapseProject for correlated subqueries in projection over aggregate correctly

(spark) branch master updated (d85e7bc0beb4 -> c54c017e9309)

2024-09-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from d85e7bc0beb4 [SPARK-49811][SQL] Rename StringTypeAnyCollation add c54c017e9309 [SPARK-49666][SQL] Add feature

[jira] [Resolved] (SPARK-49561) PIVOT + UNPIVOT operators

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49561. - Fix Version/s: 4.0.0 Resolution: Fixed > PIVOT + UNPIVOT operat

[jira] [Assigned] (SPARK-49561) PIVOT + UNPIVOT operators

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49561: --- Assignee: Daniel > PIVOT + UNPIVOT operat

[jira] [Resolved] (SPARK-49816) should only update out-going-ref-count for referenced outer CTE relation

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49816. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48284 [https

[jira] [Resolved] (SPARK-49819) Disable CollapseProject for correlated subqueries in projection over aggregate correctly

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49819. - Fix Version/s: 3.5.4 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-49819) Disable CollapseProject for correlated subqueries in projection over aggregate correctly

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49819: --- Assignee: Nick Young > Disable CollapseProject for correlated subqueries in project

[jira] [Assigned] (SPARK-49666) Put trim sensitivity under a flag

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49666: --- Assignee: Uroš Bojanić > Put trim sensitivity under a f

[jira] [Resolved] (SPARK-49666) Put trim sensitivity under a flag

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49666. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48222 [https

[jira] [Assigned] (SPARK-49811) Rename StringTypeAnyCollation to reflect all collation identifiers supported

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49811: --- Assignee: Vladan Vasić > Rename StringTypeAnyCollation to reflect all collat

[jira] [Resolved] (SPARK-49811) Rename StringTypeAnyCollation to reflect all collation identifiers supported

2024-09-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49811. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48265 [https

[jira] [Created] (SPARK-49816) should only update out-going-ref-count for referenced outer CTE relation

2024-09-27 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-49816: --- Summary: should only update out-going-ref-count for referenced outer CTE relation Key: SPARK-49816 URL: https://issues.apache.org/jira/browse/SPARK-49816 Project

[jira] [Assigned] (SPARK-49757) SET CATALOG syntax doesn't support IDENTIFIER

2024-09-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49757: --- Assignee: Mikhail Nikoliukin > SET CATALOG syntax doesn't support ID

[jira] [Resolved] (SPARK-49808) Fix a deadlock in subquery execution due to lazy vals

2024-09-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49808. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48279 [https

(spark) branch master updated: [SPARK-49808][SQL] Fix a deadlock in subquery execution due to lazy vals

2024-09-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d7abddc454ff [SPARK-49808][SQL] Fix a deadlock

[jira] [Resolved] (SPARK-49757) SET CATALOG syntax doesn't support IDENTIFIER

2024-09-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49757. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48228 [https

(spark) branch master updated: [SPARK-49757][SQL] Support IDENTIFIER expression in SET CATALOG statement

2024-09-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9b739d415cd5 [SPARK-49757][SQL] Support

(spark) branch branch-3.5 updated: [SPARK-49211][SQL][FOLLOW-UP] Support catalog in QualifiedTableName

2024-09-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 1040657ecbce [SPARK-49211][SQL][FOLLOW

(spark) branch master updated: [SPARK-49211][SQL][FOLLOW-UP] Support catalog in QualifiedTableName

2024-09-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new fc9d421a2345 [SPARK-49211][SQL][FOLLOW-UP

[jira] [Assigned] (SPARK-49444) Univocity parser handles ArrayIndexOutOfBounds exception

2024-09-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49444: --- Assignee: Vladan Vasić > Univocity parser handles ArrayIndexOutOfBounds except

[jira] [Resolved] (SPARK-49444) Univocity parser handles ArrayIndexOutOfBounds exception

2024-09-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49444. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47906 [https

(spark) branch master updated: [SPARK-49444][SQL] Modified UnivocityParser to throw runtime exceptions caused by ArrayIndexOutOfBounds with more user-oriented messages

2024-09-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 624eda5030eb [SPARK-49444][SQL] Modified

Re: [DISCUSS] Spark 3.5.3 breaks Iceberg SparkSessionCatalog

2024-09-25 Thread Wenchen Fan
it should be minimally difficult to switch this around on the >> Iceberg side, we only have to move the initialize code out and duplicate >> it. Not a huge cost >> >> On Sun, Sep 22, 2024 at 11:39 PM Wenchen Fan wrote: >> >>> It's a buggy behavior that a custo

Re: [DISCUSS] Spark 3.5.3 breaks Iceberg SparkSessionCatalog

2024-09-25 Thread Wenchen Fan
it should be minimally difficult to switch this around on the >> Iceberg side, we only have to move the initialize code out and duplicate >> it. Not a huge cost >> >> On Sun, Sep 22, 2024 at 11:39 PM Wenchen Fan wrote: >> >>> It's a buggy behavior that a custo

[jira] [Created] (SPARK-49791) Make DelegatingCatalogExtension more extendable

2024-09-25 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-49791: --- Summary: Make DelegatingCatalogExtension more extendable Key: SPARK-49791 URL: https://issues.apache.org/jira/browse/SPARK-49791 Project: Spark Issue Type

(spark) branch master updated: [SPARK-48195][FOLLOWUP] Accumulator reset() no longer needed in CollectMetricsExec.doExecute()

2024-09-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 80d6651cf6a1 [SPARK-48195][FOLLOWUP

(spark) branch master updated: [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields

2024-09-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a4fb6cbfda22 [SPARK-49743][SQL

Re: [DISCUSS][Spark SQL] Update API

2024-09-24 Thread Wenchen Fan
All the existing DML APIs we support today have a source query so they all start with the source DataFrame, e.g. sourceDf.write.insertInto... sourceDf.write.saveAsTable... sourceDf.mergeInto... However, this is not the case for UPDATE and DELETE, as there is no source query. We need a different st

Re: [DISCUSS] [Spark SQL] Single-pass Analyzer SPIP

2024-09-24 Thread Wenchen Fan
Let me add a bit more color since I'm the Shepherd. I've fixed quite some bugs in the analyzer due to rule order issues. The recent ones are https://github.com/apache/spark/pull/45718 and https://github.com/apache/spark/pull/45350 . Dealing with rule order is very tricky and making all the analyze

Re: [DISCUSS] Spark 3.5.3 breaks Iceberg SparkSessionCatalog

2024-09-22 Thread Wenchen Fan
It's a buggy behavior that a custom v2 catalog (without extending DelegatingCatalogExtension) expects Spark to still use the v1 DDL commands to operate on the tables inside it. This is also why the third-party catalogs (e.g. Unity Catalog and Apache Polaris) can not be used to overwrite `spark_cata

Re: [DISCUSS] Spark 3.5.3 breaks Iceberg SparkSessionCatalog

2024-09-22 Thread Wenchen Fan
It's a buggy behavior that a custom v2 catalog (without extending DelegatingCatalogExtension) expects Spark to still use the v1 DDL commands to operate on the tables inside it. This is also why the third-party catalogs (e.g. Unity Catalog and Apache Polaris) can not be used to overwrite `spark_cata

[jira] [Resolved] (SPARK-49653) Support single joins for full scalar subquery decorrelation

2024-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49653. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48145 [https

[jira] [Assigned] (SPARK-49653) Support single joins for full scalar subquery decorrelation

2024-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49653: --- Assignee: Andrey Gubichev > Support single joins for full scalar subquery decorrelat

(spark) branch master updated: [SPARK-49653][SQL] Single join for correlated scalar subqueries

2024-09-22 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3c81f076ab9c [SPARK-49653][SQL] Single join for

[jira] [Assigned] (SPARK-49667) Spark expressions that use StringSearch do not behave properly with CS_AI collators

2024-09-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49667: --- Assignee: Vladan Vasić > Spark expressions that use StringSearch do not behave prope

[jira] [Resolved] (SPARK-49667) Spark expressions that use StringSearch do not behave properly with CS_AI collators

2024-09-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49667. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48121 [https

(spark) branch master updated: [SPARK-49667][SQL] Disallowed CS_AI collators with expressions that use StringSearch

2024-09-19 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a060c236d314 [SPARK-49667][SQL] Disallowed

(spark) branch master updated: [SPARK-48280][SQL][FOLLOW-UP] Add expressions that are built via expressionBuilder to Expression Walker

2024-09-19 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ac34f1de92c6 [SPARK-48280][SQL][FOLLOW-UP] Add

[jira] [Assigned] (SPARK-48782) Add support for loading stored procedures in catalogs

2024-09-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48782: --- Assignee: Anton Okolnychyi > Add support for loading stored procedures in catal

(spark) branch master updated: [SPARK-48782][SQL] Add support for executing procedures in catalogs

2024-09-19 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 492d1b14c0d1 [SPARK-48782][SQL] Add support for

[jira] [Resolved] (SPARK-48782) Add support for loading stored procedures in catalogs

2024-09-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48782. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47943 [https

Re: [VOTE] Release Spark 4.0.0-preview2 (RC1)

2024-09-18 Thread Wenchen Fan
+1 On Wed, Sep 18, 2024 at 1:21 AM John Zhuge wrote: > +1 non-binding > > John Zhuge > > > On Mon, Sep 16, 2024 at 11:07 PM Xinrong Meng wrote: > >> +1 >> >> Thank you @Dongjoon Hyun ! >> >> On Tue, Sep 17, 2024 at 11:31 AM huaxin gao >> wrote: >> >>> +1 >>> >>> On Mon, Sep 16, 2024 at 6:20 P

[jira] [Resolved] (SPARK-49611) Introduce TVF all_collations()

2024-09-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49611. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48087 [https

(spark) branch master updated: [SPARK-49611][SQL] Introduce TVF `collations()` & remove the `SHOW COLLATIONS` command

2024-09-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2113f109b8d7 [SPARK-49611][SQL] Introduce TVF

[jira] [Resolved] (SPARK-49646) fix subquery decorrelation for union / set operations when parentOuterReferences has references not covered in collectedChildOuterReferences

2024-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49646. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48109 [https

[jira] [Assigned] (SPARK-49646) fix subquery decorrelation for union / set operations when parentOuterReferences has references not covered in collectedChildOuterReferences

2024-09-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49646: --- Assignee: Avery Qi > fix subquery decorrelation for union / set operations w

(spark) branch master updated: [SPARK-49646][SQL] fix subquery decorrelation for union/set operations when parentOuterReferences has references not covered in collectedChildOuterReferences

2024-09-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 738db079c0b6 [SPARK-49646][SQL] fix subquery

[jira] [Resolved] (SPARK-48824) Add SQL syntax in create/replace table to create an identity column

2024-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48824. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47614 [https

[jira] [Assigned] (SPARK-48824) Add SQL syntax in create/replace table to create an identity column

2024-09-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48824: --- Assignee: Carmen Kwan > Add SQL syntax in create/replace table to create an ident

(spark) branch master updated: [SPARK-48824][SQL] Add Identity Column SQL syntax

2024-09-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 931ab065df39 [SPARK-48824][SQL] Add Identity

[jira] [Resolved] (SPARK-49556) SELECT operator

2024-09-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49556. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48047 [https

(spark) branch master updated: [SPARK-49556][SQL] Add SQL pipe syntax for the SELECT operator

2024-09-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 017b0ea71e03 [SPARK-49556][SQL] Add SQL pipe

[jira] [Resolved] (SPARK-49488) MySQL dialect supports pushdown datetime functions.

2024-09-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49488. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47951 [https

(spark) branch master updated (5533c81e3453 -> 9fc58aa4c075)

2024-09-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 5533c81e3453 [SPARK-48355][SQL] Support for CASE statement add 9fc58aa4c075 [SPARK-49488][SQL] MySQL dialect

[jira] [Assigned] (SPARK-49591) Distinguish logical and physical types in variant spec

2024-09-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49591: --- Assignee: David Cashman > Distinguish logical and physical types in variant s

[jira] [Resolved] (SPARK-49591) Distinguish logical and physical types in variant spec

2024-09-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49591. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48064 [https

(spark) branch master updated: [SPARK-49591][SQL] Add Logical Type column to variant readme

2024-09-12 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c5c880e690c3 [SPARK-49591][SQL] Add Logical

(spark) branch master updated (c5fd509ad3c0 -> d72e8f9e1263)

2024-09-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from c5fd509ad3c0 [SPARK-49085][CONNECT][BUILD][FOLLOWUP] Remove the erroneous `type` definition for `spark-protobuf

Re: [VOTE] Document and Feature Preview via GitHub Pages

2024-09-11 Thread Wenchen Fan
+1 On Wed, Sep 11, 2024 at 5:15 PM Martin Grund wrote: > +1 > > On Wed, Sep 11, 2024 at 9:39 AM Kent Yao wrote: > >> Hi all, >> >> Following the discussion[1], I'd like to start the vote for 'Document and >> Feature Preview via GitHub Pages' >> >> >> Please vote for the next 72 hours:(excluding

[jira] [Assigned] (SPARK-49443) Implement to_variant_object expression and make schema_of_variant expressions print OBJECT for for Variant Objects

2024-09-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49443: --- Assignee: Harsh Motwani > Implement to_variant_object expression and m

[jira] [Resolved] (SPARK-49501) Catalog createTable API is double-escaping paths

2024-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49501. - Fix Version/s: 3.5.4 4.0.0 Resolution: Fixed > Catalog createTa

(spark) branch master updated (532aaafec4f1 -> dc3333bcc599)

2024-09-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 532aaafec4f1 [SPARK-49006] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files add

Re: Apache Spark 4.0.0-preview2 (?)

2024-09-08 Thread Wenchen Fan
+1, thanks Dongjoon! On Mon, Sep 9, 2024 at 9:44 AM Xinrong Meng wrote: > +1 > > Thank you @Dongjoon Hyun ! > > On Sat, Sep 7, 2024 at 8:05 PM Hyukjin Kwon wrote: > >> +1 >> >> On Sat, Sep 7, 2024 at 9:04 AM huaxin gao wrote: >> >>> +1 >>> >>> On Fri, Sep 6, 2024 at 1:12 PM L. C. Hsieh wrote

(spark) branch branch-3.5 updated: [SPARK-49246][SQL][FOLLOW-UP] The behavior of SaveAsTable should not be changed by falling back to v1 command

2024-09-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 3f22ef172173 [SPARK-49246][SQL][FOLLOW

(spark) branch master updated: [SPARK-49246][SQL][FOLLOW-UP] The behavior of SaveAsTable should not be changed by falling back to v1 command

2024-09-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 37b39b41d07c [SPARK-49246][SQL][FOLLOW-UP] The

[jira] [Resolved] (SPARK-49383) Support Transpose DataFrame API

2024-09-05 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49383. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47884 [https

[jira] [Assigned] (SPARK-49396) PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals rule

2024-09-05 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49396: --- Assignee: Avery Qi > PLAN_VALIDATION_FAILED_RULE_IN_BATCH in SimplifyConditionals r

(spark) branch master updated: [SPARK-49152][SQL][FOLLOWUP] DelegatingCatalogExtension should also use V1 commands

2024-09-05 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f7cfeb534d92 [SPARK-49152][SQL][FOLLOWUP

(spark) branch master updated: [SPARK-48348][SPARK-48376][SQL] Introduce `LEAVE` and `ITERATE` statements

2024-09-05 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 9676b1c48cba [SPARK-48348][SPARK-48376][SQL

[jira] [Assigned] (SPARK-48348) [M0] Support for LEAVE statement

2024-09-05 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48348: --- Assignee: David Milicevic > [M0] Support for LEAVE statem

Re: [DISCUSS] release Spark 3.5.3?

2024-09-01 Thread Wenchen Fan
+1 >> >> Yuming Wang 于2024年8月30日周五 02:34写道: >> >>> +1, Could we include two additional issues: >>> https://issues.apache.org/jira/browse/SPARK-49472 >>> https://issues.apache.org/jira/browse/SPARK-49349 >>> >>> On Wed, Aug 28, 2024 at

[jira] [Assigned] (SPARK-49451) Allow duplicate keys in parse_json.

2024-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49451: --- Assignee: Chenhao Li > Allow duplicate keys in parse_j

[jira] [Resolved] (SPARK-49451) Allow duplicate keys in parse_json.

2024-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49451. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47920 [https

(spark) branch master updated: [SPARK-49451] Allow duplicate keys in parse_json

2024-09-01 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8879df5fc12b [SPARK-49451] Allow duplicate keys

[jira] [Resolved] (SPARK-49480) NullPointerException from SparkThrowableHelper.isInternalError method

2024-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49480. - Fix Version/s: 4.0.0 3.5.3 Resolution: Fixed > NullPointerExcept

[jira] [Assigned] (SPARK-49480) NullPointerException from SparkThrowableHelper.isInternalError method

2024-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49480: --- Assignee: Xi Chen > NullPointerException from SparkThrowableHelper.isInternalError met

(spark) branch branch-3.5 updated: [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError`

2024-09-01 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new d5c3437b [SPARK-49480][CORE] Fix

(spark) branch master updated: [SPARK-49480][CORE] Fix NullPointerException from `SparkThrowableHelper.isInternalError`

2024-09-01 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new cef3c86e046e [SPARK-49480][CORE] Fix

[jira] [Assigned] (SPARK-46037) When Left Join build Left, ShuffledHashJoinExec may result in incorrect results

2024-08-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-46037: --- Assignee: mcdull_zhang > When Left Join build Left, ShuffledHashJoinExec may result

[jira] [Resolved] (SPARK-46037) When Left Join build Left, ShuffledHashJoinExec may result in incorrect results

2024-08-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-46037. - Fix Version/s: 4.0.0 3.5.3 Resolution: Fixed Issue resolved by pull

(spark) branch branch-3.5 updated: [SPARK-46037][SQL] Correctness fix for Shuffled Hash Join build left without codegen

2024-08-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 2ad11b632e07 [SPARK-46037][SQL

(spark) branch master updated: [SPARK-46037][SQL] Correctness fix for Shuffled Hash Join build left without codegen

2024-08-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new af5e0a267e5a [SPARK-46037][SQL] Correctness fix

[DISCUSS] release Spark 3.5.3?

2024-08-28 Thread Wenchen Fan
after 3.5.2 was released. Shall we do a 3.5.3 release now? Thanks, Wenchen

[jira] [Created] (SPARK-49393) fail by default in deprecated catalog plugin APIs

2024-08-26 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-49393: --- Summary: fail by default in deprecated catalog plugin APIs Key: SPARK-49393 URL: https://issues.apache.org/jira/browse/SPARK-49393 Project: Spark Issue Type

Re: [DISCUSS] [Spark SQL] A single-pass resolution approach for the Catalyst Analyzer

2024-08-26 Thread Wenchen Fan
+1. The analyzer rule order issue has bitten me multiple times and it's very hard to make your analyzer rule bug-free if it interacts with other rules. On Wed, Aug 21, 2024 at 2:49 AM Reynold Xin wrote: > +1 on this too > > When I implemented "group by all", I introduced at least two subtle bugs

[jira] [Updated] (SPARK-49359) allow StagedTableCatalog implementations to fall back to non-atomic write

2024-08-24 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-49359: Fix Version/s: 3.5.3 > allow StagedTableCatalog implementations to fall back to non-atomic wr

(spark) branch branch-3.5 updated: [SPARK-49359][SQL] Allow StagedTableCatalog implementations to fall back to non-atomic write

2024-08-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 5eca9530309c [SPARK-49359][SQL] Allow

[jira] [Updated] (SPARK-49359) allow StagedTableCatalog implementations to fall back to non-atomic write

2024-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-49359: Summary: allow StagedTableCatalog implementations to fall back to non-atomic write (was: allow

  1   2   3   4   5   6   7   8   9   10   >