Re: Welcome to Our New Apache Spark Committer and PMCs

2023-10-03 Thread Wenchen Fan
Congrats! On Wed, Oct 4, 2023 at 8:25 AM Hyukjin Kwon wrote: > Woohoo! > > On Tue, 3 Oct 2023 at 22:47, Hussein Awala wrote: > >> Congrats to all of you! >> >> On Tue 3 Oct 2023 at 08:15, Rui Wang wrote: >> >>> Congratulations! Well deserved! >>> >>> -Rui >>> >>> >>> On Mon, Oct 2, 2023 at

[spark] branch branch-3.5 updated: [SPARK-43380][SQL] Fix Avro data type conversion issues without causing performance regression

2023-09-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new e6e0c074636 [SPARK-43380][SQL] Fix Avro

[spark] branch master updated: [SPARK-43380][SQL] Fix Avro data type conversion issues without causing performance regression

2023-09-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f8c87f03297 [SPARK-43380][SQL] Fix Avro data

[jira] [Resolved] (SPARK-45346) Parquet schema inference should respect case sensitive flag when merging schema

2023-09-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45346. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-45346) Parquet schema inference should respect case sensitive flag when merging schema

2023-09-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45346: --- Assignee: Wenchen Fan > Parquet schema inference should respect case sensitive flag w

[spark] branch branch-3.5 updated: [SPARK-45346][SQL] Parquet schema inference should respect case sensitive flag when merging schema

2023-09-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new eb6b68daa4e [SPARK-45346][SQL] Parquet

[spark] branch master updated: [SPARK-45346][SQL] Parquet schema inference should respect case sensitive flag when merging schema

2023-09-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1cbc424ae2a [SPARK-45346][SQL] Parquet schema

[jira] [Resolved] (SPARK-44780) Document SQL Session variables

2023-09-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44780. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42467 [https

[jira] [Assigned] (SPARK-44780) Document SQL Session variables

2023-09-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44780: --- Assignee: Serge Rielau > Document SQL Session variab

[spark] branch master updated: [SPARK-44780][DOC] SQL temporary variables

2023-09-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f6c6acbc00d [SPARK-44780][DOC] SQL temporary

[jira] [Created] (SPARK-45346) Parquet schema inference should respect case sensitive flag when merging schema

2023-09-26 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-45346: --- Summary: Parquet schema inference should respect case sensitive flag when merging schema Key: SPARK-45346 URL: https://issues.apache.org/jira/browse/SPARK-45346

[jira] [Assigned] (SPARK-45271) Merge _LEGACY_ERROR_TEMP_1113 into TABLE_OPERATION & delete some unused method in QueryCompilationErrors

2023-09-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45271: --- Assignee: BingKun Pan > Merge _LEGACY_ERROR_TEMP_1113 into TABLE_OPERATION & dele

[jira] [Resolved] (SPARK-45271) Merge _LEGACY_ERROR_TEMP_1113 into TABLE_OPERATION & delete some unused method in QueryCompilationErrors

2023-09-26 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45271. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43044 [https

[spark] branch master updated: [SPARK-45271][SQL] Merge _LEGACY_ERROR_TEMP_1113 into TABLE_OPERATION & delete some unused method in QueryCompilationErrors

2023-09-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2aa06fcf160 [SPARK-45271][SQL] Merge

[jira] [Resolved] (SPARK-45182) Ignore task completion from old stage after retrying indeterminate stages

2023-09-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45182. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-45182) Ignore task completion from old stage after retrying indeterminate stages

2023-09-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45182: --- Assignee: Mayur Bhosale > Ignore task completion from old stage after retry

[spark] branch branch-3.5 updated: [SPARK-45182][CORE] Ignore task completion from old stage after retrying indeterminate stages

2023-09-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 252970bab65d [SPARK-45182][CORE] Ignore

[spark] branch master updated: [SPARK-45182][CORE] Ignore task completion from old stage after retrying indeterminate stages

2023-09-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7ffc0b71aa3e [SPARK-45182][CORE] Ignore task

[jira] [Resolved] (SPARK-44550) Wrong semantics for null IN (empty list)

2023-09-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44550. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43068 [https

[spark] branch master updated: [SPARK-44550][SQL] Enable correctness fixes for `null IN (empty list)` under ANSI

2023-09-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7741fe79e20 [SPARK-44550][SQL] Enable

[jira] [Assigned] (SPARK-45306) Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans

2023-09-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45306: --- Assignee: Yang Jie > Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect pl

[jira] [Resolved] (SPARK-45306) Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans

2023-09-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45306. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[spark] branch branch-3.5 updated: [SPARK-45306][SQL][TESTS] Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans

2023-09-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 3551f8b89f1 [SPARK-45306][SQL][TESTS

[spark] branch master updated: [SPARK-45306][SQL][TESTS] Make `InMemoryColumnarBenchmark` use AQE-aware utils to collect plans

2023-09-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7e9666be15b [SPARK-45306][SQL][TESTS] Make

[jira] [Resolved] (SPARK-41086) Consolidate SecondArgumentXXX error to INVALID_PARAMETER_VALUE

2023-09-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-41086. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[spark] branch branch-3.5 updated: [SPARK-41086][SQL] Use DataFrame ID to semantically validate CollectMetrics

2023-09-21 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 5af0819654a [SPARK-41086][SQL] Use

[spark] branch master updated: [SPARK-41086][SQL] Use DataFrame ID to semantically validate CollectMetrics

2023-09-21 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7c3c7c5a4bd [SPARK-41086][SQL] Use DataFrame ID

[jira] [Assigned] (SPARK-45163) Merge TABLE_OPERATION & _LEGACY_ERROR_TEMP_1113 into UNSUPPORTED_TABLE_OPERATION and refactor some logic

2023-09-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45163: --- Assignee: BingKun Pan > Merge TABLE_OPERATION & _LEGACY_ERROR_TE

[jira] [Resolved] (SPARK-45163) Merge TABLE_OPERATION & _LEGACY_ERROR_TEMP_1113 into UNSUPPORTED_TABLE_OPERATION and refactor some logic

2023-09-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45163. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42917 [https

[spark] branch master updated: [SPARK-45163][SQL] Merge UNSUPPORTED_VIEW_OPERATION & UNSUPPORTED_TABLE_OPERATION & fix some issue

2023-09-21 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 39d6fdabb48 [SPARK-45163][SQL] Merge

[jira] [Resolved] (SPARK-45108) Improve the InjectRuntimeFilter for check probably shuffle

2023-09-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45108. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42861 [https

[jira] [Assigned] (SPARK-45108) Improve the InjectRuntimeFilter for check probably shuffle

2023-09-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45108: --- Assignee: jiaan.geng > Improve the InjectRuntimeFilter for check probably shuf

[spark] branch master updated: [SPARK-45108][SQL] Improve the InjectRuntimeFilter for check probably shuffle

2023-09-21 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 892fdc53269 [SPARK-45108][SQL] Improve

[jira] [Assigned] (SPARK-45112) Use UnresolvedFunction based resolution in SQL Dataset functions

2023-09-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45112: --- Assignee: Peter Toth > Use UnresolvedFunction based resolution in SQL Dataset functi

[spark] branch master updated (d92d6f60342c -> 288a92b9f41e)

2023-09-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from d92d6f60342c [SPARK-43979][SQL][FOLLOWUP] Handle non alias-only project case add 288a92b9f41e [SPARK-45112][SQL

[spark] branch branch-3.5 updated: [SPARK-43979][SQL][FOLLOWUP] Handle non alias-only project case

2023-09-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 6ab870f6a991 [SPARK-43979][SQL

[jira] [Resolved] (SPARK-45112) Use UnresolvedFunction based resolution in SQL Dataset functions

2023-09-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45112. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42864 [https

[spark] branch master updated: [SPARK-43979][SQL][FOLLOWUP] Handle non alias-only project case

2023-09-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d92d6f60342c [SPARK-43979][SQL][FOLLOWUP

[jira] [Created] (SPARK-45203) join tests in KeyGroupedPartitioningSuite should use merge join hint

2023-09-18 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-45203: --- Summary: join tests in KeyGroupedPartitioningSuite should use merge join hint Key: SPARK-45203 URL: https://issues.apache.org/jira/browse/SPARK-45203 Project: Spark

[jira] [Resolved] (SPARK-45184) Remove orphaned error class documents

2023-09-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45184. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42952 [https

[jira] [Assigned] (SPARK-45184) Remove orphaned error class documents

2023-09-17 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45184: --- Assignee: BingKun Pan > Remove orphaned error class docume

[spark] branch master updated (f5365d0dc59 -> fd155189226)

2023-09-17 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from f5365d0dc59 [SPARK-45034][SQL] Support deterministic mode function add fd155189226 [SPARK-45184][SQL][DOCS

[jira] [Resolved] (SPARK-36191) Support ORDER BY and LIMIT to be on the correlation path

2023-09-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-36191. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42705 [https

[jira] [Assigned] (SPARK-36191) Support ORDER BY and LIMIT to be on the correlation path

2023-09-13 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-36191: --- Assignee: Allison Wang > Support ORDER BY and LIMIT to be on the correlation p

[spark] branch master updated: [SPARK-36191][SQL] Handle limit and order by in correlated scalar (lateral) subqueries

2023-09-13 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 21c27d55b34 [SPARK-36191][SQL] Handle limit

[jira] [Resolved] (SPARK-45117) Implement missing otherCopyArgs for the MultiCommutativeOp expression

2023-09-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45117. - Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by pull

[spark] branch branch-3.5 updated: [SPARK-45117][SQL] Implement missing otherCopyArgs for the MultiCommutativeOp expression

2023-09-12 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 4e44d929005 [SPARK-45117][SQL

[spark] branch master updated: [SPARK-45117][SQL] Implement missing otherCopyArgs for the MultiCommutativeOp expression

2023-09-12 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d999f622dc6 [SPARK-45117][SQL] Implement

[jira] [Assigned] (SPARK-45117) Implement missing otherCopyArgs for the MultiCommutativeOp expression

2023-09-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45117: --- Assignee: Supun Nakandala > Implement missing otherCopyArgs for the MultiCommutativ

Re: [VOTE] Release Apache Spark 3.5.0 (RC5)

2023-09-11 Thread Wenchen Fan
+1 On Tue, Sep 12, 2023 at 9:00 AM Yuanjian Li wrote: > +1 (non-binding) > > Yuanjian Li 于2023年9月11日周一 09:36写道: > >> @Peter Toth I've looked into the details of this >> issue, and it appears that it's neither a regression in version 3.5.0 nor a >> correctness issue. It's a bug related to a

[jira] [Resolved] (SPARK-45069) SQL variable should always be resolved after outer reference

2023-09-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45069. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42803 [https

[jira] [Assigned] (SPARK-45069) SQL variable should always be resolved after outer reference

2023-09-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-45069: --- Assignee: Wenchen Fan > SQL variable should always be resolved after outer refere

[spark] branch master updated: [SPARK-45069][SQL] SQL variable should always be resolved after outer reference

2023-09-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2a10c8d93aa [SPARK-45069][SQL] SQL variable

[jira] [Created] (SPARK-45069) SQL variable should always be resolved after outer reference

2023-09-04 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-45069: --- Summary: SQL variable should always be resolved after outer reference Key: SPARK-45069 URL: https://issues.apache.org/jira/browse/SPARK-45069 Project: Spark

[jira] [Updated] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-44577: Fix Version/s: (was: 4.0.0) > INSERT BY NAME returns non-sensical error mess

[jira] [Resolved] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44577. - Fix Version/s: 3.5.0 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-44577) INSERT BY NAME returns non-sensical error message

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44577: --- Assignee: Jia Fan > INSERT BY NAME returns non-sensical error mess

[spark] branch branch-3.5 updated: [SPARK-44577][SQL] Fix INSERT BY NAME returns nonsensical error message

2023-09-01 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new b41fd4c454a [SPARK-44577][SQL] Fix

[spark] branch master updated: [SPARK-44577][SQL] Fix INSERT BY NAME returns nonsensical error message

2023-09-01 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 00f66994c80 [SPARK-44577][SQL] Fix INSERT

[jira] [Assigned] (SPARK-44743) Reflect function behavior different from Hive

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44743: --- Assignee: Jia Fan > Reflect function behavior different from H

[jira] [Assigned] (SPARK-44743) Reflect function behavior different from Hive

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44743: --- Assignee: (was: Fan Jiang) > Reflect function behavior different from H

[jira] [Assigned] (SPARK-44743) Reflect function behavior different from Hive

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44743: --- Assignee: Fan Jiang > Reflect function behavior different from H

[jira] [Resolved] (SPARK-44743) Reflect function behavior different from Hive

2023-09-01 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44743. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42661 [https

[spark] branch master updated (fd7acd32895 -> 9d28bef2f70)

2023-09-01 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from fd7acd32895 [SPARK-44238][CORE][SQL] Introduce a new `readFrom` method with byte array input for `BloomFilter

[spark] branch master updated: [SPARK-44990][SQL][FOLLOWUP] Remove lazy of `nullAsQuotedEmptyString`

2023-08-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c783e5abb58 [SPARK-44990][SQL][FOLLOWUP] Remove

Re: [VOTE] Release Apache Spark 3.5.0 (RC3)

2023-08-31 Thread Wenchen Fan
but one day after the RC3 cut: https://github.com/apache/spark/pull/42393 . I'm -1 and let's include this fix in 3.5. Thanks, Wenchen On Thu, Aug 31, 2023 at 9:09 PM Ian Manning wrote: > +1 (non-binding) > > Using Spark Core, Spark SQL, Structured Streaming. > > On Tue, Aug 29,

[jira] [Resolved] (SPARK-45012) CheckAnalysis should throw inlined plan in AnalysisException

2023-08-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-45012. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42729 [https

[spark] branch master updated: [SPARK-45012][SQL] CheckAnalysis should throw inlined plan in AnalysisException

2023-08-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a45a3a3d60cb [SPARK-45012][SQL] CheckAnalysis

[jira] [Resolved] (SPARK-44239) Free memory allocated by large vectors when vectors are reset

2023-08-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44239. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41782 [https

[jira] [Assigned] (SPARK-44239) Free memory allocated by large vectors when vectors are reset

2023-08-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44239: --- Assignee: Wan Kun > Free memory allocated by large vectors when vectors are re

[spark] branch master updated: [SPARK-44239][SQL] Free memory allocated by large vectors when vectors are reset

2023-08-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 27f9ac20675 [SPARK-44239][SQL] Free memory

[jira] [Resolved] (SPARK-44967) Unit should be considered first before using Boolean for TreeNodeTag

2023-08-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44967. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42683 [https

[jira] [Assigned] (SPARK-44967) Unit should be considered first before using Boolean for TreeNodeTag

2023-08-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44967: --- Assignee: jiaan.geng > Unit should be considered first before using Bool

[spark] branch master updated: [SPARK-44967][SQL][CONNECT] Unit should be considered first before using Boolean for TreeNodeTag

2023-08-29 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 0a8a29807b5 [SPARK-44967][SQL][CONNECT] Unit

[jira] [Resolved] (SPARK-44897) Local Property Propagation to Subquery Broadcast Exec

2023-08-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44897. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42587 [https

[jira] [Assigned] (SPARK-44897) Local Property Propagation to Subquery Broadcast Exec

2023-08-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44897: --- Assignee: Michael Chen > Local Property Propagation to Subquery Broadcast E

[spark] branch branch-3.5 updated: [SPARK-44897][SQL] Propagating local properties to subquery broadcast exec

2023-08-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 2f4a71252cc [SPARK-44897][SQL

[spark] branch master updated (eacd234a887 -> 4a4856207d4)

2023-08-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from eacd234a887 [SPARK-44963][PYTHON][ML][TESTS] Make PySpark (pyspark-ml module) tests passing without any optional

[jira] [Assigned] (SPARK-44865) Make StreamingRelationV2 support metadata column

2023-08-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44865: --- Assignee: Zerui Bao > Make StreamingRelationV2 support metadata col

[jira] [Resolved] (SPARK-44865) Make StreamingRelationV2 support metadata column

2023-08-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44865. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42554 [https

[spark] branch master updated: [SPARK-44865][SS] Make StreamingRelationV2 support metadata column

2023-08-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 428a7a1f981 [SPARK-44865][SS] Make

[jira] [Resolved] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

2023-08-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44930. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42629 [https

[jira] [Assigned] (SPARK-44930) Deterministic ApplyFunctionExpression should be foldable

2023-08-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44930: --- Assignee: Xianyang Liu > Deterministic ApplyFunctionExpression should be folda

[spark] branch branch-3.5 updated: [SPARK-44930][SQL] Deterministic ApplyFunctionExpression should be foldable

2023-08-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new cf3d10156ff [SPARK-44930][SQL

[spark] branch master updated (fb45476d58c -> 994389f42a4)

2023-08-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from fb45476d58c [SPARK-44957][PYTHON][SQL][TESTS] Make PySpark (pyspark-sql module) tests passing without any dependency

[jira] [Resolved] (SPARK-44549) Support correlated references under window functions

2023-08-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44549. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42383 [https

[jira] [Assigned] (SPARK-44549) Support correlated references under window functions

2023-08-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44549: --- Assignee: Andrey Gubichev > Support correlated references under window functi

[spark] branch master updated: [SPARK-44549][SQL] Support window functions in correlated scalar subqueries

2023-08-23 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 75e6b731cce [SPARK-44549][SQL] Support window

[jira] [Resolved] (SPARK-42768) Enable cached plan apply AQE by default

2023-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42768. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40390 [https

[jira] [Assigned] (SPARK-42768) Enable cached plan apply AQE by default

2023-08-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42768: --- Assignee: XiDuo You > Enable cached plan apply AQE by defa

[spark] branch master updated: [SPARK-42768][SQL] Enable cached plan apply AQE by default

2023-08-22 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1569ab543d2 [SPARK-42768][SQL] Enable cached

[spark] branch branch-3.5 updated: [SPARK-42768][SQL] Enable cached plan apply AQE by default

2023-08-22 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new d52582bfdf3 [SPARK-42768][SQL] Enable

[jira] [Resolved] (SPARK-43327) Trigger `committer.setupJob` before plan execute in `FileFormatWriter`

2023-08-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43327. - Fix Version/s: 3.3.4 Resolution: Fixed Issue resolved by pull request 41154 [https

[jira] [Assigned] (SPARK-43327) Trigger `committer.setupJob` before plan execute in `FileFormatWriter`

2023-08-21 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43327: --- Assignee: ming95 > Trigger `committer.setupJob` before plan exec

[spark] branch branch-3.3 updated: [SPARK-43327][CORE][3.3] Trigger `committer.setupJob` before plan execute in `FileFormatWriter#write`

2023-08-21 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.3 by this push: new 557f9f58817 [SPARK-43327][CORE][3.3

[spark] branch master updated: [SPARK-43205][DOCS][SQL][FOLLOWUP] IDENTIFIER clause docs

2023-08-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f351e7b4edd [SPARK-43205][DOCS][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-43205][DOC] identifier clause docs

2023-08-17 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 02a07cd6adc [SPARK-43205][DOC] identifier

[spark] branch branch-3.5 updated: [SPARK-43205][DOC] identifier clause docs

2023-08-17 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 7786d0b2f35 [SPARK-43205][DOC

Re: Spark writing API

2023-08-17 Thread Wenchen Fan
for it? Also note that: there is not always a shuffle right before the writing, if it's shuffle read -> filter -> data source write, we don't know the final output data size unless we know the selectivity of the filter. On Thu, Aug 17, 2023 at 1:02 PM Andrew Melo wrote: > Hell

[jira] [Assigned] (SPARK-44822) Make Python UDTFs by default non-deterministic

2023-08-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44822: --- Assignee: Allison Wang > Make Python UDTFs by default non-determinis

<    4   5   6   7   8   9   10   11   12   13   >