[jira] [Resolved] (SPARK-44822) Make Python UDTFs by default non-deterministic

2023-08-16 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44822. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42519 [https

[spark] branch branch-3.5 updated: [SPARK-44822][PYTHON][SQL] Make Python UDTFs by default non-deterministic

2023-08-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 92ecbaf600b [SPARK-44822][PYTHON][SQL

[spark] branch master updated: [SPARK-44822][PYTHON][SQL] Make Python UDTFs by default non-deterministic

2023-08-16 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new fce83d49993 [SPARK-44822][PYTHON][SQL] Make

Re: Spark writing API

2023-08-16 Thread Wenchen Fan
> is there a way to hint to the downstream users on the number of rows expected to write? It will be very hard to do. Spark pipelines the execution (within shuffle boundaries) and we can't predict the number of final output rows. On Mon, Aug 7, 2023 at 8:27 PM Steve Loughran wrote: > > > On

[spark] branch branch-3.5 updated: [SPARK-43205][SQL][FOLLOWUP] IDENTIFIER clause should accept alias and RuntimeReplaceable

2023-08-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new d3d3cf3c0dd [SPARK-43205][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-43205][SQL][FOLLOWUP] IDENTIFIER clause should accept alias and RuntimeReplaceable

2023-08-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1ae1d89d08a [SPARK-43205][SQL][FOLLOWUP

[jira] [Assigned] (SPARK-44718) High On-heap memory usage is detected while doing parquet-file reading with Off-Heap memory mode enabled on spark

2023-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44718: --- Assignee: Zamil Majdy > High On-heap memory usage is detected while doing parquet-f

[jira] [Resolved] (SPARK-44718) High On-heap memory usage is detected while doing parquet-file reading with Off-Heap memory mode enabled on spark

2023-08-15 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44718. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42394 [https

[spark] branch master updated: [SPARK-44718][SQL] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value

2023-08-15 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new afcccb42c96 [SPARK-44718][SQL] Match

[spark] branch branch-3.5 updated: [SPARK-43885][SQL][FOLLOWUP] Instruction#dataType should not fail

2023-08-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 4fd5e770209 [SPARK-43885][SQL][FOLLOWUP

[spark] branch master updated (46580ab4cb0 -> c9ff7025399)

2023-08-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 46580ab4cb0 [SPARK-43380][SQL] Revert `Fix Avro data type conversion issues` add c9ff7025399 [SPARK-43885][SQL

[spark] branch branch-3.5 updated: [SPARK-43063][SQL][FOLLOWUP] Add a space between -> and value when first value is null

2023-08-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 247948194a0 [SPARK-43063][SQL][FOLLOWUP

[spark] branch master updated (420e6878c68 -> 7e521694335)

2023-08-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 420e6878c68 [SPARK-43780][SQL] Support correlated references in join predicates for scalar and lateral subqueries

[jira] [Assigned] (SPARK-43780) Support correlated columns in join ON conditions

2023-08-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43780: --- Assignee: Andrey Gubichev > Support correlated columns in join ON conditi

[spark] branch master updated: [SPARK-43780][SQL] Support correlated references in join predicates for scalar and lateral subqueries

2023-08-14 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 420e6878c68 [SPARK-43780][SQL] Support

[jira] [Resolved] (SPARK-43780) Support correlated columns in join ON conditions

2023-08-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43780. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 41301 [https

[spark] branch master updated: [MINOR][SQL] Rename shouldBroadcast to isDynamicPruning in InSubqueryExec

2023-08-12 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7070b3672d8 [MINOR][SQL] Rename shouldBroadcast

[spark] branch branch-3.5 updated: [SPARK-43979][SQL][FOLLOWUP] transformUpWithNewOutput should only be used with new outputs

2023-08-11 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new dae1314003e [SPARK-43979][SQL][FOLLOWUP

[jira] [Resolved] (SPARK-43781) IllegalStateException when cogrouping two datasets derived from the same source

2023-08-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43781. - Fix Version/s: 3.5.0 Assignee: Jia Fan Resolution: Fixed

[spark] branch master updated: [SPARK-43781][SQL] Fix IllegalStateException when cogrouping two datasets derived from the same source

2023-08-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 5db87787d5c [SPARK-43781][SQL] Fix

[jira] [Assigned] (SPARK-44691) Move Subclasses of Analysis to sql/api

2023-08-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44691: --- Assignee: Yihong He > Move Subclasses of Analysis to sql/

[jira] [Resolved] (SPARK-44691) Move Subclasses of Analysis to sql/api

2023-08-10 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44691. - Fix Version/s: 3.5.0 Resolution: Fixed > Move Subclasses of Analysis to sql/

[spark] branch branch-3.5 updated: [SPARK-44691][SQL][CONNECT] Move Subclasses of AnalysisException to sql/api

2023-08-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 1db23d263a1 [SPARK-44691][SQL][CONNECT

[spark] branch master updated: [SPARK-44691][SQL][CONNECT] Move Subclasses of AnalysisException to sql/api

2023-08-10 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d5f1f17278a [SPARK-44691][SQL][CONNECT] Move

[jira] [Resolved] (SPARK-42849) Session variables

2023-08-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42849. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 40474 [https

[jira] [Assigned] (SPARK-42849) Session variables

2023-08-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42849: --- Assignee: Serge Rielau > Session variables > - > >

[jira] [Resolved] (SPARK-44551) Wrong semantics for null IN (empty list) - IN expression execution

2023-08-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44551. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42163 [https

[jira] [Assigned] (SPARK-44551) Wrong semantics for null IN (empty list) - IN expression execution

2023-08-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44551: --- Assignee: Jack Chen > Wrong semantics for null IN (empty list) - IN expression execut

[spark] branch branch-3.5 updated: [SPARK-44551][SQL] Fix behavior of null IN (empty list) in expression execution

2023-08-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 7520fc160aa [SPARK-44551][SQL] Fix

[spark] branch master updated: [SPARK-44551][SQL] Fix behavior of null IN (empty list) in expression execution

2023-08-09 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f9058d69b2a [SPARK-44551][SQL] Fix behavior

[jira] [Assigned] (SPARK-44236) Even `spark.sql.codegen.factoryMode` is NO_CODEGEN, the WholeStageCodegen also will be generated.

2023-08-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44236: --- Assignee: Jia Fan > Even `spark.sql.codegen.factoryMode` is NO_CODE

[spark] branch branch-3.5 updated: [SPARK-44236][SQL] Disable WholeStageCodegen when set `spark.sql.codegen.factoryMode` to NO_CODEGEN

2023-08-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 0e252195635 [SPARK-44236][SQL] Disable

[spark] branch master updated: [SPARK-44236][SQL] Disable WholeStageCodegen when set `spark.sql.codegen.factoryMode` to NO_CODEGEN

2023-08-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 74fa07c5702 [SPARK-44236][SQL] Disable

[jira] [Resolved] (SPARK-44236) Even `spark.sql.codegen.factoryMode` is NO_CODEGEN, the WholeStageCodegen also will be generated.

2023-08-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44236. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41779 [https

[jira] [Resolved] (SPARK-44714) Ease restriction of LCA resolution regarding queries with having

2023-08-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44714. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42276 [https

[jira] [Assigned] (SPARK-44714) Ease restriction of LCA resolution regarding queries with having

2023-08-08 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44714: --- Assignee: Xinyi Yu > Ease restriction of LCA resolution regarding queries with hav

[spark] branch branch-3.5 updated: [SPARK-44714] Ease restriction of LCA resolution regarding queries with having

2023-08-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 763c448759d [SPARK-44714] Ease

[spark] branch master updated: [SPARK-44714] Ease restriction of LCA resolution regarding queries with having

2023-08-08 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 29e8331681c [SPARK-44714] Ease restriction

Re: What else could be removed in Spark 4?

2023-08-07 Thread Wenchen Fan
I think the principle is we should remove things that block us from supporting new things like Java 21, or come with a significant maintenance cost. If there is no benefit to removing deprecated APIs (just to keep the codebase clean?), I'd prefer to leave them there and not bother. On Tue, Aug 8,

Welcome two new Apache Spark committers

2023-08-06 Thread Wenchen Fan
Hi all, The Spark PMC recently voted to add two new committers. Please join me in welcoming them to their new role! - Peter Toth (Spark SQL) - Xiduo You (Spark SQL) They consistently make contributions to the project and clearly showed their expertise. We are very excited to have them join as

[jira] [Resolved] (SPARK-44653) non-trivial DataFrame unions should not break caching

2023-08-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44653. - Fix Version/s: 3.3.3 3.5.0 3.4.2 Resolution: Fixed

[jira] [Assigned] (SPARK-44653) non-trivial DataFrame unions should not break caching

2023-08-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44653: --- Assignee: Wenchen Fan > non-trivial DataFrame unions should not break cach

[spark] branch branch-3.3 updated: [SPARK-44653][SQL] Non-trivial DataFrame unions should not break caching

2023-08-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.3 by this push: new 407bb57ac22 [SPARK-44653][SQL] Non

[spark] branch branch-3.4 updated: [SPARK-44653][SQL] Non-trivial DataFrame unions should not break caching

2023-08-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new b888ea87723 [SPARK-44653][SQL] Non

[spark] branch branch-3.5 updated: [SPARK-44653][SQL] Non-trivial DataFrame unions should not break caching

2023-08-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new aeb0cabbea7 [SPARK-44653][SQL] Non

[spark] branch master updated: [SPARK-44653][SQL] Non-trivial DataFrame unions should not break caching

2023-08-03 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ce1fe57cdd7 [SPARK-44653][SQL] Non-trivial

[jira] [Created] (SPARK-44655) make the code cleaner about static and dynamc data/partition filters

2023-08-03 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-44655: --- Summary: make the code cleaner about static and dynamc data/partition filters Key: SPARK-44655 URL: https://issues.apache.org/jira/browse/SPARK-44655 Project: Spark

[jira] [Created] (SPARK-44653) non-trivial DataFrame unions should not break caching

2023-08-03 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-44653: --- Summary: non-trivial DataFrame unions should not break caching Key: SPARK-44653 URL: https://issues.apache.org/jira/browse/SPARK-44653 Project: Spark Issue

[jira] [Resolved] (SPARK-40664) Union in query can remove cache from the plan

2023-08-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40664. - Resolution: Not A Problem > Union in query can remove cache from the p

[jira] [Assigned] (SPARK-44280) Add convertJavaTimestampToTimestamp in JDBCDialect API

2023-08-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44280: --- Assignee: Mingkang Li > Add convertJavaTimestampToTimestamp in JDBCDialect

[jira] [Resolved] (SPARK-44280) Add convertJavaTimestampToTimestamp in JDBCDialect API

2023-08-02 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44280. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41843 [https

[spark] branch master updated (4ec27c3801a -> e91c024cf3b)

2023-08-02 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 4ec27c3801a [SPARK-44555][SQL] Use checkError() to check Exception in command Suite & assign some error class n

[spark] branch branch-3.5 updated: [SPARK-44280][SQL] Add convertJavaTimestampToTimestamp in JDBCDialect API

2023-08-02 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new c26bc3a98ac [SPARK-44280][SQL] Add

[jira] [Created] (SPARK-44610) DeduplicateRelations should retain Alias metadata when creating a new instance

2023-07-31 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-44610: --- Summary: DeduplicateRelations should retain Alias metadata when creating a new instance Key: SPARK-44610 URL: https://issues.apache.org/jira/browse/SPARK-44610 Project

[spark] branch branch-3.5 updated: [SPARK-44340][SQL][FOLLOWUP][3.5] Set partition index correctly for WindowGroupLimitExec

2023-07-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new ee3ddf8536c [SPARK-44340][SQL][FOLLOWUP

[jira] [Resolved] (SPARK-44605) refine internal ShuffleWriteProcessor API

2023-07-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44605. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42234 [https

[jira] [Assigned] (SPARK-44605) refine internal ShuffleWriteProcessor API

2023-07-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44605: --- Assignee: Wenchen Fan > refine internal ShuffleWriteProcessor

[spark] branch master updated: [SPARK-44605][CORE] Refine internal ShuffleWriteProcessor API

2023-07-31 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8f44033dfce [SPARK-44605][CORE] Refine internal

[jira] [Created] (SPARK-44605) refine internal ShuffleWriteProcessor API

2023-07-31 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-44605: --- Summary: refine internal ShuffleWriteProcessor API Key: SPARK-44605 URL: https://issues.apache.org/jira/browse/SPARK-44605 Project: Spark Issue Type

[spark] branch master updated: [SPARK-44340][SPARK-44341][SQL][PYTHON][FOLLOWUP] Set partition index correctly for WindowGroupLimitExec,WindowExec and WindowInPandasExec

2023-07-30 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ea3061beedf [SPARK-44340][SPARK-44341][SQL

[spark] branch branch-3.5 updated: [SPARK-37377][SQL][FOLLOWUP] Fix the partitioned join of one side test case not match

2023-07-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 6879519cafa [SPARK-37377][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-37377][SQL][FOLLOWUP] Fix the partitioned join of one side test case not match

2023-07-28 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 20bb6c0c5a8 [SPARK-37377][SQL][FOLLOWUP] Fix

[spark] branch branch-3.5 updated: [SPARK-44361][SQL][FOLLOWUP] Use PartitionEvaluator API in MapInBatchExec

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 75679f72173 [SPARK-44361][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-44361][SQL][FOLLOWUP] Use PartitionEvaluator API in MapInBatchExec

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3cf88cb6c42 [SPARK-44361][SQL][FOLLOWUP] Use

[jira] [Assigned] (SPARK-42098) ResolveInlineTables should handle RuntimeReplaceable

2023-07-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42098: --- Assignee: Jia Fan > ResolveInlineTables should handle RuntimeReplacea

[jira] [Resolved] (SPARK-42098) ResolveInlineTables should handle RuntimeReplaceable

2023-07-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42098. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42110 [https

[spark] branch branch-3.5 updated: [SPARK-42098][SQL] Fix ResolveInlineTables can not handle with RuntimeReplaceable expression

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 81d56603b30 [SPARK-42098][SQL] Fix

[spark] branch master updated: [SPARK-42098][SQL] Fix ResolveInlineTables can not handle with RuntimeReplaceable expression

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new f235c9f622d [SPARK-42098][SQL] Fix

[spark] branch branch-3.5 updated: [SPARK-44287][SQL][FOLLOWUP] Set partition index correctly

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 4f7187885a6 [SPARK-44287][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-44287][SQL][FOLLOWUP] Set partition index correctly

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new bf1bbc514eb [SPARK-44287][SQL][FOLLOWUP] Set

[jira] [Assigned] (SPARK-44505) DataSource v2 Scans should not require planning the input partitions on explain

2023-07-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44505: --- Assignee: Martin Grund > DataSource v2 Scans should not require planning the in

[jira] [Resolved] (SPARK-44505) DataSource v2 Scans should not require planning the input partitions on explain

2023-07-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44505. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42099 [https

[spark] branch branch-3.5 updated: [SPARK-44505][SQL] Provide override for columnar support in Scan for DSv2

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 48494e1fa49 [SPARK-44505][SQL] Provide

[spark] branch master updated: [SPARK-44505][SQL] Provide override for columnar support in Scan for DSv2

2023-07-27 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 01191c83f8c [SPARK-44505][SQL] Provide override

[spark] branch branch-3.5 updated: [SPARK-44154][SQL][FOLLOWUP] `BitmapCount` and `BitmapOrAgg` should use `DataTypeMismatch` to indicate unexpected input data type

2023-07-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 5cb7d2b81af [SPARK-44154][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-44154][SQL][FOLLOWUP] `BitmapCount` and `BitmapOrAgg` should use `DataTypeMismatch` to indicate unexpected input data type

2023-07-26 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d0d4aab4378 [SPARK-44154][SQL][FOLLOWUP

[jira] [Assigned] (SPARK-44355) Move commands to CTEDef code path and deprecate CTE inline path

2023-07-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44355: --- Assignee: Wenchen Fan > Move commands to CTEDef code path and deprecate CTE inline p

[jira] [Resolved] (SPARK-44355) Move commands to CTEDef code path and deprecate CTE inline path

2023-07-25 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44355. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 42036 [https

[spark] branch master updated: [SPARK-44355][SQL] Move WithCTE into command queries

2023-07-25 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new da84f81b1e2 [SPARK-44355][SQL] Move WithCTE

[spark] branch branch-3.5 updated: [SPARK-44131][SQL][PYTHON][CONNECT][FOLLOWUP] Support qualified function name for call_function

2023-07-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new f1377a856e8 [SPARK-44131][SQL][PYTHON

[spark] branch master updated (a6ac63d14b5 -> d97a4e214c7)

2023-07-24 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from a6ac63d14b5 [SPARK-9][CONNECT] Upcasting for direct Arrow Deserialization add d97a4e214c7 [SPARK-44131][SQL

[spark] branch master updated: [SPARK-44341][SQL][PYTHON][FOLLOWUP] Move the base trait WindowEvaluatorFactoryBase to a single file

2023-07-21 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new bdeae870674 [SPARK-44341][SQL][PYTHON][FOLLOWUP

[jira] [Resolved] (SPARK-39634) Allow file splitting in combination with row index generation

2023-07-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-39634. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40728 [https

[spark] branch branch-3.5 updated: [SPARK-39634][SQL] Allow file splitting in combination with row index generation

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new e8dd144abca [SPARK-39634][SQL] Allow

[spark] branch master updated (7606ad32d61 -> 679ea56dc8b)

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 7606ad32d61 [SPARK-44484][SS] Add batchDuration to StreamingQueryProgress json method add 679ea56dc8b [SPARK

[jira] [Assigned] (SPARK-39634) Allow file splitting in combination with row index generation

2023-07-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-39634: --- Assignee: Ala Luszczak > Allow file splitting in combination with row index generat

[jira] [Assigned] (SPARK-43966) Support non-deterministic Python UDTFs

2023-07-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43966: --- Assignee: Allison Wang > Support non-deterministic Python UD

[jira] [Resolved] (SPARK-43966) Support non-deterministic Python UDTFs

2023-07-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43966. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42075 [https

[spark] branch master updated: [SPARK-43966][SQL][PYTHON] Support non-deterministic table-valued functions

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1fb3e16a48d [SPARK-43966][SQL][PYTHON] Support

[spark] branch branch-3.5 updated: [SPARK-43966][SQL][PYTHON] Support non-deterministic table-valued functions

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 40f30dd036c [SPARK-43966][SQL][PYTHON

[spark] branch branch-3.5 updated: [SPARK-44292][SQL][FOLLOWUP] Make TYPE_CHECK_FAILURE_WITH_HINT use correct name

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new b2d9b62ea5a [SPARK-44292][SQL][FOLLOWUP

[spark] branch master updated (bde87d66d0d -> 325888bc521)

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from bde87d66d0d [SPARK-44252][SS] Define a new error class and apply for the case where loading state from DFS fails

[spark] branch branch-3.5 updated: [SPARK-43839][SQL][FOLLOWUP] Convert _LEGACY_ERROR_TEMP_1337 to UNSUPPORTED_FEATURE.TIME_TRAVEL

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 39c732f89cd [SPARK-43839][SQL][FOLLOWUP

[spark] branch master updated: [SPARK-43839][SQL][FOLLOWUP] Convert _LEGACY_ERROR_TEMP_1337 to UNSUPPORTED_FEATURE.TIME_TRAVEL

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 6700f3ce8b1 [SPARK-43839][SQL][FOLLOWUP

[jira] [Resolved] (SPARK-44475) Relocate DataType and Parser to sql/api

2023-07-20 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44475. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 41928 [https

[spark] branch master updated (24bf76a088b -> 8ff6b7a04cb)

2023-07-20 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 24bf76a088b [SPARK-44491][INFRA] Add `branch-3.5` to `publish_snapshot` GitHub Action job add 8ff6b7a04cb

[jira] [Assigned] (SPARK-44431) Wrong semantics for null IN (empty list)

2023-07-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-44431: --- Assignee: Jack Chen > Wrong semantics for null IN (empty l

[jira] [Resolved] (SPARK-44431) Wrong semantics for null IN (empty list)

2023-07-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-44431. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 42007 [https

[spark] branch branch-3.5 updated: [SPARK-44431][SQL] Fix behavior of null IN (empty list) in optimization rules

2023-07-19 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.5 by this push: new 5511a0bd1d3 [SPARK-44431][SQL] Fix

[spark] branch master updated (e0c79c637c1 -> db357edb7b2)

2023-07-19 Thread wenchen
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from e0c79c637c1 [SPARK-43838][SQL] Fix subquery on single table with having clause can't be optimized add

<    5   6   7   8   9   10   11   12   13   14   >