[jira] [Updated] (SPARK-48302) Preserve nulls in map columns in PyArrow Tables

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Parent: SPARK-44111 Issue Type: Sub-task (was: Bug) > Preserve nulls in map columns in PyAr

[jira] [Updated] (SPARK-48302) Preserve nulls in map columns in PyArrow Tables

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Parent: (was: SPARK-44111) Issue Type: Bug (was: Sub-task) > Preserve nulls in map colu

[jira] [Updated] (SPARK-48302) Preserve nulls in map columns in PyArrow Tables

2024-06-02 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48302: --- Labels: pull-request-available (was: ) > Preserve nulls in map columns in PyArrow Tab

[jira] [Updated] (SPARK-48302) Preserve nulls in map columns in PyArrow Tables

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Summary: Preserve nulls in map columns in PyArrow Tables (was: Null values in map columns of PyArrow

[jira] [Updated] (SPARK-48302) Null values in map columns of PyArrow tables are replaced with empty lists

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Description: Because of a limitation in PyArrow, when PyArrow Tables containing MapArray columns

[jira] [Updated] (SPARK-48302) Null values in map columns of PyArrow tables are replaced with empty lists

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Description: Because of a limitation in PyArrow, when PyArrow Tables are passed

[jira] [Updated] (SPARK-48302) Null values in map columns of PyArrow tables are replaced with empty lists

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Description: Because of a limitation in PyArrow, when PyArrow Tables are passed

[jira] [Updated] (SPARK-48302) Null values in map columns of PyArrow tables are replaced with empty lists

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Description: Because of a limitation in PyArrow, when PyArrow Tables are passed

[jira] [Updated] (SPARK-47466) Add PySpark DataFrame method to return iterator of PyArrow RecordBatches

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-47466: - Component/s: Connect Input/Output SQL > Add PySpark DataFrame met

[jira] [Updated] (SPARK-47466) Add PySpark DataFrame method to return iterator of PyArrow RecordBatches

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-47466: - Affects Version/s: 4.0.0 > Add PySpark DataFrame method to return iterator of PyArrow RecordBatc

[jira] [Updated] (SPARK-48302) Null values in map columns of PyArrow tables are replaced with empty lists

2024-06-02 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Cook updated SPARK-48302: - Language: Python > Null values in map columns of PyArrow tables are replaced with empty li

[jira] [Assigned] (SPARK-48220) Allow passing PyArrow Table to createDataFrame()

2024-06-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48220: Assignee: Ian Cook > Allow passing PyArrow Table to createDataFr

[jira] [Resolved] (SPARK-48220) Allow passing PyArrow Table to createDataFrame()

2024-06-02 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48220. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46529 [https

[jira] [Updated] (SPARK-48499) Use Math.abs to get positive numbers

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48499: --- Labels: pull-request-available (was: ) > Use Math.abs to get positive numb

[jira] [Updated] (SPARK-48499) Use Math.abs to get positive numbers

2024-05-31 Thread Junqing Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junqing Li updated SPARK-48499: --- Summary: Use Math.abs to get positive numbers (was: Use Math.abs to get Unsigned Numbers) >

[jira] [Created] (SPARK-48499) Use Math.abs to get Unsigned Numbers

2024-05-31 Thread Junqing Li (Jira)
Junqing Li created SPARK-48499: -- Summary: Use Math.abs to get Unsigned Numbers Key: SPARK-48499 URL: https://issues.apache.org/jira/browse/SPARK-48499 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-48497) Add user guide for batch data source write API

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48497: --- Labels: pull-request-available (was: ) > Add user guide for batch data source write

[jira] [Created] (SPARK-48497) Add user guide for batch data source write API

2024-05-31 Thread Allison Wang (Jira)
Allison Wang created SPARK-48497: Summary: Add user guide for batch data source write API Key: SPARK-48497 URL: https://issues.apache.org/jira/browse/SPARK-48497 Project: Spark Issue Type

[jira] [Assigned] (SPARK-48490) Unescapes any literals for message of MessageWithContext

2024-05-31 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48490: -- Assignee: BingKun Pan > Unescapes any literals for message of MessageWithCont

[jira] [Resolved] (SPARK-48490) Unescapes any literals for message of MessageWithContext

2024-05-31 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48490. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46824 [https

[jira] [Assigned] (SPARK-48391) use addAll instead of add function in TaskMetrics to accelerate

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48391: --- Assignee: jiahong.li > use addAll instead of add function in TaskMetrics to acceler

[jira] [Resolved] (SPARK-48391) use addAll instead of add function in TaskMetrics to accelerate

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48391. - Fix Version/s: 3.5.2 4.0.0 Resolution: Fixed Issue resolved by pull

[jira] [Updated] (SPARK-48466) Wrap empty relation propagation in a dedicated node

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48466: --- Labels: pull-request-available (was: ) > Wrap empty relation propagation in a dedica

[jira] [Updated] (SPARK-48491) Refactor HiveWindowFunctionQuerySuite in BeforeAll and AfterAll

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48491: --- Labels: pull-request-available (was: ) > Refactor HiveWindowFunctionQuerySu

[jira] [Created] (SPARK-48495) Document planned approach to shredding

2024-05-31 Thread David Cashman (Jira)
David Cashman created SPARK-48495: - Summary: Document planned approach to shredding Key: SPARK-48495 URL: https://issues.apache.org/jira/browse/SPARK-48495 Project: Spark Issue Type: Sub

[jira] [Resolved] (SPARK-48465) Avoid no-op empty relation propagation in AQE

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48465. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46814 [https

[jira] [Updated] (SPARK-48494) Update airlift:aircompressor to 0.27

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48494: --- Labels: pull-request-available (was: ) > Update airlift:aircompressor to 0

[jira] [Updated] (SPARK-48493) Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance

2024-05-31 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-48493: Description: This enhancement adds an option to the Python Datasource Reader to yield Arrow

[jira] [Updated] (SPARK-48493) Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48493: --- Labels: pull-request-available (was: ) > Enhance Python Datasource Reader with Arrow Ba

[jira] [Created] (SPARK-48493) Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance

2024-05-31 Thread Luca Canali (Jira)
Luca Canali created SPARK-48493: --- Summary: Enhance Python Datasource Reader with Arrow Batch Support for Improved Performance Key: SPARK-48493 URL: https://issues.apache.org/jira/browse/SPARK-48493

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Updated] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Peloton updated SPARK-48492: --- Description: Hello, In the documentation, it is stated that > When reading Parquet fi

[jira] [Created] (SPARK-48492) batch-read parquet files written by streaming returns non-nullable fields in schema

2024-05-31 Thread Julien Peloton (Jira)
Julien Peloton created SPARK-48492: -- Summary: batch-read parquet files written by streaming returns non-nullable fields in schema Key: SPARK-48492 URL: https://issues.apache.org/jira/browse/SPARK-48492

[jira] [Created] (SPARK-48491) Refactor HiveWindowFunctionQuerySuite in BeforeAll and AfterAll

2024-05-31 Thread Rui Wang (Jira)
Rui Wang created SPARK-48491: Summary: Refactor HiveWindowFunctionQuerySuite in BeforeAll and AfterAll Key: SPARK-48491 URL: https://issues.apache.org/jira/browse/SPARK-48491 Project: Spark

[jira] [Updated] (SPARK-47172) Upgrade Transport block cipher mode to GCM

2024-05-31 Thread Steve Weis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Weis updated SPARK-47172: --- Due Date: 12/Jun/24 (was: 26/Mar/24) > Upgrade Transport block cipher mode to

[jira] [Resolved] (SPARK-48430) Fix map value extraction when map contains collated strings

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48430. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46758 [https

[jira] [Resolved] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48476. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46810 [https

[jira] [Assigned] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48476: --- Assignee: Milan Stefanovic > NPE thrown when delimiter set to null in

[jira] [Updated] (SPARK-48490) Unescapes any literals for message of MessageWithContext

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48490: --- Labels: pull-request-available (was: ) > Unescapes any literals for mess

[jira] [Assigned] (SPARK-48484) V2Write use the same TaskAttemptId for different task attempts

2024-05-31 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-48484: Assignee: Jackey Lee > V2Write use the same TaskAttemptId for different task attem

[jira] [Created] (SPARK-48490) Unescapes any literals for message of MessageWithContext

2024-05-31 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48490: --- Summary: Unescapes any literals for message of MessageWithContext Key: SPARK-48490 URL: https://issues.apache.org/jira/browse/SPARK-48490 Project: Spark Issue

[jira] [Resolved] (SPARK-48484) V2Write use the same TaskAttemptId for different task attempts

2024-05-31 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-48484. -- Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution: Fixed Issue

[jira] [Updated] (SPARK-48489) Throw an user-facing error when reading invalid schema from text DataSource

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48489: --- Labels: pull-request-available (was: ) > Throw an user-facing error when reading inva

[jira] [Created] (SPARK-48489) Throw an user-facing error when reading invalid schema from text DataSource

2024-05-31 Thread Stefan Bukorovic (Jira)
Stefan Bukorovic created SPARK-48489: Summary: Throw an user-facing error when reading invalid schema from text DataSource Key: SPARK-48489 URL: https://issues.apache.org/jira/browse/SPARK-48489

[jira] [Updated] (SPARK-47690) Hash aggregate support for strings with collation

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47690: --- Labels: pull-request-available (was: ) > Hash aggregate support for strings with collat

[jira] [Updated] (SPARK-48488) Restore the original logic of methods `log[info|warning|error]` in `SparkSubmit`

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48488: --- Labels: pull-request-available (was: ) > Restore the original logic of methods `log[i

[jira] [Created] (SPARK-48488) Restore the original logic of methods `log[info|warning|error]` in `SparkSubmit`

2024-05-31 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-48488: --- Summary: Restore the original logic of methods `log[info|warning|error]` in `SparkSubmit` Key: SPARK-48488 URL: https://issues.apache.org/jira/browse/SPARK-48488

[jira] [Updated] (SPARK-48487) Update License & Notice according to the dependency changes

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48487: --- Labels: pull-request-available (was: ) > Update License & Notice ac

[jira] [Created] (SPARK-48487) Update License & Notice according to the dependency changes

2024-05-31 Thread Kent Yao (Jira)
Kent Yao created SPARK-48487: Summary: Update License & Notice according to the dependency changes Key: SPARK-48487 URL: https://issues.apache.org/jira/browse/SPARK-48487 Project: Spark I

[jira] [Assigned] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48476: -- Assignee: (was: Apache Spark) > NPE thrown when delimiter set to null in

[jira] [Assigned] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48476: -- Assignee: Apache Spark > NPE thrown when delimiter set to null in

[jira] [Assigned] (SPARK-47258) Assign error classes to SHOW CREATE TABLE errors

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47258: -- Assignee: Apache Spark > Assign error classes to SHOW CREATE TABLE err

[jira] [Assigned] (SPARK-47258) Assign error classes to SHOW CREATE TABLE errors

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-47258: -- Assignee: (was: Apache Spark) > Assign error classes to SHOW CREATE TABLE err

[jira] [Commented] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17851000#comment-17851000 ] chenfengbin commented on SPARK-48486: - The reason for the problem starting from Spark 3.3

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Attachment: create_table.sql > The Dynamic Partition Pruning (DPP) feature in Spark can ca

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Attachment: create_table.sql.rtf > The Dynamic Partition Pruning (DPP) feature in Spark can ca

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Attachment: (was: create_table.sql.rtf) > The Dynamic Partition Pruning (DPP) feat

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Description: The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Description: The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Description: The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated

[jira] [Updated] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenfengbin updated SPARK-48486: Description: The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated

[jira] [Created] (SPARK-48486) The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand.

2024-05-31 Thread chenfengbin (Jira)
chenfengbin created SPARK-48486: --- Summary: The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated Directed Acyclic Graph (DAG) to expand. Key: SPARK-48486 URL: https://issues.apache.org/jira

[jira] [Updated] (SPARK-48485) Support interruptTag and interruptAll in streaming queries

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48485: --- Labels: pull-request-available (was: ) > Support interruptTag and interrupt

[jira] [Created] (SPARK-48485) Support interruptTag and interruptAll in streaming queries

2024-05-31 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-48485: Summary: Support interruptTag and interruptAll in streaming queries Key: SPARK-48485 URL: https://issues.apache.org/jira/browse/SPARK-48485 Project: Spark

[jira] [Updated] (SPARK-38506) Push partial aggregation through join

2024-05-31 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-38506: --- Labels: pull-request-available (was: ) > Push partial aggregation through j

[jira] [Updated] (SPARK-48484) V2Write use the same TaskAttemptId for different task attempts

2024-05-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48484: --- Labels: pull-request-available (was: ) > V2Write use the same TaskAttemptId for differ

[jira] [Created] (SPARK-48484) V2Write use the same TaskAttemptId for different task attempts

2024-05-30 Thread Yang Jie (Jira)
Yang Jie created SPARK-48484: Summary: V2Write use the same TaskAttemptId for different task attempts Key: SPARK-48484 URL: https://issues.apache.org/jira/browse/SPARK-48484 Project: Spark

[jira] [Created] (SPARK-48483) Allow UnsafeExternalSorter to spill when other consumer request memory

2024-05-30 Thread Jin Chengcheng (Jira)
Jin Chengcheng created SPARK-48483: -- Summary: Allow UnsafeExternalSorter to spill when other consumer request memory Key: SPARK-48483 URL: https://issues.apache.org/jira/browse/SPARK-48483 Project

[jira] [Updated] (SPARK-48483) Allow UnsafeExternalSorter to spill when other consumer requests memory

2024-05-30 Thread Jin Chengcheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jin Chengcheng updated SPARK-48483: --- Summary: Allow UnsafeExternalSorter to spill when other consumer requests memory

[jira] [Updated] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-05-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48482: --- Labels: pull-request-available (was: ) > dropDuplicates and dropDuplicatesWithinWaterm

[jira] [Created] (SPARK-48482) dropDuplicates and dropDuplicatesWithinWatermark should accept varargs

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48482: --- Summary: dropDuplicates and dropDuplicatesWithinWatermark should accept varargs Key: SPARK-48482 URL: https://issues.apache.org/jira/browse/SPARK-48482 Project: Spark

[jira] [Updated] (SPARK-48476) NPE thrown when delimiter set to null in CSV

2024-05-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48476: --- Labels: pull-request-available (was: ) > NPE thrown when delimiter set to null in

[jira] [Assigned] (SPARK-48474) Fix the class name of the log in `SparkSubmitArguments` & `SparkSubmit`

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48474: Assignee: BingKun Pan > Fix the class name of the log in `SparkSubmitArgume

[jira] [Resolved] (SPARK-48474) Fix the class name of the log in `SparkSubmitArguments` & `SparkSubmit`

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48474. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46808 [https

[jira] [Assigned] (SPARK-48467) Upgrade Maven to 3.9.7

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48467: Assignee: BingKun Pan > Upgrade Maven to 3.

[jira] [Resolved] (SPARK-48467) Upgrade Maven to 3.9.7

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48467. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46798 [https

[jira] [Assigned] (SPARK-47716) SQLQueryTestSuite flaky case due to view name conflict

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-47716: Assignee: Jack Chen > SQLQueryTestSuite flaky case due to view name confl

[jira] [Resolved] (SPARK-47716) SQLQueryTestSuite flaky case due to view name conflict

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-47716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-47716. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 45855 [https

[jira] [Assigned] (SPARK-48419) Foldable propagation replace foldable column should use origin column

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48419: --- Assignee: KnightChess > Foldable propagation replace foldable column should use ori

[jira] [Resolved] (SPARK-48419) Foldable propagation replace foldable column should use origin column

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48419. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46742 [https

[jira] [Resolved] (SPARK-48461) Replace NullPointerExceptions with proper error classes in AssertNotNull expression

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48461. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46793 [https

[jira] [Created] (SPARK-48480) StreamingQueryListener thread should not be interruptable

2024-05-30 Thread Wei Liu (Jira)
Wei Liu created SPARK-48480: --- Summary: StreamingQueryListener thread should not be interruptable Key: SPARK-48480 URL: https://issues.apache.org/jira/browse/SPARK-48480 Project: Spark Issue Type

[jira] [Assigned] (SPARK-48446) Update SS Doc of dropDuplicatesWithinWatermark to use the right syntax

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-48446: Assignee: Yuchen Liu > Update SS Doc of dropDuplicatesWithinWatermark to use the ri

[jira] [Updated] (SPARK-48479) Support creating SQL functions in parser

2024-05-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48479: --- Labels: pull-request-available (was: ) > Support creating SQL functions in par

[jira] [Resolved] (SPARK-48446) Update SS Doc of dropDuplicatesWithinWatermark to use the right syntax

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48446. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46797 [https

[jira] [Resolved] (SPARK-48475) Optimize _get_jvm_function in PySpark.

2024-05-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48475. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46809 [https

[jira] [Updated] (SPARK-48479) Support creating SQL functions in parser

2024-05-30 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-48479: - Summary: Support creating SQL functions in parser (was: Support ccreating SQL functions

[jira] [Updated] (SPARK-48479) Support ccreating SQL functions in parser

2024-05-30 Thread Allison Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-48479: - Summary: Support ccreating SQL functions in parser (was: Add support for creating SQL

[jira] [Updated] (SPARK-48465) Avoid no-op empty relation propagation in AQE

2024-05-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48465: --- Labels: pull-request-available (was: ) > Avoid no-op empty relation propagation in

[jira] [Created] (SPARK-48479) Add support for creating SQL functions in parser

2024-05-30 Thread Allison Wang (Jira)
Allison Wang created SPARK-48479: Summary: Add support for creating SQL functions in parser Key: SPARK-48479 URL: https://issues.apache.org/jira/browse/SPARK-48479 Project: Spark Issue Type

[jira] [Resolved] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48468. - Fix Version/s: 4.0.0 Resolution: Fixed > Add LogicalQueryStage interface in catal

[jira] [Commented] (SPARK-48478) Allow passing iterator of PyArrow RecordBatches to createDataFrame()

2024-05-30 Thread Ian Cook (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17850865#comment-17850865 ] Ian Cook commented on SPARK-48478: -- For Connect, see class {{LocalRelation}} in {{{}python/pyspark/sql

[jira] [Assigned] (SPARK-48468) Add LogicalQueryStage interface in catalyst

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-48468: --- Assignee: Ziqi Liu > Add LogicalQueryStage interface in catal

[jira] [Resolved] (SPARK-48477) Refactor CollationSuite, CoalesceShufflePartitionsSuite, SQLExecutionSuite

2024-05-30 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-48477. - Fix Version/s: 4.0.0 Resolution: Fixed > Refactor CollationSu

[jira] [Created] (SPARK-48478) Allow passing iterator of PyArrow RecordBatches to createDataFrame()

2024-05-30 Thread Ian Cook (Jira)
Ian Cook created SPARK-48478: Summary: Allow passing iterator of PyArrow RecordBatches to createDataFrame() Key: SPARK-48478 URL: https://issues.apache.org/jira/browse/SPARK-48478 Project: Spark

  1   2   3   4   5   6   7   8   9   10   >