[jira] [Created] (SPARK-42635) Several counter-intuitive behaviours in the TimestampAdd expression

2023-03-01 Thread Chenhao Li (Jira)
Chenhao Li created SPARK-42635: -- Summary: Several counter-intuitive behaviours in the TimestampAdd expression Key: SPARK-42635 URL: https://issues.apache.org/jira/browse/SPARK-42635 Project: Spark

[jira] [Created] (SPARK-42634) Several counter-intuitive behaviours in the TimestampAdd expression

2023-03-01 Thread Chenhao Li (Jira)
Chenhao Li created SPARK-42634: -- Summary: Several counter-intuitive behaviours in the TimestampAdd expression Key: SPARK-42634 URL: https://issues.apache.org/jira/browse/SPARK-42634 Project: Spark

[jira] [Commented] (SPARK-42632) Fix scala paths in tests

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695232#comment-17695232 ] Apache Spark commented on SPARK-42632: -- User 'hvanhovell' has created a pull request for this

[jira] [Assigned] (SPARK-42632) Fix scala paths in tests

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42632: Assignee: Apache Spark (was: Herman van Hövell) > Fix scala paths in tests >

[jira] [Assigned] (SPARK-42632) Fix scala paths in tests

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42632: Assignee: Herman van Hövell (was: Apache Spark) > Fix scala paths in tests >

[jira] [Assigned] (SPARK-42633) Use the actual schema in a LocalRelation

2023-03-01 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-42633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-42633: - Assignee: Herman van Hövell > Use the actual schema in a LocalRelation >

[jira] [Created] (SPARK-42633) Use the actual schema in a LocalRelation

2023-03-01 Thread Jira
Herman van Hövell created SPARK-42633: - Summary: Use the actual schema in a LocalRelation Key: SPARK-42633 URL: https://issues.apache.org/jira/browse/SPARK-42633 Project: Spark Issue

[jira] [Created] (SPARK-42632) Fix scala paths in tests

2023-03-01 Thread Jira
Herman van Hövell created SPARK-42632: - Summary: Fix scala paths in tests Key: SPARK-42632 URL: https://issues.apache.org/jira/browse/SPARK-42632 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695222#comment-17695222 ] Dongjoon Hyun commented on SPARK-34827: --- I removed `Target Version` field value, `3.3.0`, because

[jira] [Updated] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34827: -- Affects Version/s: 3.4.0 > Support fetching shuffle blocks in batch with i/o encryption >

[jira] [Commented] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695223#comment-17695223 ] Dongjoon Hyun commented on SPARK-34827: --- Also, I lower the priority from `Blocker` to `Major`

[jira] [Updated] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34827: -- Priority: Major (was: Blocker) > Support fetching shuffle blocks in batch with i/o

[jira] [Updated] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-34827: -- Target Version/s: (was: 3.3.0) > Support fetching shuffle blocks in batch with i/o

[jira] [Updated] (SPARK-39993) Spark on Kubernetes doesn't filter data by date

2023-03-01 Thread Hanna Liashchuk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanna Liashchuk updated SPARK-39993: Affects Version/s: 3.3.2 > Spark on Kubernetes doesn't filter data by date >

[jira] [Updated] (SPARK-42609) Add tests for grouping() and grouping_id() functions

2023-03-01 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-42609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-42609: -- Summary: Add tests for grouping() and grouping_id() functions (was: Support

[jira] [Updated] (SPARK-42543) Specify protocol for UDF artifact transfer in JVM/Scala client

2023-03-01 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-42543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-42543: -- Epic Link: SPARK-42554 > Specify protocol for UDF artifact transfer in JVM/Scala

[jira] [Updated] (SPARK-42410) Support Scala 2.12/2.13 tests in connect module

2023-03-01 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-42410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-42410: -- Epic Link: SPARK-42554 > Support Scala 2.12/2.13 tests in connect module >

[jira] [Resolved] (SPARK-42581) Add SparkSession implicits

2023-03-01 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-42581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-42581. --- Fix Version/s: 3.4.1 Resolution: Fixed > Add SparkSession implicits >

[jira] [Assigned] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34827: Assignee: Apache Spark > Support fetching shuffle blocks in batch with i/o encryption >

[jira] [Assigned] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34827: Assignee: (was: Apache Spark) > Support fetching shuffle blocks in batch with i/o

[jira] [Commented] (SPARK-34827) Support fetching shuffle blocks in batch with i/o encryption

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695157#comment-17695157 ] Apache Spark commented on SPARK-34827: -- User 'tomvanbussel' has created a pull request for this

[jira] [Created] (SPARK-42631) Support custom extensions in Spark Connect Scala client

2023-03-01 Thread Tom van Bussel (Jira)
Tom van Bussel created SPARK-42631: -- Summary: Support custom extensions in Spark Connect Scala client Key: SPARK-42631 URL: https://issues.apache.org/jira/browse/SPARK-42631 Project: Spark

[jira] [Assigned] (SPARK-42630) Make `parse_data_type` use new proto message `DDLParse`

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42630: Assignee: (was: Apache Spark) > Make `parse_data_type` use new proto message

[jira] [Assigned] (SPARK-42630) Make `parse_data_type` use new proto message `DDLParse`

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42630: Assignee: Apache Spark > Make `parse_data_type` use new proto message `DDLParse` >

[jira] [Commented] (SPARK-42630) Make `parse_data_type` use new proto message `DDLParse`

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695056#comment-17695056 ] Apache Spark commented on SPARK-42630: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-42630) Make `parse_data_type` use new proto message `DDLParse`

2023-03-01 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-42630: - Summary: Make `parse_data_type` use new proto message `DDLParse` Key: SPARK-42630 URL: https://issues.apache.org/jira/browse/SPARK-42630 Project: Spark

[jira] [Commented] (SPARK-42629) Update the description of default data source in the document

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17695046#comment-17695046 ] Apache Spark commented on SPARK-42629: -- User 'huangxiaopingRD' has created a pull request for this

[jira] [Assigned] (SPARK-42629) Update the description of default data source in the document

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42629: Assignee: (was: Apache Spark) > Update the description of default data source in the

[jira] [Assigned] (SPARK-42629) Update the description of default data source in the document

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42629: Assignee: Apache Spark > Update the description of default data source in the document >

[jira] [Created] (SPARK-42629) Update the description of default data source in the document

2023-03-01 Thread xiaoping.huang (Jira)
xiaoping.huang created SPARK-42629: -- Summary: Update the description of default data source in the document Key: SPARK-42629 URL: https://issues.apache.org/jira/browse/SPARK-42629 Project: Spark

[jira] [Commented] (SPARK-40610) Spark fall back to use getPartitions instead of getPartitionsByFilter when date_add functions used in where clause

2023-03-01 Thread icyjhl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694993#comment-17694993 ] icyjhl commented on SPARK-40610: Hi [~yumwang], it's string. > Spark fall back to use getPartitions

[jira] [Commented] (SPARK-40034) PathOutputCommitters to work with dynamic partition overwrite

2023-03-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694969#comment-17694969 ] Steve Loughran commented on SPARK-40034: thanks for the update. I will get that new pr done

[jira] [Updated] (SPARK-42525) Collapse two adjacent windows with semantically-same partition/order

2023-03-01 Thread zhuml (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuml updated SPARK-42525: -- Description: Extend the CollapseWindow rule to collapse Window nodes with semantically-same partition/order

[jira] [Updated] (SPARK-42525) Collapse two adjacent windows with semantically-same partition/order

2023-03-01 Thread zhuml (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuml updated SPARK-42525: -- Description: Extend the CollapseWindow rule to collapse Window nodes, when one window in subquery and the

[jira] [Updated] (SPARK-42525) Collapse two adjacent windows with semantically-same partition/order

2023-03-01 Thread zhuml (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuml updated SPARK-42525: -- Summary: Collapse two adjacent windows with semantically-same partition/order (was: collapse two adjacent

[jira] [Resolved] (SPARK-42593) Deprecate & remove the APIs that will be removed in pandas 2.0.

2023-03-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42593. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40216

[jira] [Assigned] (SPARK-42593) Deprecate & remove the APIs that will be removed in pandas 2.0.

2023-03-01 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42593: Assignee: Haejoon Lee > Deprecate & remove the APIs that will be removed in pandas 2.0.

[jira] [Assigned] (SPARK-42628) Add a migration note for bloom filter join

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42628: Assignee: (was: Apache Spark) > Add a migration note for bloom filter join >

[jira] [Commented] (SPARK-42628) Add a migration note for bloom filter join

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694958#comment-17694958 ] Apache Spark commented on SPARK-42628: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Resolved] (SPARK-42628) Add a migration note for bloom filter join

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-42628. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40231

[jira] [Assigned] (SPARK-42628) Add a migration note for bloom filter join

2023-03-01 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42628: Assignee: Apache Spark > Add a migration note for bloom filter join >

[jira] [Assigned] (SPARK-42628) Add a migration note for bloom filter join

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-42628: - Assignee: Dongjoon Hyun > Add a migration note for bloom filter join >

[jira] [Resolved] (SPARK-41870) Handle duplicate columns in `createDataFrame`

2023-03-01 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41870. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40227

[jira] [Updated] (SPARK-38841) Enable Bloom filter join by default

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38841: -- Issue Type: Improvement (was: Bug) > Enable Bloom filter join by default >

[jira] [Updated] (SPARK-42628) Add a migration note for bloom filter join

2023-03-01 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-42628: -- Summary: Add a migration note for bloom filter join (was: Add a migration node for bloom

[jira] [Created] (SPARK-42628) Add a migration node for bloom filter join

2023-03-01 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-42628: - Summary: Add a migration node for bloom filter join Key: SPARK-42628 URL: https://issues.apache.org/jira/browse/SPARK-42628 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-41725) Remove the workaround of sql(...).collect back in PySpark tests

2023-03-01 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-41725. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40160

<    1   2