[jira] [Assigned] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40545: Assignee: Apache Spark > SparkSQLEnvSuite failed to clean the `spark_derby` directory aft

[jira] [Assigned] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40545: Assignee: (was: Apache Spark) > SparkSQLEnvSuite failed to clean the `spark_derby` di

[jira] [Assigned] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40543: - Assignee: Ruifeng Zheng > Make `ddof` in `DataFrame.var` and `Series.var` accept arbita

[jira] [Resolved] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40543. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37975 [https://

[jira] [Commented] (SPARK-40330) Implement `Series.searchsorted`.

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608572#comment-17608572 ] Apache Spark commented on SPARK-40330: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-40330) Implement `Series.searchsorted`.

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40330: Assignee: Apache Spark (was: Ruifeng Zheng) > Implement `Series.searchsorted`. > ---

[jira] [Commented] (SPARK-40330) Implement `Series.searchsorted`.

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608571#comment-17608571 ] Apache Spark commented on SPARK-40330: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-40330) Implement `Series.searchsorted`.

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40330: Assignee: Ruifeng Zheng (was: Apache Spark) > Implement `Series.searchsorted`. > ---

[jira] [Created] (SPARK-40545) SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution

2022-09-22 Thread Yang Jie (Jira)
Yang Jie created SPARK-40545: Summary: SparkSQLEnvSuite failed to clean the `spark_derby` directory after execution Key: SPARK-40545 URL: https://issues.apache.org/jira/browse/SPARK-40545 Project: Spark

[jira] [Commented] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608551#comment-17608551 ] Apache Spark commented on SPARK-40535: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40535: Assignee: Apache Spark > NPE from observe of collect_list > -

[jira] [Commented] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608550#comment-17608550 ] Apache Spark commented on SPARK-40535: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40535: Assignee: (was: Apache Spark) > NPE from observe of collect_list > --

[jira] [Commented] (SPARK-37203) Fix NotSerializableException when observe with TypedImperativeAggregate

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608549#comment-17608549 ] Apache Spark commented on SPARK-37203: -- User 'beliefer' has created a pull request

[jira] [Resolved] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-40462. - Resolution: Duplicate Already supported > Support np.ndarray for functions.lit > --

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Description: Currently we doesn't support NumPy type, `np.ndarray` for `functions.lit` We should

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Summary: Support np.ndarray for functions.lit (was: Support np.ndarray for functions.lit for mult

[jira] [Assigned] (SPARK-40544) The file size of `sql/hive/target/unit-tests.log` is too big

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40544: Assignee: (was: Apache Spark) > The file size of `sql/hive/target/unit-tests.log` is

[jira] [Assigned] (SPARK-40544) The file size of `sql/hive/target/unit-tests.log` is too big

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40544: Assignee: Apache Spark > The file size of `sql/hive/target/unit-tests.log` is too big > -

[jira] [Created] (SPARK-40544) The file size of `sql/hive/target/unit-tests.log` is too big

2022-09-22 Thread Yang Jie (Jira)
Yang Jie created SPARK-40544: Summary: The file size of `sql/hive/target/unit-tests.log` is too big Key: SPARK-40544 URL: https://issues.apache.org/jira/browse/SPARK-40544 Project: Spark Issue T

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit for multi dimensions.

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Description: Currently we doesn't support NumPy type, `np.ndarray` for `functions.lit` when the `

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit for multi dimensions.

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Summary: Support np.ndarray for functions.lit for multi dimensions. (was: Support np.ndarray for

[jira] [Commented] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608474#comment-17608474 ] Apache Spark commented on SPARK-40543: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40543: Assignee: (was: Apache Spark) > Make `ddof` in `DataFrame.var` and `Series.var` accep

[jira] [Assigned] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40543: Assignee: Apache Spark > Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary

[jira] [Commented] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608473#comment-17608473 ] Apache Spark commented on SPARK-40543: -- User 'zhengruifeng' has created a pull requ

[jira] [Created] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40543: - Summary: Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers Key: SPARK-40543 URL: https://issues.apache.org/jira/browse/SPARK-40543 Project: S

[jira] [Resolved] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40542. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37974 [https://

[jira] [Assigned] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40542: - Assignee: Ruifeng Zheng > Make `ddof` in `DataFrame.std` and `Series.std` accept arbita

[jira] [Updated] (SPARK-40501) Add PushProjectionThroughLimit for Optimizer

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40501: Summary: Add PushProjectionThroughLimit for Optimizer (was: Add pushProjectionThroughLimit for op

[jira] [Updated] (SPARK-40501) Add pushProjectionThroughLimit for optimizer

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40501: Summary: Add pushProjectionThroughLimit for optimizer (was: Add PushProjectionThroughLimit for Op

[jira] [Updated] (SPARK-40501) Add PushProjectionThroughLimit for Optimizer

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40501: Summary: Add PushProjectionThroughLimit for Optimizer (was: Enhance 'SpecialLimits' to support pr

[jira] [Updated] (SPARK-40541) NullPointerException with UTF8String.getBaseObject() when UDF

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40541: - Component/s: SQL (was: Spark Core) > NullPointerException with UTF8String.g

[jira] [Assigned] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-09-22 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40096: --- Assignee: Wan Kun > Finalize shuffle merge slow due to connection creation

[jira] [Resolved] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-09-22 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40096. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 375

[jira] [Assigned] (SPARK-40527) Keep struct field names or map keys in CreateStruct

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40527: Assignee: Ivan Sadikov > Keep struct field names or map keys in CreateStruct > --

[jira] [Resolved] (SPARK-40527) Keep struct field names or map keys in CreateStruct

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40527. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37965 [https://gi

[jira] [Resolved] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40531. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37970 [https://gi

[jira] [Assigned] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40531: Assignee: BingKun Pan > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 > --

[jira] [Commented] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608447#comment-17608447 ] Apache Spark commented on SPARK-40542: -- User 'zhengruifeng' has created a pull requ

[jira] [Assigned] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40542: Assignee: (was: Apache Spark) > Make `ddof` in `DataFrame.std` and `Series.std` accep

[jira] [Assigned] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40542: Assignee: Apache Spark > Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary

[jira] [Updated] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-40542: -- Component/s: SQL > Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers >

[jira] [Created] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40542: - Summary: Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers Key: SPARK-40542 URL: https://issues.apache.org/jira/browse/SPARK-40542 Project: S

[jira] [Updated] (SPARK-40541) NullPointerException with UTF8String.getBaseObject() when UDF

2022-09-22 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40541: -- Description: I'm using Spark 3.3.0 on Windows with Java 17. I have a UDF that returns several

[jira] [Created] (SPARK-40541) NullPointerException with UTF8String.getBaseObject() when UDF

2022-09-22 Thread Garret Wilson (Jira)
Garret Wilson created SPARK-40541: - Summary: NullPointerException with UTF8String.getBaseObject() when UDF Key: SPARK-40541 URL: https://issues.apache.org/jira/browse/SPARK-40541 Project: Spark

[jira] [Commented] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608396#comment-17608396 ] Apache Spark commented on SPARK-40540: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40540: Assignee: Max Gekk (was: Apache Spark) > Migrate compilation errors onto error classes >

[jira] [Commented] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608395#comment-17608395 ] Apache Spark commented on SPARK-40540: -- User 'MaxGekk' has created a pull request f

[jira] [Assigned] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40540: Assignee: Apache Spark (was: Max Gekk) > Migrate compilation errors onto error classes >

[jira] [Updated] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40540: - Description: Use temporary error classes in the compilation exceptions. (was: Use temporary error class

[jira] [Created] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Max Gekk (Jira)
Max Gekk created SPARK-40540: Summary: Migrate compilation errors onto error classes Key: SPARK-40540 URL: https://issues.apache.org/jira/browse/SPARK-40540 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-38098) Add support for ArrayType of nested StructType to arrow-based conversion

2022-09-22 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-38098: Assignee: Luca Canali > Add support for ArrayType of nested StructType to arrow-based con

[jira] [Resolved] (SPARK-38098) Add support for ArrayType of nested StructType to arrow-based conversion

2022-09-22 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-38098. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35391 [https://gi

[jira] [Resolved] (SPARK-40476) Reduce the shuffle size of ALS

2022-09-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40476. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37918 [https://gi

[jira] [Assigned] (SPARK-40476) Reduce the shuffle size of ALS

2022-09-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-40476: Assignee: Ruifeng Zheng > Reduce the shuffle size of ALS > --

[jira] [Created] (SPARK-40539) PySpark readwriter API parity for Spark Connect

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40539: Summary: PySpark readwriter API parity for Spark Connect Key: SPARK-40539 URL: https://issues.apache.org/jira/browse/SPARK-40539 Project: Spark Issue Type: S

[jira] [Created] (SPARK-40538) Add missing PySpark functions to Spark Connect

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40538: Summary: Add missing PySpark functions to Spark Connect Key: SPARK-40538 URL: https://issues.apache.org/jira/browse/SPARK-40538 Project: Spark Issue Type: Su

[jira] [Created] (SPARK-40537) Re-enable mypi supoprt

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40537: Summary: Re-enable mypi supoprt Key: SPARK-40537 URL: https://issues.apache.org/jira/browse/SPARK-40537 Project: Spark Issue Type: Sub-task Compone

[jira] [Created] (SPARK-40536) Make Spark Connect port configurable.

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40536: Summary: Make Spark Connect port configurable. Key: SPARK-40536 URL: https://issues.apache.org/jira/browse/SPARK-40536 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40535: - Description: The code below reproduces the issue: {code:scala} import org.apache.spark.sql.functions._

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40535: - Description: The code below reproduces the issue: {code:scala} import org.apache.spark.sql.functions._

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40535: - Description: The code below reproduces the issue: {code:scala} import org.apache.spark.sql.functions._

[jira] [Created] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
Max Gekk created SPARK-40535: Summary: NPE from observe of collect_list Key: SPARK-40535 URL: https://issues.apache.org/jira/browse/SPARK-40535 Project: Spark Issue Type: Bug Components

[jira] [Assigned] (SPARK-40407) Repartition of DataFrame can result in severe data skew in some special case

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40407: --- Assignee: Bobby Wang > Repartition of DataFrame can result in severe data skew in some spec

[jira] [Created] (SPARK-40534) Extend support for Join Relation

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40534: Summary: Extend support for Join Relation Key: SPARK-40534 URL: https://issues.apache.org/jira/browse/SPARK-40534 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-40533) Extend type support for Spark Connect literals

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40533: Summary: Extend type support for Spark Connect literals Key: SPARK-40533 URL: https://issues.apache.org/jira/browse/SPARK-40533 Project: Spark Issue Type: Su

[jira] [Resolved] (SPARK-40407) Repartition of DataFrame can result in severe data skew in some special case

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40407. - Fix Version/s: 3.3.1 3.2.3 3.4.0 Resolution: Fixed

[jira] [Created] (SPARK-40532) Python version for UDF should follow the servers version

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40532: Summary: Python version for UDF should follow the servers version Key: SPARK-40532 URL: https://issues.apache.org/jira/browse/SPARK-40532 Project: Spark Issu

[jira] [Assigned] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40488: --- Assignee: Bo Zhang > Do not wrap exceptions thrown in FileFormatWriter.write with SparkExce

[jira] [Resolved] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40488. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37931 [https://gith

[jira] [Commented] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

2022-09-22 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608268#comment-17608268 ] wuyi commented on SPARK-40320: -- I see. Thanks for the explaination.  > When the Executor p

[jira] [Updated] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40531: Description: *1.5.2-3 VS 1.5.2-4* !image-2022-09-22-20-03-44-348.png|width=833,height=251! was

[jira] [Resolved] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40529. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37968 [https://gi

[jira] [Assigned] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40529: Assignee: Ruifeng Zheng > Remove `pyspark.pandas.ml` > -- > >

[jira] [Updated] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40531: Description: *1.5.2-3 VS 1.5.2-4* !image-2022-09-22-20-03-44-348.png! > Upgrade zstd-jni from 1.

[jira] [Updated] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40531: Attachment: image-2022-09-22-20-03-44-348.png > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 > ---

[jira] [Assigned] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40531: Assignee: (was: Apache Spark) > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 > --

[jira] [Commented] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608236#comment-17608236 ] Apache Spark commented on SPARK-40531: -- User 'panbingkun' has created a pull reques

[jira] [Assigned] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40531: Assignee: Apache Spark > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 > -

[jira] [Created] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-40531: --- Summary: Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 Key: SPARK-40531 URL: https://issues.apache.org/jira/browse/SPARK-40531 Project: Spark Issue Type: Improvemen

[jira] [Assigned] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40530: Assignee: (was: Apache Spark) > Add error-related developer APIs > --

[jira] [Commented] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608223#comment-17608223 ] Apache Spark commented on SPARK-40530: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40530: Assignee: Apache Spark > Add error-related developer APIs > -

[jira] [Updated] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40490: -- Fix Version/s: 3.2.3 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile`

[jira] [Comment Edited] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608174#comment-17608174 ] Dongjoon Hyun edited comment on SPARK-40490 at 9/22/22 11:15 AM: -

[jira] [Created] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-40530: --- Summary: Add error-related developer APIs Key: SPARK-40530 URL: https://issues.apache.org/jira/browse/SPARK-40530 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-40523) pyspark dataframe methods (i.e. show()) won't run in VSCode debug console

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608213#comment-17608213 ] Hyukjin Kwon commented on SPARK-40523: -- Is this a pyspark issue? or VSCode issue?

[jira] [Assigned] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40529: Assignee: (was: Apache Spark) > Remove `pyspark.pandas.ml` >

[jira] [Assigned] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40529: Assignee: Apache Spark > Remove `pyspark.pandas.ml` > -- > >

[jira] [Commented] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608198#comment-17608198 ] Apache Spark commented on SPARK-40529: -- User 'zhengruifeng' has created a pull requ

[jira] [Created] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40529: - Summary: Remove `pyspark.pandas.ml` Key: SPARK-40529 URL: https://issues.apache.org/jira/browse/SPARK-40529 Project: Spark Issue Type: Sub-task C

[jira] [Assigned] (SPARK-40503) Add resampling to API references

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40503: - Assignee: Ruifeng Zheng > Add resampling to API references > --

[jira] [Reopened] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reopened SPARK-40327: --- > Increase pandas API coverage for pandas API on Spark > ---

[jira] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327 ] Ruifeng Zheng deleted comment on SPARK-40327: --- was (Author: podongfeng): Issue resolved by pull request 37948 [https://github.com/apache/spark/pull/37948] > Increase pandas API coverage fo

[jira] [Assigned] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40327: - Assignee: (was: Ruifeng Zheng) > Increase pandas API coverage for pandas API on Spa

[jira] [Resolved] (SPARK-40503) Add resampling to API references

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40503. --- Fix Version/s: 3.4.0 Target Version/s: 3.4.0 Resolution: Resolved Resolved

[jira] [Resolved] (SPARK-40359) Migrate JSON type check failures onto error classes

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40359. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37902 [https://github.com

[jira] [Assigned] (SPARK-40359) Migrate JSON type check failures onto error classes

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40359: Assignee: Max Gekk > Migrate JSON type check failures onto error classes > --

[jira] [Resolved] (SPARK-40510) Implement `ddof` in `Series.cov`

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40510. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37953 [https://

  1   2   >