[jira] [Commented] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608551#comment-17608551 ] Apache Spark commented on SPARK-40535: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40535: Assignee: Apache Spark > NPE from observe of collect_list >

[jira] [Commented] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608550#comment-17608550 ] Apache Spark commented on SPARK-40535: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40535: Assignee: (was: Apache Spark) > NPE from observe of collect_list >

[jira] [Commented] (SPARK-37203) Fix NotSerializableException when observe with TypedImperativeAggregate

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608549#comment-17608549 ] Apache Spark commented on SPARK-37203: -- User 'beliefer' has created a pull request for this issue:

[jira] [Resolved] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-40462. - Resolution: Duplicate Already supported > Support np.ndarray for functions.lit >

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Description: Currently we doesn't support NumPy type, `np.ndarray` for `functions.lit` We should

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Summary: Support np.ndarray for functions.lit (was: Support np.ndarray for functions.lit for

[jira] [Assigned] (SPARK-40544) The file size of `sql/hive/target/unit-tests.log` is too big

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40544: Assignee: (was: Apache Spark) > The file size of `sql/hive/target/unit-tests.log` is

[jira] [Assigned] (SPARK-40544) The file size of `sql/hive/target/unit-tests.log` is too big

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40544: Assignee: Apache Spark > The file size of `sql/hive/target/unit-tests.log` is too big >

[jira] [Created] (SPARK-40544) The file size of `sql/hive/target/unit-tests.log` is too big

2022-09-22 Thread Yang Jie (Jira)
Yang Jie created SPARK-40544: Summary: The file size of `sql/hive/target/unit-tests.log` is too big Key: SPARK-40544 URL: https://issues.apache.org/jira/browse/SPARK-40544 Project: Spark Issue

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit for multi dimensions.

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Description: Currently we doesn't support NumPy type, `np.ndarray` for `functions.lit` when the

[jira] [Updated] (SPARK-40462) Support np.ndarray for functions.lit for multi dimensions.

2022-09-22 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-40462: Summary: Support np.ndarray for functions.lit for multi dimensions. (was: Support np.ndarray for

[jira] [Commented] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608474#comment-17608474 ] Apache Spark commented on SPARK-40543: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40543: Assignee: (was: Apache Spark) > Make `ddof` in `DataFrame.var` and `Series.var`

[jira] [Assigned] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40543: Assignee: Apache Spark > Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary

[jira] [Commented] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608473#comment-17608473 ] Apache Spark commented on SPARK-40543: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-40543) Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40543: - Summary: Make `ddof` in `DataFrame.var` and `Series.var` accept arbitary integers Key: SPARK-40543 URL: https://issues.apache.org/jira/browse/SPARK-40543 Project:

[jira] [Resolved] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40542. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37974

[jira] [Assigned] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40542: - Assignee: Ruifeng Zheng > Make `ddof` in `DataFrame.std` and `Series.std` accept

[jira] [Updated] (SPARK-40501) Add PushProjectionThroughLimit for Optimizer

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40501: Summary: Add PushProjectionThroughLimit for Optimizer (was: Add pushProjectionThroughLimit for

[jira] [Updated] (SPARK-40501) Add pushProjectionThroughLimit for optimizer

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40501: Summary: Add pushProjectionThroughLimit for optimizer (was: Add PushProjectionThroughLimit for

[jira] [Updated] (SPARK-40501) Add PushProjectionThroughLimit for Optimizer

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40501: Summary: Add PushProjectionThroughLimit for Optimizer (was: Enhance 'SpecialLimits' to support

[jira] [Updated] (SPARK-40541) NullPointerException with UTF8String.getBaseObject() when UDF

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-40541: - Component/s: SQL (was: Spark Core) > NullPointerException with

[jira] [Assigned] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-09-22 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40096: --- Assignee: Wan Kun > Finalize shuffle merge slow due to connection creation

[jira] [Resolved] (SPARK-40096) Finalize shuffle merge slow due to connection creation fails

2022-09-22 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40096. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-40527) Keep struct field names or map keys in CreateStruct

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40527: Assignee: Ivan Sadikov > Keep struct field names or map keys in CreateStruct >

[jira] [Resolved] (SPARK-40527) Keep struct field names or map keys in CreateStruct

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40527. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37965

[jira] [Resolved] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40531. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37970

[jira] [Assigned] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40531: Assignee: BingKun Pan > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 >

[jira] [Commented] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608447#comment-17608447 ] Apache Spark commented on SPARK-40542: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40542: Assignee: (was: Apache Spark) > Make `ddof` in `DataFrame.std` and `Series.std`

[jira] [Assigned] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40542: Assignee: Apache Spark > Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary

[jira] [Updated] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-40542: -- Component/s: SQL > Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers >

[jira] [Created] (SPARK-40542) Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers

2022-09-22 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40542: - Summary: Make `ddof` in `DataFrame.std` and `Series.std` accept arbitary integers Key: SPARK-40542 URL: https://issues.apache.org/jira/browse/SPARK-40542 Project:

[jira] [Updated] (SPARK-40541) NullPointerException with UTF8String.getBaseObject() when UDF

2022-09-22 Thread Garret Wilson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Garret Wilson updated SPARK-40541: -- Description: I'm using Spark 3.3.0 on Windows with Java 17. I have a UDF that returns

[jira] [Created] (SPARK-40541) NullPointerException with UTF8String.getBaseObject() when UDF

2022-09-22 Thread Garret Wilson (Jira)
Garret Wilson created SPARK-40541: - Summary: NullPointerException with UTF8String.getBaseObject() when UDF Key: SPARK-40541 URL: https://issues.apache.org/jira/browse/SPARK-40541 Project: Spark

[jira] [Commented] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608396#comment-17608396 ] Apache Spark commented on SPARK-40540: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40540: Assignee: Max Gekk (was: Apache Spark) > Migrate compilation errors onto error classes

[jira] [Commented] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608395#comment-17608395 ] Apache Spark commented on SPARK-40540: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40540: Assignee: Apache Spark (was: Max Gekk) > Migrate compilation errors onto error classes

[jira] [Updated] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40540: - Description: Use temporary error classes in the compilation exceptions. (was: Use temporary error

[jira] [Created] (SPARK-40540) Migrate compilation errors onto error classes

2022-09-22 Thread Max Gekk (Jira)
Max Gekk created SPARK-40540: Summary: Migrate compilation errors onto error classes Key: SPARK-40540 URL: https://issues.apache.org/jira/browse/SPARK-40540 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-38098) Add support for ArrayType of nested StructType to arrow-based conversion

2022-09-22 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-38098: Assignee: Luca Canali > Add support for ArrayType of nested StructType to arrow-based

[jira] [Resolved] (SPARK-38098) Add support for ArrayType of nested StructType to arrow-based conversion

2022-09-22 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-38098. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 35391

[jira] [Resolved] (SPARK-40476) Reduce the shuffle size of ALS

2022-09-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-40476. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37918

[jira] [Assigned] (SPARK-40476) Reduce the shuffle size of ALS

2022-09-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-40476: Assignee: Ruifeng Zheng > Reduce the shuffle size of ALS >

[jira] [Created] (SPARK-40539) PySpark readwriter API parity for Spark Connect

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40539: Summary: PySpark readwriter API parity for Spark Connect Key: SPARK-40539 URL: https://issues.apache.org/jira/browse/SPARK-40539 Project: Spark Issue Type:

[jira] [Created] (SPARK-40538) Add missing PySpark functions to Spark Connect

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40538: Summary: Add missing PySpark functions to Spark Connect Key: SPARK-40538 URL: https://issues.apache.org/jira/browse/SPARK-40538 Project: Spark Issue Type:

[jira] [Created] (SPARK-40537) Re-enable mypi supoprt

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40537: Summary: Re-enable mypi supoprt Key: SPARK-40537 URL: https://issues.apache.org/jira/browse/SPARK-40537 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-40536) Make Spark Connect port configurable.

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40536: Summary: Make Spark Connect port configurable. Key: SPARK-40536 URL: https://issues.apache.org/jira/browse/SPARK-40536 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40535: - Description: The code below reproduces the issue: {code:scala} import org.apache.spark.sql.functions._

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40535: - Description: The code below reproduces the issue: {code:scala} import org.apache.spark.sql.functions._

[jira] [Updated] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk updated SPARK-40535: - Description: The code below reproduces the issue: {code:scala} import org.apache.spark.sql.functions._

[jira] [Created] (SPARK-40535) NPE from observe of collect_list

2022-09-22 Thread Max Gekk (Jira)
Max Gekk created SPARK-40535: Summary: NPE from observe of collect_list Key: SPARK-40535 URL: https://issues.apache.org/jira/browse/SPARK-40535 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-40407) Repartition of DataFrame can result in severe data skew in some special case

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40407: --- Assignee: Bobby Wang > Repartition of DataFrame can result in severe data skew in some

[jira] [Created] (SPARK-40534) Extend support for Join Relation

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40534: Summary: Extend support for Join Relation Key: SPARK-40534 URL: https://issues.apache.org/jira/browse/SPARK-40534 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-40533) Extend type support for Spark Connect literals

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40533: Summary: Extend type support for Spark Connect literals Key: SPARK-40533 URL: https://issues.apache.org/jira/browse/SPARK-40533 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-40407) Repartition of DataFrame can result in severe data skew in some special case

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40407. - Fix Version/s: 3.3.1 3.2.3 3.4.0 Resolution: Fixed

[jira] [Created] (SPARK-40532) Python version for UDF should follow the servers version

2022-09-22 Thread Martin Grund (Jira)
Martin Grund created SPARK-40532: Summary: Python version for UDF should follow the servers version Key: SPARK-40532 URL: https://issues.apache.org/jira/browse/SPARK-40532 Project: Spark

[jira] [Assigned] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-40488: --- Assignee: Bo Zhang > Do not wrap exceptions thrown in FileFormatWriter.write with

[jira] [Resolved] (SPARK-40488) Do not wrap exceptions thrown in FileFormatWriter.write with SparkException

2022-09-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-40488. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37931

[jira] [Commented] (SPARK-40320) When the Executor plugin fails to initialize, the Executor shows active but does not accept tasks forever, just like being hung

2022-09-22 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608268#comment-17608268 ] wuyi commented on SPARK-40320: -- I see. Thanks for the explaination.  > When the Executor plugin fails to

[jira] [Updated] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40531: Description: *1.5.2-3 VS 1.5.2-4* !image-2022-09-22-20-03-44-348.png|width=833,height=251!

[jira] [Resolved] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-40529. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37968

[jira] [Assigned] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-40529: Assignee: Ruifeng Zheng > Remove `pyspark.pandas.ml` > -- > >

[jira] [Updated] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40531: Description: *1.5.2-3 VS 1.5.2-4* !image-2022-09-22-20-03-44-348.png! > Upgrade zstd-jni from

[jira] [Updated] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BingKun Pan updated SPARK-40531: Attachment: image-2022-09-22-20-03-44-348.png > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 >

[jira] [Assigned] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40531: Assignee: (was: Apache Spark) > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 >

[jira] [Commented] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608236#comment-17608236 ] Apache Spark commented on SPARK-40531: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40531: Assignee: Apache Spark > Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 >

[jira] [Created] (SPARK-40531) Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4

2022-09-22 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-40531: --- Summary: Upgrade zstd-jni from 1.5.2-3 to 1.5.2-4 Key: SPARK-40531 URL: https://issues.apache.org/jira/browse/SPARK-40531 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40530: Assignee: (was: Apache Spark) > Add error-related developer APIs >

[jira] [Commented] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608223#comment-17608223 ] Apache Spark commented on SPARK-40530: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40530: Assignee: Apache Spark > Add error-related developer APIs >

[jira] [Updated] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40490: -- Fix Version/s: 3.2.3 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile`

[jira] [Comment Edited] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608174#comment-17608174 ] Dongjoon Hyun edited comment on SPARK-40490 at 9/22/22 11:15 AM: - This

[jira] [Created] (SPARK-40530) Add error-related developer APIs

2022-09-22 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-40530: --- Summary: Add error-related developer APIs Key: SPARK-40530 URL: https://issues.apache.org/jira/browse/SPARK-40530 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-40523) pyspark dataframe methods (i.e. show()) won't run in VSCode debug console

2022-09-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608213#comment-17608213 ] Hyukjin Kwon commented on SPARK-40523: -- Is this a pyspark issue? or VSCode issue? > pyspark

[jira] [Assigned] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40529: Assignee: (was: Apache Spark) > Remove `pyspark.pandas.ml` >

[jira] [Assigned] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40529: Assignee: Apache Spark > Remove `pyspark.pandas.ml` > -- > >

[jira] [Commented] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608198#comment-17608198 ] Apache Spark commented on SPARK-40529: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-40529) Remove `pyspark.pandas.ml`

2022-09-22 Thread Ruifeng Zheng (Jira)
Ruifeng Zheng created SPARK-40529: - Summary: Remove `pyspark.pandas.ml` Key: SPARK-40529 URL: https://issues.apache.org/jira/browse/SPARK-40529 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-40503) Add resampling to API references

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40503: - Assignee: Ruifeng Zheng > Add resampling to API references >

[jira] [Reopened] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reopened SPARK-40327: --- > Increase pandas API coverage for pandas API on Spark >

[jira] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327 ] Ruifeng Zheng deleted comment on SPARK-40327: --- was (Author: podongfeng): Issue resolved by pull request 37948 [https://github.com/apache/spark/pull/37948] > Increase pandas API coverage

[jira] [Assigned] (SPARK-40327) Increase pandas API coverage for pandas API on Spark

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40327: - Assignee: (was: Ruifeng Zheng) > Increase pandas API coverage for pandas API on

[jira] [Resolved] (SPARK-40503) Add resampling to API references

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40503. --- Fix Version/s: 3.4.0 Target Version/s: 3.4.0 Resolution: Resolved Resolved

[jira] [Resolved] (SPARK-40359) Migrate JSON type check failures onto error classes

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-40359. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37902

[jira] [Assigned] (SPARK-40359) Migrate JSON type check failures onto error classes

2022-09-22 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-40359: Assignee: Max Gekk > Migrate JSON type check failures onto error classes >

[jira] [Resolved] (SPARK-40510) Implement `ddof` in `Series.cov`

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-40510. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 37953

[jira] [Assigned] (SPARK-40510) Implement `ddof` in `Series.cov`

2022-09-22 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-40510: - Assignee: Ruifeng Zheng > Implement `ddof` in `Series.cov` >

[jira] [Commented] (SPARK-40437) Support string representation of durationMs in GroupState.setTimeoutDuration

2022-09-22 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608179#comment-17608179 ] Jungtaek Lim commented on SPARK-40437: -- It doesn't seem to be easy one to solve... We have to do

[jira] [Commented] (SPARK-40438) Support additionalDuration parameter in GroupState.setTimeoutTimestamp

2022-09-22 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608180#comment-17608180 ] Jungtaek Lim commented on SPARK-40438: -- Same comment with SPARK-40437 {quote} It doesn't seem to

[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608174#comment-17608174 ] Dongjoon Hyun commented on SPARK-40490: --- This landed at branch-3.3 via

[jira] [Updated] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

2022-09-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40490: -- Fix Version/s: 3.3.2 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile`

[jira] [Commented] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608151#comment-17608151 ] Apache Spark commented on SPARK-40462: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40462: Assignee: (was: Apache Spark) > Support np.ndarray for functions.lit >

[jira] [Commented] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608150#comment-17608150 ] Apache Spark commented on SPARK-40462: -- User 'itholic' has created a pull request for this issue:

[jira] [Assigned] (SPARK-40462) Support np.ndarray for functions.lit

2022-09-22 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-40462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-40462: Assignee: Apache Spark > Support np.ndarray for functions.lit >

  1   2   >