[jira] [Resolved] (SPARK-38085) DataSource V2: Handle DELETE commands for group-based sources

2022-04-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-38085. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35395

[jira] [Assigned] (SPARK-38085) DataSource V2: Handle DELETE commands for group-based sources

2022-04-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38085: --- Assignee: Anton Okolnychyi > DataSource V2: Handle DELETE commands for group-based sources

[jira] [Commented] (SPARK-38788) More comprehensive DSV2 push down capabilities

2022-04-12 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521458#comment-17521458 ] Max Gekk commented on SPARK-38788: -- [~xkrogen] This epic ticket gets together all related activities in

[jira] [Resolved] (SPARK-38865) Update document of JDBC options for pushDownAggregate and pushDownLimit

2022-04-12 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-38865. Fix Version/s: 3.3.0 3.4.0 Assignee: jiaan.geng Resolution:

[jira] [Assigned] (SPARK-38885) Upgrade netty to 4.1.76

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38885: Assignee: Apache Spark > Upgrade netty to 4.1.76 > --- > >

[jira] [Commented] (SPARK-38885) Upgrade netty to 4.1.76

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521454#comment-17521454 ] Apache Spark commented on SPARK-38885: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-38885) Upgrade netty to 4.1.76

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38885: Assignee: (was: Apache Spark) > Upgrade netty to 4.1.76 > --- >

[jira] [Commented] (SPARK-38885) Upgrade netty to 4.1.76

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521452#comment-17521452 ] Apache Spark commented on SPARK-38885: -- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-38885) Upgrade netty to 4.1.76

2022-04-12 Thread Yang Jie (Jira)
Yang Jie created SPARK-38885: Summary: Upgrade netty to 4.1.76 Key: SPARK-38885 URL: https://issues.apache.org/jira/browse/SPARK-38885 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-38677) pyspark hangs in local mode running rdd map operation

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521446#comment-17521446 ] Apache Spark commented on SPARK-38677: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-38785) impl Series.ewm and DataFrame.ewm

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38785: Assignee: zhengruifeng > impl Series.ewm and DataFrame.ewm >

[jira] [Resolved] (SPARK-38785) impl Series.ewm and DataFrame.ewm

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38785. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36063

[jira] [Updated] (SPARK-38884) java.util.NoSuchElementException: key not found: numPartitions

2022-04-12 Thread chopperChen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chopperChen updated SPARK-38884: Description: When running function spark.sql("sql").isEmpty, the logs print

[jira] [Created] (SPARK-38884) java.util.NoSuchElementException: key not found: numPartitions

2022-04-12 Thread chopperChen (Jira)
chopperChen created SPARK-38884: --- Summary: java.util.NoSuchElementException: key not found: numPartitions Key: SPARK-38884 URL: https://issues.apache.org/jira/browse/SPARK-38884 Project: Spark

[jira] [Commented] (SPARK-37643) when charVarcharAsString is true, char datatype partition table query incorrect

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521434#comment-17521434 ] Apache Spark commented on SPARK-37643: -- User 'fhygh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38863) Implement `skipna` parameter of `DataFrame.all`

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38863: Assignee: Xinrong Meng > Implement `skipna` parameter of `DataFrame.all` >

[jira] [Resolved] (SPARK-38863) Implement `skipna` parameter of `DataFrame.all`

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38863. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36104

[jira] [Assigned] (SPARK-38793) Support `return_indexer` parameter of `Index/MultiIndex.sort_values`

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38793: Assignee: Xinrong Meng > Support `return_indexer` parameter of

[jira] [Resolved] (SPARK-38793) Support `return_indexer` parameter of `Index/MultiIndex.sort_values`

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38793. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36076

[jira] [Commented] (SPARK-38804) Add StreamingQueryManager.removeListener in PySpark

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521427#comment-17521427 ] Apache Spark commented on SPARK-38804: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-38804) Add StreamingQueryManager.removeListener in PySpark

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521423#comment-17521423 ] Apache Spark commented on SPARK-38804: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-38721) Test the error class: CANNOT_PARSE_DECIMAL

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521425#comment-17521425 ] Apache Spark commented on SPARK-38721: -- User 'panbingkun' has created a pull request for this

[jira] [Commented] (SPARK-38721) Test the error class: CANNOT_PARSE_DECIMAL

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521422#comment-17521422 ] Apache Spark commented on SPARK-38721: -- User 'panbingkun' has created a pull request for this

[jira] [Assigned] (SPARK-38721) Test the error class: CANNOT_PARSE_DECIMAL

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38721: Assignee: (was: Apache Spark) > Test the error class: CANNOT_PARSE_DECIMAL >

[jira] [Assigned] (SPARK-38721) Test the error class: CANNOT_PARSE_DECIMAL

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38721: Assignee: Apache Spark > Test the error class: CANNOT_PARSE_DECIMAL >

[jira] [Created] (SPARK-38883) smaller pyspark install if not using streaming?

2022-04-12 Thread t oo (Jira)
t oo created SPARK-38883: Summary: smaller pyspark install if not using streaming? Key: SPARK-38883 URL: https://issues.apache.org/jira/browse/SPARK-38883 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-38823) Incorrect result of dataset reduceGroups in java

2022-04-12 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38823: -- Affects Version/s: 3.3.0 > Incorrect result of dataset reduceGroups in java >

[jira] [Updated] (SPARK-38823) Incorrect result of dataset reduceGroups in java

2022-04-12 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38823: -- Labels: correctness (was: ) > Incorrect result of dataset reduceGroups in java >

[jira] [Commented] (SPARK-38823) Incorrect result of dataset reduceGroups in java

2022-04-12 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521414#comment-17521414 ] Bruce Robbins commented on SPARK-38823: --- This appears to be an optimization bug that results in

[jira] [Commented] (SPARK-38822) Raise indexError when insert loc is out of bounds

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521412#comment-17521412 ] Apache Spark commented on SPARK-38822: -- User 'Yikun' has created a pull request for this issue:

[jira] [Commented] (SPARK-38822) Raise indexError when insert loc is out of bounds

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521411#comment-17521411 ] Apache Spark commented on SPARK-38822: -- User 'Yikun' has created a pull request for this issue:

[jira] [Resolved] (SPARK-38822) Raise indexError when insert loc is out of bounds

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38822. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36115

[jira] [Assigned] (SPARK-38822) Raise indexError when insert loc is out of bounds

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38822: Assignee: Yikun Jiang > Raise indexError when insert loc is out of bounds >

[jira] [Updated] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38879: - Fix Version/s: (was: 3.4.0) > Improve the test coverage for pyspark/rddsampler.py >

[jira] [Assigned] (SPARK-38854) Improve the test coverage for pyspark/statcounter.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38854: Assignee: pralabhkumar (was: Hyukjin Kwon) > Improve the test coverage for

[jira] [Commented] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521398#comment-17521398 ] Hyukjin Kwon commented on SPARK-38879: -- [~pralabhkumar] please just go ahead. no need to ask :-).

[jira] [Assigned] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38879: Assignee: (was: Hyukjin Kwon) > Improve the test coverage for pyspark/rddsampler.py

[jira] [Resolved] (SPARK-38872) CLONE - Improve the test coverage for pyspark/pandas module

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38872. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/pandas module >

[jira] [Resolved] (SPARK-38873) CLONE - Improve the test coverage for pyspark/mllib module

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38873. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/mllib module >

[jira] [Resolved] (SPARK-38874) CLONE - Improve the test coverage for pyspark/ml module

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38874. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/ml module >

[jira] [Resolved] (SPARK-38877) CLONE - Improve the test coverage for pyspark/find_spark_home.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38877. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/find_spark_home.py >

[jira] [Resolved] (SPARK-38875) CLONE - Improve the test coverage for pyspark/sql module

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38875. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/sql module >

[jira] [Resolved] (SPARK-38876) CLONE - Improve the test coverage for pyspark/*.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38876. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/*.py >

[jira] [Resolved] (SPARK-38878) CLONE - Improve the test coverage for pyspark/statcounter.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38878. -- Resolution: Invalid > CLONE - Improve the test coverage for pyspark/statcounter.py >

[jira] [Updated] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38882: - Fix Version/s: 3.3.0 > The usage logger attachment logic should handle static methods properly.

[jira] [Resolved] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38882. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/36167 > The usage logger

[jira] [Commented] (SPARK-38788) More comprehensive DSV2 push down capabilities

2022-04-12 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521363#comment-17521363 ] Erik Krogen commented on SPARK-38788: - What's the relationship between this and SPARK-38852? Seems

[jira] [Commented] (SPARK-38852) Better Data Source V2 operator pushdown framework

2022-04-12 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521364#comment-17521364 ] Erik Krogen commented on SPARK-38852: - What's the relationship between this and SPARK-38788? Seems

[jira] [Assigned] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38882: Assignee: Apache Spark > The usage logger attachment logic should handle static methods

[jira] [Assigned] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38882: Assignee: (was: Apache Spark) > The usage logger attachment logic should handle

[jira] [Commented] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521355#comment-17521355 ] Apache Spark commented on SPARK-38882: -- User 'ueshin' has created a pull request for this issue:

[jira] [Created] (SPARK-38882) The usage logger attachment logic should handle static methods properly.

2022-04-12 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-38882: - Summary: The usage logger attachment logic should handle static methods properly. Key: SPARK-38882 URL: https://issues.apache.org/jira/browse/SPARK-38882 Project:

[jira] [Updated] (SPARK-38792) Regression in time executor takes to do work sometime after v3.0.1 ?

2022-04-12 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Guinther updated SPARK-38792: --- Attachment: what-is-this-code.jpg > Regression in time executor takes to do work sometime

[jira] [Commented] (SPARK-38792) Regression in time executor takes to do work sometime after v3.0.1 ?

2022-04-12 Thread Danny Guinther (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521342#comment-17521342 ] Danny Guinther commented on SPARK-38792: Where does

[jira] [Assigned] (SPARK-38767) Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options

2022-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38767: - Assignee: Yaohua Zhao (was: Yaohua Cui) > Support ignoreCorruptFiles and

[jira] [Resolved] (SPARK-38767) Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options

2022-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38767. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36069

[jira] [Assigned] (SPARK-38767) Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options

2022-04-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38767: - Assignee: Yaohua Cui > Support ignoreCorruptFiles and ignoreMissingFiles in Data

[jira] [Assigned] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38881: Assignee: Apache Spark > PySpark Kinesis Streaming should expose metricsLevel CloudWatch

[jira] [Updated] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-12 Thread Mark Khaitman (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Khaitman updated SPARK-38881: -- Description: This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was

[jira] [Commented] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521293#comment-17521293 ] Apache Spark commented on SPARK-38881: -- User 'mkman84' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38881: Assignee: (was: Apache Spark) > PySpark Kinesis Streaming should expose metricsLevel

[jira] [Created] (SPARK-38881) PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs

2022-04-12 Thread Mark Khaitman (Jira)
Mark Khaitman created SPARK-38881: - Summary: PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that is already supported in the Scala/Java APIs Key: SPARK-38881 URL:

[jira] [Commented] (SPARK-36620) Client side related push-based shuffle metrics

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521277#comment-17521277 ] Apache Spark commented on SPARK-36620: -- User 'thejdeep' has created a pull request for this issue:

[jira] [Commented] (SPARK-36620) Client side related push-based shuffle metrics

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521276#comment-17521276 ] Apache Spark commented on SPARK-36620: -- User 'thejdeep' has created a pull request for this issue:

[jira] [Commented] (SPARK-38880) Implement `numeric_only` parameter of `GroupBy.max/min`

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521245#comment-17521245 ] Apache Spark commented on SPARK-38880: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38880) Implement `numeric_only` parameter of `GroupBy.max/min`

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38880: Assignee: Apache Spark > Implement `numeric_only` parameter of `GroupBy.max/min` >

[jira] [Commented] (SPARK-38880) Implement `numeric_only` parameter of `GroupBy.max/min`

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521244#comment-17521244 ] Apache Spark commented on SPARK-38880: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38880) Implement `numeric_only` parameter of `GroupBy.max/min`

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38880: Assignee: (was: Apache Spark) > Implement `numeric_only` parameter of

[jira] [Created] (SPARK-38880) Implement `numeric_only` parameter of `GroupBy.max/min`

2022-04-12 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38880: Summary: Implement `numeric_only` parameter of `GroupBy.max/min` Key: SPARK-38880 URL: https://issues.apache.org/jira/browse/SPARK-38880 Project: Spark

[jira] [Assigned] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38689: Assignee: (was: Apache Spark) > Use error classes in the compilation errors of not

[jira] [Commented] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521227#comment-17521227 ] Apache Spark commented on SPARK-38689: -- User 'ivoson' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38689) Use error classes in the compilation errors of not allowed DESC PARTITION

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38689: Assignee: Apache Spark > Use error classes in the compilation errors of not allowed DESC

[jira] [Assigned] (SPARK-38847) Introduce a `viewToSeq` function for `KVUtils`

2022-04-12 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-38847: Assignee: Yang Jie > Introduce a `viewToSeq` function for `KVUtils` >

[jira] [Resolved] (SPARK-38847) Introduce a `viewToSeq` function for `KVUtils`

2022-04-12 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38847. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36132

[jira] [Resolved] (SPARK-38848) Replcace all `@Test(expected = XXException)` with assertThrows

2022-04-12 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38848. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36133

[jira] [Assigned] (SPARK-38848) Replcace all `@Test(expected = XXException)` with assertThrows

2022-04-12 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-38848: Assignee: Yang Jie > Replcace all `@Test(expected = XXException)` with assertThrows >

[jira] [Created] (SPARK-38874) CLONE - Improve the test coverage for pyspark/ml module

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38874: Summary: CLONE - Improve the test coverage for pyspark/ml module Key: SPARK-38874 URL: https://issues.apache.org/jira/browse/SPARK-38874 Project: Spark

[jira] [Closed] (SPARK-38871) Improve the test coverage for PySpark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar closed SPARK-38871. This issue is wrongly created , hence closing it  > Improve the test coverage for

[jira] [Resolved] (SPARK-38871) Improve the test coverage for PySpark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar resolved SPARK-38871. -- Resolution: Invalid > Improve the test coverage for PySpark/rddsampler.py >

[jira] [Commented] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521135#comment-17521135 ] pralabhkumar commented on SPARK-38879: -- I will be working on this .  > Improve the test coverage

[jira] [Comment Edited] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521135#comment-17521135 ] pralabhkumar edited comment on SPARK-38879 at 4/12/22 1:07 PM: --- Please

[jira] [Updated] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-38879: - Description: Improve the test coverage of rddsampler.py (was: Improve the test coverage of

[jira] [Created] (SPARK-38879) Improve the test coverage for pyspark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38879: Summary: Improve the test coverage for pyspark/rddsampler.py Key: SPARK-38879 URL: https://issues.apache.org/jira/browse/SPARK-38879 Project: Spark Issue

[jira] [Commented] (SPARK-38871) Improve the test coverage for PySpark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521134#comment-17521134 ] pralabhkumar commented on SPARK-38871: -- Please close this one , wrongly cloned  > Improve the test

[jira] [Created] (SPARK-38876) CLONE - Improve the test coverage for pyspark/*.py

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38876: Summary: CLONE - Improve the test coverage for pyspark/*.py Key: SPARK-38876 URL: https://issues.apache.org/jira/browse/SPARK-38876 Project: Spark Issue

[jira] [Created] (SPARK-38872) CLONE - Improve the test coverage for pyspark/pandas module

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38872: Summary: CLONE - Improve the test coverage for pyspark/pandas module Key: SPARK-38872 URL: https://issues.apache.org/jira/browse/SPARK-38872 Project: Spark

[jira] [Created] (SPARK-38878) CLONE - Improve the test coverage for pyspark/statcounter.py

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38878: Summary: CLONE - Improve the test coverage for pyspark/statcounter.py Key: SPARK-38878 URL: https://issues.apache.org/jira/browse/SPARK-38878 Project: Spark

[jira] [Created] (SPARK-38875) CLONE - Improve the test coverage for pyspark/sql module

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38875: Summary: CLONE - Improve the test coverage for pyspark/sql module Key: SPARK-38875 URL: https://issues.apache.org/jira/browse/SPARK-38875 Project: Spark

[jira] [Created] (SPARK-38877) CLONE - Improve the test coverage for pyspark/find_spark_home.py

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38877: Summary: CLONE - Improve the test coverage for pyspark/find_spark_home.py Key: SPARK-38877 URL: https://issues.apache.org/jira/browse/SPARK-38877 Project: Spark

[jira] [Created] (SPARK-38871) Improve the test coverage for PySpark/rddsampler.py

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38871: Summary: Improve the test coverage for PySpark/rddsampler.py Key: SPARK-38871 URL: https://issues.apache.org/jira/browse/SPARK-38871 Project: Spark Issue

[jira] [Created] (SPARK-38873) CLONE - Improve the test coverage for pyspark/mllib module

2022-04-12 Thread pralabhkumar (Jira)
pralabhkumar created SPARK-38873: Summary: CLONE - Improve the test coverage for pyspark/mllib module Key: SPARK-38873 URL: https://issues.apache.org/jira/browse/SPARK-38873 Project: Spark

[jira] [Resolved] (SPARK-38589) New SQL function: try_avg

2022-04-12 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38589. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35896

[jira] [Resolved] (SPARK-38854) Improve the test coverage for pyspark/statcounter.py

2022-04-12 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38854. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 36145

[jira] [Commented] (SPARK-32170) Improve the speculation for the inefficient tasks by the task metrics.

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521084#comment-17521084 ] Apache Spark commented on SPARK-32170: -- User 'weixiuli' has created a pull request for this issue:

[jira] [Commented] (SPARK-38870) SparkSession.builder returns a new builder in Scala, but not in Python

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521064#comment-17521064 ] Apache Spark commented on SPARK-38870: -- User 'FurcyPin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38870) SparkSession.builder returns a new builder in Scala, but not in Python

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38870: Assignee: Apache Spark > SparkSession.builder returns a new builder in Scala, but not in

[jira] [Assigned] (SPARK-38870) SparkSession.builder returns a new builder in Scala, but not in Python

2022-04-12 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38870: Assignee: (was: Apache Spark) > SparkSession.builder returns a new builder in Scala,

[jira] [Updated] (SPARK-38870) SparkSession.builder returns a new builder in Scala, but not in Python

2022-04-12 Thread Furcy Pin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Furcy Pin updated SPARK-38870: -- Description: In pyspark, _SparkSession.builder_ always returns the same static builder, while the

[jira] [Updated] (SPARK-38870) SparkSession.builder returns a new builder in Scala, but not in Python

2022-04-12 Thread Furcy Pin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Furcy Pin updated SPARK-38870: -- Description: In pyspark, _SparkSession.builder_ always returns the same static builder, while the

[jira] [Created] (SPARK-38870) SparkSession.builder returns a new builder in Scala, but not in Python

2022-04-12 Thread Furcy Pin (Jira)
Furcy Pin created SPARK-38870: - Summary: SparkSession.builder returns a new builder in Scala, but not in Python Key: SPARK-38870 URL: https://issues.apache.org/jira/browse/SPARK-38870 Project: Spark

  1   2   >