[jira] [Assigned] (SPARK-38498) Support add StreamingListener by conf

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38498: Assignee: Apache Spark > Support add StreamingListener by conf >

[jira] [Commented] (SPARK-38498) Support add StreamingListener by conf

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504054#comment-17504054 ] Apache Spark commented on SPARK-38498: -- User 'AngersZh' has created a pull request for this

[jira] [Assigned] (SPARK-38498) Support add StreamingListener by conf

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38498: Assignee: (was: Apache Spark) > Support add StreamingListener by conf >

[jira] [Updated] (SPARK-38498) Support add StreamingListener by conf

2022-03-09 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-38498: -- Description: Currently, if user want to add an customized StreamingListener to StreamingContext, we

[jira] [Assigned] (SPARK-38451) Fix R tests under ANSI mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38451: Assignee: Hyukjin Kwon (was: Apache Spark) > Fix R tests under ANSI mode >

[jira] [Commented] (SPARK-38451) Fix R tests under ANSI mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504033#comment-17504033 ] Apache Spark commented on SPARK-38451: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-38451) Fix R tests under ANSI mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38451: Assignee: Apache Spark (was: Hyukjin Kwon) > Fix R tests under ANSI mode >

[jira] [Commented] (SPARK-38101) MetadataFetchFailedException due to decommission block migrations

2022-03-09 Thread Emil Ejbyfeldt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504028#comment-17504028 ] Emil Ejbyfeldt commented on SPARK-38101: The race condition only exists when broadcast is used

[jira] [Created] (SPARK-38498) Support add StreamingListener by conf

2022-03-09 Thread angerszhu (Jira)
angerszhu created SPARK-38498: - Summary: Support add StreamingListener by conf Key: SPARK-38498 URL: https://issues.apache.org/jira/browse/SPARK-38498 Project: Spark Issue Type: Task

[jira] [Resolved] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38487. -- Fix Version/s: 3.3.0 3.2.2 Assignee: Xinrong Meng

[jira] [Comment Edited] (SPARK-9213) Improve regular expression performance (via joni)

2022-03-09 Thread tonydoen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504016#comment-17504016 ] tonydoen edited comment on SPARK-9213 at 3/10/22, 5:58 AM: --- [~rxin] [~waterman]

[jira] [Commented] (SPARK-38492) Improve the test coverage for PySpark

2022-03-09 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504017#comment-17504017 ] Haejoon Lee commented on SPARK-38492: - Each sub-task again contains several Python files. So, if

[jira] [Comment Edited] (SPARK-9213) Improve regular expression performance (via joni)

2022-03-09 Thread tonydoen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504016#comment-17504016 ] tonydoen edited comment on SPARK-9213 at 3/10/22, 5:58 AM: --- [~rxin] [~waterman]

[jira] [Commented] (SPARK-9213) Improve regular expression performance (via joni)

2022-03-09 Thread tonydoen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504016#comment-17504016 ] tonydoen commented on SPARK-9213: - [~rxin] [~waterman] [~mridulm80]  > Improve regular expression

[jira] [Comment Edited] (SPARK-9213) Improve regular expression performance (via joni)

2022-03-09 Thread tonydoen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504016#comment-17504016 ] tonydoen edited comment on SPARK-9213 at 3/10/22, 5:53 AM: --- [~rxin] [~waterman]

[jira] [Created] (SPARK-38497) Improve the test coverage for pyspark/*.py

2022-03-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-38497: --- Summary: Improve the test coverage for pyspark/*.py Key: SPARK-38497 URL: https://issues.apache.org/jira/browse/SPARK-38497 Project: Spark Issue Type:

[jira] [Created] (SPARK-38496) Improve the test coverage for pyspark/sql module

2022-03-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-38496: --- Summary: Improve the test coverage for pyspark/sql module Key: SPARK-38496 URL: https://issues.apache.org/jira/browse/SPARK-38496 Project: Spark Issue Type:

[jira] [Created] (SPARK-38495) Improve the test coverage for pyspark/ml module

2022-03-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-38495: --- Summary: Improve the test coverage for pyspark/ml module Key: SPARK-38495 URL: https://issues.apache.org/jira/browse/SPARK-38495 Project: Spark Issue Type:

[jira] [Created] (SPARK-38494) Improve the test coverage for pyspark/mllib module

2022-03-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-38494: --- Summary: Improve the test coverage for pyspark/mllib module Key: SPARK-38494 URL: https://issues.apache.org/jira/browse/SPARK-38494 Project: Spark Issue Type:

[jira] [Created] (SPARK-38493) Improve the test coverage for pyspark/pandas module

2022-03-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-38493: --- Summary: Improve the test coverage for pyspark/pandas module Key: SPARK-38493 URL: https://issues.apache.org/jira/browse/SPARK-38493 Project: Spark Issue

[jira] [Created] (SPARK-38492) Improve the test coverage for PySpark

2022-03-09 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-38492: --- Summary: Improve the test coverage for PySpark Key: SPARK-38492 URL: https://issues.apache.org/jira/browse/SPARK-38492 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38379: -- Fix Version/s: (was: 3.2.2) > Kubernetes: NoSuchElementException: spark.app.id when using

[jira] [Updated] (SPARK-38455) Support driver/executor PodGroup templates

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38455: -- Parent: SPARK-36057 Issue Type: Sub-task (was: Improvement) > Support

[jira] [Resolved] (SPARK-38429) Why the latest release named spark-3.2.1-bin-hadoop3.2.tgz when I choose hadoop 3.3.1

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38429. -- Resolution: Duplicate It's fixed in SPARK-37445 > Why the latest release named

[jira] [Updated] (SPARK-38480) Remove spark.kubernetes.job.queue in favor of spark.kubernetes.driver.podGroupTemplateFile

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38480: -- Parent: SPARK-36057 Issue Type: Sub-task (was: Improvement) > Remove

[jira] [Updated] (SPARK-38453) Add volcano section to K8s IT README.md

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38453: -- Summary: Add volcano section to K8s IT README.md (was: Make Volcano IT can be completed in

[jira] [Resolved] (SPARK-38453) Make Volcano IT can be completed in 4U6G env

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38453. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35773

[jira] [Assigned] (SPARK-38453) Make Volcano IT can be completed in 4U6G env

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38453: - Assignee: Yikun Jiang > Make Volcano IT can be completed in 4U6G env >

[jira] [Commented] (SPARK-38490) Add Github action test job for ANSI SQL mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503991#comment-17503991 ] Apache Spark commented on SPARK-38490: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503992#comment-17503992 ] Apache Spark commented on SPARK-38491: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38490) Add Github action test job for ANSI SQL mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38490: Assignee: Gengliang Wang (was: Apache Spark) > Add Github action test job for ANSI SQL

[jira] [Commented] (SPARK-38490) Add Github action test job for ANSI SQL mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503993#comment-17503993 ] Apache Spark commented on SPARK-38490: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-38148) Do not add dynamic partition pruning if there exists static partition pruning

2022-03-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-38148: --- Assignee: XiDuo You > Do not add dynamic partition pruning if there exists static

[jira] [Commented] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503994#comment-17503994 ] Apache Spark commented on SPARK-38491: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38491: Assignee: Apache Spark > Support `ignore_index` of `Series.sort_values` >

[jira] [Assigned] (SPARK-38490) Add Github action test job for ANSI SQL mode

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38490: Assignee: Apache Spark (was: Gengliang Wang) > Add Github action test job for ANSI SQL

[jira] [Resolved] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38379. --- Fix Version/s: 3.3.0 3.2.2 Assignee: Thomas Graves

[jira] [Assigned] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38491: Assignee: (was: Apache Spark) > Support `ignore_index` of `Series.sort_values` >

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Description: {code:scala} spark.range(200L).selectExpr("id AS a").createTempView("t1")

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Description: {code:scala} spark.range(200L).selectExpr("id AS a").createTempView("t1")

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Description: {code:scala} spark.range(200L).selectExpr("id AS a").createTempView("t1")

[jira] [Updated] (SPARK-38439) Add Braces with if,else,for,do and while statements

2022-03-09 Thread qian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qian updated SPARK-38439: - Priority: Trivial (was: Minor) > Add Braces with if,else,for,do and while statements >

[jira] [Commented] (SPARK-38439) Add Braces with if,else,for,do and while statements

2022-03-09 Thread qian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503989#comment-17503989 ] qian commented on SPARK-38439: -- This is useless. Please ignore it > Add Braces with if,else,for,do and

[jira] [Created] (SPARK-38491) Support `ignore_index` of `Series.sort_values`

2022-03-09 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38491: Summary: Support `ignore_index` of `Series.sort_values` Key: SPARK-38491 URL: https://issues.apache.org/jira/browse/SPARK-38491 Project: Spark Issue Type:

[jira] [Created] (SPARK-38490) Add Github action test job for ANSI SQL mode

2022-03-09 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-38490: -- Summary: Add Github action test job for ANSI SQL mode Key: SPARK-38490 URL: https://issues.apache.org/jira/browse/SPARK-38490 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37853) Clean up deprecation compilation warning related to log4j2

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503988#comment-17503988 ] Apache Spark commented on SPARK-37853: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38489: Assignee: (was: Apache Spark) > Aggregate.groupOnly support foldable expressions >

[jira] [Commented] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503984#comment-17503984 ] Apache Spark commented on SPARK-38489: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38489: Assignee: Apache Spark > Aggregate.groupOnly support foldable expressions >

[jira] [Commented] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503978#comment-17503978 ] Yikun Jiang commented on SPARK-38488: - [~hyukjin.kwon] Sure, let me check and test it both in

[jira] [Resolved] (SPARK-38457) spark.sql.catalyst.analysis

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38457. -- Resolution: Invalid no response from the author. > spark.sql.catalyst.analysis >

[jira] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353 ] Hyukjin Kwon deleted comment on SPARK-38353: -- was (Author: apachespark): User 'heyihong' has created a pull request for this issue: https://github.com/apache/spark/pull/35790 > Instrument

[jira] [Updated] (SPARK-38454) Partition Data Type Prevents Filtering Sporadically

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38454: - Component/s: SQL (was: PySpark) > Partition Data Type Prevents Filtering

[jira] [Commented] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503962#comment-17503962 ] Apache Spark commented on SPARK-38484: -- User 'heyihong' has created a pull request for this issue:

[jira] [Updated] (SPARK-38454) Partition Data Type Prevents Filtering Sporadically

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38454: - Priority: Major (was: Critical) > Partition Data Type Prevents Filtering Sporadically >

[jira] [Assigned] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38484: Assignee: (was: Apache Spark) > Move usage logging instrumentation util functions

[jira] [Commented] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503963#comment-17503963 ] Hyukjin Kwon commented on SPARK-38483: -- The real column names can only be known after resolving it

[jira] [Assigned] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38484: Assignee: Apache Spark > Move usage logging instrumentation util functions from pandas

[jira] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353 ] Hyukjin Kwon deleted comment on SPARK-38353: -- was (Author: apachespark): User 'heyihong' has created a pull request for this issue: https://github.com/apache/spark/pull/35790 > Instrument

[jira] [Commented] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503960#comment-17503960 ] Hyukjin Kwon commented on SPARK-38488: -- [~yikunkero] please go ahead for a PR. > Spark doc build

[jira] [Created] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38489: --- Summary: Aggregate.groupOnly support foldable expressions Key: SPARK-38489 URL: https://issues.apache.org/jira/browse/SPARK-38489 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38489: Fix Version/s: (was: 3.3.0) > Aggregate.groupOnly support foldable expressions >

[jira] [Assigned] (SPARK-38489) Aggregate.groupOnly support foldable expressions

2022-03-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-38489: --- Assignee: (was: Yuming Wang) > Aggregate.groupOnly support foldable expressions >

[jira] [Updated] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang updated SPARK-38488: Description:   {code:java} diff --git a/docs/.bundle/config b/docs/.bundle/config index

[jira] [Created] (SPARK-38488) Spark doc build not work on Mac OS M1

2022-03-09 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38488: --- Summary: Spark doc build not work on Mac OS M1 Key: SPARK-38488 URL: https://issues.apache.org/jira/browse/SPARK-38488 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503945#comment-17503945 ] Apache Spark commented on SPARK-38487: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38487: Assignee: Apache Spark > Fix docstrings of nlargest/nsmallest of DataFrame >

[jira] [Assigned] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38487: Assignee: (was: Apache Spark) > Fix docstrings of nlargest/nsmallest of DataFrame >

[jira] [Commented] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503943#comment-17503943 ] Apache Spark commented on SPARK-38487: -- User 'xinrong-databricks' has created a pull request for

[jira] [Created] (SPARK-38487) Fix docstrings of nlargest/nsmallest of DataFrame

2022-03-09 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38487: Summary: Fix docstrings of nlargest/nsmallest of DataFrame Key: SPARK-38487 URL: https://issues.apache.org/jira/browse/SPARK-38487 Project: Spark Issue

[jira] [Assigned] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38379: Assignee: (was: Apache Spark) > Kubernetes: NoSuchElementException: spark.app.id

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503938#comment-17503938 ] Apache Spark commented on SPARK-38379: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38379: Assignee: Apache Spark > Kubernetes: NoSuchElementException: spark.app.id when using

[jira] [Resolved] (SPARK-36060) Support backing off dynamic allocation increases if resources are "stuck"

2022-03-09 Thread Yikun Jiang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yikun Jiang resolved SPARK-36060. - Resolution: Fixed Now minRes and queue supported have been supported by

[jira] [Resolved] (SPARK-38387) Support `na_action` and Series input correspondence in `Series.map`

2022-03-09 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-38387. --- Fix Version/s: 3.3.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Resolved] (SPARK-38355) Change mktemp() to mkstemp()

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38355. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35775

[jira] [Assigned] (SPARK-38355) Change mktemp() to mkstemp()

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38355: Assignee: Bjørn Jørgensen > Change mktemp() to mkstemp() >

[jira] [Commented] (SPARK-38457) spark.sql.catalyst.analysis

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503928#comment-17503928 ] Hyukjin Kwon commented on SPARK-38457: -- Spark 2.X is EOL. Can you see if that bug persists in Spark

[jira] [Updated] (SPARK-38457) spark.sql.catalyst.analysis

2022-03-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-38457: - Priority: Major (was: Blocker) > spark.sql.catalyst.analysis > --- > >

[jira] [Resolved] (SPARK-38486) Upgrade the minimum Minikube version to 1.18.0

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38486. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35791

[jira] [Commented] (SPARK-32268) Bloom Filter Join

2022-03-09 Thread Abhishek Somani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503876#comment-17503876 ] Abhishek Somani commented on SPARK-32268: - We have created a Design doc and PR for this Design

[jira] [Commented] (SPARK-32268) Bloom Filter Join

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503874#comment-17503874 ] Apache Spark commented on SPARK-32268: -- User 'somani' has created a pull request for this issue:

[jira] [Commented] (SPARK-32268) Bloom Filter Join

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503875#comment-17503875 ] Apache Spark commented on SPARK-32268: -- User 'somani' has created a pull request for this issue:

[jira] [Commented] (SPARK-38486) Upgrade the minimum Minikube version to 1.18.0

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503804#comment-17503804 ] Apache Spark commented on SPARK-38486: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Description: This JIRA aims to improve K8s integration tests for the following recent

[jira] [Assigned] (SPARK-38486) Upgrade the minimum Minikube version to 1.18.0

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38486: - Assignee: Dongjoon Hyun > Upgrade the minimum Minikube version to 1.18.0 >

[jira] [Created] (SPARK-38486) Upgrade the minimum Minikube version to 1.18.0

2022-03-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38486: - Summary: Upgrade the minimum Minikube version to 1.18.0 Key: SPARK-38486 URL: https://issues.apache.org/jira/browse/SPARK-38486 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-38101) MetadataFetchFailedException due to decommission block migrations

2022-03-09 Thread Igor Dvorzhak (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503775#comment-17503775 ] Igor Dvorzhak edited comment on SPARK-38101 at 3/9/22, 6:42 PM: Is there

[jira] [Commented] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-09 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503782#comment-17503782 ] Brian Schaefer commented on SPARK-38483: Extracting the column name from the {{Column.__repr__}}

[jira] [Commented] (SPARK-38101) MetadataFetchFailedException due to decommission block migrations

2022-03-09 Thread Igor Dvorzhak (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503775#comment-17503775 ] Igor Dvorzhak commented on SPARK-38101: --- Are there any workaround for this issue? >

[jira] [Updated] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-09 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38484: -- Description: It will be helpful to attach the usage logger to other modules (e.g. sql) besides Pandas

[jira] [Updated] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-09 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38484: -- Description: It will be helpful to attach the usage logger to other modules (e.g. sql) besides Pandas

[jira] [Commented] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503771#comment-17503771 ] Apache Spark commented on SPARK-38353: -- User 'heyihong' has created a pull request for this issue:

[jira] [Commented] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-09 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503770#comment-17503770 ] Apache Spark commented on SPARK-38353: -- User 'heyihong' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38480) Remove spark.kubernetes.job.queue in favor of spark.kubernetes.driver.podGroupTemplateFile

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38480: - Assignee: Dongjoon Hyun > Remove spark.kubernetes.job.queue in favor of >

[jira] [Resolved] (SPARK-38480) Remove spark.kubernetes.job.queue in favor of spark.kubernetes.driver.podGroupTemplateFile

2022-03-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38480. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35783

[jira] [Updated] (SPARK-38485) Non-deterministic UDF executed multiple times when combined with withField

2022-03-09 Thread Tanel Kiis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanel Kiis updated SPARK-38485: --- Description: When adding fields to a result of a non-deterministic UDF, that returns a struct,

[jira] [Created] (SPARK-38485) Non-deterministic UDF executed multiple times when combined with withField

2022-03-09 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-38485: -- Summary: Non-deterministic UDF executed multiple times when combined with withField Key: SPARK-38485 URL: https://issues.apache.org/jira/browse/SPARK-38485 Project:

[jira] [Created] (SPARK-38484) Move usage logging instrumentation util functions from pandas module to pyspark.util module

2022-03-09 Thread Yihong He (Jira)
Yihong He created SPARK-38484: - Summary: Move usage logging instrumentation util functions from pandas module to pyspark.util module Key: SPARK-38484 URL: https://issues.apache.org/jira/browse/SPARK-38484

[jira] [Updated] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-09 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brian Schaefer updated SPARK-38483: --- Description: Having the name of a column as an attribute of PySpark {{Column}} class

  1   2   >