[jira] [Assigned] (SPARK-38521) Throw Exception if overwriting hive partition table with dynamic and staticPartitionOverwriteMode

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38521: Assignee: (was: Apache Spark) > Throw Exception if overwriting hive partition table

[jira] [Commented] (SPARK-38521) Throw Exception if overwriting hive partition table with dynamic and staticPartitionOverwriteMode

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504771#comment-17504771 ] Apache Spark commented on SPARK-38521: -- User 'jackylee-ch' has created a pull request for this

[jira] [Assigned] (SPARK-38521) Throw Exception if overwriting hive partition table with dynamic and staticPartitionOverwriteMode

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38521: Assignee: Apache Spark > Throw Exception if overwriting hive partition table with

[jira] [Created] (SPARK-38522) Strengthen the contract on iterator method in StateStore

2022-03-10 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-38522: Summary: Strengthen the contract on iterator method in StateStore Key: SPARK-38522 URL: https://issues.apache.org/jira/browse/SPARK-38522 Project: Spark

[jira] [Resolved] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38509. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35805

[jira] [Assigned] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-38509: Assignee: Max Gekk > Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF >

[jira] [Created] (SPARK-38521) Throw Exception if overwriting hive partition table with dynamic and staticPartitionOverwriteMode

2022-03-10 Thread Jackey Lee (Jira)
Jackey Lee created SPARK-38521: -- Summary: Throw Exception if overwriting hive partition table with dynamic and staticPartitionOverwriteMode Key: SPARK-38521 URL: https://issues.apache.org/jira/browse/SPARK-38521

[jira] [Resolved] (SPARK-38502) Distribution with hadoop-provided is missing log4j2

2022-03-10 Thread Emil Ejbyfeldt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emil Ejbyfeldt resolved SPARK-38502. Resolution: Duplicate > Distribution with hadoop-provided is missing log4j2 >

[jira] [Commented] (SPARK-38502) Distribution with hadoop-provided is missing log4j2

2022-03-10 Thread Emil Ejbyfeldt (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504749#comment-17504749 ] Emil Ejbyfeldt commented on SPARK-38502: Duplicate of SPARK-38516 > Distribution with

[jira] [Created] (SPARK-38520) Overflow occurs when reading ANSI day time interval from CSV file

2022-03-10 Thread chong (Jira)
chong created SPARK-38520: - Summary: Overflow occurs when reading ANSI day time interval from CSV file Key: SPARK-38520 URL: https://issues.apache.org/jira/browse/SPARK-38520 Project: Spark Issue

[jira] [Assigned] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38519: Assignee: (was: Apache Spark) > AQE throw exception should respect

[jira] [Commented] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504734#comment-17504734 ] Apache Spark commented on SPARK-38519: -- User 'ulysses-you' has created a pull request for this

[jira] [Assigned] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38519: Assignee: Apache Spark > AQE throw exception should respect SparkFatalException >

[jira] [Updated] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38519: -- Description: BroadcastExchangeExec will wrap fatal exception inside SparkFatalException and unwarp

[jira] [Updated] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38519: -- Description: BroadcastExchangeExec will wrap fatal exception in SparkFatalException and unwarp it 

[jira] [Updated] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiDuo You updated SPARK-38519: -- Description: BroadcastExchangeExec will wrap fatal exception in SparkFatalException and unwarp it in

[jira] [Created] (SPARK-38519) AQE throw exception should respect SparkFatalException

2022-03-10 Thread XiDuo You (Jira)
XiDuo You created SPARK-38519: - Summary: AQE throw exception should respect SparkFatalException Key: SPARK-38519 URL: https://issues.apache.org/jira/browse/SPARK-38519 Project: Spark Issue Type:

[jira] [Updated] (SPARK-37273) Hidden File Metadata Support for Spark SQL

2022-03-10 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-37273: Labels: release-notes (was: ) > Hidden File Metadata Support for Spark SQL >

[jira] [Commented] (SPARK-38518) Implement `skipna` of `Series.all/Index.all` to exclude NA/null values

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504721#comment-17504721 ] Apache Spark commented on SPARK-38518: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38518) Implement `skipna` of `Series.all/Index.all` to exclude NA/null values

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38518: Assignee: (was: Apache Spark) > Implement `skipna` of `Series.all/Index.all` to

[jira] [Assigned] (SPARK-38518) Implement `skipna` of `Series.all/Index.all` to exclude NA/null values

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38518: Assignee: Apache Spark > Implement `skipna` of `Series.all/Index.all` to exclude NA/null

[jira] [Created] (SPARK-38518) Implement `skipna` of `Series.all/Index.all` to exclude NA/null values

2022-03-10 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38518: Summary: Implement `skipna` of `Series.all/Index.all` to exclude NA/null values Key: SPARK-38518 URL: https://issues.apache.org/jira/browse/SPARK-38518 Project:

[jira] [Updated] (SPARK-38516) Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided

2022-03-10 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-38516: Summary: Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided

[jira] [Commented] (SPARK-38507) DataFrame withColumn method not adding or replacing columns when alias is used

2022-03-10 Thread qian (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504703#comment-17504703 ] qian commented on SPARK-38507: -- Hi [~amavrommatis] The reason for this problem is because you alias

[jira] [Assigned] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38511: - Assignee: Dongjoon Hyun > Remove priorityClassName propagation in favor of explicit

[jira] [Resolved] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38511. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35807

[jira] [Assigned] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-38517: Assignee: Hyukjin Kwon > Fix PySpark documentation generation (missing ipython_genutils)

[jira] [Resolved] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-38517. -- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504685#comment-17504685 ] Apache Spark commented on SPARK-38517: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38517: Assignee: (was: Apache Spark) > Fix PySpark documentation generation (missing

[jira] [Assigned] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38517: Assignee: Apache Spark > Fix PySpark documentation generation (missing ipython_genutils)

[jira] [Commented] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504684#comment-17504684 ] Apache Spark commented on SPARK-38517: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-38516) Add log4j-core and log4j-api to classpath if active hadoop-provided

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504682#comment-17504682 ] Apache Spark commented on SPARK-38516: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38516) Add log4j-core and log4j-api to classpath if active hadoop-provided

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38516: Assignee: (was: Apache Spark) > Add log4j-core and log4j-api to classpath if active

[jira] [Created] (SPARK-38517) Fix PySpark documentation generation (missing ipython_genutils)

2022-03-10 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-38517: Summary: Fix PySpark documentation generation (missing ipython_genutils) Key: SPARK-38517 URL: https://issues.apache.org/jira/browse/SPARK-38517 Project: Spark

[jira] [Assigned] (SPARK-38516) Add log4j-core and log4j-api to classpath if active hadoop-provided

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38516: Assignee: Apache Spark > Add log4j-core and log4j-api to classpath if active

[jira] [Commented] (SPARK-38516) Add log4j-core and log4j-api to classpath if active hadoop-provided

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504683#comment-17504683 ] Apache Spark commented on SPARK-38516: -- User 'wangyum' has created a pull request for this issue:

[jira] [Created] (SPARK-38516) Add log4j-core and log4j-api to classpath if active hadoop-provided

2022-03-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38516: --- Summary: Add log4j-core and log4j-api to classpath if active hadoop-provided Key: SPARK-38516 URL: https://issues.apache.org/jira/browse/SPARK-38516 Project: Spark

[jira] [Updated] (SPARK-38515) Volcano queue is not deleted

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38515: -- Priority: Critical (was: Blocker) > Volcano queue is not deleted >

[jira] [Updated] (SPARK-38515) Volcano test fails at deleting queue

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38515: -- Description: {code} $ k delete queue queue0 Error from server: admission webhook

[jira] [Updated] (SPARK-38515) Volcano queue is not deleted

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38515: -- Priority: Blocker (was: Major) > Volcano queue is not deleted >

[jira] [Updated] (SPARK-38515) Volcano test fails at deleting queue

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38515: -- Component/s: (was: Tests) > Volcano test fails at deleting queue >

[jira] [Updated] (SPARK-38515) Volcano queue is not deleted

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38515: -- Summary: Volcano queue is not deleted (was: Volcano test fails at deleting queue) > Volcano

[jira] [Commented] (SPARK-38515) Volcano test fails at deleting queue

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504670#comment-17504670 ] Dongjoon Hyun commented on SPARK-38515: --- cc [~yikunkero] this is happening Intel architecture EKS.

[jira] [Created] (SPARK-38515) Volcano test fails at deleting queue

2022-03-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38515: - Summary: Volcano test fails at deleting queue Key: SPARK-38515 URL: https://issues.apache.org/jira/browse/SPARK-38515 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38513. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35809

[jira] [Assigned] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38513: - Assignee: Dongjoon Hyun > Move custom scheduler-specific configs to under >

[jira] [Created] (SPARK-38514) Download link on spark 3.2.1 for hadoop 3.2 is wrong

2022-03-10 Thread Brett Ryan (Jira)
Brett Ryan created SPARK-38514: -- Summary: Download link on spark 3.2.1 for hadoop 3.2 is wrong Key: SPARK-38514 URL: https://issues.apache.org/jira/browse/SPARK-38514 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-38320) (flat)MapGroupsWithState can timeout groups which just received inputs in the same microbatch

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38320: Assignee: (was: Apache Spark) > (flat)MapGroupsWithState can timeout groups which

[jira] [Commented] (SPARK-38320) (flat)MapGroupsWithState can timeout groups which just received inputs in the same microbatch

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504642#comment-17504642 ] Apache Spark commented on SPARK-38320: -- User 'alex-balikov' has created a pull request for this

[jira] [Assigned] (SPARK-38320) (flat)MapGroupsWithState can timeout groups which just received inputs in the same microbatch

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38320: Assignee: Apache Spark > (flat)MapGroupsWithState can timeout groups which just received

[jira] [Commented] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504619#comment-17504619 ] Apache Spark commented on SPARK-38513: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38513: Assignee: (was: Apache Spark) > Move custom scheduler-specific configs to under >

[jira] [Assigned] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38513: Assignee: Apache Spark > Move custom scheduler-specific configs to under >

[jira] [Commented] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504618#comment-17504618 ] Apache Spark commented on SPARK-38513: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-38513) Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix

2022-03-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38513: - Summary: Move custom scheduler-specific configs to under `spark.kubernetes.scheduler.NAME` prefix Key: SPARK-38513 URL: https://issues.apache.org/jira/browse/SPARK-38513

[jira] [Assigned] (SPARK-38512) ResolveFunctions implemented incorrectly requiring multiple passes to Resolve Nested Expressions

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38512: Assignee: (was: Apache Spark) > ResolveFunctions implemented incorrectly requiring

[jira] [Assigned] (SPARK-38512) ResolveFunctions implemented incorrectly requiring multiple passes to Resolve Nested Expressions

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38512: Assignee: Apache Spark > ResolveFunctions implemented incorrectly requiring multiple

[jira] [Commented] (SPARK-38512) ResolveFunctions implemented incorrectly requiring multiple passes to Resolve Nested Expressions

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504617#comment-17504617 ] Apache Spark commented on SPARK-38512: -- User 'alexeykudinkin' has created a pull request for this

[jira] [Updated] (SPARK-38492) Improve the test coverage for PySpark

2022-03-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38492: Description: Currently, PySpark test coverage is around 91% according to codecov report:

[jira] [Created] (SPARK-38512) ResolveFunctions implemented incorrectly requiring multiple passes to Resolve Nested Expressions

2022-03-10 Thread Alexey Kudinkin (Jira)
Alexey Kudinkin created SPARK-38512: --- Summary: ResolveFunctions implemented incorrectly requiring multiple passes to Resolve Nested Expressions Key: SPARK-38512 URL:

[jira] [Updated] (SPARK-38492) Improve the test coverage for PySpark

2022-03-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38492: Description: Currently, PySpark test coverage is around 91% according to codecov report:

[jira] [Updated] (SPARK-38492) Improve the test coverage for PySpark

2022-03-10 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38492: Description: Currently, PySpark test coverage is around 91% according to codecov report:

[jira] [Commented] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504552#comment-17504552 ] Apache Spark commented on SPARK-38511: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38511: Assignee: (was: Apache Spark) > Remove priorityClassName propagation in favor of

[jira] [Commented] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504550#comment-17504550 ] Apache Spark commented on SPARK-38511: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38511: Assignee: Apache Spark > Remove priorityClassName propagation in favor of explicit

[jira] [Created] (SPARK-38511) Remove priorityClassName propagation in favor of explicit settings

2022-03-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38511: - Summary: Remove priorityClassName propagation in favor of explicit settings Key: SPARK-38511 URL: https://issues.apache.org/jira/browse/SPARK-38511 Project: Spark

[jira] [Created] (SPARK-38510) Failure fetching JSON representation of Spark plans with Hive UDFs

2022-03-10 Thread Shardul Mahadik (Jira)
Shardul Mahadik created SPARK-38510: --- Summary: Failure fetching JSON representation of Spark plans with Hive UDFs Key: SPARK-38510 URL: https://issues.apache.org/jira/browse/SPARK-38510 Project:

[jira] [Comment Edited] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-10 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504529#comment-17504529 ] Brian Schaefer edited comment on SPARK-38483 at 3/10/22, 7:13 PM: -- The

[jira] [Commented] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-10 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504529#comment-17504529 ] Brian Schaefer commented on SPARK-38483: The column name does differ between the two when

[jira] [Commented] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504522#comment-17504522 ] Apache Spark commented on SPARK-38509: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504521#comment-17504521 ] Apache Spark commented on SPARK-38509: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38509: Assignee: Apache Spark > Unregister the TIMESTAMPADD/DIFF functions and remove

[jira] [Assigned] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38509: Assignee: (was: Apache Spark) > Unregister the TIMESTAMPADD/DIFF functions and

[jira] [Created] (SPARK-38509) Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF

2022-03-10 Thread Max Gekk (Jira)
Max Gekk created SPARK-38509: Summary: Unregister the TIMESTAMPADD/DIFF functions and remove DATE_ADD/DIFF Key: SPARK-38509 URL: https://issues.apache.org/jira/browse/SPARK-38509 Project: Spark

[jira] [Created] (SPARK-38508) Volcano feature doesn't work on EKS graviton instances

2022-03-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38508: - Summary: Volcano feature doesn't work on EKS graviton instances Key: SPARK-38508 URL: https://issues.apache.org/jira/browse/SPARK-38508 Project: Spark

[jira] [Commented] (SPARK-34960) Aggregate (Min/Max/Count) push down for ORC

2022-03-10 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504500#comment-17504500 ] Cheng Su commented on SPARK-34960: -- Thanks [~tgraves] and [~ahussein] for commenting, and yes, if any

[jira] [Updated] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38379: -- Fix Version/s: 3.2.2 (was: 3.3.0) > Kubernetes:

[jira] [Updated] (SPARK-37735) Add appId interface to KubernetesConf

2022-03-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37735: -- Fix Version/s: 3.2.2 > Add appId interface to KubernetesConf >

[jira] [Created] (SPARK-38507) DataFrame withColumn method not adding or replacing columns when alias is used

2022-03-10 Thread Alexandros Mavrommatis (Jira)
Alexandros Mavrommatis created SPARK-38507: -- Summary: DataFrame withColumn method not adding or replacing columns when alias is used Key: SPARK-38507 URL:

[jira] [Resolved] (SPARK-38501) Fix thriftserver test failures under ANSI mode

2022-03-10 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38501. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35802

[jira] [Commented] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-10 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504448#comment-17504448 ] Brian Schaefer commented on SPARK-38483: Could you provide an example of when the real column

[jira] [Comment Edited] (SPARK-38483) Column name or alias as an attribute of the PySpark Column class

2022-03-10 Thread Brian Schaefer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503782#comment-17503782 ] Brian Schaefer edited comment on SPARK-38483 at 3/10/22, 5:08 PM: --

[jira] [Resolved] (SPARK-38451) Fix R tests under ANSI mode

2022-03-10 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-38451. Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35798

[jira] [Comment Edited] (SPARK-34960) Aggregate (Min/Max/Count) push down for ORC

2022-03-10 Thread Ahmed Hussein (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504374#comment-17504374 ] Ahmed Hussein edited comment on SPARK-34960 at 3/10/22, 3:57 PM: - Thanks

[jira] [Commented] (SPARK-34960) Aggregate (Min/Max/Count) push down for ORC

2022-03-10 Thread Ahmed Hussein (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504374#comment-17504374 ] Ahmed Hussein commented on SPARK-34960: --- Thanks [~chengsu] for putting up the optimization on

[jira] [Updated] (SPARK-34960) Aggregate (Min/Max/Count) push down for ORC

2022-03-10 Thread Ahmed Hussein (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated SPARK-34960: -- Attachment: file_no_stats-orc.tar.gz > Aggregate (Min/Max/Count) push down for ORC >

[jira] [Commented] (SPARK-38505) Make partial aggregation adaptive

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504349#comment-17504349 ] Apache Spark commented on SPARK-38505: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38505) Make partial aggregation adaptive

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38505: Assignee: Apache Spark > Make partial aggregation adaptive >

[jira] [Assigned] (SPARK-38505) Make partial aggregation adaptive

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38505: Assignee: (was: Apache Spark) > Make partial aggregation adaptive >

[jira] [Commented] (SPARK-38505) Make partial aggregation adaptive

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504350#comment-17504350 ] Apache Spark commented on SPARK-38505: -- User 'wangyum' has created a pull request for this issue:

[jira] [Created] (SPARK-38506) Push partial aggregation through join

2022-03-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38506: --- Summary: Push partial aggregation through join Key: SPARK-38506 URL: https://issues.apache.org/jira/browse/SPARK-38506 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37735) Add appId interface to KubernetesConf

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504336#comment-17504336 ] Apache Spark commented on SPARK-37735: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Commented] (SPARK-37735) Add appId interface to KubernetesConf

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504337#comment-17504337 ] Apache Spark commented on SPARK-37735: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Created] (SPARK-38505) Make partial aggregation adaptive

2022-03-10 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-38505: --- Summary: Make partial aggregation adaptive Key: SPARK-38505 URL: https://issues.apache.org/jira/browse/SPARK-38505 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-37735) Add appId interface to KubernetesConf

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504335#comment-17504335 ] Apache Spark commented on SPARK-37735: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504334#comment-17504334 ] Apache Spark commented on SPARK-38379: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504333#comment-17504333 ] Apache Spark commented on SPARK-38379: -- User 'tgravescs' has created a pull request for this issue:

[jira] [Commented] (SPARK-38330) Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

2022-03-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504286#comment-17504286 ] Steve Loughran commented on SPARK-38330: this is a hadoop issue -create a Jira there and file as

  1   2   >