[jira] [Commented] (SPARK-34648) Reading Parquet Files in Spark Extremely Slow for Large Number of Files?

[ https://issues.apache.org/jira/browse/SPARK-34648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296471#comment-17296471 ] Pankaj Bhootra commented on SPARK-34648: [~marmbrus] - I understand you were involved in

[jira] [Created] (SPARK-34648) Reading Parquet Files in Spark Extremely Slow for Large Number of Files?

Pankaj Bhootra created SPARK-34648: -- Summary: Reading Parquet Files in Spark Extremely Slow for Large Number of Files? Key: SPARK-34648 URL: https://issues.apache.org/jira/browse/SPARK-34648

[jira] [Assigned] (SPARK-34647) Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes

[ https://issues.apache.org/jira/browse/SPARK-34647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34647: Assignee: Apache Spark > Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes >

[jira] [Commented] (SPARK-34647) Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes

[ https://issues.apache.org/jira/browse/SPARK-34647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296443#comment-17296443 ] Apache Spark commented on SPARK-34647: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-34647) Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes

[ https://issues.apache.org/jira/browse/SPARK-34647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34647: Assignee: (was: Apache Spark) > Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer

[jira] [Created] (SPARK-34647) Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes

Dongjoon Hyun created SPARK-34647: - Summary: Upgrade ZSTD-JNI to 1.4.8-7 and use NoFinalizer classes Key: SPARK-34647 URL: https://issues.apache.org/jira/browse/SPARK-34647 Project: Spark

[jira] [Resolved] (SPARK-30303) Percentile will hit match err when percentage is null

[ https://issues.apache.org/jira/browse/SPARK-30303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-30303. - Fix Version/s: 3.0.0 Resolution: Fixed > Percentile will hit match err when percentage

[jira] [Updated] (SPARK-33539) Standardize exception messages in Spark

[ https://issues.apache.org/jira/browse/SPARK-33539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-33539: - Description: In the SPIP: Standardize Exception Messages in Spark, there are three major

[jira] [Updated] (SPARK-33539) Standardize exception messages in Spark

[ https://issues.apache.org/jira/browse/SPARK-33539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-33539: - Description: In the SPIP: Standardize Exception Messages in Spark, there are three major

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296393#comment-17296393 ] Dongjoon Hyun commented on SPARK-34645: --- Thank you for investigation. I'm looking forward to

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296391#comment-17296391 ] Andy Grove commented on SPARK-34645: I can reproduce this with Spark 3.1.1 + JDK 11 as well (and

[jira] [Resolved] (SPARK-34601) Do not delete shuffle file on executor lost event when using remote shuffle service

[ https://issues.apache.org/jira/browse/SPARK-34601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BoYang resolved SPARK-34601. Resolution: Won't Fix > Do not delete shuffle file on executor lost event when using remote shuffle >

[jira] [Commented] (SPARK-34601) Do not delete shuffle file on executor lost event when using remote shuffle service

[ https://issues.apache.org/jira/browse/SPARK-34601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296390#comment-17296390 ] BoYang commented on SPARK-34601: It looks Spark already checks and matches the executor id when it tries

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296385#comment-17296385 ] Dongjoon Hyun commented on SPARK-34645: --- BTW, since Apache Spark 3.0.2 is released before Apache

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296374#comment-17296374 ] Dongjoon Hyun commented on SPARK-34645: --- Could you provide a reproducer for this? > [K8S] Driver

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296370#comment-17296370 ] Dongjoon Hyun commented on SPARK-34645: --- The above my result and my local K8s IT integration test

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296368#comment-17296368 ] Dongjoon Hyun commented on SPARK-34645: --- First of all, both Spark 3.0.1/3.0.2 has the same K8s

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296358#comment-17296358 ] Andy Grove commented on SPARK-34645: I just confirmed that this only happens with JDK 11 and works

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296356#comment-17296356 ] Dongjoon Hyun commented on SPARK-34645: --- So, it happens only at Apache Spark 3.0.2? Let me take a

[jira] [Updated] (SPARK-34646) TreeNode bind issue for duplicate column name.

[ https://issues.apache.org/jira/browse/SPARK-34646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] loc nguyen updated SPARK-34646: --- Description: I received a Spark {{TreeNodeException}} executing a union of two data frames. When I

[jira] [Updated] (SPARK-34646) TreeNode bind issue for duplicate column name.

[ https://issues.apache.org/jira/browse/SPARK-34646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] loc nguyen updated SPARK-34646: --- Description: I received a Spark {{TreeNodeException}} executing a union of two data frames. When I

[jira] [Updated] (SPARK-34646) TreeNode bind issue for duplicate column name.

[ https://issues.apache.org/jira/browse/SPARK-34646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] loc nguyen updated SPARK-34646: --- Description: I received a Spark {{TreeNodeException}} executing a union of two data frames. When I

[jira] [Updated] (SPARK-34646) TreeNode bind issue for duplicate column name.

[ https://issues.apache.org/jira/browse/SPARK-34646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] loc nguyen updated SPARK-34646: --- Summary: TreeNode bind issue for duplicate column name. (was: TreeNode bind issue) > TreeNode

[jira] [Updated] (SPARK-34646) TreeNode bind issue

[ https://issues.apache.org/jira/browse/SPARK-34646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] loc nguyen updated SPARK-34646: --- Description: I received a Spark {{TreeNodeException}} executing a union of two data frames. When I

[jira] [Updated] (SPARK-34646) TreeNode bind issue

[ https://issues.apache.org/jira/browse/SPARK-34646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] loc nguyen updated SPARK-34646: --- Description: I received a Spark {{TreeNodeException}} executing a union of two data frames. When I

[jira] [Created] (SPARK-34646) TreeNode bind issue

loc nguyen created SPARK-34646: -- Summary: TreeNode bind issue Key: SPARK-34646 URL: https://issues.apache.org/jira/browse/SPARK-34646 Project: Spark Issue Type: Bug Components: Spark

[jira] [Commented] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

[ https://issues.apache.org/jira/browse/SPARK-34645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296340#comment-17296340 ] Thomas Graves commented on SPARK-34645: --- [~dongjoon] [~holden]. have you seen this at all? >

[jira] [Created] (SPARK-34645) [K8S] Driver pod stuck in Running state after job completes

Andy Grove created SPARK-34645: -- Summary: [K8S] Driver pod stuck in Running state after job completes Key: SPARK-34645 URL: https://issues.apache.org/jira/browse/SPARK-34645 Project: Spark

[jira] [Assigned] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34295: Assignee: L. C. Hsieh (was: Apache Spark) > Allow option similar to

[jira] [Assigned] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34295: Assignee: Apache Spark (was: L. C. Hsieh) > Allow option similar to

[jira] [Commented] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296324#comment-17296324 ] Apache Spark commented on SPARK-34295: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

[ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296327#comment-17296327 ] Apache Spark commented on SPARK-34295: -- User 'viirya' has created a pull request for this issue:

[jira] [Updated] (SPARK-34644) UDF returning array followed by explode returns wrong results

[ https://issues.apache.org/jira/browse/SPARK-34644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gavrilescu Laurentiu updated SPARK-34644: - Description: *Applying an UDF followed by explode looks to be calling the UDF

[jira] [Created] (SPARK-34644) UDF returning array followed by explode returns wrong results

Gavrilescu Laurentiu created SPARK-34644: Summary: UDF returning array followed by explode returns wrong results Key: SPARK-34644 URL: https://issues.apache.org/jira/browse/SPARK-34644

[jira] [Updated] (SPARK-34392) Invalid ID for offset-based ZoneId since Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-34392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-34392: - Fix Version/s: 3.0.3 > Invalid ID for offset-based ZoneId since Spark 3.0 >

[jira] [Updated] (SPARK-34563) Checkpointing a union with another checkpoint fails

[ https://issues.apache.org/jira/browse/SPARK-34563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kamprath updated SPARK-34563: - Affects Version/s: 3.1.1 > Checkpointing a union with another checkpoint fails >

[jira] [Commented] (SPARK-34563) Checkpointing a union with another checkpoint fails

[ https://issues.apache.org/jira/browse/SPARK-34563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296250#comment-17296250 ] Michael Kamprath commented on SPARK-34563: -- I just tested this under Spark 3.1.1 keep

[jira] [Assigned] (SPARK-34643) Use CRAN URL in canonical form

[ https://issues.apache.org/jira/browse/SPARK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-34643: - Assignee: Hyukjin Kwon > Use CRAN URL in canonical form >

[jira] [Resolved] (SPARK-34643) Use CRAN URL in canonical form

[ https://issues.apache.org/jira/browse/SPARK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34643. --- Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue resolved by

[jira] [Resolved] (SPARK-34624) Filter non-jar dependencies from ivy/maven coordinates

[ https://issues.apache.org/jira/browse/SPARK-34624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-34624. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31741

[jira] [Assigned] (SPARK-34624) Filter non-jar dependencies from ivy/maven coordinates

[ https://issues.apache.org/jira/browse/SPARK-34624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-34624: - Assignee: Shardul Mahadik > Filter non-jar dependencies from ivy/maven coordinates >

[jira] [Commented] (SPARK-34641) ARM CI failed due to download hive-exec failed

[ https://issues.apache.org/jira/browse/SPARK-34641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296239#comment-17296239 ] Shane Knapp commented on SPARK-34641: - looks like hive-exec downloaded fine in my recent build:

[jira] [Commented] (SPARK-34641) ARM CI failed due to download hive-exec failed

[ https://issues.apache.org/jira/browse/SPARK-34641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296235#comment-17296235 ] Shane Knapp commented on SPARK-34641: - no clue, tbh.  i'll try and take a closer look at the build

[jira] [Assigned] (SPARK-34643) Use CRAN URL in canonical form

[ https://issues.apache.org/jira/browse/SPARK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34643: Assignee: Apache Spark > Use CRAN URL in canonical form > --

[jira] [Assigned] (SPARK-34643) Use CRAN URL in canonical form

[ https://issues.apache.org/jira/browse/SPARK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34643: Assignee: (was: Apache Spark) > Use CRAN URL in canonical form >

[jira] [Commented] (SPARK-34643) Use CRAN URL in canonical form

[ https://issues.apache.org/jira/browse/SPARK-34643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296227#comment-17296227 ] Apache Spark commented on SPARK-34643: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-34642) TypeError in Pyspark Linear Regression docs

[ https://issues.apache.org/jira/browse/SPARK-34642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296226#comment-17296226 ] Apache Spark commented on SPARK-34642: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34642) TypeError in Pyspark Linear Regression docs

[ https://issues.apache.org/jira/browse/SPARK-34642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34642: Assignee: Apache Spark > TypeError in Pyspark Linear Regression docs >

[jira] [Assigned] (SPARK-34642) TypeError in Pyspark Linear Regression docs

[ https://issues.apache.org/jira/browse/SPARK-34642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34642: Assignee: (was: Apache Spark) > TypeError in Pyspark Linear Regression docs >

[jira] [Resolved] (SPARK-34592) Mark indeterminate RDD in Web UI

[ https://issues.apache.org/jira/browse/SPARK-34592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-34592. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 31707

[jira] [Created] (SPARK-34643) Use CRAN URL in canonical form

Dongjoon Hyun created SPARK-34643: - Summary: Use CRAN URL in canonical form Key: SPARK-34643 URL: https://issues.apache.org/jira/browse/SPARK-34643 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-34642) TypeError in Pyspark Linear Regression docs

Sean R. Owen created SPARK-34642: Summary: TypeError in Pyspark Linear Regression docs Key: SPARK-34642 URL: https://issues.apache.org/jira/browse/SPARK-34642 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-34636) sql method in UnresolvedAttribute, AttributeReference and Alias don't quote qualified names properly.

[ https://issues.apache.org/jira/browse/SPARK-34636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-34636: --- Description: UnresolvedReference, AttributeReference and Alias which take qualifier don't

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296125#comment-17296125 ] Itay Bittan commented on SPARK-33288: - [~tgraves] you are right! I used my compiled version (3.0.1)

[jira] [Commented] (SPARK-34641) ARM CI failed due to download hive-exec failed

[ https://issues.apache.org/jira/browse/SPARK-34641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296072#comment-17296072 ] Yikun Jiang commented on SPARK-34641: - [~shaneknapp] Do you have any idea on it? > ARM CI failed

[jira] [Created] (SPARK-34641) ARM CI failed due to download hive-exec failed

Yikun Jiang created SPARK-34641: --- Summary: ARM CI failed due to download hive-exec failed Key: SPARK-34641 URL: https://issues.apache.org/jira/browse/SPARK-34641 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296060#comment-17296060 ] Thomas Graves commented on SPARK-33288: --- It looks like the version you are trying to launch

[jira] [Commented] (SPARK-33288) Support k8s cluster manager with stage level scheduling

[ https://issues.apache.org/jira/browse/SPARK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296027#comment-17296027 ] Itay Bittan commented on SPARK-33288: - Hi! I just upgraded from spark 3.0.1 to 3.1.1 and I'm having

[jira] [Created] (SPARK-34640) unable to access grouping column after groupBy

Jiri Humpolicek created SPARK-34640: --- Summary: unable to access grouping column after groupBy Key: SPARK-34640 URL: https://issues.apache.org/jira/browse/SPARK-34640 Project: Spark Issue

[jira] [Updated] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly

[ https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] EdisonWang updated SPARK-34634: --- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Commented] (SPARK-34639) always remove unnecessary Alias in Analyzer.resolveExpression

[ https://issues.apache.org/jira/browse/SPARK-34639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295901#comment-17295901 ] Apache Spark commented on SPARK-34639: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-34639) always remove unnecessary Alias in Analyzer.resolveExpression

[ https://issues.apache.org/jira/browse/SPARK-34639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34639: Assignee: (was: Apache Spark) > always remove unnecessary Alias in

[jira] [Assigned] (SPARK-34639) always remove unnecessary Alias in Analyzer.resolveExpression

[ https://issues.apache.org/jira/browse/SPARK-34639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34639: Assignee: Apache Spark > always remove unnecessary Alias in Analyzer.resolveExpression >

[jira] [Commented] (SPARK-33602) Group exception messages in execution/datasources

[ https://issues.apache.org/jira/browse/SPARK-33602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295900#comment-17295900 ] Apache Spark commented on SPARK-33602: -- User 'beliefer' has created a pull request for this issue:

[jira] [Created] (SPARK-34639) always remove unnecessary Alias in Analyzer.resolveExpression

Wenchen Fan created SPARK-34639: --- Summary: always remove unnecessary Alias in Analyzer.resolveExpression Key: SPARK-34639 URL: https://issues.apache.org/jira/browse/SPARK-34639 Project: Spark

[jira] [Commented] (SPARK-29721) Spark SQL reads unnecessary nested fields after using explode

[ https://issues.apache.org/jira/browse/SPARK-29721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295859#comment-17295859 ] Jiri Humpolicek commented on SPARK-29721: - here it is

[jira] [Created] (SPARK-34638) Spark SQL reads unnecessary nested fields (another type of pruning case)

Jiri Humpolicek created SPARK-34638: --- Summary: Spark SQL reads unnecessary nested fields (another type of pruning case) Key: SPARK-34638 URL: https://issues.apache.org/jira/browse/SPARK-34638