[jira] [Created] (SPARK-38404) Spark does not find CTE inside nested CTE

2022-03-02 Thread Joan Heredia Rius (Jira)
Joan Heredia Rius created SPARK-38404: - Summary: Spark does not find CTE inside nested CTE Key: SPARK-38404 URL: https://issues.apache.org/jira/browse/SPARK-38404 Project: Spark Issue

[jira] [Commented] (SPARK-38104) Use error classes in the parsing errors of windows

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500556#comment-17500556 ] Apache Spark commented on SPARK-38104: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38104) Use error classes in the parsing errors of windows

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38104: Assignee: (was: Apache Spark) > Use error classes in the parsing errors of windows >

[jira] [Commented] (SPARK-38104) Use error classes in the parsing errors of windows

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500555#comment-17500555 ] Apache Spark commented on SPARK-38104: -- User 'yutoacts' has created a pull request for this issue:

[jira] [Assigned] (SPARK-38104) Use error classes in the parsing errors of windows

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38104: Assignee: Apache Spark > Use error classes in the parsing errors of windows >

[jira] [Commented] (SPARK-38403) `running-on-kubernetes` page render bad in v3.2.1(latest) website

2022-03-02 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500553#comment-17500553 ] Kent Yao commented on SPARK-38403: -- I guess it has been fixed via

[jira] [Created] (SPARK-38403) `running-on-kubernetes` page render bad in v3.2.1(latest) website

2022-03-02 Thread Yikun Jiang (Jira)
Yikun Jiang created SPARK-38403: --- Summary: `running-on-kubernetes` page render bad in v3.2.1(latest) website Key: SPARK-38403 URL: https://issues.apache.org/jira/browse/SPARK-38403 Project: Spark

[jira] [Created] (SPARK-38402) Improve user experience when working on data frames created from CSV and JSON in PERMISSIVE mode.

2022-03-02 Thread Dilip Biswal (Jira)
Dilip Biswal created SPARK-38402: Summary: Improve user experience when working on data frames created from CSV and JSON in PERMISSIVE mode. Key: SPARK-38402 URL: https://issues.apache.org/jira/browse/SPARK-38402

[jira] [Created] (SPARK-38401) Unify get preferred locations for shuffle in AQE

2022-03-02 Thread XiDuo You (Jira)
XiDuo You created SPARK-38401: - Summary: Unify get preferred locations for shuffle in AQE Key: SPARK-38401 URL: https://issues.apache.org/jira/browse/SPARK-38401 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38104) Use error classes in the parsing errors of windows

2022-03-02 Thread Yuto Akutsu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500535#comment-17500535 ] Yuto Akutsu commented on SPARK-38104: - I'm working on this. > Use error classes in the parsing

[jira] [Commented] (SPARK-38400) Enable Series.rename to change index labels

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500530#comment-17500530 ] Apache Spark commented on SPARK-38400: -- User 'xinrong-databricks' has created a pull request for

[jira] [Assigned] (SPARK-38400) Enable Series.rename to change index labels

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38400: Assignee: (was: Apache Spark) > Enable Series.rename to change index labels >

[jira] [Assigned] (SPARK-38400) Enable Series.rename to change index labels

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38400: Assignee: Apache Spark > Enable Series.rename to change index labels >

[jira] [Commented] (SPARK-38398) Add `priorityClassName` integration test case

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500531#comment-17500531 ] Apache Spark commented on SPARK-38398: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-38383) Support APP_ID and EXECUTOR_ID placeholder in annotations

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38383: -- Parent: SPARK-36057 Issue Type: Sub-task (was: Improvement) > Support APP_ID and

[jira] [Created] (SPARK-38400) Enable Series.rename to change index labels

2022-03-02 Thread Xinrong Meng (Jira)
Xinrong Meng created SPARK-38400: Summary: Enable Series.rename to change index labels Key: SPARK-38400 URL: https://issues.apache.org/jira/browse/SPARK-38400 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-38398) Add `priorityClassName` integration test case

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38398: - Assignee: Dongjoon Hyun > Add `priorityClassName` integration test case >

[jira] [Created] (SPARK-38399) Why doesn't shuffle hash join support build left table for left-outer-join but full-outer-join?

2022-03-02 Thread mengdou (Jira)
mengdou created SPARK-38399: --- Summary: Why doesn't shuffle hash join support build left table for left-outer-join but full-outer-join? Key: SPARK-38399 URL: https://issues.apache.org/jira/browse/SPARK-38399

[jira] [Created] (SPARK-38398) Add `priorityClassName` integration test case

2022-03-02 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38398: - Summary: Add `priorityClassName` integration test case Key: SPARK-38398 URL: https://issues.apache.org/jira/browse/SPARK-38398 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38334) Implement support for DEFAULT values for columns in tables

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38334: -- Affects Version/s: 3.3.0 (was: 3.2.1) > Implement support for

[jira] [Updated] (SPARK-38334) Implement support for DEFAULT values for columns in tables

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38334: -- Issue Type: Improvement (was: Story) > Implement support for DEFAULT values for columns in

[jira] [Updated] (SPARK-38395) Pyspark issue in resolving column when there is dot (.)

2022-03-02 Thread Krishna Sangeeth KS (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Sangeeth KS updated SPARK-38395: Description: Pyspark apply in pandas have some difficulty in resolving columns

[jira] [Commented] (SPARK-38340) Upgrade protobuf-java from 2.5.0 to 3.16.1

2022-03-02 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500487#comment-17500487 ] Yang Jie commented on SPARK-38340: -- cc [~hyukjin.kwon] , do you know how GA tests code that doesn't

[jira] [Updated] (SPARK-38397) Support Kueue: K8s-native Job Queueing

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38397: -- Description: There are several ways to run Spark on K8s including vanilla `spark-submit`

[jira] [Updated] (SPARK-38397) Support Kueue: K8s-native Job Queueing

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38397: -- Description: This issue is tracking K8s-native Job Queueing related work. *

[jira] [Updated] (SPARK-38397) Support Kueue: K8s-native Job Queueing

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38397: -- Description: This issue is tracking K8s-native Job Queueing related work. *

[jira] [Created] (SPARK-38397) Support Kueue: K8s-native Job Queueing

2022-03-02 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38397: - Summary: Support Kueue: K8s-native Job Queueing Key: SPARK-38397 URL: https://issues.apache.org/jira/browse/SPARK-38397 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38310) Support job queue in YuniKorn feature step

2022-03-02 Thread Wilfred Spiegelenburg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500448#comment-17500448 ] Wilfred Spiegelenburg commented on SPARK-38310: --- Spark on yarn uses the --queue option as

[jira] [Commented] (SPARK-37753) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should apply to left outer with join many empty partitions on the left

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500446#comment-17500446 ] Apache Spark commented on SPARK-37753: -- User 'ekoifman' has created a pull request for this issue:

[jira] [Assigned] (SPARK-37753) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should apply to left outer with join many empty partitions on the left

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37753: Assignee: Apache Spark > DynamicJoinSelection.shouldDemoteBroadcastHashJoin should apply

[jira] [Assigned] (SPARK-37753) DynamicJoinSelection.shouldDemoteBroadcastHashJoin should apply to left outer with join many empty partitions on the left

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-37753: Assignee: (was: Apache Spark) > DynamicJoinSelection.shouldDemoteBroadcastHashJoin

[jira] [Resolved] (SPARK-38191) The staging directory of write job only needs to be initialized once in HadoopMapReduceCommitProtocol.

2022-03-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38191. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35693

[jira] [Commented] (SPARK-37895) Error while joining two tables with non-english field names

2022-03-02 Thread Pablo Langa Blanco (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500419#comment-17500419 ] Pablo Langa Blanco commented on SPARK-37895: I'm working on a fix > Error while joining two

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Description: This JIRA aims to improve K8s integration tests for the following recent

[jira] [Updated] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38081: -- Fix Version/s: 3.2.2 > Support cloud-backend in K8s IT with SBT >

[jira] [Comment Edited] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500416#comment-17500416 ] Dongjoon Hyun edited comment on SPARK-38379 at 3/2/22, 10:54 PM: - Could

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500416#comment-17500416 ] Dongjoon Hyun commented on SPARK-38379: --- Could you describe your procedure about this, [~tgraves]?

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-02 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38353: Description: Create the ticket since instrumenting _{_}enter{_}_ and _{_}exit{_}_ magic methods

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-02 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38353: Description: Create the ticket since instrumenting __enter__ and __exit__ magic methods for

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for pandas API on Spark

2022-03-02 Thread Haejoon Lee (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-38353: Summary: Instrument __enter__ and __exit__ magic methods for pandas API on Spark (was:

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Description: This JIRA aims to improve K8s integration tests for the following recent

[jira] [Updated] (SPARK-33206) Spark Shuffle Index Cache calculates memory usage wrong

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33206: -- Fix Version/s: 3.2.2 > Spark Shuffle Index Cache calculates memory usage wrong >

[jira] [Updated] (SPARK-38272) Use docker-desktop instead of docker-for-desktop for Docker K8S IT deployMode and context name

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38272: -- Fix Version/s: 3.2.2 > Use docker-desktop instead of docker-for-desktop for Docker K8S IT

[jira] [Resolved] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38396. --- Fix Version/s: 3.3.0 Resolution: Fixed > Improve K8s Integration Tests >

[jira] [Assigned] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38396: - Assignee: William Hyun > Improve K8s Integration Tests > -

[jira] [Updated] (SPARK-37354) Make the Java version installed on the container image used by the K8s integration tests with SBT configurable

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37354: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Bug) > Make the Java version

[jira] [Updated] (SPARK-37319) Support K8s image building with Java 17

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37319: -- Parent Issue: SPARK-38396 (was: SPARK-33772) > Support K8s image building with Java 17 >

[jira] [Updated] (SPARK-38302) Use Java 17 in K8S integration tests when setting spark-tgz

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38302: -- Parent Issue: SPARK-38396 (was: SPARK-33772) > Use Java 17 in K8S integration tests when

[jira] [Resolved] (SPARK-38269) Clean up redundant type cast

2022-03-02 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-38269. Fix Version/s: 3.3.0 Assignee: Yang Jie Resolution: Fixed > Clean up redundant

[jira] [Updated] (SPARK-37529) Support K8s integration tests for Java 17

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37529: -- Parent Issue: SPARK-38396 (was: SPARK-33772) > Support K8s integration tests for Java 17 >

[jira] [Updated] (SPARK-37645) Word spell error - "labeled" spells as "labled"

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37645: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Improvement) > Word spell error -

[jira] [Updated] (SPARK-37998) Use `rbac.authorization.k8s.io/v1` instead of `v1beta1`

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-37998: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Use

[jira] [Updated] (SPARK-38081) Support cloud-backend in K8s IT with SBT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38081: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Support cloud-backend in K8s

[jira] [Updated] (SPARK-38241) Close KubernetesClient in K8S integrations tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38241: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Task) > Close KubernetesClient in K8S

[jira] [Resolved] (SPARK-36553) KMeans fails with NegativeArraySizeException for K = 50000 after issue #27758 was introduced

2022-03-02 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-36553. Fix Version/s: 3.1.3 3.3.0 3.2.2 Assignee:

[jira] [Updated] (SPARK-38272) Use docker-desktop instead of docker-for-desktop for Docker K8S IT deployMode and context name

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38272: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Improvement) > Use docker-desktop

[jira] [Updated] (SPARK-38022) Use relativePath for K8s remote file test in BasicTestsSuite

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38022: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Use relativePath for K8s

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Description: This JIRA aims to improve K8s integration tests for the following recent

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Description: This JIRA aims to improve K8s integration tests for the following recent

[jira] [Updated] (SPARK-38244) Upgrade kubernetes-client to 5.12.1

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38244: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Bug) > Upgrade kubernetes-client to

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Labels: releasenotes (was: ) > Improve K8s Integration Tests > -

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Target Version/s: 3.3.0 > Improve K8s Integration Tests > - > >

[jira] [Updated] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38396: -- Affects Version/s: (was: 3.2.2) > Improve K8s Integration Tests >

[jira] [Updated] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38048: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Add

[jira] [Updated] (SPARK-38029) Support docker-desktop K8S integration test in SBT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38029: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Support docker-desktop K8S

[jira] [Updated] (SPARK-38392) Add `spark-` prefix to namespaces and `-driver` suffix to drivers during IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38392: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Improvement) > Add `spark-` prefix to

[jira] [Updated] (SPARK-38072) Support K8s imageTag parameter in SBT K8s IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38072: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Support K8s imageTag

[jira] [Updated] (SPARK-38071) Support K8s namespace parameter in SBT K8s IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38071: -- Parent: SPARK-38396 Issue Type: Sub-task (was: Test) > Support K8s namespace

[jira] [Created] (SPARK-38396) Improve K8s Integration Tests

2022-03-02 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-38396: - Summary: Improve K8s Integration Tests Key: SPARK-38396 URL: https://issues.apache.org/jira/browse/SPARK-38396 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-38048) Add IntegrationTestBackend.describePods to support all K8s test backends

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38048: -- Fix Version/s: 3.2.2 > Add IntegrationTestBackend.describePods to support all K8s test

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

2022-03-02 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500354#comment-17500354 ] Thomas Graves commented on SPARK-38379: --- just going by the stack trace this looks related to

[jira] [Updated] (SPARK-38072) Support K8s imageTag parameter in SBT K8s IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38072: -- Fix Version/s: 3.2.2 > Support K8s imageTag parameter in SBT K8s IT >

[jira] [Updated] (SPARK-38071) Support K8s namespace parameter in SBT K8s IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38071: -- Fix Version/s: 3.2.2 > Support K8s namespace parameter in SBT K8s IT >

[jira] [Updated] (SPARK-38029) Support docker-desktop K8S integration test in SBT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38029: -- Fix Version/s: 3.2.2 > Support docker-desktop K8S integration test in SBT >

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for Pandas module

2022-03-02 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38353: -- Description: Create the ticket since instrumenting {_}{{_}}enter{{_}}{_} and {_}{{_}}exit{{_}}{_}

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for Pandas module

2022-03-02 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38353: -- Description: Create the ticket since instrumenting {_}{{_}}enter{{_}}{_} and {_}{{_}}exit{{_}}{_}

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for Pandas module

2022-03-02 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38353: -- Description: Create the ticket since instrumenting _{_}enter{_}_ and _{_}exit{_}_ magic methods for

[jira] [Updated] (SPARK-38392) [K8S] Improve the names used for K8S namespace and driver pod in integration tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38392: -- Component/s: Tests > [K8S] Improve the names used for K8S namespace and driver pod in

[jira] [Commented] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2022-03-02 Thread James Grinter (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500321#comment-17500321 ] James Grinter commented on SPARK-20050: --- You're just working around a genuine bug, and behaviour

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for Pandas module

2022-03-02 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38353: -- Description: For example, for the following code:   {code:java} pdf = pd.DataFrame( [(0.2, 0.3),

[jira] [Updated] (SPARK-38392) Add spark- prefix to namespaces and -driver suffix to drivers during IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38392: -- Summary: Add spark- prefix to namespaces and -driver suffix to drivers during IT (was: [K8S]

[jira] [Updated] (SPARK-38353) Instrument __enter__ and __exit__ magic methods for Pandas module

2022-03-02 Thread Yihong He (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yihong He updated SPARK-38353: -- Description: For example, for the following code:   {code:java} pdf = pd.DataFrame( [(0.2, 0.3),

[jira] [Updated] (SPARK-38392) Add `spark-` prefix to namespaces and `-driver` suffix to drivers during IT

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38392: -- Summary: Add `spark-` prefix to namespaces and `-driver` suffix to drivers during IT (was:

[jira] [Assigned] (SPARK-38392) [K8S] Improve the names used for K8S namespace and driver pod in integration tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-38392: - Assignee: Martin Tzvetanov Grigorov > [K8S] Improve the names used for K8S namespace

[jira] [Resolved] (SPARK-38392) [K8S] Improve the names used for K8S namespace and driver pod in integration tests

2022-03-02 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-38392. --- Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue resolved by

[jira] [Created] (SPARK-38395) Pyspark issue in resolving column when there is dot (.)

2022-03-02 Thread Krishna Sangeeth KS (Jira)
Krishna Sangeeth KS created SPARK-38395: --- Summary: Pyspark issue in resolving column when there is dot (.) Key: SPARK-38395 URL: https://issues.apache.org/jira/browse/SPARK-38395 Project: Spark

[jira] [Updated] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

2022-03-02 Thread Jason Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Xu updated SPARK-38388: - Description: Spark repartition uses RoundRobinPartitioning, the generated results is non-deterministic

[jira] [Updated] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

2022-03-02 Thread Jason Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Xu updated SPARK-38388: - Description: Spark repartition uses RoundRobinPartitioning, the generated results is non-deterministic

[jira] [Resolved] (SPARK-38389) Add the `DATEDIFF` and `DATE_DIFF` aliases for `TIMESTAMPDIFF()`

2022-03-02 Thread Max Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-38389. -- Resolution: Fixed Issue resolved by pull request 35709 [https://github.com/apache/spark/pull/35709]

[jira] [Commented] (SPARK-33206) Spark Shuffle Index Cache calculates memory usage wrong

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500272#comment-17500272 ] Apache Spark commented on SPARK-33206: -- User 'attilapiros' has created a pull request for this

[jira] [Commented] (SPARK-33206) Spark Shuffle Index Cache calculates memory usage wrong

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500271#comment-17500271 ] Apache Spark commented on SPARK-33206: -- User 'attilapiros' has created a pull request for this

[jira] [Created] (SPARK-38394) build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error

2022-03-02 Thread Steve Loughran (Jira)
Steve Loughran created SPARK-38394: -- Summary: build of spark sql against hadoop-3.4.0-snapshot failing with bouncycastle classpath error Key: SPARK-38394 URL: https://issues.apache.org/jira/browse/SPARK-38394

[jira] [Commented] (SPARK-38393) Clean up deprecated usage of GenSeq/GenMap

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500207#comment-17500207 ] Apache Spark commented on SPARK-38393: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-38393) Clean up deprecated usage of GenSeq/GenMap

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38393: Assignee: Apache Spark > Clean up deprecated usage of GenSeq/GenMap >

[jira] [Assigned] (SPARK-38393) Clean up deprecated usage of GenSeq/GenMap

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38393: Assignee: (was: Apache Spark) > Clean up deprecated usage of GenSeq/GenMap >

[jira] [Commented] (SPARK-38393) Clean up deprecated usage of GenSeq/GenMap

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500206#comment-17500206 ] Apache Spark commented on SPARK-38393: -- User 'LuciferYang' has created a pull request for this

[jira] [Created] (SPARK-38393) Clean up deprecated usage of GenSeq/GenMap

2022-03-02 Thread Yang Jie (Jira)
Yang Jie created SPARK-38393: Summary: Clean up deprecated usage of GenSeq/GenMap Key: SPARK-38393 URL: https://issues.apache.org/jira/browse/SPARK-38393 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-38342) Clean up deprecated api usage of Ivy

2022-03-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-38342. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 35672

[jira] [Assigned] (SPARK-38342) Clean up deprecated api usage of Ivy

2022-03-02 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-38342: Assignee: Yang Jie > Clean up deprecated api usage of Ivy >

[jira] [Assigned] (SPARK-38392) [K8S] Improve the names used for K8S namespace and driver pod in integration tests

2022-03-02 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38392: Assignee: (was: Apache Spark) > [K8S] Improve the names used for K8S namespace and

  1   2   >