[jira] [Commented] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843123#comment-17843123 ] Anish Shrigondekar commented on SPARK-48105: Thanks [~huanli.wang]  Worth noting that we

[jira] [Updated] (SPARK-48103) Promote ` KubernetesDriverBuilder` to `DeveloperApi`

[ https://issues.apache.org/jira/browse/SPARK-48103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48103: --- Labels: pull-request-available (was: ) > Promote ` KubernetesDriverBuilder` to

[jira] [Updated] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

[ https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48105: --- Labels: pull-request-available (was: ) > Fix the data corruption issue when state store

[jira] [Resolved] (SPARK-48108) Skip `tpcds-1g` and `docker-integration-tests` tests from `RocksDB UI-Backend` job

[ https://issues.apache.org/jira/browse/SPARK-48108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48108. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46355

[jira] [Assigned] (SPARK-48109) Enable `k8s-integration-tests` only for `kubernetes` module change

[ https://issues.apache.org/jira/browse/SPARK-48109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48109: - Assignee: Dongjoon Hyun > Enable `k8s-integration-tests` only for `kubernetes` module

[jira] [Updated] (SPARK-48109) Enable `k8s-integration-tests` only for `kubernetes` module change

[ https://issues.apache.org/jira/browse/SPARK-48109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48109: --- Labels: pull-request-available (was: ) > Enable `k8s-integration-tests` only for

[jira] [Created] (SPARK-48109) Enable `k8s-integration-tests` only for `kubernetes` module change

Dongjoon Hyun created SPARK-48109: - Summary: Enable `k8s-integration-tests` only for `kubernetes` module change Key: SPARK-48109 URL: https://issues.apache.org/jira/browse/SPARK-48109 Project: Spark

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Attachment: (was: Screenshot 2024-05-02 at 13.18.42.png) > Reduce GitHub Action usage

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Attachment: Screenshot 2024-05-02 at 20.59.18.png > Reduce GitHub Action usage according to

[jira] [Updated] (SPARK-48107) Exclude tests from Python distribution

[ https://issues.apache.org/jira/browse/SPARK-48107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48107: --- Labels: pull-request-available (was: ) > Exclude tests from Python distribution >

[jira] [Created] (SPARK-48107) Exclude tests from Python distribution

Nicholas Chammas created SPARK-48107: Summary: Exclude tests from Python distribution Key: SPARK-48107 URL: https://issues.apache.org/jira/browse/SPARK-48107 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-48106) Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48106: - Assignee: Dongjoon Hyun > Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

[jira] [Updated] (SPARK-48106) Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48106: -- Description: - https://docs.python.org/3/whatsnew/3.11.html#summary-release-highlights bq.

[jira] [Updated] (SPARK-48106) Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48106: -- Description: - https://docs.python.org/3/whatsnew/3.11.html#summary-release-highlights >

[jira] [Updated] (SPARK-48106) Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48106: --- Labels: pull-request-available (was: ) > Use `Python 3.11` in `pyspark` tests of

[jira] [Created] (SPARK-48106) Use `Python 3.11` in `pyspark` tests of `build_and_test.yml`

Dongjoon Hyun created SPARK-48106: - Summary: Use `Python 3.11` in `pyspark` tests of `build_and_test.yml` Key: SPARK-48106 URL: https://issues.apache.org/jira/browse/SPARK-48106 Project: Spark

[jira] [Assigned] (SPARK-48104) Run `publish_snapshot.yml` once per day

[ https://issues.apache.org/jira/browse/SPARK-48104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48104: - Assignee: Dongjoon Hyun > Run `publish_snapshot.yml` once per day >

[jira] [Resolved] (SPARK-48104) Run `publish_snapshot.yml` once per day

[ https://issues.apache.org/jira/browse/SPARK-48104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48104. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46352

[jira] [Created] (SPARK-48105) Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store

Huanli Wang created SPARK-48105: --- Summary: Fix the data corruption issue when state store unload and snapshotting happens concurrently for HDFS state store Key: SPARK-48105 URL:

[jira] [Updated] (SPARK-48104) Run `publish_snapshot.yml` once per day

[ https://issues.apache.org/jira/browse/SPARK-48104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48104: --- Labels: pull-request-available (was: ) > Run `publish_snapshot.yml` once per day >

[jira] [Created] (SPARK-48104) Run `publish_snapshot.yml` once per day

Dongjoon Hyun created SPARK-48104: - Summary: Run `publish_snapshot.yml` once per day Key: SPARK-48104 URL: https://issues.apache.org/jira/browse/SPARK-48104 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-47671) Enable structured logging in log4j2.properties.template and update `configuration.md`

[ https://issues.apache.org/jira/browse/SPARK-47671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-47671. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46349

[jira] [Resolved] (SPARK-48102) Track time to acquire source progress metrics for streaming triggers

[ https://issues.apache.org/jira/browse/SPARK-48102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-48102. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46350

[jira] [Assigned] (SPARK-48102) Track time to acquire source progress metrics for streaming triggers

[ https://issues.apache.org/jira/browse/SPARK-48102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-48102: Assignee: Anish Shrigondekar > Track time to acquire source progress metrics for

[jira] [Resolved] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48099. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46347

[jira] [Updated] (SPARK-47920) Add documentation for python streaming data source

[ https://issues.apache.org/jira/browse/SPARK-47920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-47920: - Affects Version/s: 4.0.0 (was: 3.5.1) > Add documentation for python

[jira] [Created] (SPARK-48103) Promote ` KubernetesDriverBuilder` to `DeveloperApi`

Zhou JIANG created SPARK-48103: -- Summary: Promote ` KubernetesDriverBuilder` to `DeveloperApi` Key: SPARK-48103 URL: https://issues.apache.org/jira/browse/SPARK-48103 Project: Spark Issue Type:

[jira] [Updated] (SPARK-48102) Track time to acquire source progress metrics for streaming triggers

[ https://issues.apache.org/jira/browse/SPARK-48102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48102: --- Labels: pull-request-available (was: ) > Track time to acquire source progress metrics for

[jira] [Created] (SPARK-48102) Track time to acquire source progress metrics for streaming triggers

Anish Shrigondekar created SPARK-48102: -- Summary: Track time to acquire source progress metrics for streaming triggers Key: SPARK-48102 URL: https://issues.apache.org/jira/browse/SPARK-48102

[jira] [Resolved] (SPARK-48065) SPJ: allowJoinKeysSubsetOfPartitionKeys is too strict

[ https://issues.apache.org/jira/browse/SPARK-48065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved SPARK-48065. -- Resolution: Fixed Issue resolved by pull request 46325 [https://github.com/apache/spark/pull/46325]

[jira] [Assigned] (SPARK-48065) SPJ: allowJoinKeysSubsetOfPartitionKeys is too strict

[ https://issues.apache.org/jira/browse/SPARK-48065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned SPARK-48065: Assignee: Szehon Ho > SPJ: allowJoinKeysSubsetOfPartitionKeys is too strict >

[jira] [Updated] (SPARK-47671) Enable structured logging in log4j2.properties.template and update `configuration.md`

[ https://issues.apache.org/jira/browse/SPARK-47671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-47671: --- Labels: pull-request-available (was: ) > Enable structured logging in

[jira] [Updated] (SPARK-47671) Enable structured logging in log4j2.properties.template and update `configuration.md`

[ https://issues.apache.org/jira/browse/SPARK-47671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-47671: --- Description: # rename the current log4j2.properties.template as

[jira] [Updated] (SPARK-48101) When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved

[ https://issues.apache.org/jira/browse/SPARK-48101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-48101: - Priority: Minor (was: Major) > When using INSERT OVERWRITE with Spark CTEs they may not be

[jira] [Assigned] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48099: - Assignee: Dongjoon Hyun > Run `maven-build` test only on `Java 21 on MacOS 14 (Apple

[jira] [Created] (SPARK-48101) When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved

Holden Karau created SPARK-48101: Summary: When using INSERT OVERWRITE with Spark CTEs they may not be fully resolved Key: SPARK-48101 URL: https://issues.apache.org/jira/browse/SPARK-48101 Project:

[jira] [Updated] (SPARK-48100) [SQL][XML] Fix issues in skipping nested structure fields not selected in schema

[ https://issues.apache.org/jira/browse/SPARK-48100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shujing Yang updated SPARK-48100: - Description: Previously, the XML parser can't skip nested structure data fields effectively

[jira] [Updated] (SPARK-48100) [SQL][XML] Fix issues in skipping nested structure fields not selected in schema

[ https://issues.apache.org/jira/browse/SPARK-48100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48100: --- Labels: pull-request-available (was: ) > [SQL][XML] Fix issues in skipping nested

[jira] [Updated] (SPARK-48100) [SQL][XML] Fix issues in skipping nested structure fields not selected in schema

[ https://issues.apache.org/jira/browse/SPARK-48100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shujing Yang updated SPARK-48100: - Summary: [SQL][XML] Fix issues in skipping nested structure fields not selected in schema

[jira] [Created] (SPARK-48100) [SQL][XML] Fix projection issue when there's a nested struct

Shujing Yang created SPARK-48100: Summary: [SQL][XML] Fix projection issue when there's a nested struct Key: SPARK-48100 URL: https://issues.apache.org/jira/browse/SPARK-48100 Project: Spark

[jira] [Assigned] (SPARK-48097) Limit GHA job execution time to up to 3 hours in `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48097: - Assignee: Dongjoon Hyun > Limit GHA job execution time to up to 3 hours in

[jira] [Resolved] (SPARK-48097) Limit GHA job execution time to up to 3 hours in `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48097. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46344

[jira] [Updated] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48099: --- Labels: pull-request-available (was: ) > Run `maven-build` test only on `Java 21 on MacOS

[jira] [Updated] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48099: -- Description: `Java 21 on MacOS 14` is the fastest Maven test and covers both Java 17 and

[jira] [Updated] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48099: -- Description: !Screenshot 2024-05-02 at 14.59.14.png!width=100%! > Run `maven-build` test

[jira] [Updated] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48099: -- Description: !Screenshot 2024-05-02 at 14.59.14.png! (was: !Screenshot 2024-05-02 at

[jira] [Updated] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48099: -- Description: !Screenshot 2024-05-02 at 14.59.14.png|width=100%! (was: !Screenshot

[jira] [Created] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

Dongjoon Hyun created SPARK-48099: - Summary: Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)` Key: SPARK-48099 URL: https://issues.apache.org/jira/browse/SPARK-48099 Project:

[jira] [Updated] (SPARK-48099) Run `maven-build` test only on `Java 21 on MacOS 14 (Apple Silicon)`

[ https://issues.apache.org/jira/browse/SPARK-48099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48099: -- Attachment: Screenshot 2024-05-02 at 14.59.14.png > Run `maven-build` test only on `Java 21

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Issue Type: Umbrella (was: Task) > Reduce GitHub Action usage according to ASF project

[jira] [Created] (SPARK-48098) Enable `NOLINT_ON_COMPILE` for all except `linter` job

Dongjoon Hyun created SPARK-48098: - Summary: Enable `NOLINT_ON_COMPILE` for all except `linter` job Key: SPARK-48098 URL: https://issues.apache.org/jira/browse/SPARK-48098 Project: Spark

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2.

[jira] [Updated] (SPARK-48097) Limit GHA job execution time to up to 3 hours in `build_and_test.yml`

[ https://issues.apache.org/jira/browse/SPARK-48097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48097: --- Labels: pull-request-available (was: ) > Limit GHA job execution time to up to 3 hours in

[jira] [Created] (SPARK-48097) Limit GHA job execution time to up to 3 hours in `build_and_test.yml`

Dongjoon Hyun created SPARK-48097: - Summary: Limit GHA job execution time to up to 3 hours in `build_and_test.yml` Key: SPARK-48097 URL: https://issues.apache.org/jira/browse/SPARK-48097 Project:

[jira] [Assigned] (SPARK-48095) Run `build_non_ansi.yml` once per day

[ https://issues.apache.org/jira/browse/SPARK-48095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48095: - Assignee: Dongjoon Hyun > Run `build_non_ansi.yml` once per day >

[jira] [Resolved] (SPARK-48095) Run `build_non_ansi.yml` once per day

[ https://issues.apache.org/jira/browse/SPARK-48095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48095. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46342

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. ASF INFRA POLICY - https://infra.apache.org/github-actions-policy.html h2.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h2. MONITORING [https://infra-reports.apache.org/#ghactions=spark=168] h2.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h1. MONITORING [https://infra-reports.apache.org/#ghactions=spark=168] h1.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: *{*}MONITORING{*}* [https://infra-reports.apache.org/#ghactions=spark=168]

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Description: h1. MONITORING [https://infra-reports.apache.org/#ghactions=spark=168] h1.

[jira] [Updated] (SPARK-48094) Reduce GitHub Action usage according to ASF project allowance

[ https://issues.apache.org/jira/browse/SPARK-48094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48094: -- Attachment: Screenshot 2024-05-02 at 13.18.42.png > Reduce GitHub Action usage according to

[jira] [Updated] (SPARK-48095) Run `build_non_ansi.yml` once per day

[ https://issues.apache.org/jira/browse/SPARK-48095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48095: --- Labels: pull-request-available (was: ) > Run `build_non_ansi.yml` once per day >

[jira] [Created] (SPARK-48095) Run `build_non_ansi.yml` once per day

Dongjoon Hyun created SPARK-48095: - Summary: Run `build_non_ansi.yml` once per day Key: SPARK-48095 URL: https://issues.apache.org/jira/browse/SPARK-48095 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-48093) Add config to switch between client side listener and server side listener

[ https://issues.apache.org/jira/browse/SPARK-48093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48093: --- Labels: pull-request-available (was: ) > Add config to switch between client side listener

[jira] [Updated] (SPARK-48081) Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type

[ https://issues.apache.org/jira/browse/SPARK-48081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48081: -- Fix Version/s: 3.5.2 3.4.4 > Fix ClassCastException in

[jira] [Updated] (SPARK-48043) Kryo serialization issue with push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-48043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Ardiet updated SPARK-48043: -- Description: I'm running a spark job on AWS EMR. I wanted to test the new push-based shuffle

[jira] [Updated] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-36705: --- Labels: pull-request-available (was: ) > Disable push based shuffle when IO encryption is

[jira] [Updated] (SPARK-48043) Kryo serialization issue with push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-48043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Ardiet updated SPARK-48043: -- Description: I'm running a spark job on AWS EMR. I wanted to test the new push-based shuffle

[jira] [Updated] (SPARK-48043) Kryo serialization issue with push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-48043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Ardiet updated SPARK-48043: -- Description: I'm running a spark job on AWS EMR. I wanted to test the new push-based shuffle

[jira] [Updated] (SPARK-48043) Kryo serialization issue with push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-48043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Ardiet updated SPARK-48043: -- Description: I'm running a spark job on AWS EMR. I wanted to test the new push-based shuffle

[jira] [Updated] (SPARK-48043) Kryo serialization issue with push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-48043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romain Ardiet updated SPARK-48043: -- Description: I'm running a spark job on AWS EMR. I wanted to test the new push-based shuffle

[jira] [Created] (SPARK-48093) Add config to switch between client side listener and server side listener

Wei Liu created SPARK-48093: --- Summary: Add config to switch between client side listener and server side listener Key: SPARK-48093 URL: https://issues.apache.org/jira/browse/SPARK-48093 Project: Spark

[jira] [Resolved] (SPARK-48067) Fix Variant default columns for more complex default variants

[ https://issues.apache.org/jira/browse/SPARK-48067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-48067. Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46312

[jira] [Assigned] (SPARK-48067) Fix Variant default columns for more complex default variants

[ https://issues.apache.org/jira/browse/SPARK-48067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-48067: -- Assignee: Richard Chen > Fix Variant default columns for more complex default

[jira] [Updated] (SPARK-48089) Streaming query listener not working in 3.5 client <> 4.0 server

[ https://issues.apache.org/jira/browse/SPARK-48089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48089: --- Labels: pull-request-available (was: ) > Streaming query listener not working in 3.5

[jira] [Updated] (SPARK-45988) Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11

[ https://issues.apache.org/jira/browse/SPARK-45988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45988: -- Fix Version/s: 3.4.4 > Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11

[jira] [Updated] (SPARK-45989) Fix `pyspark.pandas.tests.connect.computation.test_parity_apply_func` in Python 3.11

[ https://issues.apache.org/jira/browse/SPARK-45989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45989: -- Fix Version/s: 3.4.4 > Fix `pyspark.pandas.tests.connect.computation.test_parity_apply_func`

[jira] [Updated] (SPARK-48081) Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type

[ https://issues.apache.org/jira/browse/SPARK-48081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48081: -- Fix Version/s: (was: 3.5.2) (was: 3.4.4) > Fix ClassCastException

[jira] [Updated] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL

[ https://issues.apache.org/jira/browse/SPARK-48092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Madnani updated SPARK-48092: -- Description: Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on

[jira] [Updated] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL

[ https://issues.apache.org/jira/browse/SPARK-48092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Madnani updated SPARK-48092: -- Attachment: Screenshot 2024-05-02 at 21.00.18.png Screenshot 2024-05-02 at

[jira] [Updated] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL

[ https://issues.apache.org/jira/browse/SPARK-48092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayur Madnani updated SPARK-48092: -- Description: Python 3.8 will be EOL in Oct 2024 and all the Spark docker images are based on

[jira] [Created] (SPARK-48092) Spark images are based of Python 3.8 which is soon EOL

Mayur Madnani created SPARK-48092: - Summary: Spark images are based of Python 3.8 which is soon EOL Key: SPARK-48092 URL: https://issues.apache.org/jira/browse/SPARK-48092 Project: Spark

[jira] [Resolved] (SPARK-48079) Upgrade maven-install/deploy-plugin to 3.1.2

[ https://issues.apache.org/jira/browse/SPARK-48079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48079. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46330

[jira] [Resolved] (SPARK-48081) Fix ClassCastException in NTile.checkInputDataTypes() when argument is non-foldable or of wrong type

[ https://issues.apache.org/jira/browse/SPARK-48081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48081. --- Fix Version/s: 3.4.4 3.5.2 4.0.0 Resolution:

[jira] [Updated] (SPARK-48072) Improve SQLQuerySuite test output - use `===` instead of `sameElements` for Arrays

[ https://issues.apache.org/jira/browse/SPARK-48072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48072: -- Summary: Improve SQLQuerySuite test output - use `===` instead of `sameElements` for Arrays

[jira] [Assigned] (SPARK-48072) Test output is not descriptive for some Array comparisons in SQLQuerySuite

[ https://issues.apache.org/jira/browse/SPARK-48072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-48072: - Assignee: Vladimir Golubev > Test output is not descriptive for some Array comparisons

[jira] [Resolved] (SPARK-48072) Test output is not descriptive for some Array comparisons in SQLQuerySuite

[ https://issues.apache.org/jira/browse/SPARK-48072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-48072. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46318

[jira] [Updated] (SPARK-48091) Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored

[ https://issues.apache.org/jira/browse/SPARK-48091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Serruya updated SPARK-48091: Description: When using an `explode` function, and `transform` function in the same select

[jira] [Created] (SPARK-48091) Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored

Ron Serruya created SPARK-48091: --- Summary: Using `explode` together with `transform` in the same select statement causes aliases in the transformed column to be ignored Key: SPARK-48091 URL:

[jira] [Resolved] (SPARK-48056) [CONNECT][PYTHON] Session not found error should automatically retry during reattach

[ https://issues.apache.org/jira/browse/SPARK-48056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48056. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 46297

[jira] [Updated] (SPARK-48088) Skip tests being failed in client 3.5 <> server 4.0

[ https://issues.apache.org/jira/browse/SPARK-48088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-48088: --- Labels: pull-request-available (was: ) > Skip tests being failed in client 3.5 <> server

[jira] [Created] (SPARK-48090) Streaming exception catch failure in 3.5 client <> 4.0 server

Hyukjin Kwon created SPARK-48090: Summary: Streaming exception catch failure in 3.5 client <> 4.0 server Key: SPARK-48090 URL: https://issues.apache.org/jira/browse/SPARK-48090 Project: Spark

[jira] [Created] (SPARK-48089) Streaming query listener not working in 3.5 client <> 4.0 server

Hyukjin Kwon created SPARK-48089: Summary: Streaming query listener not working in 3.5 client <> 4.0 server Key: SPARK-48089 URL: https://issues.apache.org/jira/browse/SPARK-48089 Project: Spark

[jira] [Resolved] (SPARK-48085) ANSI enabled by default brings different results in the tests in 3.5 client <> 4.0 server

[ https://issues.apache.org/jira/browse/SPARK-48085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-48085. -- Resolution: Invalid We actually made this all compatible in master branch. I will avoid

[jira] [Updated] (SPARK-48085) ANSI enabled by default brings different results in the tests in 3.5 client <> 4.0 server

[ https://issues.apache.org/jira/browse/SPARK-48085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-48085: - Description: {code} == FAIL

[jira] [Updated] (SPARK-48088) Skip tests being failed in client 3.5 <> server 4.0

[ https://issues.apache.org/jira/browse/SPARK-48088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-48088: - Affects Version/s: 3.5.2 (was: 4.0.0) > Skip tests being failed in

  1   2   >