[jira] [Created] (SPARK-49853) Increase test timeout of PythonForeachWriterSuite

Dongjoon Hyun created SPARK-49853: - Summary: Increase test timeout of PythonForeachWriterSuite Key: SPARK-49853 URL: https://issues.apache.org/jira/browse/SPARK-49853 Project: Spark Issue Typ

[jira] [Updated] (SPARK-49853) Increase test timeout of PythonForeachWriterSuite to 60s

[ https://issues.apache.org/jira/browse/SPARK-49853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49853: -- Summary: Increase test timeout of PythonForeachWriterSuite to 60s (was: Increase test timeout

[jira] [Updated] (SPARK-47866) Deflaky PythonForeachWriterSuite

[ https://issues.apache.org/jira/browse/SPARK-47866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47866: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Bug) > Deflaky PythonForeachWriterSuit

[jira] [Updated] (SPARK-44707) Use INFO log in ExecutorPodsWatcher.onClose if SparkContext is stopped

[ https://issues.apache.org/jira/browse/SPARK-44707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44707: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Improvement) > Use INFO log in Executo

[jira] [Updated] (SPARK-44115) Upgrade ORC to 2.0

[ https://issues.apache.org/jira/browse/SPARK-44115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44115: -- Summary: Upgrade ORC to 2.0 (was: Upgrade Apache ORC to 2.0) > Upgrade ORC to 2.0 > -

[jira] [Updated] (SPARK-46998) The SQL config spark.sql.legacy.allowZeroIndexInFormatString doesn't work

[ https://issues.apache.org/jira/browse/SPARK-46998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46998: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Bug) > The SQL config spark.sql.legacy

[jira] [Updated] (SPARK-46998) Deprecate `spark.sql.legacy.allowZeroIndexInFormatString`

[ https://issues.apache.org/jira/browse/SPARK-46998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46998: -- Summary: Deprecate `spark.sql.legacy.allowZeroIndexInFormatString` (was: The SQL config spark

[jira] [Updated] (SPARK-44996) VolcanoFeatureStep should not create `DefaultVolcanoClient` if not needed

[ https://issues.apache.org/jira/browse/SPARK-44996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44996: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Improvement) > VolcanoFeatureStep shou

[jira] [Updated] (SPARK-45407) Skip Unidoc in SparkR GitHub Action Job

[ https://issues.apache.org/jira/browse/SPARK-45407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45407: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Test) > Skip Unidoc in SparkR GitHub A

[jira] [Updated] (SPARK-45010) Limit GHA job execution time to up to 5 hours in build_and_test.yml

[ https://issues.apache.org/jira/browse/SPARK-45010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45010: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Task) > Limit GHA job execution time t

[jira] [Updated] (SPARK-44995) Promote SparkKubernetesClientFactory to DeveloperApi

[ https://issues.apache.org/jira/browse/SPARK-44995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44995: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Improvement) > Promote SparkKubernetes

[jira] [Updated] (SPARK-44993) Add ShuffleChecksumUtils.compareChecksums by reusing ShuffleChecksumTestHelp.compareChecksums

[ https://issues.apache.org/jira/browse/SPARK-44993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44993: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Improvement) > Add ShuffleChecksumUtil

[jira] [Updated] (SPARK-44936) Simplify the log when Spark HybridStore hits the memory limit

[ https://issues.apache.org/jira/browse/SPARK-44936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44936: -- Parent: SPARK-45869 Issue Type: Sub-task (was: Improvement) > Simplify the log when S

[jira] [Updated] (SPARK-44925) K8s default service token file should not be materialized into token

[ https://issues.apache.org/jira/browse/SPARK-44925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44925: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Bug) > K8s default service token file

[jira] [Updated] (SPARK-44745) Document shuffle data recovery from the remounted K8s PVCs

[ https://issues.apache.org/jira/browse/SPARK-44745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44745: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Documentation) > Document shuffle data

[jira] [Updated] (SPARK-44821) Upgrade `kubernetes-client` to 6.8.1

[ https://issues.apache.org/jira/browse/SPARK-44821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44821: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Dependency upgrade) > Upgrade `kuberne

[jira] [Updated] (SPARK-44725) Document spark.network.timeoutInterval

[ https://issues.apache.org/jira/browse/SPARK-44725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-44725: -- Parent: SPARK-45869 Issue Type: Sub-task (was: Documentation) > Document spark.networ

[jira] [Updated] (SPARK-45145) Add JavaSparkSQLCli example

[ https://issues.apache.org/jira/browse/SPARK-45145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45145: -- Parent: SPARK-45869 Issue Type: Sub-task (was: Improvement) > Add JavaSparkSQLCli exa

[jira] [Updated] (SPARK-47875) Remove `spark.deploy.recoverySerializer`

[ https://issues.apache.org/jira/browse/SPARK-47875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47875: -- Parent: (was: SPARK-45869) Issue Type: Task (was: Sub-task) > Remove `spark.deplo

[jira] [Updated] (SPARK-45462) Show `Duration` in `ApplicationPage`

[ https://issues.apache.org/jira/browse/SPARK-45462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45462: -- Parent: SPARK-45869 Issue Type: Sub-task (was: Improvement) > Show `Duration` in `App

[jira] [Updated] (SPARK-45587) Skip UNIDOC and MIMA in build GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-45587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45587: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Improvement) > Skip UNIDOC and MIMA in

[jira] [Updated] (SPARK-45741) Upgrade Netty to 4.1.100.Final

[ https://issues.apache.org/jira/browse/SPARK-45741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45741: -- Parent: SPARK-47046 Issue Type: Sub-task (was: Bug) > Upgrade Netty to 4.1.100.Final

[jira] [Updated] (SPARK-46735) `pyspark.sql.tests.test_group` should skip Pandas/PyArrow tests if not available

[ https://issues.apache.org/jira/browse/SPARK-46735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46735: -- Parent: SPARK-45981 Issue Type: Sub-task (was: Test) > `pyspark.sql.tests.test_group`

[jira] [Updated] (SPARK-45978) Skip unidoc step in docker-integration-tests GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-45978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-45978: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Task) > Skip unidoc step in docker-int

[jira] [Updated] (SPARK-46435) Exclude `branch-3.3` from `publish_snapshot.yml`

[ https://issues.apache.org/jira/browse/SPARK-46435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46435: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Task) > Exclude `branch-3.3` from `pub

[jira] [Updated] (SPARK-46570) Run Python 3.11 and 3.12 test independently

[ https://issues.apache.org/jira/browse/SPARK-46570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46570: -- Parent: SPARK-45981 Issue Type: Sub-task (was: Test) > Run Python 3.11 and 3.12 test

[jira] [Updated] (SPARK-46657) Install `lxml` in Python 3.12

[ https://issues.apache.org/jira/browse/SPARK-46657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46657: -- Parent: SPARK-45981 Issue Type: Sub-task (was: Bug) > Install `lxml` in Python 3.12 >

[jira] [Updated] (SPARK-46316) Enable `buf-lint-action` on `core` module

[ https://issues.apache.org/jira/browse/SPARK-46316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46316: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Task) > Enable `buf-lint-action` on `c

[jira] [Updated] (SPARK-46786) Fix MountVolumesFeatureStep to use ReadWriteOncePod instead of ReadWriteOnce

[ https://issues.apache.org/jira/browse/SPARK-46786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46786: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Bug) > Fix MountVolumesFeatureStep to

[jira] [Updated] (SPARK-46628) Use SPDX short identifier in `licenses` name

[ https://issues.apache.org/jira/browse/SPARK-46628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46628: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Task) > Use SPDX short identifier in `

[jira] [Updated] (SPARK-46780) Improve SBT K8s IT to skip R image build step if not needed

[ https://issues.apache.org/jira/browse/SPARK-46780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46780: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Test) > Improve SBT K8s IT to skip R i

[jira] [Updated] (SPARK-46945) Add `spark.kubernetes.legacy.useReadWriteOnceAccessMode` for old K8s clusters

[ https://issues.apache.org/jira/browse/SPARK-46945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46945: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Bug) > Add `spark.kubernetes.legacy.us

[jira] [Updated] (SPARK-47535) Update `publish_snapshot.yml` to publish twice per day

[ https://issues.apache.org/jira/browse/SPARK-47535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47535: -- Parent: SPARK-48094 Issue Type: Sub-task (was: Task) > Update `publish_snapshot.yml`

[jira] [Updated] (SPARK-48046) Remove `clock` parameter from `DriverServiceFeatureStep`

[ https://issues.apache.org/jira/browse/SPARK-48046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48046: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Task) > Remove `clock` parameter from

[jira] [Updated] (SPARK-48855) Make ExecutorPodsAllocatorSuite independent from default allocation batch size

[ https://issues.apache.org/jira/browse/SPARK-48855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-48855: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Test) > Make ExecutorPodsAllocatorSuit

[jira] [Updated] (SPARK-49381) Fix `SPARK-41388` test case to assert the result

[ https://issues.apache.org/jira/browse/SPARK-49381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49381: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Bug) > Fix `SPARK-41388` test case to

[jira] [Updated] (SPARK-49385) Fix `getReusablePVCs` to use `podCreationTimeout` instead of `podAllocationDelay`

[ https://issues.apache.org/jira/browse/SPARK-49385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49385: -- Parent: SPARK-49524 Issue Type: Sub-task (was: Bug) > Fix `getReusablePVCs` to use `p

[jira] [Updated] (SPARK-49347) Deprecate SaprkR

[ https://issues.apache.org/jira/browse/SPARK-49347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49347: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Task) > Deprecate SaprkR > ---

[jira] [Updated] (SPARK-49732) Spark deamons should respect spark.log.structuredLogging.enabled conf

[ https://issues.apache.org/jira/browse/SPARK-49732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49732: -- Parent Issue: SPARK-45869 (was: SPARK-44111) > Spark deamons should respect spark.log.structu

[jira] [Resolved] (SPARK-49845) Make `appArgs` and `environmentVariables` optional in REST API

[ https://issues.apache.org/jira/browse/SPARK-49845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49845. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48316 [https://

[jira] [Updated] (SPARK-49836) The outer query is broken when the subquery uses window function which receives time window as parameter

[ https://issues.apache.org/jira/browse/SPARK-49836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-49836: - Component/s: SQL > The outer query is broken when the subquery uses window function which > rec

[jira] [Updated] (SPARK-49836) The outer query is broken when the subquery uses window function which receives time window as parameter

[ https://issues.apache.org/jira/browse/SPARK-49836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-49836: - Reporter: Andrzej Zera (was: Jungtaek Lim) > The outer query is broken when the subquery uses w

[jira] [Updated] (SPARK-49836) The outer query is broken when the subquery uses window function which receives time window as parameter

[ https://issues.apache.org/jira/browse/SPARK-49836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-49836: - Description: NOTE: The ticket is describing the actual issue after RCA of tests the reporter re

[jira] [Created] (SPARK-49852) Fix deadlock between shuffle dependency initialization and explain string generation

Ziqi Liu created SPARK-49852: Summary: Fix deadlock between shuffle dependency initialization and explain string generation Key: SPARK-49852 URL: https://issues.apache.org/jira/browse/SPARK-49852 Project:

[jira] [Comment Edited] (SPARK-49727) A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset

[ https://issues.apache.org/jira/browse/SPARK-49727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883147#comment-17883147 ] Asif edited comment on SPARK-49727 at 10/1/24 11:57 PM: This iss

[jira] [Updated] (SPARK-49727) A Bean class with a serializable POJO field looses data when converted back from dataframe, as dataset

[ https://issues.apache.org/jira/browse/SPARK-49727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Asif updated SPARK-49727: - Description: If a Bean class contains a serializable POJO  ( without any getter/setter) property,  the encoder

[jira] [Updated] (SPARK-49609) Add API compatibility check between Classic and Connect

[ https://issues.apache.org/jira/browse/SPARK-49609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-49609: Parent: SPARK-49847 Issue Type: Sub-task (was: Bug) > Add API compatibility check between

[jira] [Created] (SPARK-49848) API compatibility check for Spark SQL

Haejoon Lee created SPARK-49848: --- Summary: API compatibility check for Spark SQL Key: SPARK-49848 URL: https://issues.apache.org/jira/browse/SPARK-49848 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-49847) PySpark compatibility with Spark Connect

[ https://issues.apache.org/jira/browse/SPARK-49847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-49847: Description: This aims to ensure full compatibility between PySpark and Spark Connect by thorough

[jira] [Created] (SPARK-49850) API compatibility check for Pandas API on Spark

Haejoon Lee created SPARK-49850: --- Summary: API compatibility check for Pandas API on Spark Key: SPARK-49850 URL: https://issues.apache.org/jira/browse/SPARK-49850 Project: Spark Issue Type: Sub

[jira] [Created] (SPARK-49851) API compatibility check for Spark Core

Haejoon Lee created SPARK-49851: --- Summary: API compatibility check for Spark Core Key: SPARK-49851 URL: https://issues.apache.org/jira/browse/SPARK-49851 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-49847) PySpark compatibility with Spark Connect

Haejoon Lee created SPARK-49847: --- Summary: PySpark compatibility with Spark Connect Key: SPARK-49847 URL: https://issues.apache.org/jira/browse/SPARK-49847 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-49846) Update numUpdated and numRemoved state rows with transformWithState operator

Anish Shrigondekar created SPARK-49846: -- Summary: Update numUpdated and numRemoved state rows with transformWithState operator Key: SPARK-49846 URL: https://issues.apache.org/jira/browse/SPARK-49846

[jira] [Commented] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886272#comment-17886272 ] Asif commented on SPARK-46679: -- [~andoni.teso] The link to PR which fixes the issue is upda

[jira] [Updated] (SPARK-49631) Remove the ANSI config suggestion in ARITHMETIC_OVERFLOW

[ https://issues.apache.org/jira/browse/SPARK-49631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mihailo Milosevic updated SPARK-49631: -- Labels: pull-request-available (was: pull-request-available starter) > Remove the ANS

[jira] [Created] (SPARK-49845) Make `appArgs` and `environmentVariables` optional in REST API

Dongjoon Hyun created SPARK-49845: - Summary: Make `appArgs` and `environmentVariables` optional in REST API Key: SPARK-49845 URL: https://issues.apache.org/jira/browse/SPARK-49845 Project: Spark

[jira] [Created] (SPARK-49844) PySpark requiring vulnerable Apache Zookeeper version 3.6.3

Zach Barnett created SPARK-49844: Summary: PySpark requiring vulnerable Apache Zookeeper version 3.6.3 Key: SPARK-49844 URL: https://issues.apache.org/jira/browse/SPARK-49844 Project: Spark

[jira] [Created] (SPARK-49843) Fix change comment for char/varchar columns

Stefan Kandic created SPARK-49843: - Summary: Fix change comment for char/varchar columns Key: SPARK-49843 URL: https://issues.apache.org/jira/browse/SPARK-49843 Project: Spark Issue Type: Imp

[jira] [Created] (SPARK-49842) For modules that depend on `mockito-core`, `byte-buddy` should also be defined so that `sbt test` can use the correct version of `byte-buddy`.

Yang Jie created SPARK-49842: Summary: For modules that depend on `mockito-core`, `byte-buddy` should also be defined so that `sbt test` can use the correct version of `byte-buddy`. Key: SPARK-49842 URL: https://issu

[jira] [Resolved] (SPARK-48700) Mode expression for Arrays and Structs (all collations)

[ https://issues.apache.org/jira/browse/SPARK-48700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-48700. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47154 [https://github.com

[jira] [Assigned] (SPARK-48700) Mode expression for Arrays and Structs (all collations)

[ https://issues.apache.org/jira/browse/SPARK-48700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-48700: Assignee: Gideon P > Mode expression for Arrays and Structs (all collations) > --