[jira] [Comment Edited] (SPARK-53052) emptyValue option does not seem to work from pyspark

[ https://issues.apache.org/jira/browse/SPARK-53052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011540#comment-18011540 ] Vindhya G edited comment on SPARK-53052 at 8/2/25 5:46 AM: --- [R

[jira] [Comment Edited] (SPARK-53052) emptyValue option does not seem to work from pyspark

[ https://issues.apache.org/jira/browse/SPARK-53052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011540#comment-18011540 ] Vindhya G edited comment on SPARK-53052 at 8/2/25 5:32 AM: --- [R

[jira] [Comment Edited] (SPARK-53052) emptyValue option does not seem to work from pyspark

[ https://issues.apache.org/jira/browse/SPARK-53052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011540#comment-18011540 ] Vindhya G edited comment on SPARK-53052 at 8/2/25 5:15 AM: --- [R

[jira] [Comment Edited] (SPARK-53052) emptyValue option does not seem to work from pyspark

[ https://issues.apache.org/jira/browse/SPARK-53052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011540#comment-18011540 ] Vindhya G edited comment on SPARK-53052 at 8/2/25 5:14 AM: --- [R

[jira] [Commented] (SPARK-53052) emptyValue option does not seem to work from pyspark

[ https://issues.apache.org/jira/browse/SPARK-53052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011540#comment-18011540 ] Vindhya G commented on SPARK-53052: --- [Row(id='1', val=None), Row(id='2', val='emptyStr

[jira] [Assigned] (SPARK-53075) Use Java `Files.readAllLines/write` instead of `FileUtils.(read|write)Lines`

[ https://issues.apache.org/jira/browse/SPARK-53075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-53075: - Assignee: Dongjoon Hyun > Use Java `Files.readAllLines/write` instead of `FileUtils.(re

[jira] [Updated] (SPARK-53075) Use Java `Files.readAllLines/write` instead of `FileUtils.(read|write)Lines`

[ https://issues.apache.org/jira/browse/SPARK-53075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53075: --- Labels: pull-request-available (was: ) > Use Java `Files.readAllLines/write` instead of `Fi

[jira] [Created] (SPARK-53075) Use Java `Files.readAllLines/write` instead of `FileUtils.(read|write)Lines`

Dongjoon Hyun created SPARK-53075: - Summary: Use Java `Files.readAllLines/write` instead of `FileUtils.(read|write)Lines` Key: SPARK-53075 URL: https://issues.apache.org/jira/browse/SPARK-53075 Projec

[jira] [Resolved] (SPARK-53071) Support `copyFile` in `SparkFileUtils`

[ https://issues.apache.org/jira/browse/SPARK-53071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-53071. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51784 [https://

[jira] [Created] (SPARK-53074) Avoid partially clustered distribution when key-grouped partitioning used to satisfy non-JOIN required distribution

Chirag Singh created SPARK-53074: Summary: Avoid partially clustered distribution when key-grouped partitioning used to satisfy non-JOIN required distribution Key: SPARK-53074 URL: https://issues.apache.org/jira/b

[jira] [Assigned] (SPARK-53073) Support `copyDirectory` in `SparkFileUtils` and `JavaUtils`

[ https://issues.apache.org/jira/browse/SPARK-53073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-53073: - Assignee: Dongjoon Hyun > Support `copyDirectory` in `SparkFileUtils` and `JavaUtils` >

[jira] [Updated] (SPARK-53073) Support `copyDirectory` in `SparkFileUtils` and `JavaUtils`

[ https://issues.apache.org/jira/browse/SPARK-53073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53073: --- Labels: pull-request-available (was: ) > Support `copyDirectory` in `SparkFileUtils` and `J

[jira] [Created] (SPARK-53073) Support `copyDirectory` in `SparkFileUtils` and `JavaUtils`

Dongjoon Hyun created SPARK-53073: - Summary: Support `copyDirectory` in `SparkFileUtils` and `JavaUtils` Key: SPARK-53073 URL: https://issues.apache.org/jira/browse/SPARK-53073 Project: Spark

[jira] [Resolved] (SPARK-53070) Support `is(Not)?Empty` in `SparkCollectionUtils`

[ https://issues.apache.org/jira/browse/SPARK-53070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-53070. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51783 [https://

[jira] [Assigned] (SPARK-53068) Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core`

[ https://issues.apache.org/jira/browse/SPARK-53068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-53068: - Assignee: Dongjoon Hyun > Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core` > -

[jira] [Resolved] (SPARK-53068) Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core`

[ https://issues.apache.org/jira/browse/SPARK-53068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-53068. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51782 [https://

[jira] [Updated] (SPARK-53072) Only run Lock Hardening tests in RocksDBStateStoreLockHardeningSuite

[ https://issues.apache.org/jira/browse/SPARK-53072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53072: --- Labels: pull-request-available (was: ) > Only run Lock Hardening tests in RocksDBStateStore

[jira] [Created] (SPARK-53072) Only run Lock Hardening tests in RocksDBStateStoreLockHardeningSuite

Eric Marnadi created SPARK-53072: Summary: Only run Lock Hardening tests in RocksDBStateStoreLockHardeningSuite Key: SPARK-53072 URL: https://issues.apache.org/jira/browse/SPARK-53072 Project: Spark

[jira] [Assigned] (SPARK-53071) Support `copyFile` in `SparkFileUtils`

[ https://issues.apache.org/jira/browse/SPARK-53071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-53071: - Assignee: Dongjoon Hyun > Support `copyFile` in `SparkFileUtils` >

[jira] [Updated] (SPARK-53071) Support `copyFile` in `SparkFileUtils`

[ https://issues.apache.org/jira/browse/SPARK-53071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53071: --- Labels: pull-request-available (was: ) > Support `copyFile` in `SparkFileUtils` > -

[jira] [Created] (SPARK-53071) Support `copyFile` in `SparkFileUtils`

Dongjoon Hyun created SPARK-53071: - Summary: Support `copyFile` in `SparkFileUtils` Key: SPARK-53071 URL: https://issues.apache.org/jira/browse/SPARK-53071 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-53070) Support `is(Not)?Empty` in `SparkCollectionUtils`

[ https://issues.apache.org/jira/browse/SPARK-53070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-53070: - Assignee: Dongjoon Hyun > Support `is(Not)?Empty` in `SparkCollectionUtils` > -

[jira] [Resolved] (SPARK-53058) Clean up old comments related to HashClusteredDistribution

[ https://issues.apache.org/jira/browse/SPARK-53058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-53058. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51766 [https://

[jira] [Updated] (SPARK-52989) Add explicit close API to RocksDB State store iterator and fix current usage

[ https://issues.apache.org/jira/browse/SPARK-52989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52989: --- Labels: pull-request-available (was: ) > Add explicit close API to RocksDB State store iter

[jira] [Commented] (SPARK-52968) Emit additional streaming metrics

[ https://issues.apache.org/jira/browse/SPARK-52968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011491#comment-18011491 ] Anish Shrigondekar commented on SPARK-52968: [~dylanwong250] - I'm unable to

[jira] [Resolved] (SPARK-52968) Emit additional streaming metrics

[ https://issues.apache.org/jira/browse/SPARK-52968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar resolved SPARK-52968. Fix Version/s: 4.1.0 Resolution: Fixed > Emit additional streaming metrics

[jira] [Commented] (SPARK-52968) Emit additional streaming metrics

[ https://issues.apache.org/jira/browse/SPARK-52968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011490#comment-18011490 ] Anish Shrigondekar commented on SPARK-52968: PR merged here - [https://githu

[jira] [Assigned] (SPARK-52968) Emit additional streaming metrics

[ https://issues.apache.org/jira/browse/SPARK-52968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar reassigned SPARK-52968: -- Assignee: (was: Anish Shrigondekar) > Emit additional streaming metrics >

[jira] [Assigned] (SPARK-52968) Emit additional streaming metrics

[ https://issues.apache.org/jira/browse/SPARK-52968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anish Shrigondekar reassigned SPARK-52968: -- Assignee: Anish Shrigondekar > Emit additional streaming metrics > --

[jira] [Created] (SPARK-53070) Support `is(Not)?Empty` in `SparkCollectionUtils`

Dongjoon Hyun created SPARK-53070: - Summary: Support `is(Not)?Empty` in `SparkCollectionUtils` Key: SPARK-53070 URL: https://issues.apache.org/jira/browse/SPARK-53070 Project: Spark Issue Typ

[jira] [Updated] (SPARK-53070) Support `is(Not)?Empty` in `SparkCollectionUtils`

[ https://issues.apache.org/jira/browse/SPARK-53070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53070: --- Labels: pull-request-available (was: ) > Support `is(Not)?Empty` in `SparkCollectionUtils`

[jira] [Created] (SPARK-53069) Fix incorrect metrics with virtual column families

Dylan Wong created SPARK-53069: -- Summary: Fix incorrect metrics with virtual column families Key: SPARK-53069 URL: https://issues.apache.org/jira/browse/SPARK-53069 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-53052) emptyValue option does not seem to work from pyspark

[ https://issues.apache.org/jira/browse/SPARK-53052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kash Bhatt updated SPARK-53052: --- Description: According to [docs|https://spark.apache.org/docs/latest/sql-data-sources-csv.html#data

[jira] [Updated] (SPARK-53068) Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core`

[ https://issues.apache.org/jira/browse/SPARK-53068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53068: --- Labels: pull-request-available (was: ) > Mark `TransformWith*Suite` as `SlowSQLTest` in `sq

[jira] [Created] (SPARK-53068) Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core` module

Dongjoon Hyun created SPARK-53068: - Summary: Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core` module Key: SPARK-53068 URL: https://issues.apache.org/jira/browse/SPARK-53068 Project: Spark

[jira] [Updated] (SPARK-53068) Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core`

[ https://issues.apache.org/jira/browse/SPARK-53068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-53068: -- Summary: Mark `TransformWith*Suite` as `SlowSQLTest` in `sql/core` (was: Mark `TransformWith*

[jira] [Resolved] (SPARK-53061) Support `copyFileToDirectory` in `SparkFileUtils`

[ https://issues.apache.org/jira/browse/SPARK-53061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-53061. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51770 [https://

[jira] [Updated] (SPARK-53066) Improve EXPLAIN output for DSv2 Join pushdown

[ https://issues.apache.org/jira/browse/SPARK-53066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53066: --- Labels: pull-request-available (was: ) > Improve EXPLAIN output for DSv2 Join pushdown > --

[jira] [Updated] (SPARK-53067) Increase GCLockerRetryAllocationCount for SBT

[ https://issues.apache.org/jira/browse/SPARK-53067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53067: --- Labels: pull-request-available (was: ) > Increase GCLockerRetryAllocationCount for SBT > --

[jira] [Created] (SPARK-53067) Increase GCLockerRetryAllocationCount for SBT

Cheng Pan created SPARK-53067: - Summary: Increase GCLockerRetryAllocationCount for SBT Key: SPARK-53067 URL: https://issues.apache.org/jira/browse/SPARK-53067 Project: Spark Issue Type: Improveme

[jira] [Created] (SPARK-53066) Improve EXPLAIN output for DSv2 Join pushdown

Petar Vasiljevic created SPARK-53066: Summary: Improve EXPLAIN output for DSv2 Join pushdown Key: SPARK-53066 URL: https://issues.apache.org/jira/browse/SPARK-53066 Project: Spark Issue T

[jira] [Commented] (SPARK-42360) Transform LeftOuter join with IsNull filter on right side to Anti join

[ https://issues.apache.org/jira/browse/SPARK-42360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18011469#comment-18011469 ] Eugene Koifman commented on SPARK-42360: https://github.com/apache/spark/pull/51

[jira] [Resolved] (SPARK-53059) Arrow UDF no need to depend on pandas

[ https://issues.apache.org/jira/browse/SPARK-53059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-53059. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51767 [https://

[jira] [Assigned] (SPARK-53059) Arrow UDF no need to depend on pandas

[ https://issues.apache.org/jira/browse/SPARK-53059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-53059: - Assignee: Ruifeng Zheng > Arrow UDF no need to depend on pandas > -

[jira] [Resolved] (SPARK-53062) Support `deleteQuietly` in `SparkFileUtils` and `JavaUtils`

[ https://issues.apache.org/jira/browse/SPARK-53062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-53062. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51771 [https://

[jira] [Updated] (SPARK-52996) Update brace-expansion to 1.1.12

[ https://issues.apache.org/jira/browse/SPARK-52996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cameron updated SPARK-52996: Description: brace-expansion 1.1.11 contains CVE-2025-5889. Updating to 1.1.12 removes it (was: brace-exp

[jira] [Updated] (SPARK-52996) Update brace-expansion to 1.1.12

[ https://issues.apache.org/jira/browse/SPARK-52996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cameron updated SPARK-52996: Affects Version/s: 4.1.0 (was: 4.0.0) > Update brace-expansion to 1.1.12 >

[jira] [Updated] (SPARK-52469) Use JEP 223 API to process Java version

[ https://issues.apache.org/jira/browse/SPARK-52469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-52469: -- Parent Issue: SPARK-53047 (was: SPARK-51166) > Use JEP 223 API to process Java version >

[jira] [Updated] (SPARK-52793) Support `isUnix` at `o.a.s.util.Utils`

[ https://issues.apache.org/jira/browse/SPARK-52793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-52793: -- Parent Issue: SPARK-53047 (was: SPARK-51166) > Support `isUnix` at `o.a.s.util.Utils` > -

[jira] [Updated] (SPARK-52928) Upgrade the minimum version of pyarrow to 15.0.0

[ https://issues.apache.org/jira/browse/SPARK-52928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-52928: -- Parent: SPARK-51166 Issue Type: Sub-task (was: Test) > Upgrade the minimum version of

[jira] [Assigned] (SPARK-52772) Inconsistent table attribute handling during updates

[ https://issues.apache.org/jira/browse/SPARK-52772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-52772: --- Assignee: Kevin Beyer > Inconsistent table attribute handling during updates >

[jira] [Resolved] (SPARK-52772) Inconsistent table attribute handling during updates

[ https://issues.apache.org/jira/browse/SPARK-52772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-52772. - Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51466 [https://gith

[jira] [Updated] (SPARK-52942) YARN External Shuffle Service fails to start

[ https://issues.apache.org/jira/browse/SPARK-52942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-52942: -- Priority: Blocker (was: Critical) > YARN External Shuffle Service fails to start > --

[jira] [Updated] (SPARK-52942) YARN External Shuffle Service fails to start

[ https://issues.apache.org/jira/browse/SPARK-52942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-52942: -- Priority: Critical (was: Major) > YARN External Shuffle Service fails to start >

[jira] [Assigned] (SPARK-53054) The Scala Spark Connect DataFrameReader does not use the correct default format

[ https://issues.apache.org/jira/browse/SPARK-53054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-53054: Assignee: Robert Dillitz > The Scala Spark Connect DataFrameReader does not use the corre

[jira] [Resolved] (SPARK-53054) The Scala Spark Connect DataFrameReader does not use the correct default format

[ https://issues.apache.org/jira/browse/SPARK-53054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-53054. -- Fix Version/s: 4.1.0 4.0.1 3.5.7 Resolution: Fixed

[jira] [Updated] (SPARK-53065) [RISC-V Support] - Comprehensive Tracking of RISC-V Architecture Support

[ https://issues.apache.org/jira/browse/SPARK-53065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gong Xiaofei updated SPARK-53065: - Description: This JIRA serves as an umbrella issue to comprehensively track all enhancements, o

[jira] [Updated] (SPARK-53064) Rewrite MDC LogKey in Java

[ https://issues.apache.org/jira/browse/SPARK-53064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-53064: --- Labels: pull-request-available (was: ) > Rewrite MDC LogKey in Java > -

[jira] [Created] (SPARK-53065) [RISC-V Support] - Comprehensive Tracking of RISC-V Architecture Support

Gong Xiaofei created SPARK-53065: Summary: [RISC-V Support] - Comprehensive Tracking of RISC-V Architecture Support Key: SPARK-53065 URL: https://issues.apache.org/jira/browse/SPARK-53065 Project: Spa

[jira] [Created] (SPARK-53064) Rewrite MDC LogKey in Java

Cheng Pan created SPARK-53064: - Summary: Rewrite MDC LogKey in Java Key: SPARK-53064 URL: https://issues.apache.org/jira/browse/SPARK-53064 Project: Spark Issue Type: Improvement Compon