[jira] [Resolved] (SPARK-33537) Hive Metastore filter pushdown improvement

[ https://issues.apache.org/jira/browse/SPARK-33537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-33537. - Resolution: Fixed > Hive Metastore filter pushdown improvement > ---

[jira] [Updated] (SPARK-52334) In Kubernetes mode, update all files, jars, archiveFiles, and pyFiles to reference the working directory after they are downloaded.

[ https://issues.apache.org/jira/browse/SPARK-52334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52334: --- Labels: pull-request-available (was: ) > In Kubernetes mode, update all files, jars, archiv

[jira] [Updated] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

[ https://issues.apache.org/jira/browse/SPARK-52333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52333: --- Labels: pull-request-available (was: ) > Squeeze protocol for timers (list on specific grou

[jira] [Updated] (SPARK-52334) In Kubernetes mode, update all files, jars, archiveFiles, and pyFiles to reference the working directory after they are downloaded.

[ https://issues.apache.org/jira/browse/SPARK-52334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongwei updated SPARK-52334: Description: When submitting a Spark job with the {{--files}} option and also calling {{SparkContext.addF

[jira] [Updated] (SPARK-52334) In Kubernetes mode, update all files, jars, archiveFiles, and pyFiles to reference the working directory after they are downloaded.

[ https://issues.apache.org/jira/browse/SPARK-52334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongwei updated SPARK-52334: Description: When submitting a Spark job with the {{--files}} option and also calling {{SparkContext.addF

[jira] [Updated] (SPARK-52334) In Kubernetes mode, update all files, jars, archiveFiles, and pyFiles to reference the working directory after they are downloaded.

[ https://issues.apache.org/jira/browse/SPARK-52334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tongwei updated SPARK-52334: Description: When submitting a Spark job with the {{--files}} option and also calling {{SparkContext.addF

[jira] [Created] (SPARK-52334) In Kubernetes mode, update all files, jars, archiveFiles, and pyFiles to reference the working directory after they are downloaded.

Tongwei created SPARK-52334: --- Summary: In Kubernetes mode, update all files, jars, archiveFiles, and pyFiles to reference the working directory after they are downloaded. Key: SPARK-52334 URL: https://issues.apache.org/

[jira] [Commented] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

[ https://issues.apache.org/jira/browse/SPARK-52333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954468#comment-17954468 ] Jungtaek Lim commented on SPARK-52333: -- Going to submit a PR for this. Probably in

[jira] [Created] (SPARK-52333) Squeeze protocol for timers (list on specific grouping key, and expiry timers)

Jungtaek Lim created SPARK-52333: Summary: Squeeze protocol for timers (list on specific grouping key, and expiry timers) Key: SPARK-52333 URL: https://issues.apache.org/jira/browse/SPARK-52333 Projec

[jira] [Resolved] (SPARK-52329) Remove private[sql] tags for new transformWithState API

[ https://issues.apache.org/jira/browse/SPARK-52329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-52329. -- Fix Version/s: 4.1.0 4.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-52329) Remove private[sql] tags for new transformWithState API

[ https://issues.apache.org/jira/browse/SPARK-52329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-52329: Assignee: Anish Shrigondekar > Remove private[sql] tags for new transformWithState API >

[jira] [Resolved] (SPARK-52313) Correctly resolve reference data type for Views with default collation

[ https://issues.apache.org/jira/browse/SPARK-52313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-52313. - Fix Version/s: 4.1.0 4.0.1 Resolution: Fixed Issue resolved by pull re

[jira] [Assigned] (SPARK-52313) Correctly resolve reference data type for Views with default collation

[ https://issues.apache.org/jira/browse/SPARK-52313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-52313: --- Assignee: Marko Ilic > Correctly resolve reference data type for Views with default collati

[jira] [Resolved] (SPARK-52331) Adjust test for promotion from float32 to float64 during division

[ https://issues.apache.org/jira/browse/SPARK-52331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-52331. -- Assignee: Xinrong Meng Resolution: Resolved Resolved by https://github.com/apache/spark/

[jira] [Updated] (SPARK-52332) Fix promotion from float32 to float64 during division

[ https://issues.apache.org/jira/browse/SPARK-52332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-52332: - Description: {{>>> ps.set_option("compute.fail_on_ansi_mode", False)}} {{>>> spark.conf.set("spa

[jira] [Updated] (SPARK-52331) Adjust test for promotion from float32 to float64 during division

[ https://issues.apache.org/jira/browse/SPARK-52331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52331: --- Labels: pull-request-available (was: ) > Adjust test for promotion from float32 to float64

[jira] [Created] (SPARK-52332) Fix promotion from float32 to float64 during division

Xinrong Meng created SPARK-52332: Summary: Fix promotion from float32 to float64 during division Key: SPARK-52332 URL: https://issues.apache.org/jira/browse/SPARK-52332 Project: Spark Issue T

[jira] [Created] (SPARK-52331) Adjust test for promotion from float32 to float64 during division

Xinrong Meng created SPARK-52331: Summary: Adjust test for promotion from float32 to float64 during division Key: SPARK-52331 URL: https://issues.apache.org/jira/browse/SPARK-52331 Project: Spark

[jira] [Updated] (SPARK-52330) SPIP: Real-Time Mode in Apache Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-52330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-52330: -- Description: The SPIP proposes a new execution mode called “{*}Real-time Mode{*}”  in

[jira] [Created] (SPARK-52330) SPIP: Real-Time Mode in Apache Spark Structured Streaming

Boyang Jerry Peng created SPARK-52330: - Summary: SPIP: Real-Time Mode in Apache Spark Structured Streaming Key: SPARK-52330 URL: https://issues.apache.org/jira/browse/SPARK-52330 Project: Spark

[jira] [Updated] (SPARK-52330) SPIP: Real-Time Mode in Apache Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-52330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-52330: -- Description: The SPIP proposes to add a new execution mode called “{*}Real-time Mode{*

[jira] [Updated] (SPARK-52330) SPIP: Real-Time Mode in Apache Spark Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-52330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boyang Jerry Peng updated SPARK-52330: -- Description: We propose to add a *real-time mode* in Spark Structured Streaming that

[jira] [Resolved] (SPARK-52270) User guide for native plotting

[ https://issues.apache.org/jira/browse/SPARK-52270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-52270. -- Resolution: Resolved Resolved by https://github.com/apache/spark/pull/50992 > User guide for

[jira] [Resolved] (SPARK-52264) Test divide-by-zero behavior with more numeric data types

[ https://issues.apache.org/jira/browse/SPARK-52264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-52264. -- Assignee: Xinrong Meng Resolution: Resolved Resolved by https://github.com/apache/spark/

[jira] [Assigned] (SPARK-52270) User guide for native plotting

[ https://issues.apache.org/jira/browse/SPARK-52270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng reassigned SPARK-52270: Assignee: Xinrong Meng > User guide for native plotting > --

[jira] [Resolved] (SPARK-52223) [SDP] Create spark connect API for SDP

[ https://issues.apache.org/jira/browse/SPARK-52223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved SPARK-52223. Resolution: Fixed > [SDP] Create spark connect API for SDP > -

[jira] [Updated] (SPARK-52329) Remove private[sql] tags for new transformWithState API

[ https://issues.apache.org/jira/browse/SPARK-52329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52329: --- Labels: pull-request-available (was: ) > Remove private[sql] tags for new transformWithStat

[jira] [Created] (SPARK-52329) Remove private[sql] tags for new transformWithState API

Anish Shrigondekar created SPARK-52329: -- Summary: Remove private[sql] tags for new transformWithState API Key: SPARK-52329 URL: https://issues.apache.org/jira/browse/SPARK-52329 Project: Spark

[jira] [Assigned] (SPARK-52328) Use `apache/spark-connect-swift:pi` image

[ https://issues.apache.org/jira/browse/SPARK-52328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-52328: - Assignee: Dongjoon Hyun > Use `apache/spark-connect-swift:pi` image > -

[jira] [Resolved] (SPARK-52328) Use `apache/spark-connect-swift:pi` image

[ https://issues.apache.org/jira/browse/SPARK-52328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-52328. --- Fix Version/s: kubernetes-operator-0.3.0 Resolution: Fixed Issue resolved by pull req

[jira] [Updated] (SPARK-52326) Add partitions related external catalog events

[ https://issues.apache.org/jira/browse/SPARK-52326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li updated SPARK-52326: - Description: In [ExternalCatalogWithListener|https://github.com/apache/spark/blob/master/sql/catalyst/sr

[jira] [Updated] (SPARK-52328) Use `apache/spark-connect-swift:pi` image

[ https://issues.apache.org/jira/browse/SPARK-52328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52328: --- Labels: pull-request-available (was: ) > Use `apache/spark-connect-swift:pi` image > --

[jira] [Updated] (SPARK-52328) Use `apache/spark-connect-swift:pi` image

[ https://issues.apache.org/jira/browse/SPARK-52328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-52328: -- Summary: Use `apache/spark-connect-swift:pi` image (was: Use `apache/spark-connect-swift:pi`)

[jira] [Created] (SPARK-52328) Use `apache/spark-connect-swift:pi`

Dongjoon Hyun created SPARK-52328: - Summary: Use `apache/spark-connect-swift:pi` Key: SPARK-52328 URL: https://issues.apache.org/jira/browse/SPARK-52328 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-52315) Upgrade kubernetes-client version to 7.3.1

[ https://issues.apache.org/jira/browse/SPARK-52315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-52315: -- Parent: SPARK-52205 Issue Type: Sub-task (was: Bug) > Upgrade kubernetes-client versi

[jira] [Updated] (SPARK-52327) Glob based history provider

[ https://issues.apache.org/jira/browse/SPARK-52327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Waghmare updated SPARK-52327: Description: Currently, spark history server runs with one base directory immediate subdi

[jira] [Resolved] (SPARK-52300) Catalog config overrides do not make it into UDTVF resolution

[ https://issues.apache.org/jira/browse/SPARK-52300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-52300. - Fix Version/s: 4.1.0 4.0.1 Resolution: Fixed > Catalog config override

[jira] (SPARK-52286) Publish Apache Spark 4.0.0 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-52286 ] Yury Molchan deleted comment on SPARK-52286: -- was (Author: yurkom): Hello I am trying to run 4.0.0 on the Mac M1 and I am facing with the error to pull it. 4.0.0-preview2 docker image work

[jira] [Created] (SPARK-52327) Glob based history provider

Gaurav Waghmare created SPARK-52327: --- Summary: Glob based history provider Key: SPARK-52327 URL: https://issues.apache.org/jira/browse/SPARK-52327 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-52326) Add partitions related external catalog events

[ https://issues.apache.org/jira/browse/SPARK-52326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li updated SPARK-52326: - External issue URL: https://github.com/apache/spark/pull/51030 > Add partitions related external catalog

[jira] [Updated] (SPARK-52326) Add partitions related external catalog events

[ https://issues.apache.org/jira/browse/SPARK-52326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li updated SPARK-52326: - External issue URL: (was: https://github.com/apache/spark/pull/51030) > Add partitions related externa

[jira] [Created] (SPARK-52326) Add partitions related external catalog events

Xiang Li created SPARK-52326: Summary: Add partitions related external catalog events Key: SPARK-52326 URL: https://issues.apache.org/jira/browse/SPARK-52326 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-52312) Caching AppendData plan causes data to be inserted twice

[ https://issues.apache.org/jira/browse/SPARK-52312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52312: --- Labels: pull-request-available (was: ) > Caching AppendData plan causes data to be inserted

[jira] [Resolved] (SPARK-52324) move Spark docs to the release directory

[ https://issues.apache.org/jira/browse/SPARK-52324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-52324. -- Fix Version/s: 4.1.0 3.5.6 4.0.1 Resolution: Fixed

[jira] [Comment Edited] (SPARK-52286) Publish Apache Spark 4.0.0 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-52286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954309#comment-17954309 ] Yury Molchan edited comment on SPARK-52286 at 5/27/25 12:01 PM: --

[jira] [Comment Edited] (SPARK-52286) Publish Apache Spark 4.0.0 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-52286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954309#comment-17954309 ] Yury Molchan edited comment on SPARK-52286 at 5/27/25 11:52 AM: --

[jira] [Commented] (SPARK-52286) Publish Apache Spark 4.0.0 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-52286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954309#comment-17954309 ] Yury Molchan commented on SPARK-52286: -- Hello I am trying to run 4.0.0 on the Mac

[jira] [Resolved] (SPARK-52254) Adds a GitHub Actions workflow to convert RC to the official release

[ https://issues.apache.org/jira/browse/SPARK-52254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-52254. -- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 50974 [https://gi

[jira] [Assigned] (SPARK-52254) Adds a GitHub Actions workflow to convert RC to the official release

[ https://issues.apache.org/jira/browse/SPARK-52254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-52254: Assignee: Hyukjin Kwon > Adds a GitHub Actions workflow to convert RC to the official rel

[jira] [Updated] (SPARK-52325) Publish Apache Spark 3.5.6 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-52325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52325: --- Labels: pull-request-available (was: ) > Publish Apache Spark 3.5.6 to docker registry > -

[jira] [Updated] (SPARK-52088) Redesign ClosureCleaner Implementation Due to JDK-8309635's Removal of Old Core Reflection and Inability to Modify Private Final Fields in Hidden Classes

[ https://issues.apache.org/jira/browse/SPARK-52088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-52088: --- Labels: pull-request-available (was: ) > Redesign ClosureCleaner Implementation Due to JDK

[jira] [Created] (SPARK-52325) Publish Apache Spark 3.5.6 to docker registry

Hyukjin Kwon created SPARK-52325: Summary: Publish Apache Spark 3.5.6 to docker registry Key: SPARK-52325 URL: https://issues.apache.org/jira/browse/SPARK-52325 Project: Spark Issue Type: Im

[jira] [Assigned] (SPARK-52305) Refine the docstring for isnotnull, equal_null, nullif, nullifzero, nvl, nvl2, zeroifnull

[ https://issues.apache.org/jira/browse/SPARK-52305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-52305: - Assignee: Evan Wu > Refine the docstring for isnotnull, equal_null, nullif, nullifzero,

[jira] [Resolved] (SPARK-52305) Refine the docstring for isnotnull, equal_null, nullif, nullifzero, nvl, nvl2, zeroifnull

[ https://issues.apache.org/jira/browse/SPARK-52305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-52305. --- Fix Version/s: 4.1.0 Resolution: Fixed Issue resolved by pull request 51016 [https://

[jira] [Updated] (SPARK-44728) Improve PySpark documentations

[ https://issues.apache.org/jira/browse/SPARK-44728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-44728: -- Affects Version/s: 4.1.0 > Improve PySpark documentations > -- > >

[jira] [Resolved] (SPARK-52267) Match field id in ParquetToSparkSchemaConverter

[ https://issues.apache.org/jira/browse/SPARK-52267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-52267. - Fix Version/s: 4.1.0 4.0.1 Resolution: Fixed Issue resolved by pull re

[jira] [Assigned] (SPARK-52267) Match field id in ParquetToSparkSchemaConverter

[ https://issues.apache.org/jira/browse/SPARK-52267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-52267: --- Assignee: Chenhao Li > Match field id in ParquetToSparkSchemaConverter > --

[jira] [Assigned] (SPARK-52324) move Spark docs to the release directory

[ https://issues.apache.org/jira/browse/SPARK-52324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-52324: Assignee: Wenchen Fan > move Spark docs to the release directory > --