[jira] [Updated] (SPARK-49824) SparkConnectStreamingQueryCache Log improvement

[ https://issues.apache.org/jira/browse/SPARK-49824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49824: --- Labels: pull-request-available (was: ) > SparkConnectStreamingQueryCache Log improvement >

[jira] [Created] (SPARK-49824) SparkConnectStreamingQueryCache Log improvement

Wei Liu created SPARK-49824: --- Summary: SparkConnectStreamingQueryCache Log improvement Key: SPARK-49824 URL: https://issues.apache.org/jira/browse/SPARK-49824 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-49823) Set avoidFlushDuringShutdown to true used in rocksdb close

[ https://issues.apache.org/jira/browse/SPARK-49823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49823: --- Labels: pull-request-available (was: ) > Set avoidFlushDuringShutdown to true used in rocks

[jira] [Created] (SPARK-49823) Set avoidFlushDuringShutdown to true used in rocksdb close

Anish Shrigondekar created SPARK-49823: -- Summary: Set avoidFlushDuringShutdown to true used in rocksdb close Key: SPARK-49823 URL: https://issues.apache.org/jira/browse/SPARK-49823 Project: Spark

[jira] [Updated] (SPARK-49821) [PySpark] Support MapState

[ https://issues.apache.org/jira/browse/SPARK-49821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49821: --- Labels: pull-request-available (was: ) > [PySpark] Support MapState > -

[jira] [Updated] (SPARK-49822) Update postgres docker image to 17.0

[ https://issues.apache.org/jira/browse/SPARK-49822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49822: --- Labels: pull-request-available (was: ) > Update postgres docker image to 17.0 > ---

[jira] [Created] (SPARK-49821) [PySpark] Support MapState

Bo Gao created SPARK-49821: -- Summary: [PySpark] Support MapState Key: SPARK-49821 URL: https://issues.apache.org/jira/browse/SPARK-49821 Project: Spark Issue Type: Task Components: Structu

[jira] [Assigned] (SPARK-49820) change IOError to OSError

[ https://issues.apache.org/jira/browse/SPARK-49820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-49820: Assignee: Bjørn Jørgensen > change IOError to OSError > - > >

[jira] [Resolved] (SPARK-49820) change IOError to OSError

[ https://issues.apache.org/jira/browse/SPARK-49820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-49820. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48287 [https://gi

[jira] [Resolved] (SPARK-49417) Create a shared interface for StreamingQueryManager

[ https://issues.apache.org/jira/browse/SPARK-49417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-49417. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48217 [

[jira] [Commented] (SPARK-47949) MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

[ https://issues.apache.org/jira/browse/SPARK-47949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885494#comment-17885494 ] Dongjoon Hyun commented on SPARK-47949: --- This is backported to branch-3.4 via cher

[jira] [Updated] (SPARK-46525) Fix docker-integration-tests on Apple Chips

[ https://issues.apache.org/jira/browse/SPARK-46525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46525: -- Fix Version/s: 3.4.4 > Fix docker-integration-tests on Apple Chips > -

[jira] [Updated] (SPARK-47949) MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

[ https://issues.apache.org/jira/browse/SPARK-47949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47949: -- Fix Version/s: 3.4.4 > MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

[jira] [Commented] (SPARK-46525) Fix docker-integration-tests on Apple Chips

[ https://issues.apache.org/jira/browse/SPARK-46525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885493#comment-17885493 ] Dongjoon Hyun commented on SPARK-46525: --- This is backported to branch-3.4 via cher

[jira] [Commented] (SPARK-47949) MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

[ https://issues.apache.org/jira/browse/SPARK-47949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885491#comment-17885491 ] Dongjoon Hyun commented on SPARK-47949: --- This is backported to branch-3.5 via htt

[jira] [Updated] (SPARK-47949) MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

[ https://issues.apache.org/jira/browse/SPARK-47949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-47949: -- Fix Version/s: 3.5.4 > MsSQLServer: Bump up docker image version to2022-CU12-GDR1-ubuntu-22.04

[jira] [Updated] (SPARK-46525) Fix docker-integration-tests on Apple Chips

[ https://issues.apache.org/jira/browse/SPARK-46525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46525: -- Parent Issue: SPARK-47361 (was: SPARK-44111) > Fix docker-integration-tests on Apple Chips >

[jira] [Commented] (SPARK-46525) Fix docker-integration-tests on Apple Chips

[ https://issues.apache.org/jira/browse/SPARK-46525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885488#comment-17885488 ] Dongjoon Hyun commented on SPARK-46525: --- This is backported to branch-3.5 via http

[jira] [Updated] (SPARK-46525) Fix docker-integration-tests on Apple Chips

[ https://issues.apache.org/jira/browse/SPARK-46525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-46525: -- Fix Version/s: 3.5.4 > Fix docker-integration-tests on Apple Chips > -

[jira] [Resolved] (SPARK-49817) Upgrade `gcs-connector` to `2.2.25`

[ https://issues.apache.org/jira/browse/SPARK-49817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49817. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48285 [https://

[jira] [Updated] (SPARK-49601) Support Initial State Handling for TransformWithStateInPandas

[ https://issues.apache.org/jira/browse/SPARK-49601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49601: --- Labels: pull-request-available (was: ) > Support Initial State Handling for TransformWithSt

[jira] [Resolved] (SPARK-49818) Update E2E tests to use Spark 3.5.3

[ https://issues.apache.org/jira/browse/SPARK-49818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49818. --- Fix Version/s: kubernetes-operator-0.1.0 Resolution: Fixed Issue resolved by pull req

[jira] [Resolved] (SPARK-49787) Cast between UDT and other types

[ https://issues.apache.org/jira/browse/SPARK-49787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao resolved SPARK-49787. Fix Version/s: 4.0.0 Resolution: Fixed > Cast between UDT and other types > ---

[jira] [Updated] (SPARK-49820) change IOError to OSError

[ https://issues.apache.org/jira/browse/SPARK-49820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49820: --- Labels: pull-request-available (was: ) > change IOError to OSError > --

[jira] [Updated] (SPARK-49820) change IOError to OSError

[ https://issues.apache.org/jira/browse/SPARK-49820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-49820: Priority: Minor (was: Major) > change IOError to OSError > - > >

[jira] [Created] (SPARK-49820) change IOError to OSError

Bjørn Jørgensen created SPARK-49820: --- Summary: change IOError to OSError Key: SPARK-49820 URL: https://issues.apache.org/jira/browse/SPARK-49820 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-49814) When Spark Connect Client starts, show the spark version of the connect server

BingKun Pan created SPARK-49814: --- Summary: When Spark Connect Client starts, show the spark version of the connect server Key: SPARK-49814 URL: https://issues.apache.org/jira/browse/SPARK-49814 Project:

[jira] [Updated] (SPARK-49787) Cast between UDT and other types

[ https://issues.apache.org/jira/browse/SPARK-49787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49787: -- Parent: SPARK-44111 Issue Type: Sub-task (was: Improvement) > Cast between UDT and ot

[jira] [Updated] (SPARK-49819) Disable CollapseProject for correlated subqueries in projection over aggregate correctly

[ https://issues.apache.org/jira/browse/SPARK-49819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49819: --- Labels: pull-request-available (was: ) > Disable CollapseProject for correlated subqueries

[jira] [Created] (SPARK-49819) Disable CollapseProject for correlated subqueries in projection over aggregate correctly

Nick Young created SPARK-49819: -- Summary: Disable CollapseProject for correlated subqueries in projection over aggregate correctly Key: SPARK-49819 URL: https://issues.apache.org/jira/browse/SPARK-49819

[jira] [Updated] (SPARK-49818) Update E2E tests to use Spark 3.5.3

[ https://issues.apache.org/jira/browse/SPARK-49818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49818: -- Summary: Update E2E tests to use Spark 3.5.3 (was: Update E2 tests to use Spark 3.5.3) > Upd

[jira] [Created] (SPARK-49818) Update E22 tests to use Spark 3.5.3

Dongjoon Hyun created SPARK-49818: - Summary: Update E22 tests to use Spark 3.5.3 Key: SPARK-49818 URL: https://issues.apache.org/jira/browse/SPARK-49818 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-49818) Update E2 tests to use Spark 3.5.3

[ https://issues.apache.org/jira/browse/SPARK-49818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49818: -- Summary: Update E2 tests to use Spark 3.5.3 (was: Update E22 tests to use Spark 3.5.3) > Upd

[jira] [Updated] (SPARK-49818) Update E22 tests to use Spark 3.5.3

[ https://issues.apache.org/jira/browse/SPARK-49818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49818: --- Labels: pull-request-available (was: ) > Update E22 tests to use Spark 3.5.3 >

[jira] [Resolved] (SPARK-49809) Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23

[ https://issues.apache.org/jira/browse/SPARK-49809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49809. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48280 [https://

[jira] [Assigned] (SPARK-49809) Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23

[ https://issues.apache.org/jira/browse/SPARK-49809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-49809: - Assignee: Yang Jie > Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with

[jira] [Updated] (SPARK-49811) Rename StringTypeAnyCollation to reflect all collation identifiers supported

[ https://issues.apache.org/jira/browse/SPARK-49811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49811: --- Labels: pull-request-available (was: ) > Rename StringTypeAnyCollation to reflect all colla

[jira] [Updated] (SPARK-49816) should only update out-going-ref-count for referenced outer CTE relation

[ https://issues.apache.org/jira/browse/SPARK-49816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49816: --- Labels: pull-request-available (was: ) > should only update out-going-ref-count for referen

[jira] [Created] (SPARK-49816) should only update out-going-ref-count for referenced outer CTE relation

Wenchen Fan created SPARK-49816: --- Summary: should only update out-going-ref-count for referenced outer CTE relation Key: SPARK-49816 URL: https://issues.apache.org/jira/browse/SPARK-49816 Project: Spark

[jira] [Assigned] (SPARK-49757) SET CATALOG syntax doesn't support IDENTIFIER

[ https://issues.apache.org/jira/browse/SPARK-49757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-49757: --- Assignee: Mikhail Nikoliukin > SET CATALOG syntax doesn't support IDENTIFIER > ---

[jira] [Resolved] (SPARK-49808) Fix a deadlock in subquery execution due to lazy vals

[ https://issues.apache.org/jira/browse/SPARK-49808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49808. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48279 [https://gith

[jira] [Resolved] (SPARK-49757) SET CATALOG syntax doesn't support IDENTIFIER

[ https://issues.apache.org/jira/browse/SPARK-49757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-49757. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48228 [https://gith

[jira] [Created] (SPARK-49815) Refactor TypeCoercion rules for BinaryArithmetic

Mihailo Timotic created SPARK-49815: --- Summary: Refactor TypeCoercion rules for BinaryArithmetic Key: SPARK-49815 URL: https://issues.apache.org/jira/browse/SPARK-49815 Project: Spark Issue

[jira] [Created] (SPARK-49813) Publish Apache Spark 3.5.3 to docker registry

Haejoon Lee created SPARK-49813: --- Summary: Publish Apache Spark 3.5.3 to docker registry Key: SPARK-49813 URL: https://issues.apache.org/jira/browse/SPARK-49813 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-49814) When Spark Connect Client starts, show the spark version of the connect server

[ https://issues.apache.org/jira/browse/SPARK-49814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49814: --- Labels: pull-request-available (was: ) > When Spark Connect Client starts, show the spark v

[jira] [Resolved] (SPARK-49813) Publish Apache Spark 3.5.3 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-49813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee resolved SPARK-49813. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 72 [https://github.

[jira] [Updated] (SPARK-49813) Publish Apache Spark 3.5.3 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-49813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-49813: Fix Version/s: 3.5.3 (was: 4.0.0) > Publish Apache Spark 3.5.3 to docker re

[jira] [Updated] (SPARK-49813) Publish Apache Spark 3.5.3 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-49813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49813: --- Labels: pull-request-available (was: ) > Publish Apache Spark 3.5.3 to docker registry > --

[jira] [Assigned] (SPARK-49813) Publish Apache Spark 3.5.3 to docker registry

[ https://issues.apache.org/jira/browse/SPARK-49813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee reassigned SPARK-49813: --- Assignee: Haejoon Lee > Publish Apache Spark 3.5.3 to docker registry > ---

[jira] [Updated] (SPARK-49807) Incorporate Java Virtual Threads

[ https://issues.apache.org/jira/browse/SPARK-49807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Changgyoo Park updated SPARK-49807: --- Description: The Spark driver is handling mostly IO-bound tasks, and incorporating Java Vir

[jira] [Commented] (SPARK-49812) NPE when reading empty zstd compressed csv file

[ https://issues.apache.org/jira/browse/SPARK-49812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885297#comment-17885297 ] Martin Andersson commented on SPARK-49812: -- Changing the schema to a single col

[jira] [Created] (SPARK-49812) NPE when reading empty zstd compressed csv file

Martin Andersson created SPARK-49812: Summary: NPE when reading empty zstd compressed csv file Key: SPARK-49812 URL: https://issues.apache.org/jira/browse/SPARK-49812 Project: Spark Issue

[jira] [Created] (SPARK-49811) Rename StringTypeAnyCollation to reflect all collation identifiers supported

Vladan Vasić created SPARK-49811: Summary: Rename StringTypeAnyCollation to reflect all collation identifiers supported Key: SPARK-49811 URL: https://issues.apache.org/jira/browse/SPARK-49811 Project:

[jira] [Updated] (SPARK-49810) Extract the preparation of df.sort to parent class

[ https://issues.apache.org/jira/browse/SPARK-49810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49810: --- Labels: pull-request-available (was: ) > Extract the preparation of df.sort to parent class

[jira] [Created] (SPARK-49810) Extract the preparation of df.sort to parent class

Ruifeng Zheng created SPARK-49810: - Summary: Extract the preparation of df.sort to parent class Key: SPARK-49810 URL: https://issues.apache.org/jira/browse/SPARK-49810 Project: Spark Issue T

[jira] [Created] (SPARK-49808) Fix a deadlock in subquery execution due to lazy vals

Ruifeng Zheng created SPARK-49808: - Summary: Fix a deadlock in subquery execution due to lazy vals Key: SPARK-49808 URL: https://issues.apache.org/jira/browse/SPARK-49808 Project: Spark Issue

[jira] [Updated] (SPARK-49809) Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23

[ https://issues.apache.org/jira/browse/SPARK-49809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49809: --- Labels: pull-request-available (was: ) > Use `sbt.IO` in `SparkBuild.scala` to avoid naming

[jira] [Created] (SPARK-49809) Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23

Yang Jie created SPARK-49809: Summary: Use `sbt.IO` in `SparkBuild.scala` to avoid naming conflicts with `java.io.IO` in Java 23 Key: SPARK-49809 URL: https://issues.apache.org/jira/browse/SPARK-49809 Pro

[jira] [Updated] (SPARK-49808) Fix a deadlock in subquery execution due to lazy vals

[ https://issues.apache.org/jira/browse/SPARK-49808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49808: --- Labels: pull-request-available (was: ) > Fix a deadlock in subquery execution due to lazy v

[jira] [Created] (SPARK-49807) Incorporate Java Virtual Threads

Changgyoo Park created SPARK-49807: -- Summary: Incorporate Java Virtual Threads Key: SPARK-49807 URL: https://issues.apache.org/jira/browse/SPARK-49807 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-49806) Remove redundant `blank space` after 'show' in `Scala` and `Connect` clients

BingKun Pan created SPARK-49806: --- Summary: Remove redundant `blank space` after 'show' in `Scala` and `Connect` clients Key: SPARK-49806 URL: https://issues.apache.org/jira/browse/SPARK-49806 Project: S

[jira] [Resolved] (SPARK-49805) Remove internal functions from `function.scala`

[ https://issues.apache.org/jira/browse/SPARK-49805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie resolved SPARK-49805. -- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 48276 [https://github.com

[jira] [Assigned] (SPARK-49805) Remove internal functions from `function.scala`

[ https://issues.apache.org/jira/browse/SPARK-49805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie reassigned SPARK-49805: Assignee: Ruifeng Zheng > Remove internal functions from `function.scala` > -