[jira] (SPARK-42411) Better support for Istio service mesh while running Spark on Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-42411 ] Puneet deleted comment on SPARK-42411: was (Author: puneetguptanitj): Should be able to create a PR hopefully by next week.  > Better support for Istio service mesh while running Spark on

[jira] [Commented] (SPARK-42983) Fix the error message of createDataFrame from np.array(0)

[ https://issues.apache.org/jira/browse/SPARK-42983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708645#comment-17708645 ] Snoot.io commented on SPARK-42983: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-43030) deduplicate relations with metadata columns

[ https://issues.apache.org/jira/browse/SPARK-43030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708644#comment-17708644 ] Snoot.io commented on SPARK-43030: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Resolved] (SPARK-42983) Fix the error message of createDataFrame from np.array(0)

[ https://issues.apache.org/jira/browse/SPARK-42983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42983. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40669

[jira] [Assigned] (SPARK-42983) Fix the error message of createDataFrame from np.array(0)

[ https://issues.apache.org/jira/browse/SPARK-42983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42983: Assignee: Takuya Ueshin > Fix the error message of createDataFrame from np.array(0) >

[jira] [Created] (SPARK-43035) Add error classes in Spark Connect server's ErrorInfo

Allison Wang created SPARK-43035: Summary: Add error classes in Spark Connect server's ErrorInfo Key: SPARK-43035 URL: https://issues.apache.org/jira/browse/SPARK-43035 Project: Spark Issue

[jira] [Updated] (SPARK-43035) Add error class in Spark Connect server's ErrorInfo

[ https://issues.apache.org/jira/browse/SPARK-43035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allison Wang updated SPARK-43035: - Summary: Add error class in Spark Connect server's ErrorInfo (was: Add error classes in Spark

[jira] [Resolved] (SPARK-43009) Parameterized sql() with constants

[ https://issues.apache.org/jira/browse/SPARK-43009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-43009. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40666

[jira] [Commented] (SPARK-43031) Enable tests for Python streaming spark-connect

[ https://issues.apache.org/jira/browse/SPARK-43031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708609#comment-17708609 ] Wei Liu commented on SPARK-43031: - https://github.com/apache/spark/pull/40586#discussion_r1157042286  

[jira] [Created] (SPARK-43034) Improve error message in AssertNotNull by including combined column path

Anton Okolnychyi created SPARK-43034: Summary: Improve error message in AssertNotNull by including combined column path Key: SPARK-43034 URL: https://issues.apache.org/jira/browse/SPARK-43034

[jira] [Created] (SPARK-43033) Avoid task retries due to AssertNotNull checks

Anton Okolnychyi created SPARK-43033: Summary: Avoid task retries due to AssertNotNull checks Key: SPARK-43033 URL: https://issues.apache.org/jira/browse/SPARK-43033 Project: Spark Issue

[jira] [Created] (SPARK-43032) Add StreamingQueryManager API

Raghu Angadi created SPARK-43032: Summary: Add StreamingQueryManager API Key: SPARK-43032 URL: https://issues.apache.org/jira/browse/SPARK-43032 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-43031) Enable tests for Python streaming spark-connect

[ https://issues.apache.org/jira/browse/SPARK-43031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708519#comment-17708519 ] Raghu Angadi commented on SPARK-43031: -- cc: [~WweiL]  > Enable tests for Python streaming

[jira] [Created] (SPARK-43031) Enable tests for Python streaming spark-connect

Raghu Angadi created SPARK-43031: Summary: Enable tests for Python streaming spark-connect Key: SPARK-43031 URL: https://issues.apache.org/jira/browse/SPARK-43031 Project: Spark Issue Type:

[jira] [Updated] (SPARK-42362) Upgrade kubernetes-client from 6.4.0 to 6.4.1

[ https://issues.apache.org/jira/browse/SPARK-42362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bjørn Jørgensen updated SPARK-42362: Issue Type: Dependency upgrade (was: Improvement) > Upgrade kubernetes-client from 6.4.0

[jira] [Updated] (SPARK-43011) array_insert should fail with 0 index

[ https://issues.apache.org/jira/browse/SPARK-43011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43011: - Priority: Blocker (was: Major) > array_insert should fail with 0 index >

[jira] [Updated] (SPARK-43009) Parameterized sql() with constants

[ https://issues.apache.org/jira/browse/SPARK-43009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-43009: - Priority: Blocker (was: Major) > Parameterized sql() with constants >

[jira] [Commented] (SPARK-39696) Uncaught exception in thread executor-heartbeater java.util.ConcurrentModificationException: mutation occurred during iteration

[ https://issues.apache.org/jira/browse/SPARK-39696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708453#comment-17708453 ] Emil Ejbyfeldt commented on SPARK-39696: Created a test case that consistently reproduces the

[jira] [Created] (SPARK-43030) deduplicate relations with metadata columns

Wenchen Fan created SPARK-43030: --- Summary: deduplicate relations with metadata columns Key: SPARK-43030 URL: https://issues.apache.org/jira/browse/SPARK-43030 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-42655) Incorrect ambiguous column reference error

[ https://issues.apache.org/jira/browse/SPARK-42655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-42655: --- Assignee: Shrikant Prasad > Incorrect ambiguous column reference error >

[jira] [Resolved] (SPARK-42655) Incorrect ambiguous column reference error

[ https://issues.apache.org/jira/browse/SPARK-42655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42655. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40258

[jira] [Assigned] (SPARK-43026) Apply AQE with non-exchange table cache

[ https://issues.apache.org/jira/browse/SPARK-43026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-43026: --- Assignee: XiDuo You > Apply AQE with non-exchange table cache >

[jira] [Resolved] (SPARK-43026) Apply AQE with non-exchange table cache

[ https://issues.apache.org/jira/browse/SPARK-43026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-43026. - Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40659

[jira] [Updated] (SPARK-43029) PySpark Breaks with Pandas 2.0

[ https://issues.apache.org/jira/browse/SPARK-43029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phillip Cloud updated SPARK-43029: -- Summary: PySpark Breaks with Pandas 2.0 (was: Support pandas 2.0) > PySpark Breaks with

[jira] [Created] (SPARK-43029) Support pandas 2.0

Phillip Cloud created SPARK-43029: - Summary: Support pandas 2.0 Key: SPARK-43029 URL: https://issues.apache.org/jira/browse/SPARK-43029 Project: Spark Issue Type: Bug Components:

[jira] [Updated] (SPARK-39696) Uncaught exception in thread executor-heartbeater java.util.ConcurrentModificationException: mutation occurred during iteration

[ https://issues.apache.org/jira/browse/SPARK-39696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Emil Ejbyfeldt updated SPARK-39696: --- Attachment: 0001-Spec-that-produces-race-condition-with-hearbeater-th.patch > Uncaught

[jira] [Updated] (SPARK-43025) Eliminate Union if filters have the same child plan

[ https://issues.apache.org/jira/browse/SPARK-43025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-43025: --- Summary: Eliminate Union if filters have the same child plan (was: Eliminate Union if filters on

[jira] [Commented] (SPARK-43026) Apply AQE with non-exchange table cache

[ https://issues.apache.org/jira/browse/SPARK-43026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708327#comment-17708327 ] ASF GitHub Bot commented on SPARK-43026: User 'ulysses-you' has created a pull request for this

[jira] [Commented] (SPARK-43026) Apply AQE with non-exchange table cache

[ https://issues.apache.org/jira/browse/SPARK-43026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708326#comment-17708326 ] ASF GitHub Bot commented on SPARK-43026: User 'ulysses-you' has created a pull request for this

[jira] [Resolved] (SPARK-42964) PosgresDialect SqlState 42P07 also means table already exists

[ https://issues.apache.org/jira/browse/SPARK-42964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-42964. -- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40588

[jira] [Assigned] (SPARK-42964) PosgresDialect SqlState 42P07 also means table already exists

[ https://issues.apache.org/jira/browse/SPARK-42964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-42964: Assignee: Kent Yao > PosgresDialect SqlState 42P07 also means table already exists >

[jira] [Created] (SPARK-43028) Add error class SQL_CONF_NOT_FOUND

Allison Wang created SPARK-43028: Summary: Add error class SQL_CONF_NOT_FOUND Key: SPARK-43028 URL: https://issues.apache.org/jira/browse/SPARK-43028 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-42974) Restore `Utils#createTempDir` use `ShutdownHookManager.registerShutdownDeleteDir` to cleanup tempDir

[ https://issues.apache.org/jira/browse/SPARK-42974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-42974: -- Fix Version/s: 3.4.1 (was: 3.5.0) > Restore `Utils#createTempDir` use

[jira] [Assigned] (SPARK-43011) array_insert should fail with 0 index

[ https://issues.apache.org/jira/browse/SPARK-43011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-43011: Assignee: Ruifeng Zheng > array_insert should fail with 0 index >

[jira] [Resolved] (SPARK-43011) array_insert should fail with 0 index

[ https://issues.apache.org/jira/browse/SPARK-43011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-43011. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 40641

[jira] [Updated] (SPARK-42974) Restore `Utils#createTempDir` use `ShutdownHookManager.registerShutdownDeleteDir` to cleanup tempDir

[ https://issues.apache.org/jira/browse/SPARK-42974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-42974: -- Issue Type: Bug (was: Improvement) > Restore `Utils#createTempDir` use >

[jira] [Commented] (SPARK-42974) Restore `Utils#createTempDir` use `ShutdownHookManager.registerShutdownDeleteDir` to cleanup tempDir

[ https://issues.apache.org/jira/browse/SPARK-42974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708266#comment-17708266 ] Dongjoon Hyun commented on SPARK-42974: --- This landed at branch-3.4 via

[jira] [Assigned] (SPARK-42963) Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

[ https://issues.apache.org/jira/browse/SPARK-42963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-42963: - Assignee: XiDuo You > Extend SparkSessionExtensions to inject rules into AQE query

[jira] [Resolved] (SPARK-42963) Extend SparkSessionExtensions to inject rules into AQE query stage optimizer

[ https://issues.apache.org/jira/browse/SPARK-42963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-42963. --- Fix Version/s: 3.5.0 Resolution: Fixed Issue resolved by pull request 40653

[jira] [Created] (SPARK-43027) Introduce dropDuplicatesWithinWatermark in PySpark

Jungtaek Lim created SPARK-43027: Summary: Introduce dropDuplicatesWithinWatermark in PySpark Key: SPARK-43027 URL: https://issues.apache.org/jira/browse/SPARK-43027 Project: Spark Issue

[jira] [Created] (SPARK-43026) Apply AQE with non-exchange table cache

XiDuo You created SPARK-43026: - Summary: Apply AQE with non-exchange table cache Key: SPARK-43026 URL: https://issues.apache.org/jira/browse/SPARK-43026 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-43025) Eliminate Union if filters on the same child plan

[ https://issues.apache.org/jira/browse/SPARK-43025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708229#comment-17708229 ] jiaan.geng commented on SPARK-43025: I'm working on. > Eliminate Union if filters on the same child