[jira] [Updated] (SPARK-46360) Enhance error message debugging with new `getMessage` API

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46360: --- Labels: pull-request-available (was: ) > Enhance error message debugging with new

[jira] [Created] (SPARK-46360) Enhance error message debugging with new `getMessage` API

2023-12-10 Thread Haejoon Lee (Jira)
Haejoon Lee created SPARK-46360: --- Summary: Enhance error message debugging with new `getMessage` API Key: SPARK-46360 URL: https://issues.apache.org/jira/browse/SPARK-46360 Project: Spark

[jira] [Updated] (SPARK-46359) add optional batch id _spark_metadata sink file status

2023-12-10 Thread gagan taneja (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gagan taneja updated SPARK-46359: - Summary: add optional batch id _spark_metadata sink file status (was: add optional batch id

[jira] [Created] (SPARK-46359) add optional batch id to sink file status

2023-12-10 Thread gagan taneja (Jira)
gagan taneja created SPARK-46359: Summary: add optional batch id to sink file status Key: SPARK-46359 URL: https://issues.apache.org/jira/browse/SPARK-46359 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46358) Mino refactor of `ResponseValidator#verifyResponse`

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46358: --- Labels: pull-request-available (was: ) > Mino refactor of

[jira] [Created] (SPARK-46358) Mino refactor of `ResponseValidator#verifyResponse`

2023-12-10 Thread Yang Jie (Jira)
Yang Jie created SPARK-46358: Summary: Mino refactor of `ResponseValidator#verifyResponse` Key: SPARK-46358 URL: https://issues.apache.org/jira/browse/SPARK-46358 Project: Spark Issue Type:

[jira] [Updated] (SPARK-46357) Replace use of setConf with conf.set in docs

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46357: --- Labels: pull-request-available (was: ) > Replace use of setConf with conf.set in docs >

[jira] [Commented] (SPARK-45599) Percentile can produce a wrong answer if -0.0 and 0.0 are mixed in the dataset

2023-12-10 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795162#comment-17795162 ] Nicholas Chammas commented on SPARK-45599: -- Per the [contributing

[jira] [Created] (SPARK-46357) Replace use of setConf with conf.set in docs

2023-12-10 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-46357: Summary: Replace use of setConf with conf.set in docs Key: SPARK-46357 URL: https://issues.apache.org/jira/browse/SPARK-46357 Project: Spark Issue

[jira] [Updated] (SPARK-46356) Upgrade `sbt-assembly` from 2.1.1 to 2.1.5

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46356: --- Labels: pull-request-available (was: ) > Upgrade `sbt-assembly` from 2.1.1 to 2.1.5 >

[jira] [Updated] (SPARK-46355) XML: Close InputStreamReader on read completion

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46355: --- Labels: pull-request-available (was: ) > XML: Close InputStreamReader on read completion >

[jira] [Created] (SPARK-46355) XML: Close InputStreamReader on read completion

2023-12-10 Thread Sandip Agarwala (Jira)
Sandip Agarwala created SPARK-46355: --- Summary: XML: Close InputStreamReader on read completion Key: SPARK-46355 URL: https://issues.apache.org/jira/browse/SPARK-46355 Project: Spark Issue

[jira] [Updated] (SPARK-46354) Add restriction parameters to dynamic partitions for writing datasource

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46354: --- Labels: pull-request-available (was: ) > Add restriction parameters to dynamic partitions

[jira] [Updated] (SPARK-46354) Add restriction parameters to dynamic partitions for writing datasource

2023-12-10 Thread hao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated SPARK-46354: Priority: Trivial (was: Major) > Add restriction parameters to dynamic partitions for writing datasource >

[jira] [Created] (SPARK-46354) Add restriction parameters to dynamic partitions for writing datasource

2023-12-10 Thread hao (Jira)
hao created SPARK-46354: --- Summary: Add restriction parameters to dynamic partitions for writing datasource Key: SPARK-46354 URL: https://issues.apache.org/jira/browse/SPARK-46354 Project: Spark Issue

[jira] [Updated] (SPARK-45019) Make workflow scala213 on container & clean env

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-45019: --- Labels: pull-request-available (was: ) > Make workflow scala213 on container & clean env >

[jira] [Resolved] (SPARK-45642) Fix `FileSystem.isFile & FileSystem.isDirectory is deprecated`

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45642. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 43505

[jira] [Assigned] (SPARK-45642) Fix `FileSystem.isFile & FileSystem.isDirectory is deprecated`

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45642: - Assignee: BingKun Pan > Fix `FileSystem.isFile & FileSystem.isDirectory is deprecated`

[jira] [Assigned] (SPARK-45969) Document configuration change of executor failure tracker

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-45969: - Assignee: Cheng Pan > Document configuration change of executor failure tracker >

[jira] [Resolved] (SPARK-45969) Document configuration change of executor failure tracker

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-45969. --- Fix Version/s: 3.5.1 4.0.0 Resolution: Fixed Issue resolved by

[jira] [Resolved] (SPARK-46285) Add foreachWithSubqueries

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46285. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44206

[jira] [Updated] (SPARK-46353) Refactor to improve `RegisterWorker` unit test

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-46353: --- Labels: pull-request-available (was: ) > Refactor to improve `RegisterWorker` unit test >

[jira] [Created] (SPARK-46353) Refactor to improve `RegisterWorker` unit test

2023-12-10 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-46353: - Summary: Refactor to improve `RegisterWorker` unit test Key: SPARK-46353 URL: https://issues.apache.org/jira/browse/SPARK-46353 Project: Spark Issue Type:

[jira] [Created] (SPARK-46352) Support spark conf to configure log level of specific package or class

2023-12-10 Thread Zhongwei Zhu (Jira)
Zhongwei Zhu created SPARK-46352: Summary: Support spark conf to configure log level of specific package or class Key: SPARK-46352 URL: https://issues.apache.org/jira/browse/SPARK-46352 Project:

[jira] [Resolved] (SPARK-46342) Replace IllegalStateException by SparkException.internalError in sql/core

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46342. --- Resolution: Fixed Issue resolved by pull request 44275

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46346: - Assignee: Dongjoon Hyun > Fix Master to update worker from UNKNOWN to ALIVE on

[jira] [Resolved] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46346. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44280

[jira] [Comment Edited] (SPARK-44003) DynamicPartitionDataSingleWriter is being starved by Parquet MemoryManager

2023-12-10 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795101#comment-17795101 ] Rafal Wojdyla edited comment on SPARK-44003 at 12/10/23 7:09 PM: - Coming

[jira] [Commented] (SPARK-44003) DynamicPartitionDataSingleWriter is being starved by Parquet MemoryManager

2023-12-10 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795102#comment-17795102 ] Rafal Wojdyla commented on SPARK-44003: --- In context of {{DynamicPartitionDataConcurrentWriter}}

[jira] [Comment Edited] (SPARK-44003) DynamicPartitionDataSingleWriter is being starved by Parquet MemoryManager

2023-12-10 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795101#comment-17795101 ] Rafal Wojdyla edited comment on SPARK-44003 at 12/10/23 7:02 PM: - Coming

[jira] [Commented] (SPARK-44003) DynamicPartitionDataSingleWriter is being starved by Parquet MemoryManager

2023-12-10 Thread Rafal Wojdyla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795101#comment-17795101 ] Rafal Wojdyla commented on SPARK-44003: --- Coming back briefly to this issue, and to elaborate on

[jira] [Created] (SPARK-46351) Require an error class in AnalysisException

2023-12-10 Thread Max Gekk (Jira)
Max Gekk created SPARK-46351: Summary: Require an error class in AnalysisException Key: SPARK-46351 URL: https://issues.apache.org/jira/browse/SPARK-46351 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-44900) Cached DataFrame keeps growing

2023-12-10 Thread wang fanming (Jira)
[ https://issues.apache.org/jira/browse/SPARK-44900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795061#comment-17795061 ] wang fanming commented on SPARK-44900: -- Of course, the following is my execution in the spark3.5.1

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: Apache Spark > Fix Master to update worker from UNKNOWN to ALIVE on

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: Apache Spark > Fix Master to update worker from UNKNOWN to ALIVE on

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: (was: Apache Spark) > Fix Master to update worker from UNKNOWN to

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: (was: Apache Spark) > Fix Master to update worker from UNKNOWN to

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: Apache Spark > Fix Master to update worker from UNKNOWN to ALIVE on

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: (was: Apache Spark) > Fix Master to update worker from UNKNOWN to

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: Apache Spark > Fix Master to update worker from UNKNOWN to ALIVE on

[jira] [Assigned] (SPARK-46346) Fix Master to update worker from UNKNOWN to ALIVE on RegisterWorker message

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-46346: -- Assignee: (was: Apache Spark) > Fix Master to update worker from UNKNOWN to

[jira] [Updated] (SPARK-34888) Introduce UpdatingSessionIterator adjusting session information on elements

2023-12-10 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-34888: --- Labels: pull-request-available (was: ) > Introduce UpdatingSessionIterator adjusting

[jira] [Resolved] (SPARK-46348) Support `spark.deploy.recoveryTimeout`

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-46348. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44282

[jira] [Assigned] (SPARK-46348) Support `spark.deploy.recoveryTimeout`

2023-12-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-46348: - Assignee: Dongjoon Hyun > Support `spark.deploy.recoveryTimeout` >

[jira] [Assigned] (SPARK-46345) Reorganize `DiffFramesParityCovCorrWithTests `

2023-12-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng reassigned SPARK-46345: - Assignee: Ruifeng Zheng > Reorganize `DiffFramesParityCovCorrWithTests ` >

[jira] [Resolved] (SPARK-46345) Reorganize `DiffFramesParityCovCorrWithTests `

2023-12-10 Thread Ruifeng Zheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng resolved SPARK-46345. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44279