[jira] [Assigned] (SPARK-49376) Introduce ClusterToleration and WorkerInstanceConfig

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-49376: - Assignee: Zhou JIANG > Introduce ClusterToleration and WorkerInstanceConfig > -

[jira] [Resolved] (SPARK-49376) Introduce ClusterToleration and WorkerInstanceConfig

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49376. --- Fix Version/s: kubernetes-operator-0.1.0 Resolution: Fixed Issue resolved by pull req

[jira] [Resolved] (SPARK-49377) E2E Tests Catch Step SparkApplication Kind Not found

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49377. --- Fix Version/s: kubernetes-operator-0.1.0 Resolution: Fixed Issue resolved by pull req

[jira] [Assigned] (SPARK-49377) E2E Tests Catch Step SparkApplication Kind Not found

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-49377: - Assignee: Qi Tan > E2E Tests Catch Step SparkApplication Kind Not found > -

[jira] [Updated] (SPARK-49352) Avoid redundant array transform for identical expression

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49352: -- Fix Version/s: 3.4.4 > Avoid redundant array transform for identical expression >

[jira] [Updated] (SPARK-49352) Avoid redundant array transform for identical expression

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49352: -- Affects Version/s: 3.4.3 3.5.2 3.3.4 > Avoid red

[jira] [Updated] (SPARK-49352) Avoid redundant array transform for identical expression

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49352: -- Issue Type: Bug (was: Improvement) > Avoid redundant array transform for identical expression

[jira] [Updated] (SPARK-49352) Avoid redundant array transform for identical expression

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-49352: -- Fix Version/s: 3.5.3 > Avoid redundant array transform for identical expression >

[jira] [Updated] (SPARK-49377) E2E Tests Catch Step SparkApplication Kind Not found

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49377: --- Labels: pull-request-available (was: ) > E2E Tests Catch Step SparkApplication Kind Not fou

[jira] [Created] (SPARK-49377) E2E Tests Catch Step SparkApplication Kind Not found

2024-08-23 Thread Qi Tan (Jira)
Qi Tan created SPARK-49377: -- Summary: E2E Tests Catch Step SparkApplication Kind Not found Key: SPARK-49377 URL: https://issues.apache.org/jira/browse/SPARK-49377 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would be beneficial by establi

[jira] [Updated] (SPARK-49376) Introduce ClusterToleration and WorkerInstanceConfig

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49376: --- Labels: pull-request-available (was: ) > Introduce ClusterToleration and WorkerInstanceConf

[jira] [Created] (SPARK-49376) Introduce ClusterToleration and WorkerInstanceConfig

2024-08-23 Thread Zhou JIANG (Jira)
Zhou JIANG created SPARK-49376: -- Summary: Introduce ClusterToleration and WorkerInstanceConfig Key: SPARK-49376 URL: https://issues.apache.org/jira/browse/SPARK-49376 Project: Spark Issue Type:

[jira] [Updated] (SPARK-49375) Upgrade `postgresql` to 42.7.4

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49375: --- Labels: pull-request-available (was: ) > Upgrade `postgresql` to 42.7.4 > -

[jira] [Created] (SPARK-49375) Upgrade `postgresql` to 42.7.4

2024-08-23 Thread BingKun Pan (Jira)
BingKun Pan created SPARK-49375: --- Summary: Upgrade `postgresql` to 42.7.4 Key: SPARK-49375 URL: https://issues.apache.org/jira/browse/SPARK-49375 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-49372) Ensure latestSnapshot is set to none after close

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49372: --- Labels: pull-request-available (was: ) > Ensure latestSnapshot is set to none after close >

[jira] [Updated] (SPARK-49324) E2E Test Components: State Transition

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49324: --- Labels: pull-request-available (was: ) > E2E Test Components: State Transition > -

[jira] [Commented] (SPARK-46679) Encoders with multiple inheritance - Key not found: T

2024-08-23 Thread Asif (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876379#comment-17876379 ] Asif commented on SPARK-46679: -- [~andoni.teso] Hi. I will be opening a PR for this in a day

[jira] [Resolved] (SPARK-49352) Avoid redundant array transform for identical expression

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49352. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47843 [https://

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode by est

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode by est

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Wei Liu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Liu updated SPARK-49374: Issue Type: Epic (was: Task) > RocksDB State Store Checkpoint Structure V2 >

[jira] [Updated] (SPARK-49324) E2E Test Components: State Transition

2024-08-23 Thread Qi Tan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Tan updated SPARK-49324: --- Summary: E2E Test Components: State Transition (was: E2E Test Components) > E2E Test Components: State Tra

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Attachment: image-2024-08-23-14-30-22-474.png > RocksDB State Store Checkpoint Structure V2 >

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Attachment: image-2024-08-23-14-29-19-443.png > RocksDB State Store Checkpoint Structure V2 >

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Attachment: image-2024-08-23-14-29-59-165.png > RocksDB State Store Checkpoint Structure V2 >

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Attachment: image-2024-08-23-14-28-56-418.png > RocksDB State Store Checkpoint Structure V2 >

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Description: h2. Motivation We expect the new checkpoint structure would benefit micro-batch mode

[jira] [Updated] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated SPARK-49374: Description: Umbrella ticket for developing the new RocksDB State Store checkpoint structure. Desi

[jira] [Created] (SPARK-49374) RocksDB State Store Checkpoint Structure V2

2024-08-23 Thread Siying Dong (Jira)
Siying Dong created SPARK-49374: --- Summary: RocksDB State Store Checkpoint Structure V2 Key: SPARK-49374 URL: https://issues.apache.org/jira/browse/SPARK-49374 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-49350) FoldablePropagation rule and ConstantFolding rule leads to wrong aggregated result

2024-08-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876366#comment-17876366 ] Bruce Robbins commented on SPARK-49350: --- Possibly the same as SPARK-49000? > Fold

[jira] [Resolved] (SPARK-49373) ProtobufFunctionsSuite test case should not be environment-dependent

2024-08-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-49373. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47859 [https://

[jira] [Updated] (SPARK-49373) ProtobufFunctionsSuite test case should not be environment-dependent

2024-08-23 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-49373: Component/s: Connect (was: SQL) > ProtobufFunctionsSuite test case should not

[jira] [Updated] (SPARK-49373) ProtobufFunctionsSuite test case should not be environment-dependent

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49373: --- Labels: pull-request-available (was: ) > ProtobufFunctionsSuite test case should not be env

[jira] [Created] (SPARK-49373) ProtobufFunctionsSuite test case should not be environment-dependent

2024-08-23 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-49373: --- Summary: ProtobufFunctionsSuite test case should not be environment-dependent Key: SPARK-49373 URL: https://issues.apache.org/jira/browse/SPARK-49373 Project: Spark

[jira] [Resolved] (SPARK-49027) Move Column API to sql/api

2024-08-23 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-49027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-49027. --- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 47839 [

[jira] [Updated] (SPARK-49370) maven.scaladoc.skip should not affect test code compilation

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49370: --- Labels: pull-request-available (was: ) > maven.scaladoc.skip should not affect test code co

[jira] [Updated] (SPARK-49314) Upgrade `h2` to 2.3.232, `postgresql` to 42.7.4 and `mssql` to 12.8.1.jre11

2024-08-23 Thread Wei Guo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Guo updated SPARK-49314: Summary: Upgrade `h2` to 2.3.232, `postgresql` to 42.7.4 and `mssql` to 12.8.1.jre11 (was: Upgrade `h2` t

[jira] [Created] (SPARK-49371) Reenable Scala Connect Client Doc builder

2024-08-23 Thread Jira
Herman van Hövell created SPARK-49371: - Summary: Reenable Scala Connect Client Doc builder Key: SPARK-49371 URL: https://issues.apache.org/jira/browse/SPARK-49371 Project: Spark Issue Typ

[jira] [Created] (SPARK-49370) maven.scaladoc.skip should not affect test code compilation

2024-08-23 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-49370: - Summary: maven.scaladoc.skip should not affect test code compilation Key: SPARK-49370 URL: https://issues.apache.org/jira/browse/SPARK-49370 Project: Spark Issue

[jira] [Updated] (SPARK-49368) Avoid accessing protobuf lite classes directly

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated SPARK-49368: --- Labels: pull-request-available (was: ) > Avoid accessing protobuf lite classes directly > -

[jira] [Created] (SPARK-49368) Avoid accessing protobuf lite classes directly

2024-08-23 Thread Cheng Pan (Jira)
Cheng Pan created SPARK-49368: - Summary: Avoid accessing protobuf lite classes directly Key: SPARK-49368 URL: https://issues.apache.org/jira/browse/SPARK-49368 Project: Spark Issue Type: Improvem

[jira] [Assigned] (SPARK-48867) Upgrade the kubernetes-client dependency okhttp version 4.12

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48867: -- Assignee: (was: Apache Spark) > Upgrade the kubernetes-client dependency okhttp v

[jira] [Assigned] (SPARK-48867) Upgrade the kubernetes-client dependency okhttp version 4.12

2024-08-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/SPARK-48867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot reassigned SPARK-48867: -- Assignee: Apache Spark > Upgrade the kubernetes-client dependency okhttp version 4.12

[jira] [Updated] (SPARK-49217) Support separate buffer size configuration in UnsafeShuffleWriter

2024-08-23 Thread dzcxzl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dzcxzl updated SPARK-49217: --- Description: {{UnsafeShuffleWriter#mergeSpillsWithFileStream}} uses  {{spark.shuffle.file.buffer}} as the bu

[jira] [Resolved] (SPARK-49217) Support separate buffer size configuration in UnsafeShuffleWriter

2024-08-23 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-49217. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 477

[jira] [Assigned] (SPARK-49217) Support separate buffer size configuration in UnsafeShuffleWriter

2024-08-23 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-49217: --- Assignee: dzcxzl > Support separate buffer size configuration in UnsafeShuf

[jira] [Assigned] (SPARK-49268) Log IO exceptions in SHS history provider

2024-08-23 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-49268: --- Assignee: Rob Reeves > Log IO exceptions in SHS history provider >

[jira] [Resolved] (SPARK-49268) Log IO exceptions in SHS history provider

2024-08-23 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-49268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-49268. - Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 477