[jira] [Updated] (SPARK-38418) Add PySpark cleanShuffleDependencies API

[ https://issues.apache.org/jira/browse/SPARK-38418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38418: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Add PySpark cleanShuffl

[jira] [Updated] (SPARK-38417) Remove `Experimental` from `RDD.cleanShuffleDependencies` API

[ https://issues.apache.org/jira/browse/SPARK-38417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38417: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Task) > Remove `Experimental` from `RD

[jira] [Comment Edited] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648751#comment-17648751 ] Shardul Mahadik edited comment on SPARK-41162 at 12/16/22 6:27 PM: ---

[jira] [Commented] (SPARK-41162) Anti-join must not be pushed below aggregation with ambiguous predicates

[ https://issues.apache.org/jira/browse/SPARK-41162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648751#comment-17648751 ] Shardul Mahadik commented on SPARK-41162: - [~cloud_fan] Can you help take a look

[jira] [Updated] (SPARK-33870) Enable spark.storage.replication.proactive by default

[ https://issues.apache.org/jira/browse/SPARK-33870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33870: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Enable spark.storage.re

[jira] [Updated] (SPARK-38023) ExecutorMonitor.onExecutorRemoved should handle ExecutorDecommission as finished

[ https://issues.apache.org/jira/browse/SPARK-38023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38023: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > ExecutorMonitor.onExecutorRemov

[jira] [Updated] (SPARK-39987) Support PEAK_JVM_(ON|OFF)HEAP_MEMORY executor rolling policy

[ https://issues.apache.org/jira/browse/SPARK-39987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39987: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Improvement) > Support PEAK_JVM_(ON|OF

[jira] [Created] (SPARK-41551) Improve/complete PathOutputCommitProtocol support for dynamic partitioning

Steve Loughran created SPARK-41551: -- Summary: Improve/complete PathOutputCommitProtocol support for dynamic partitioning Key: SPARK-41551 URL: https://issues.apache.org/jira/browse/SPARK-41551 Projec

[jira] [Updated] (SPARK-41543) Add `TOTAL_SHUFFLE_WRITE` executor roll policy

[ https://issues.apache.org/jira/browse/SPARK-41543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41543: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Add `TOTAL_SHUFFLE_WRITE` executor roll policy

[jira] [Updated] (SPARK-41540) Add `DISK_USED` executor roll policy

[ https://issues.apache.org/jira/browse/SPARK-41540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41540: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Add `DISK_USED` executor roll policy > ---

[jira] [Updated] (SPARK-41536) Remove `Dynamic Resource Allocation` from K8s Future Work

[ https://issues.apache.org/jira/browse/SPARK-41536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41536: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Remove `Dynamic Resource Allocation` from K8s

[jira] [Updated] (SPARK-40304) Add decomTestTag to K8s Integration Test

[ https://issues.apache.org/jira/browse/SPARK-40304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40304: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Add decomTestTag to K8s Integration Test > ---

[jira] [Updated] (SPARK-39324) Log ExecutorDecommission as INFO level in TaskSchedulerImpl

[ https://issues.apache.org/jira/browse/SPARK-39324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39324: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Log ExecutorDecommission as INFO level in Task

[jira] [Updated] (SPARK-40198) Enable spark.storage.decommission.(rdd|shuffle)Blocks.enabled by default

[ https://issues.apache.org/jira/browse/SPARK-40198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-40198: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Enable spark.storage.decommission.(rdd|shuffle

[jira] [Updated] (SPARK-39450) Reuse PVCs by default

[ https://issues.apache.org/jira/browse/SPARK-39450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39450: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > Reuse PVCs by default > ---

[jira] [Updated] (SPARK-39846) Enable spark.dynamicAllocation.shuffleTracking.enabled by default

[ https://issues.apache.org/jira/browse/SPARK-39846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39846: -- Parent Issue: SPARK-41550 (was: SPARK-41515) > Enable spark.dynamicAllocation.shuffleTracking

[jira] [Updated] (SPARK-39450) Reuse PVCs by default

[ https://issues.apache.org/jira/browse/SPARK-39450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39450: -- Parent: (was: SPARK-41515) Issue Type: Bug (was: Sub-task) > Reuse PVCs by defaul

[jira] [Resolved] (SPARK-41515) PVC-oriented executor pod allocation

[ https://issues.apache.org/jira/browse/SPARK-41515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41515. --- Fix Version/s: 3.4.0 Resolution: Fixed > PVC-oriented executor pod allocation > -

[jira] [Updated] (SPARK-38019) ExecutorMonitor.timedOutExecutors should be deterministic

[ https://issues.apache.org/jira/browse/SPARK-38019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38019: -- Parent: (was: SPARK-35781) Issue Type: Bug (was: Sub-task) > ExecutorMonitor.time

[jira] [Updated] (SPARK-38019) ExecutorMonitor.timedOutExecutors should be deterministic

[ https://issues.apache.org/jira/browse/SPARK-38019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-38019: -- Parent: SPARK-41550 Issue Type: Sub-task (was: Bug) > ExecutorMonitor.timedOutExecuto

[jira] [Updated] (SPARK-41550) Dynamic Allocation on K8S GA

[ https://issues.apache.org/jira/browse/SPARK-41550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-41550: -- Labels: releasenotes (was: ) > Dynamic Allocation on K8S GA > >

[jira] [Created] (SPARK-41550) Dynamic Allocation on K8S GA

Dongjoon Hyun created SPARK-41550: - Summary: Dynamic Allocation on K8S GA Key: SPARK-41550 URL: https://issues.apache.org/jira/browse/SPARK-41550 Project: Spark Issue Type: Umbrella

[jira] [Updated] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure

[ https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-39006: -- Parent: SPARK-41515 Issue Type: Sub-task (was: Improvement) > Show a directional erro

[jira] [Resolved] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-41548. --- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39092 [https://

[jira] [Assigned] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-41548: - Assignee: Hyukjin Kwon > Disable ANSI mode in pyspark.sql.tests.connect.test_connect_fu

[jira] [Created] (SPARK-41549) Create a TIME data type

Patrick Grandjean created SPARK-41549: - Summary: Create a TIME data type Key: SPARK-41549 URL: https://issues.apache.org/jira/browse/SPARK-41549 Project: Spark Issue Type: Wish

[jira] [Commented] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648671#comment-17648671 ] Apache Spark commented on SPARK-41420: -- User 'techaddict' has created a pull reques

[jira] [Assigned] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41420: Assignee: Apache Spark > Protobuf serializer for ApplicationInfoWrapper > ---

[jira] [Assigned] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41420: Assignee: (was: Apache Spark) > Protobuf serializer for ApplicationInfoWrapper >

[jira] [Commented] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648670#comment-17648670 ] Apache Spark commented on SPARK-41420: -- User 'techaddict' has created a pull reques

[jira] [Commented] (SPARK-41420) Protobuf serializer for ApplicationInfoWrapper

[ https://issues.apache.org/jira/browse/SPARK-41420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648667#comment-17648667 ] Sandeep Singh commented on SPARK-41420: --- I can work on this one > Protobuf serial

[jira] [Assigned] (SPARK-41523) `protoc-jar-maven-plugin` should uniformly use `protoc-jar-maven-plugin.version` as the version

[ https://issues.apache.org/jira/browse/SPARK-41523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41523: Assignee: Yang Jie > `protoc-jar-maven-plugin` should uniformly use > `protoc-jar-maven-

[jira] [Resolved] (SPARK-41523) `protoc-jar-maven-plugin` should uniformly use `protoc-jar-maven-plugin.version` as the version

[ https://issues.apache.org/jira/browse/SPARK-41523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41523. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39066 [https://gi

[jira] [Assigned] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41548: Assignee: (was: Apache Spark) > Disable ANSI mode in pyspark.sql.tests.connect.test_c

[jira] [Commented] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648618#comment-17648618 ] Apache Spark commented on SPARK-41548: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41548: Assignee: Apache Spark > Disable ANSI mode in pyspark.sql.tests.connect.test_connect_func

[jira] [Updated] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41548: - Description: There are failures in `test_connect_functions` with ANSI mode on (https://github.c

[jira] [Updated] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41548: - Description: There are failures in {{test_connect_functions}} with ANSI mode on (https://github

[jira] [Assigned] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41547: Assignee: Xinrong Meng > Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_fun

[jira] [Created] (SPARK-41548) Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

Hyukjin Kwon created SPARK-41548: Summary: Disable ANSI mode in pyspark.sql.tests.connect.test_connect_functions Key: SPARK-41548 URL: https://issues.apache.org/jira/browse/SPARK-41548 Project: Spark

[jira] [Updated] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41547: - Description: See https://issues.apache.org/jira/browse/SPARK-41548 We should fix the tests. > R

[jira] [Updated] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

[ https://issues.apache.org/jira/browse/SPARK-41547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41547: - Parent: SPARK-41285 Issue Type: Sub-task (was: Test) > Reenable ANSI mode in pyspark.sq

[jira] [Created] (SPARK-41547) Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions

Hyukjin Kwon created SPARK-41547: Summary: Reenable ANSI mode in pyspark.sql.tests.connect.test_connect_functions Key: SPARK-41547 URL: https://issues.apache.org/jira/browse/SPARK-41547 Project: Spark

[jira] [Commented] (SPARK-24497) ANSI SQL: Recursive query

[ https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648616#comment-17648616 ] gabrywu commented on SPARK-24497: - this is a useful feature, when will it be merged to m

[jira] (SPARK-41527) Implement DataFrame.observe

[ https://issues.apache.org/jira/browse/SPARK-41527 ] jiaan.geng deleted comment on SPARK-41527: was (Author: beliefer): I'm working on. > Implement DataFrame.observe > --- > > Key: SPARK-41527 >

[jira] [Created] (SPARK-41546) pyspark_types_to_proto_types should supports StructType.

jiaan.geng created SPARK-41546: -- Summary: pyspark_types_to_proto_types should supports StructType. Key: SPARK-41546 URL: https://issues.apache.org/jira/browse/SPARK-41546 Project: Spark Issue Ty

[jira] [Commented] (SPARK-41546) pyspark_types_to_proto_types should supports StructType.

[ https://issues.apache.org/jira/browse/SPARK-41546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648615#comment-17648615 ] jiaan.geng commented on SPARK-41546: I'm working on. > pyspark_types_to_proto_types

[jira] [Commented] (SPARK-41527) Implement DataFrame.observe

[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648606#comment-17648606 ] Apache Spark commented on SPARK-41527: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-41527) Implement DataFrame.observe

[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41527: Assignee: Apache Spark > Implement DataFrame.observe > --- > >

[jira] [Assigned] (SPARK-41527) Implement DataFrame.observe

[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41527: Assignee: (was: Apache Spark) > Implement DataFrame.observe > ---

[jira] [Commented] (SPARK-41527) Implement DataFrame.observe

[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648604#comment-17648604 ] Apache Spark commented on SPARK-41527: -- User 'beliefer' has created a pull request

[jira] [Assigned] (SPARK-41334) move SortField from relations.proto to expressions.proto

[ https://issues.apache.org/jira/browse/SPARK-41334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41334: Assignee: (was: Apache Spark) > move SortField from relations.proto to expressions.pr

[jira] [Assigned] (SPARK-41334) move SortField from relations.proto to expressions.proto

[ https://issues.apache.org/jira/browse/SPARK-41334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41334: Assignee: Apache Spark > move SortField from relations.proto to expressions.proto > -

[jira] [Commented] (SPARK-41334) move SortField from relations.proto to expressions.proto

[ https://issues.apache.org/jira/browse/SPARK-41334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648571#comment-17648571 ] Apache Spark commented on SPARK-41334: -- User 'zhengruifeng' has created a pull requ

[jira] [Resolved] (SPARK-41541) Fix wrong child call in SQLShuffleWriteMetricsReporter.decRecordsWritten()

[ https://issues.apache.org/jira/browse/SPARK-41541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41541. -- Fix Version/s: 3.3.2 3.1.4 3.2.3 3.4.0

[jira] [Resolved] (SPARK-41453) Implement DataFrame.subtract

[ https://issues.apache.org/jira/browse/SPARK-41453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41453. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39073 [https://gi

[jira] [Assigned] (SPARK-41453) Implement DataFrame.subtract

[ https://issues.apache.org/jira/browse/SPARK-41453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41453: Assignee: jiaan.geng > Implement DataFrame.subtract > > >

[jira] [Assigned] (SPARK-41544) Upgrade `versions-maven-plugin` to 2.14.1

[ https://issues.apache.org/jira/browse/SPARK-41544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41544: Assignee: Yang Jie > Upgrade `versions-maven-plugin` to 2.14.1 >

[jira] [Resolved] (SPARK-41544) Upgrade `versions-maven-plugin` to 2.14.1

[ https://issues.apache.org/jira/browse/SPARK-41544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41544. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39089 [https://gi

[jira] [Commented] (SPARK-41527) Implement DataFrame.observe

[ https://issues.apache.org/jira/browse/SPARK-41527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648395#comment-17648395 ] jiaan.geng commented on SPARK-41527: I'm working on. > Implement DataFrame.observe

<    1   2