[jira] [Assigned] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-3: --- Assignee: Yang Jie > Add `RocksDBProvider` similar to `LevelDBProvider` > -

[jira] [Resolved] (SPARK-40186) mergedShuffleCleaner should have been shutdown before db closed

[ https://issues.apache.org/jira/browse/SPARK-40186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40186. - Target Version/s: 3.4.0 Assignee: Yang Jie Resolution: Fixe

[jira] [Resolved] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40094. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 375

[jira] [Assigned] (SPARK-40094) Send TaskEnd event when task failed with NotSerializableException or TaskOutputFileAlreadyExistException to release executors for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-40094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40094: --- Assignee: wangshengjie > Send TaskEnd event when task failed with NotSeria

[jira] [Assigned] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38909: --- Assignee: Yang Jie > Encapsulate LevelDB used by ExternalShuffleBlockResolv

[jira] [Resolved] (SPARK-38909) Encapsulate LevelDB used by ExternalShuffleBlockResolver and YarnShuffleService as LocalDB

[ https://issues.apache.org/jira/browse/SPARK-38909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38909. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 362

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Kai-Hsun Chen) > Improve LaunchTask process

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Kai-Hsun Chen (was: Mridul Muralidharan) > Improve LaunchTask pr

[jira] [Assigned] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39955: --- Assignee: Mridul Muralidharan > Improve LaunchTask process to avoid Stage f

[jira] [Resolved] (SPARK-39955) Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages

[ https://issues.apache.org/jira/browse/SPARK-39955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39955. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 373

[jira] [Resolved] (SPARK-40004) Redundant `LevelDB.get` in `RemoteBlockPushResolver`

[ https://issues.apache.org/jira/browse/SPARK-40004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-40004. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 374

[jira] [Assigned] (SPARK-40004) Redundant `LevelDB.get` in `RemoteBlockPushResolver`

[ https://issues.apache.org/jira/browse/SPARK-40004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-40004: --- Assignee: Yang Jie > Redundant `LevelDB.get` in `RemoteBlockPushResolver` >

[jira] [Commented] (SPARK-39696) Uncaught exception in thread executor-heartbeater java.util.ConcurrentModificationException: mutation occurred during iteration

[ https://issues.apache.org/jira/browse/SPARK-39696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571169#comment-17571169 ] Mridul Muralidharan commented on SPARK-39696: - Can you give more details on

[jira] [Resolved] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

[ https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33236. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 359

[jira] [Assigned] (SPARK-33236) Enable Push-based shuffle service to store state in NM level DB for work preserving restart

[ https://issues.apache.org/jira/browse/SPARK-33236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33236: --- Assignee: Ye Zhou > Enable Push-based shuffle service to store state in NM

[jira] [Resolved] (SPARK-39647) Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been

[ https://issues.apache.org/jira/browse/SPARK-39647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-39647. - Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue res

[jira] [Assigned] (SPARK-39647) Block push fails with java.lang.IllegalArgumentException: Active local dirs list has not been updated by any executor registration even when the NodeManager hasn't been

[ https://issues.apache.org/jira/browse/SPARK-39647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-39647: --- Assignee: Chandni Singh > Block push fails with java.lang.IllegalArgumentEx

[jira] [Resolved] (SPARK-30835) Add support for YARN decommissioning & pre-emption

[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-30835. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 356

[jira] [Assigned] (SPARK-30835) Add support for YARN decommissioning & pre-emption

[ https://issues.apache.org/jira/browse/SPARK-30835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30835: --- Assignee: Abhishek Dixit > Add support for YARN decommissioning & pre-empti

[jira] [Commented] (SPARK-38987) Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

[ https://issues.apache.org/jira/browse/SPARK-38987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550681#comment-17550681 ] Mridul Muralidharan commented on SPARK-38987: - Manually marking as resolved

[jira] [Resolved] (SPARK-38987) Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true

[ https://issues.apache.org/jira/browse/SPARK-38987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38987. - Fix Version/s: 3.4.0 Assignee: Aravind Patnam Resolution: Fixed

[jira] [Assigned] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried

[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38973: --- Assignee: Chandni Singh > When push-based shuffle is enabled, a stage may n

[jira] [Resolved] (SPARK-38973) When push-based shuffle is enabled, a stage may not complete when retried

[ https://issues.apache.org/jira/browse/SPARK-38973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38973. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 362

[jira] [Resolved] (SPARK-38856) Fix a rejectedExecutionException error when push-based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-38856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38856. - Target Version/s: 3.4.0 Assignee: weixiuli Resolution: Fixe

[jira] [Resolved] (SPARK-38555) Avoid contention and get or create clientPools quickly in the TransportClientFactory

[ https://issues.apache.org/jira/browse/SPARK-38555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38555. - Assignee: weixiuli Resolution: Fixed > Avoid contention and get or create

[jira] [Updated] (SPARK-38555) Avoid contention and get or create clientPools quickly in the TransportClientFactory

[ https://issues.apache.org/jira/browse/SPARK-38555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-38555: Fix Version/s: 3.4.0 > Avoid contention and get or create clientPools quickly in

[jira] [Commented] (SPARK-38388) Repartition + Stage retries could lead to incorrect data

[ https://issues.apache.org/jira/browse/SPARK-38388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507257#comment-17507257 ] Mridul Muralidharan commented on SPARK-38388: - Agree with [~jiangxb1987], ei

[jira] [Resolved] (SPARK-38428) Check the FetchShuffleBlocks message only once to improve iteration in external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-38428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38428. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 357

[jira] [Resolved] (SPARK-38309) SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics

[ https://issues.apache.org/jira/browse/SPARK-38309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38309. - Fix Version/s: 3.3.0 3.2.2 3.1.3 Res

[jira] [Assigned] (SPARK-38309) SHS has incorrect percentiles for shuffle read bytes and shuffle total blocks metrics

[ https://issues.apache.org/jira/browse/SPARK-38309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38309: --- Assignee: Rob Reeves > SHS has incorrect percentiles for shuffle read bytes

[jira] [Resolved] (SPARK-38344) Avoid to submit task when there are no requests to push up in push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-38344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-38344. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 356

[jira] [Assigned] (SPARK-38344) Avoid to submit task when there are no requests to push up in push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-38344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-38344: --- Assignee: weixiuli > Avoid to submit task when there are no requests to pus

[jira] [Resolved] (SPARK-37688) ExecutorMonitor should ignore SparkListenerBlockUpdated event if executor was not active

[ https://issues.apache.org/jira/browse/SPARK-37688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37688. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 349

[jira] [Assigned] (SPARK-37688) ExecutorMonitor should ignore SparkListenerBlockUpdated event if executor was not active

[ https://issues.apache.org/jira/browse/SPARK-37688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37688: --- Assignee: hujiahua > ExecutorMonitor should ignore SparkListenerBlockUpdate

[jira] [Resolved] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

[ https://issues.apache.org/jira/browse/SPARK-34826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34826. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 341

[jira] [Assigned] (SPARK-34826) Adaptive fetch of shuffle mergers for Push based shuffle

[ https://issues.apache.org/jira/browse/SPARK-34826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34826: --- Assignee: Venkata krishnan Sowrirajan > Adaptive fetch of shuffle mergers f

[jira] [Assigned] (SPARK-37793) Invalid LocalMergedBlockData cause task hang

[ https://issues.apache.org/jira/browse/SPARK-37793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37793: --- Assignee: Chandni Singh > Invalid LocalMergedBlockData cause task hang > --

[jira] [Resolved] (SPARK-37793) Invalid LocalMergedBlockData cause task hang

[ https://issues.apache.org/jira/browse/SPARK-37793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37793. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed > Invalid

[jira] [Resolved] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37675. - Fix Version/s: 3.3.0 3.2.2 Resolution: Fixed Issue res

[jira] [Assigned] (SPARK-37675) Push-based merge finalization bugs in the RemoteBlockPushResolver

[ https://issues.apache.org/jira/browse/SPARK-37675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37675: --- Assignee: Chandni Singh > Push-based merge finalization bugs in the RemoteB

[jira] [Assigned] (SPARK-37712) Spark request yarn cluster metrics slow and cause unnecessary delay

[ https://issues.apache.org/jira/browse/SPARK-37712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37712: --- Assignee: angerszhu > Spark request yarn cluster metrics slow and cause unn

[jira] [Resolved] (SPARK-37712) Spark request yarn cluster metrics slow and cause unnecessary delay

[ https://issues.apache.org/jira/browse/SPARK-37712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37712. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 349

[jira] [Assigned] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37847: --- Assignee: Cheng Pan > PushBlockStreamCallback should check isTooLate first

[jira] [Resolved] (SPARK-37847) PushBlockStreamCallback should check isTooLate first to avoid NPE

[ https://issues.apache.org/jira/browse/SPARK-37847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37847. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 351

[jira] [Assigned] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow

[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37710: --- Assignee: Eren Avsarogullari > Add detailed log message for java.io.IOExcep

[jira] [Resolved] (SPARK-37710) Add detailed log message for java.io.IOException occurring on Kryo flow

[ https://issues.apache.org/jira/browse/SPARK-37710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37710. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 349

[jira] [Resolved] (SPARK-33701) Adaptive shuffle merge finalization for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-33701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-33701. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 338

[jira] [Assigned] (SPARK-33701) Adaptive shuffle merge finalization for push-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-33701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-33701: --- Assignee: Venkata krishnan Sowrirajan > Adaptive shuffle merge finalization

[jira] [Resolved] (SPARK-37023) Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[ https://issues.apache.org/jira/browse/SPARK-37023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-37023. - Fix Version/s: 3.3.0 3.2.1 Resolution: Fixed Issue res

[jira] [Assigned] (SPARK-37023) Avoid fetching merge status when shuffleMergeEnabled is false for a shuffleDependency during retry

[ https://issues.apache.org/jira/browse/SPARK-37023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-37023: --- Assignee: Minchu Yang > Avoid fetching merge status when shuffleMergeEnable

[jira] [Resolved] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36834. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 340

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: Thejdeep Gudivada > Namespace log lines in External Shuffle Servi

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: (was: Thejdeep Gudivada) > Namespace log lines in External Sh

[jira] [Assigned] (SPARK-36834) Namespace log lines in External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-36834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36834: --- Assignee: Thejdeep Gudivada > Namespace log lines in External Shuffle Servi

[jira] [Assigned] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36798: --- Assignee: Harsh Panchal > When SparkContext is stopped, metrics system shou

[jira] [Resolved] (SPARK-36798) When SparkContext is stopped, metrics system should be flushed after listeners have finished processing

[ https://issues.apache.org/jira/browse/SPARK-36798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36798. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 340

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425016#comment-17425016 ] Mridul Muralidharan commented on SPARK-36892: - Sounds good [~Gengliang.Wang]

[jira] [Commented] (SPARK-36926) Discrepancy in Q22 of TPCH for Spark 3.2

[ https://issues.apache.org/jira/browse/SPARK-36926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424097#comment-17424097 ] Mridul Muralidharan commented on SPARK-36926: - +CC [~gengliang], [~cloud_fan

[jira] [Comment Edited] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424052#comment-17424052 ] Mridul Muralidharan edited comment on SPARK-36892 at 10/4/21, 5:26 PM: ---

[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424052#comment-17424052 ] Mridul Muralidharan commented on SPARK-36892: - [~Gengliang.Wang] Thanks for

[jira] [Created] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled

Mridul Muralidharan created SPARK-36892: --- Summary: Disable batch fetch for a shuffle when push based shuffle is enabled Key: SPARK-36892 URL: https://issues.apache.org/jira/browse/SPARK-36892 Pr

[jira] [Resolved] (SPARK-36615) SparkContext should register shutdown hook earlier

[ https://issues.apache.org/jira/browse/SPARK-36615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36615. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 338

[jira] [Assigned] (SPARK-36615) SparkContext should register shutdown hook earlier

[ https://issues.apache.org/jira/browse/SPARK-36615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36615: --- Assignee: angerszhu > SparkContext should register shutdown hook earlier >

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415891#comment-17415891 ] Mridul Muralidharan commented on SPARK-36772: - Thanks [~Gengliang.Wang] ! An

[jira] [Comment Edited] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415803#comment-17415803 ] Mridul Muralidharan edited comment on SPARK-36772 at 9/15/21, 11:55 PM: --

[jira] [Commented] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17415803#comment-17415803 ] Mridul Muralidharan commented on SPARK-36772: - +CC [~Gengliang.Wang] > Fina

[jira] [Created] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

Mridul Muralidharan created SPARK-36772: --- Summary: FinalizeShuffleMerge fails with an exception due to attempt id not matching Key: SPARK-36772 URL: https://issues.apache.org/jira/browse/SPARK-36772

[jira] [Assigned] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36705: --- Assignee: Minchu Yang > Disable push based shuffle when IO encryption is en

[jira] [Resolved] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

[ https://issues.apache.org/jira/browse/SPARK-36705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36705. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 339

[jira] [Commented] (SPARK-36700) BlockManager re-registration is broken due to deferred removal of BlockManager

[ https://issues.apache.org/jira/browse/SPARK-36700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412732#comment-17412732 ] Mridul Muralidharan commented on SPARK-36700: - Thanks for chasing this down

[jira] [Created] (SPARK-36705) Disable push based shuffle when IO encryption is enabled or serializer is not relocatable

Mridul Muralidharan created SPARK-36705: --- Summary: Disable push based shuffle when IO encryption is enabled or serializer is not relocatable Key: SPARK-36705 URL: https://issues.apache.org/jira/browse/SPARK-

[jira] [Assigned] (SPARK-35548) Handling new attempt has started error message in BlockPushErrorHandler in client

[ https://issues.apache.org/jira/browse/SPARK-35548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35548: --- Assignee: Qi Zhu (was: zhuqi) > Handling new attempt has started error mes

[jira] [Assigned] (SPARK-36344) Fix some typos in ShuffleBlockPusher class.

[ https://issues.apache.org/jira/browse/SPARK-36344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36344: --- Assignee: Qi Zhu (was: zhuqi) > Fix some typos in ShuffleBlockPusher class

[jira] [Assigned] (SPARK-35548) Handling new attempt has started error message in BlockPushErrorHandler in client

[ https://issues.apache.org/jira/browse/SPARK-35548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35548: --- Assignee: zhuqi > Handling new attempt has started error message in BlockPu

[jira] [Resolved] (SPARK-35548) Handling new attempt has started error message in BlockPushErrorHandler in client

[ https://issues.apache.org/jira/browse/SPARK-35548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35548. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 336

[jira] [Resolved] (SPARK-36374) Push-based shuffle documentation

[ https://issues.apache.org/jira/browse/SPARK-36374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36374. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 336

[jira] [Assigned] (SPARK-36374) Push-based shuffle documentation

[ https://issues.apache.org/jira/browse/SPARK-36374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36374: --- Assignee: Venkata krishnan Sowrirajan > Push-based shuffle documentation >

[jira] [Resolved] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36483. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 337

[jira] [Assigned] (SPARK-36483) Fix intermittent test failure due to netty dependency version bump

[ https://issues.apache.org/jira/browse/SPARK-36483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36483: --- Assignee: Min Shen > Fix intermittent test failure due to netty dependency

[jira] [Commented] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397725#comment-17397725 ] Mridul Muralidharan commented on SPARK-36378: - Thanks for marking this resol

[jira] [Resolved] (SPARK-36423) Randomize blocks within a push request before pushing to improve block merge ratio

[ https://issues.apache.org/jira/browse/SPARK-36423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36423. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 336

[jira] [Assigned] (SPARK-36423) Randomize blocks within a push request before pushing to improve block merge ratio

[ https://issues.apache.org/jira/browse/SPARK-36423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36423: --- Assignee: Min Shen > Randomize blocks within a push request before pushing

[jira] [Assigned] (SPARK-36173) [CORE] Support getting CPU number in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-36173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36173: --- Assignee: Xiaochang Wu > [CORE] Support getting CPU number in TaskContext >

[jira] [Resolved] (SPARK-36173) [CORE] Support getting CPU number in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-36173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36173. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 333

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30602: --- Assignee: Min Shen > SPIP: Support push-based shuffle to improve shuffle ef

[jira] [Commented] (SPARK-35275) Add checksum for shuffle blocks

[ https://issues.apache.org/jira/browse/SPARK-35275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391640#comment-17391640 ] Mridul Muralidharan commented on SPARK-35275: - Do we want to add a documenta

[jira] [Assigned] (SPARK-36206) Diagnose shuffle data corruption by checksum

[ https://issues.apache.org/jira/browse/SPARK-36206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36206: --- Assignee: wuyi > Diagnose shuffle data corruption by checksum > ---

[jira] [Resolved] (SPARK-36206) Diagnose shuffle data corruption by checksum

[ https://issues.apache.org/jira/browse/SPARK-36206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36206. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 334

[jira] [Updated] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-35917: Fix Version/s: (was: 3.2.0) > Disable push-based shuffle until the feature is

[jira] [Updated] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-35917: Fix Version/s: 3.2.0 > Disable push-based shuffle until the feature is complete >

[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete

[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35917: --- Assignee: (was: Mridul Muralidharan) > Disable push-based shuffle until

[jira] [Assigned] (SPARK-36378) Minor changes to address a few identified server side inefficiencies

[ https://issues.apache.org/jira/browse/SPARK-36378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36378: --- Assignee: (was: Mridul Muralidharan) > Minor changes to address a few i

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30602: --- Shepherd: Mridul Muralidharan Assignee: (was: Mridul Muralidharan)

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-30602: Fix Version/s: 3.2.0 > SPIP: Support push-based shuffle to improve shuffle efficie

[jira] [Comment Edited] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391318#comment-17391318 ] Mridul Muralidharan edited comment on SPARK-30602 at 8/2/21, 5:30 AM:

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391324#comment-17391324 ] Mridul Muralidharan commented on SPARK-30602: - Thanks for all the work in ge

[jira] [Assigned] (SPARK-36266) Rename classes in shuffle RPC used for block push operations

[ https://issues.apache.org/jira/browse/SPARK-36266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-36266: --- Assignee: Min Shen > Rename classes in shuffle RPC used for block push oper

[jira] [Resolved] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-30602. - Resolution: Fixed The only pending task here is documentation. [~vsowrirajan] ha

[jira] [Assigned] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-30602: --- Assignee: Mridul Muralidharan > SPIP: Support push-based shuffle to improve

[jira] [Updated] (SPARK-32923) Add support to properly handle different type of stage retries

[ https://issues.apache.org/jira/browse/SPARK-32923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32923: Fix Version/s: 3.2.0 > Add support to properly handle different type of stage retr

<    1   2   3   4   5   6   7   8   9   >