[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r927806886 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -656,6 +787,236 @@ public void registerExecutor(String

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r927276943 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -1245,11 +1696,24 @@ int getNumIOExceptions() { /** *

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r914055427 ## common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java: ## @@ -621,9 +626,10 @@ public void

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r903254647 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -993,6 +1311,42 @@ AppShufflePartitionInfo

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r903122602 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -632,6 +737,14 @@ public void registerExecutor(String appId,

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r903122602 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -632,6 +737,14 @@ public void registerExecutor(String appId,

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r903112657 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -583,6 +686,7 @@ public MergeStatuses

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r898325741 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -656,6 +771,206 @@ public void registerExecutor(String

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r897489934 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -180,12 +215,15 @@ private AppShufflePartitionInfo

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r890584847 ## common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java: ## @@ -230,11 +241,14 @@ protected void serviceInit(Configuration

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r888604583 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -209,9 +243,13 @@ private AppShufflePartitionInfo

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r888251830 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -342,6 +389,29 @@ void closeAndDeletePartitionFilesIfNeeded(

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r886137214 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -536,9 +619,11 @@ public MergeStatuses

[GitHub] [spark] otterc commented on a diff in pull request #35906: [SPARK-33236][shuffle] Enable Push-based shuffle service to store state in NM level DB for work preserving restart

otterc commented on code in PR #35906: URL: https://github.com/apache/spark/pull/35906#discussion_r837698827 ## common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java: ## @@ -88,13 +103,28 @@ private static final ByteBuffer