(incubator-gluten) branch main updated: [GLUTEN-4926][CELEBORN] CelebornShuffleManager should remove shuffleId from columnarShuffleIds after unregistering shuffle (#4927)

[ulyssesyou]

This is an automated email from the ASF dual-hosted git repository.

ulyssesyou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git


The following commit(s) were added to refs/heads/main by this push:
     new 494692d3f [GLUTEN-4926][CELEBORN] CelebornShuffleManager should remove 
shuffleId from columnarShuffleIds after unregistering shuffle (#4927)
494692d3f is described below

commit 494692d3f3eba5aad8b2425c120b0a83fa282c02
Author: Nicholas Jiang <programg...@163.com>
AuthorDate: Tue Mar 12 15:29:15 2024 +0800

    [GLUTEN-4926][CELEBORN] CelebornShuffleManager should remove shuffleId from 
columnarShuffleIds after unregistering shuffle (#4927)
---
 .../spark/shuffle/gluten/celeborn/CelebornShuffleManager.java       | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git 
a/gluten-celeborn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
 
b/gluten-celeborn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
index b0f74fad2..c447e7ade 100644
--- 
a/gluten-celeborn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
+++ 
b/gluten-celeborn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
@@ -242,7 +242,11 @@ public class CelebornShuffleManager implements 
ShuffleManager {
   @Override
   public boolean unregisterShuffle(int shuffleId) {
     if (columnarShuffleIds.contains(shuffleId)) {
-      return columnarShuffleManager().unregisterShuffle(shuffleId);
+      if (columnarShuffleManager().unregisterShuffle(shuffleId)) {
+        return columnarShuffleIds.remove(shuffleId);
+      } else {
+        return false;
+      }
     }
     if (appUniqueId == null) {
       return true;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org
For additional commands, e-mail: commits-h...@gluten.apache.org