This is an automated email from the ASF dual-hosted git repository.
yuanzhou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new de2c94f5ab [VL] Fast fail or resend data to uniffle ASSP (#11213)
de2c94f5ab is described below
commit de2c94f5abab37797f443ec64bd7a4a521aa2913
Author: Junfan Zhang <[email protected]>
AuthorDate: Tue Dec 2 17:37:14 2025 +0800
[VL] Fast fail or resend data to uniffle ASSP (#11213)
Uniffle’s partition reassignment mechanism enables fast switching to
alternative shuffle servers, improving performance and stability. However, in
the previous Uniffle integration with Gluten, this mechanism did not function
properly due to a missing trigger, which could cause significant partition
backpressure without activating multi shuffle-server load balancing.
This PR also enables fast fail for the shuffle writer in case any data is
lost.
---
.../apache/spark/shuffle/writer/VeloxUniffleColumnarShuffleWriter.java | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/backends-velox/src-uniffle/main/java/org/apache/spark/shuffle/writer/VeloxUniffleColumnarShuffleWriter.java
b/backends-velox/src-uniffle/main/java/org/apache/spark/shuffle/writer/VeloxUniffleColumnarShuffleWriter.java
index da2cb20c9a..ef35818c7b 100644
---
a/backends-velox/src-uniffle/main/java/org/apache/spark/shuffle/writer/VeloxUniffleColumnarShuffleWriter.java
+++
b/backends-velox/src-uniffle/main/java/org/apache/spark/shuffle/writer/VeloxUniffleColumnarShuffleWriter.java
@@ -310,6 +310,8 @@ public class VeloxUniffleColumnarShuffleWriter<K, V>
extends RssShuffleWriter<K,
super.getBufferManager()
.addPartitionData(partitionId, data, length,
System.currentTimeMillis());
super.processShuffleBlockInfos(shuffleBlockInfos);
+ // fast fail or resend data
+ super.checkDataIfAnyFailure();
return length;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
