RexXiong commented on code in PR #2891:
URL: https://github.com/apache/celeborn/pull/2891#discussion_r1833616602
##########
common/src/main/proto/TransportMessages.proto:
##########
@@ -443,6 +443,7 @@ message PbHeartbeatFromApplication {
repeated PbWorkerInfo needCheckedWorkerList = 5;
bool shouldResponse = 6;
int64 shuffleFallbackCount = 7;
+ map<string, int64> shuffleFallbackCounts = 8;
Review Comment:
We can change to totalShuffleCount
##########
client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala:
##########
@@ -35,12 +37,25 @@ class CelebornShuffleFallbackPolicyRunner(conf:
CelebornConf) extends Logging {
def applyFallbackPolicies[K, V, C](
dependency: ShuffleDependency[K, V, C],
lifecycleManager: LifecycleManager): Boolean = {
- val needFallback =
- shuffleFallbackPolicies.exists(_.needFallback(dependency, conf,
lifecycleManager))
- if (needFallback && FallbackPolicy.NEVER.equals(shuffleFallbackPolicy)) {
- throw new CelebornIOException(
- "Fallback to spark built-in shuffle implementation is prohibited.")
+ val fallbackPolicy =
Review Comment:
Could you also add a total shuffle count metric?
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/AbstractMetaManager.java:
##########
@@ -550,4 +561,19 @@ public void removeApplicationMeta(String appId) {
public int registeredShuffleCount() {
return
registeredAppAndShuffles.values().stream().mapToInt(Set::size).sum();
}
+
+ private void addShuffleFallbackCounts(Map<String, Long> fallbackCounts) {
+ for (String fallbackPolicy : fallbackCounts.keySet()) {
+ shuffleFallbackCounts.compute(
+ fallbackPolicy,
+ (k, v) -> {
+ LongAdder fallbackCount = v;
+ if (fallbackCount == null) {
+ fallbackCount = new LongAdder();
Review Comment:
Use Long is enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
