shangxinli commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4569059982


##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/async/SparkStreamingAsyncClusteringService.java:
##########
@@ -31,8 +33,27 @@ public class SparkStreamingAsyncClusteringService extends 
AsyncClusteringService
 
   private static final long serialVersionUID = 1L;
 
+  // whether a write client was retired by the sink, i.e. closed by the sink 
when it replaced its client
+  private final transient Predicate<BaseHoodieWriteClient> isRetiredClient;
+
   public SparkStreamingAsyncClusteringService(HoodieEngineContext context, 
BaseHoodieWriteClient writeClient) {
+    this(context, writeClient, c -> false);
+  }
+
+  public SparkStreamingAsyncClusteringService(HoodieEngineContext context, 
BaseHoodieWriteClient writeClient,
+      Predicate<BaseHoodieWriteClient> isRetiredClient) {
     super(context, writeClient, true);
+    this.isRetiredClient = isRetiredClient;
+  }
+
+  /**
+   * A clustering run that failed while using a write client the sink has 
since retired is retryable: the sink closes
+   * the previous client when it re-creates its client (e.g. on a schema 
change), which can fail a run still using it.
+   * Any other failure is not retryable and stops the service, which fails the 
next micro-batch with that failure.
+   */
+  @Override
+  protected boolean isRetryableFailure(BaseHoodieWriteClient runClient, 
Exception failure) {
+    return isRetiredClient.test(runClient);

Review Comment:
   Fair point, since a retired client's failures are all attributable to the 
close, the bounded retry makes sense without checking exception type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to