shangxinli commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4569059982
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/async/SparkStreamingAsyncClusteringService.java:
##########
@@ -31,8 +33,27 @@ public class SparkStreamingAsyncClusteringService extends
AsyncClusteringService
private static final long serialVersionUID = 1L;
+ // whether a write client was retired by the sink, i.e. closed by the sink
when it replaced its client
+ private final transient Predicate<BaseHoodieWriteClient> isRetiredClient;
+
public SparkStreamingAsyncClusteringService(HoodieEngineContext context,
BaseHoodieWriteClient writeClient) {
+ this(context, writeClient, c -> false);
+ }
+
+ public SparkStreamingAsyncClusteringService(HoodieEngineContext context,
BaseHoodieWriteClient writeClient,
+ Predicate<BaseHoodieWriteClient> isRetiredClient) {
super(context, writeClient, true);
+ this.isRetiredClient = isRetiredClient;
+ }
+
+ /**
+ * A clustering run that failed while using a write client the sink has
since retired is retryable: the sink closes
+ * the previous client when it re-creates its client (e.g. on a schema
change), which can fail a run still using it.
+ * Any other failure is not retryable and stops the service, which fails the
next micro-batch with that failure.
+ */
+ @Override
+ protected boolean isRetryableFailure(BaseHoodieWriteClient runClient,
Exception failure) {
+ return isRetiredClient.test(runClient);
Review Comment:
Fair point, since a retired client's failures are all attributable to the
close, the bounded retry makes sense without checking exception type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]