nsivabalan commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4522052515


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,7 +80,23 @@ protected Pair<CompletableFuture, ExecutorService> 
startService() {
 
           if (null != instantTime) {
             log.info("Starting Compaction for instant {}", instantTime);
-            compactor.compact(instantTime);
+            BaseHoodieWriteClient runClient = compactor.compactionClient;
+            try {
+              compactor.compact(instantTime);

Review Comment:
   Fixed in 9b228b04. `BaseCompactor` / `BaseClusterer` now take the write 
client to run with (`compact(client, instant)` / `cluster(client, instant)`; 
the old one-argument methods delegate with the current client). The service 
reads its client **once** per run and passes that same reference both to the 
run and to `isRetryableFailure`, so a switch between two reads can no longer 
happen. No lock is held across the operation. 
`testFailureClassifiedAgainstTheClientOfTheRun` switches clients while the run 
is in progress and asserts that the client that ran equals the client the 
failure was classified against.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,7 +80,23 @@ protected Pair<CompletableFuture, ExecutorService> 
startService() {
 
           if (null != instantTime) {
             log.info("Starting Compaction for instant {}", instantTime);
-            compactor.compact(instantTime);
+            BaseHoodieWriteClient runClient = compactor.compactionClient;
+            try {
+              compactor.compact(instantTime);
+            } catch (Exception e) {
+              if (!isRetryableFailure(runClient, e)) {
+                throw e;
+              }
+              log.warn("Compaction for instant {} failed with a write client 
that was replaced, retrying it with the new write client",
+                  instantTime, e);
+              awaitWriteClientReplaced(runClient);
+              if (isPendingCompaction(instantTime)) {
+                enqueuePendingAsyncServiceInstant(instantTime);

Review Comment:
   Good catch, thanks for reproducing it. Fixed in 9b228b04: with a 
multi-writer concurrency mode, before re-enqueueing the compaction the service 
waits until the instant's heartbeat has expired, using the same expiry as 
`HoodieHeartbeatClient#isHeartbeatExpired` (interval × tolerable misses), which 
`compact()` checks via `validateHeartBeat`. It re-reads the heartbeat at least 
once per interval.
   
   I went with waiting rather than deleting the file, because the retry path 
can't prove whose heartbeat it is: a heartbeat orphaned by our closed client 
expires within the timeout, and one a concurrent writer is genuinely refreshing 
is waited out, after which the instant is usually no longer pending and isn't 
retried. The cost is up to one heartbeat timeout of delay, only on this rare 
swap-during-compaction path. Clustering doesn't validate a heartbeat at 
execution, so it needs no wait.
   
   Test: `testCompactionRetryWaitsForHeartbeatToExpire` (OCC config, a fresh 
heartbeat left by the failed run → the retry runs only after expiry). An 
end-to-end OCC variant of the streaming client-swap test isn't included yet; 
the unit test pins the wait. Happy to add the integration variant if you think 
it's worth the runtime.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncClusteringService.java:
##########
@@ -76,7 +78,23 @@ protected Pair<CompletableFuture, ExecutorService> 
startService() {
           final String instant = fetchNextAsyncServiceInstant();
           if (null != instant) {
             log.info("Starting clustering for instant {}", instant);
-            clusteringClient.cluster(instant);
+            BaseHoodieWriteClient runClient = 
clusteringClient.clusteringClient;
+            try {
+              clusteringClient.cluster(instant);
+            } catch (Exception e) {
+              if (!isRetryableFailure(runClient, e)) {
+                throw e;
+              }
+              log.warn("Clustering for instant {} failed with a write client 
that was replaced, retrying it with the new write client",
+                  instant, e);
+              awaitWriteClientReplaced(runClient);
+              if (isPendingClustering(instant)) {
+                enqueuePendingAsyncServiceInstant(instant);
+              } else {
+                log.info("Clustering for instant {} is no longer pending, not 
retrying it", instant);
+              }
+              continue;

Review Comment:
   Done in 9b228b04. The shared part (failure classification against the run's 
client, waiting for the replacement client, pending check, re-enqueue) is now 
`HoodieAsyncTableService#runWithClientHandoff`, with `isRetryableFailure`, 
`isPendingInstant` and `prepareRetry` as hooks. Compaction overrides 
`prepareRetry` for the heartbeat wait. Execution stays service-specific.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala:
##########
@@ -315,18 +333,20 @@ class HoodieStreamingSink(sqlContext: SQLContext,
   }
 
   /**
-   * Same as HoodieStreamer on a schema change (StreamSync#reInitWriteClient 
and
-   * HoodieStreamer#onInitializingWriteClient): the async table services 
switch to the new client for their next
-   * runs, and the previous client is closed right away.
+   * Closes the previous client right away, then switches the async table 
services to the new client for their next
+   * runs; the switch does not wait for a running compaction or clustering. A 
run still using the previous client can
+   * fail once it is closed: the previous client is marked as retired first, 
so that its service retries such a
+   * failure with the new client instead of stopping.
    */
   private def retireWriteClient(previousClient: SparkRDDWriteClient[_], 
newClient: SparkRDDWriteClient[_]): Unit = {

Review Comment:
   Addressed in 9b228b04: the sink now switches its services to the new client 
first, then closes the previous client in a `finally`. That keeps the guarantee 
that the previous client is always closed even if a switch throws, which is why 
the close had moved up in the first place, while a slow `close()` no longer 
delays the handoff.



##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/async/SparkStreamingAsyncClusteringService.java:
##########
@@ -31,8 +33,27 @@ public class SparkStreamingAsyncClusteringService extends 
AsyncClusteringService
 
   private static final long serialVersionUID = 1L;
 
+  // whether a write client was retired by the sink, i.e. closed by the sink 
when it replaced its client
+  private final transient Predicate<BaseHoodieWriteClient> isRetiredClient;
+
   public SparkStreamingAsyncClusteringService(HoodieEngineContext context, 
BaseHoodieWriteClient writeClient) {
+    this(context, writeClient, c -> false);
+  }
+
+  public SparkStreamingAsyncClusteringService(HoodieEngineContext context, 
BaseHoodieWriteClient writeClient,
+      Predicate<BaseHoodieWriteClient> isRetiredClient) {
     super(context, writeClient, true);
+    this.isRetiredClient = isRetiredClient;
+  }
+
+  /**
+   * A clustering run that failed while using a write client the sink has 
since retired is retryable: the sink closes
+   * the previous client when it re-creates its client (e.g. on a schema 
change), which can fail a run still using it.
+   * Any other failure is not retryable and stops the service, which fails the 
next micro-batch with that failure.
+   */
+  @Override
+  protected boolean isRetryableFailure(BaseHoodieWriteClient runClient, 
Exception failure) {
+    return isRetiredClient.test(runClient);

Review Comment:
   This is intentional, and it's bounded. A retired client has been closed, so 
any failure of a run on it is attributable to the close; the exception types 
that surface (closed timeline server, closed heartbeat client, closed storage, 
a cancelled job) vary too much to match reliably. If the failure was a real 
error that happened to coincide with the switch, the retry runs on the new, 
non-retired client, fails again, and that failure is fatal with its cause. So 
an unrelated error costs at most one extra attempt. With 9b228b04 the 
classification uses the exact client the run executed on, and 
`testRetryFailingOnNewClientStopsTheService` covers the "fails again on the new 
client" path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to