nsivabalan commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4608320907


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean> 
onShutdownCallback) {
     }
     super.start(onShutdownCallback);
   }
+
+  /**
+   * Runs a table service instant with one write client.
+   */
+  @FunctionalInterface
+  protected interface WriteClientRun {
+    void run(BaseHoodieWriteClient client, String instantTime) throws 
IOException;
+  }
+
+  /**
+   * Runs the table service for the instant with the write client the service 
has when the run starts. That same
+   * client runs the instant and is passed to {@link #isRetryableFailure}, so 
a failure is never attributed to a
+   * different client than the one it happened on. On a retryable failure, 
waits until the writer has switched the
+   * service to another client, lets the service prepare the retry, and 
re-enqueues the instant if it is still pending.
+   *
+   * @param action        name of the table service, for logging
+   * @param currentClient the write client the service currently runs with
+   * @param run           runs the instant with the given client
+   * @return whether the run completed; false if it failed and was handed back 
for a retry or dropped
+   */
+  protected boolean runWithClientHandoff(String action, String instantTime, 
Supplier<BaseHoodieWriteClient> currentClient,
+                                         WriteClientRun run) throws 
IOException, InterruptedException {
+    BaseHoodieWriteClient runClient = currentClient.get();
+    try {
+      run.run(runClient, instantTime);
+      return true;
+    } catch (Exception e) {
+      if (!isRetryableFailure(runClient, e)) {
+        throw e;
+      }
+      log.warn("{} for instant {} failed with a write client that was 
replaced, retrying it with the new write client",
+          action, instantTime, e);
+      awaitWriteClientReplaced(runClient, currentClient);
+      if (isShutdownRequested()) {
+        return false;
+      }
+      BaseHoodieWriteClient retryClient = currentClient.get();
+      // checked before preparing the retry too: preparing it may wait, which 
an instant completed meanwhile doesn't need
+      if (isPendingInstant(retryClient, instantTime)) {
+        prepareRetry(retryClient, instantTime);
+        if (isShutdownRequested()) {
+          return false;
+        }
+        if (isPendingInstant(retryClient, instantTime)) {
+          enqueuePendingAsyncServiceInstant(instantTime);

Review Comment:
   Good catch, fixed in 056e55ff. The service now remembers which queued 
instants are retries, and checks them again when they come up:
   - **When the retry is dequeued**, it's skipped if the instant is no longer 
pending, or, with multi-writer compaction, if another writer holds a fresh 
heartbeat on it.
   - **If the retry then fails** on a client that wasn't retired, the same 
check runs again. If the instant was completed or taken over between the 
dequeue check and `compact()`'s locked validation, the instant is dropped with 
a warning and the service keeps running. Otherwise the failure stops the 
service, as before. If the check itself fails, its exception is added as 
suppressed to the original failure.
   - **Why a fresh heartbeat after a failure means another writer:** a failed 
run of our own deletes its heartbeat in `compact()`'s catch block. The 
heartbeat left behind by a closed client has already been waited out in 
`prepareRetry` before re-enqueueing.
   
   Taken-over instants are dropped, not waited out. If the other writer dies, 
the instant stays pending and the next restart re-enqueues it, the same as an 
instant another writer takes before our first run.
   
   I kept this to retried instants. An instant's first run fails the same way 
it does on master, so this PR doesn't change how first runs are handled.
   
   Regression test: 
`TestAsyncTableServiceRetry#testRetrySkippedWhenConcurrentWriterCompletesOrTakesOverTheInstant`.
 B is held, and A is re-enqueued behind it. It covers three cases:
   - A completes while queued;
   - another writer starts A (fresh heartbeat) while it is queued;
   - the takeover lands after A is dequeued, and `compact()` rejects it with 
the heartbeat `HoodieLockException`.
   
   Each case asserts that A isn't run again and the service stays up.
   



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala:
##########
@@ -805,6 +810,111 @@ class TestStructuredStreaming extends 
HoodieSparkClientTestBase {
     }
   }
 
+  private def compactionControlledOpts: Map[String, String] =
+    getCompactionOpts(HoodieTableType.MERGE_ON_READ, isAsyncCompaction = true) 
++ Map(
+      DataSourceWriteOptions.OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+      DataSourceWriteOptions.RECORD_MERGE_MODE.key -> "CUSTOM",
+      DataSourceWriteOptions.RECORD_MERGE_STRATEGY_ID.key -> 
CompactionControlledSparkRecordMerger.STRATEGY_ID,
+      DataSourceWriteOptions.RECORD_MERGE_IMPL_CLASSES.key -> 
classOf[CompactionControlledSparkRecordMerger].getName,
+      HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet")
+
+  private def writeUpdatesToSource(sourcePath: String, instantTime: String, 
numRecords: Int): Unit = {
+    spark.read.json(spark.sparkContext.parallelize(
+      recordsToStrings(dataGen.generateUniqueUpdates(instantTime, 
numRecords)).asScala.toList, 1))
+      .write.mode(SaveMode.Append).json(sourcePath)
+  }
+
+  /**
+   * An async compaction running on the sink's write client when a schema 
change makes the sink re-create its client
+   * may fail once that client is closed. Its service retries it with the new 
client, and the query keeps running.
+   */
+  @Test
+  def testStructuredStreamingRetriesAsyncCompactionFailedByWriteClientSwap(): 
Unit = {
+    val (sourcePath, destPath) = initStreamingSourceAndDestPath("source", 
"dest")
+    val opts = compactionControlledOpts + 
(DataSourceWriteOptions.SET_NULL_FOR_MISSING_COLUMNS.key -> "true")
+    val inputDF = 
spark.read.json(spark.sparkContext.parallelize(recordsToStrings(dataGen.generateInsertsForPartition(
+      "000", 20, 
HoodieTestDataGenerator.DEFAULT_FIRST_PARTITION_PATH)).asScala.toList, 1))
+    inputDF.write.mode(SaveMode.Append).json(sourcePath)
+    val query = initWritingStreamingQuery(inputDF.schema, sourcePath, 
destPath, opts)
+    val compactionStarted = new CountDownLatch(1)
+    val releaseCompaction = new CountDownLatch(1)
+    try {
+      query.processAllAvailable()
+      
CompactionControlledSparkRecordMerger.holdNextCompactionThenFail(compactionStarted,
 releaseCompaction)
+      writeUpdatesToSource(sourcePath, "001", 10)
+      query.processAllAvailable()
+      assertTrue(compactionStarted.await(2, TimeUnit.MINUTES), "the async 
compaction must start")
+
+      // evolve the table schema outside of the query, so that the next 
micro-batch re-creates the sink's write client
+      spark.read.json(spark.sparkContext.parallelize(
+        recordsToStrings(dataGen.generateUniqueUpdates("002", 
5)).asScala.toList, 1))
+        .withColumn("new_col", lit("evolved"))
+        
.write.format("org.apache.hudi").options(opts).mode(SaveMode.Append).save(destPath)
+      writeUpdatesToSource(sourcePath, "003", 10)
+      query.processAllAvailable()
+      // the micro-batch that replaced the write client completed without 
waiting for the running compaction
+      assertTrue(CompactionControlledSparkRecordMerger.holdingCompaction,
+        "the micro-batch must not wait for the running async compaction")
+
+      // the compaction held on the previous write client now fails, and is 
retried with the new one
+      releaseCompaction.countDown()
+      waitTillAtleastNCompletedCompactions(destPath, 1)

Review Comment:
   Done in 056e55ff. While the run is held, the test reads the one inflight 
compaction from the timeline: the service runs one compaction at a time, so the 
held run is the only inflight one. It then waits for that specific instant to 
complete. The held run always fails, so its completion proves the same instant 
was run again, on the replacement client. 
`waitTillAtleastNCompletedCompactions` is replaced by 
`waitTillCompactionCompleted(tablePath, instant)`.
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean> 
onShutdownCallback) {
     }
     super.start(onShutdownCallback);
   }
+
+  /**
+   * Runs a table service instant with one write client.
+   */
+  @FunctionalInterface
+  protected interface WriteClientRun {
+    void run(BaseHoodieWriteClient client, String instantTime) throws 
IOException;
+  }
+
+  /**
+   * Runs the table service for the instant with the write client the service 
has when the run starts. That same
+   * client runs the instant and is passed to {@link #isRetryableFailure}, so 
a failure is never attributed to a
+   * different client than the one it happened on. On a retryable failure, 
waits until the writer has switched the
+   * service to another client, lets the service prepare the retry, and 
re-enqueues the instant if it is still pending.
+   *
+   * @param action        name of the table service, for logging
+   * @param currentClient the write client the service currently runs with
+   * @param run           runs the instant with the given client
+   * @return whether the run completed; false if it failed and was handed back 
for a retry or dropped
+   */
+  protected boolean runWithClientHandoff(String action, String instantTime, 
Supplier<BaseHoodieWriteClient> currentClient,
+                                         WriteClientRun run) throws 
IOException, InterruptedException {
+    BaseHoodieWriteClient runClient = currentClient.get();
+    try {
+      run.run(runClient, instantTime);
+      return true;
+    } catch (Exception e) {
+      if (!isRetryableFailure(runClient, e)) {
+        throw e;
+      }
+      log.warn("{} for instant {} failed with a write client that was 
replaced, retrying it with the new write client",
+          action, instantTime, e);
+      awaitWriteClientReplaced(runClient, currentClient);
+      if (isShutdownRequested()) {
+        return false;
+      }
+      BaseHoodieWriteClient retryClient = currentClient.get();
+      // checked before preparing the retry too: preparing it may wait, which 
an instant completed meanwhile doesn't need
+      if (isPendingInstant(retryClient, instantTime)) {
+        prepareRetry(retryClient, instantTime);
+        if (isShutdownRequested()) {
+          return false;
+        }
+        if (isPendingInstant(retryClient, instantTime)) {
+          enqueuePendingAsyncServiceInstant(instantTime);
+          return false;
+        }
+      }
+      log.info("{} for instant {} is no longer pending, not retrying it", 
action, instantTime);
+      return false;
+    }
+  }
+
+  /**
+   * Whether a run that failed with the given exception is retried. The run 
used {@code runClient}, the client of the
+   * service when the run started. By default no failure is retried and the 
service stops with it.
+   */
+  protected boolean isRetryableFailure(BaseHoodieWriteClient runClient, 
Exception failure) {
+    return false;
+  }
+
+  /**
+   * Whether the instant is still pending, i.e. a retry of it has work to do. 
Only consulted for retryable failures.
+   */
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return true;
+  }
+
+  /**
+   * Prepares the retry of a failed run of the instant with {@code 
retryClient}, before it is re-enqueued.
+   */
+  protected void prepareRetry(BaseHoodieWriteClient retryClient, String 
instantTime) throws IOException, InterruptedException {
+  }
+
+  /**
+   * Waits up to the given time, returning early when shutdown is requested 
(see {@link #onShutdownRequested}) or the
+   * writer switches this service to another client. Callers re-check what 
they wait for after it returns.
+   */
+  protected synchronized void waitUnlessShutdown(long millis) throws 
InterruptedException {
+    if (!isShutdownRequested()) {
+      wait(millis);
+    }
+  }
+
+  @Override
+  protected synchronized void onShutdownRequested() {
+    notifyAll();
+  }
+
+  /**
+   * Waits until the writer has switched this service away from the given 
client. Services notify on this monitor
+   * when they switch clients.
+   */
+  private synchronized void awaitWriteClientReplaced(BaseHoodieWriteClient 
client, Supplier<BaseHoodieWriteClient> currentClient)
+      throws InterruptedException {
+    while (!isShutdownRequested() && currentClient.get() == client) {
+      wait(WRITE_CLIENT_POLL_MS);

Review Comment:
   Agreed, removed in 056e55ff. All client switches go through 
`AsyncCompactService` / `AsyncClusteringService#updateWriteClient`, which 
update the client under this monitor and call `notifyAll()`. That includes the 
Spark streaming sink and HoodieStreamer. `onShutdownRequested()` notifies too, 
and the wait loop re-checks its condition under the monitor, so a switch made 
just before the wait starts isn't missed. `WRITE_CLIENT_POLL_MS` is gone.
   
   Local: `TestAsyncTableService*` 23/23 (Retry 17/17, incl. the 3 new cases), 
`TestStructuredStreaming` 29/29, checkstyle and scalastyle clean.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to