danny0405 commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4594726237


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean> 
onShutdownCallback) {
     }
     super.start(onShutdownCallback);
   }
+
+  /**
+   * Runs a table service instant with one write client.
+   */
+  @FunctionalInterface
+  protected interface WriteClientRun {
+    void run(BaseHoodieWriteClient client, String instantTime) throws 
IOException;
+  }
+
+  /**
+   * Runs the table service for the instant with the write client the service 
has when the run starts. That same
+   * client runs the instant and is passed to {@link #isRetryableFailure}, so 
a failure is never attributed to a
+   * different client than the one it happened on. On a retryable failure, 
waits until the writer has switched the
+   * service to another client, lets the service prepare the retry, and 
re-enqueues the instant if it is still pending.
+   *
+   * @param action        name of the table service, for logging
+   * @param currentClient the write client the service currently runs with
+   * @param run           runs the instant with the given client
+   * @return whether the run completed; false if it failed and was handed back 
for a retry or dropped
+   */
+  protected boolean runWithClientHandoff(String action, String instantTime, 
Supplier<BaseHoodieWriteClient> currentClient,
+                                         WriteClientRun run) throws 
IOException, InterruptedException {
+    BaseHoodieWriteClient runClient = currentClient.get();
+    try {
+      run.run(runClient, instantTime);
+      return true;
+    } catch (Exception e) {
+      if (!isRetryableFailure(runClient, e)) {
+        throw e;
+      }
+      log.warn("{} for instant {} failed with a write client that was 
replaced, retrying it with the new write client",
+          action, instantTime, e);
+      awaitWriteClientReplaced(runClient, currentClient);
+      if (isShutdownRequested()) {
+        return false;
+      }
+      BaseHoodieWriteClient retryClient = currentClient.get();
+      // checked before preparing the retry too: preparing it may wait, which 
an instant completed meanwhile doesn't need
+      if (isPendingInstant(retryClient, instantTime)) {
+        prepareRetry(retryClient, instantTime);
+        if (isShutdownRequested()) {
+          return false;
+        }
+        if (isPendingInstant(retryClient, instantTime)) {
+          enqueuePendingAsyncServiceInstant(instantTime);

Review Comment:
   [P2] Preserve retry eligibility checks through dequeue.
   
   The heartbeat/pending checks happen before this instant is appended to the 
queue, but the next execution goes directly to run.run() without repeating 
them. For example, failed instant A is requeued behind B; while B runs, a 
concurrent writer acquires A or completes it. When A reaches the front, 
compact() rejects the fresh heartbeat or the no-longer-pending instant. That 
attempt uses the replacement, non-retired client, so isRetryableFailure() 
returns false and the service/query stops despite this being a recoverable 
ownership/completion change.
   
   Could we preserve retry state in the queued work and revalidate when 
executing it, with handling for concurrent acquisition/completion during the 
locked compaction validation as well? A regression test can hold B, requeue A 
behind it, mark A completed, then release B and assert that A is skipped and 
the service stays alive.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean> 
onShutdownCallback) {
     }
     super.start(onShutdownCallback);
   }
+
+  /**
+   * Runs a table service instant with one write client.
+   */
+  @FunctionalInterface
+  protected interface WriteClientRun {
+    void run(BaseHoodieWriteClient client, String instantTime) throws 
IOException;
+  }
+
+  /**
+   * Runs the table service for the instant with the write client the service 
has when the run starts. That same
+   * client runs the instant and is passed to {@link #isRetryableFailure}, so 
a failure is never attributed to a
+   * different client than the one it happened on. On a retryable failure, 
waits until the writer has switched the
+   * service to another client, lets the service prepare the retry, and 
re-enqueues the instant if it is still pending.
+   *
+   * @param action        name of the table service, for logging
+   * @param currentClient the write client the service currently runs with
+   * @param run           runs the instant with the given client
+   * @return whether the run completed; false if it failed and was handed back 
for a retry or dropped
+   */
+  protected boolean runWithClientHandoff(String action, String instantTime, 
Supplier<BaseHoodieWriteClient> currentClient,
+                                         WriteClientRun run) throws 
IOException, InterruptedException {
+    BaseHoodieWriteClient runClient = currentClient.get();
+    try {
+      run.run(runClient, instantTime);
+      return true;
+    } catch (Exception e) {
+      if (!isRetryableFailure(runClient, e)) {
+        throw e;
+      }
+      log.warn("{} for instant {} failed with a write client that was 
replaced, retrying it with the new write client",
+          action, instantTime, e);
+      awaitWriteClientReplaced(runClient, currentClient);
+      if (isShutdownRequested()) {
+        return false;
+      }
+      BaseHoodieWriteClient retryClient = currentClient.get();
+      // checked before preparing the retry too: preparing it may wait, which 
an instant completed meanwhile doesn't need
+      if (isPendingInstant(retryClient, instantTime)) {
+        prepareRetry(retryClient, instantTime);
+        if (isShutdownRequested()) {
+          return false;
+        }
+        if (isPendingInstant(retryClient, instantTime)) {
+          enqueuePendingAsyncServiceInstant(instantTime);
+          return false;
+        }
+      }
+      log.info("{} for instant {} is no longer pending, not retrying it", 
action, instantTime);
+      return false;
+    }
+  }
+
+  /**
+   * Whether a run that failed with the given exception is retried. The run 
used {@code runClient}, the client of the
+   * service when the run started. By default no failure is retried and the 
service stops with it.
+   */
+  protected boolean isRetryableFailure(BaseHoodieWriteClient runClient, 
Exception failure) {
+    return false;
+  }
+
+  /**
+   * Whether the instant is still pending, i.e. a retry of it has work to do. 
Only consulted for retryable failures.
+   */
+  protected boolean isPendingInstant(BaseHoodieWriteClient client, String 
instantTime) {
+    return true;
+  }
+
+  /**
+   * Prepares the retry of a failed run of the instant with {@code 
retryClient}, before it is re-enqueued.
+   */
+  protected void prepareRetry(BaseHoodieWriteClient retryClient, String 
instantTime) throws IOException, InterruptedException {
+  }
+
+  /**
+   * Waits up to the given time, returning early when shutdown is requested 
(see {@link #onShutdownRequested}) or the
+   * writer switches this service to another client. Callers re-check what 
they wait for after it returns.
+   */
+  protected synchronized void waitUnlessShutdown(long millis) throws 
InterruptedException {
+    if (!isShutdownRequested()) {
+      wait(millis);
+    }
+  }
+
+  @Override
+  protected synchronized void onShutdownRequested() {
+    notifyAll();
+  }
+
+  /**
+   * Waits until the writer has switched this service away from the given 
client. Services notify on this monitor
+   * when they switch clients.
+   */
+  private synchronized void awaitWriteClientReplaced(BaseHoodieWriteClient 
client, Supplier<BaseHoodieWriteClient> currentClient)
+      throws InterruptedException {
+    while (!isShutdownRequested() && currentClient.get() == client) {
+      wait(WRITE_CLIENT_POLL_MS);

Review Comment:
   Optional simplification: now that updateWriteClient() and 
onShutdownRequested() both notifyAll() on this monitor, is the one-second 
polling still needed? The condition is checked under the same monitor and the 
while loop already handles spurious wakeups, so wait() should suffice for the 
built-in compaction/clustering services. This would remove WRITE_CLIENT_POLL_MS 
and periodic wakeups while awaiting a handoff.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala:
##########
@@ -805,6 +810,111 @@ class TestStructuredStreaming extends 
HoodieSparkClientTestBase {
     }
   }
 
+  private def compactionControlledOpts: Map[String, String] =
+    getCompactionOpts(HoodieTableType.MERGE_ON_READ, isAsyncCompaction = true) 
++ Map(
+      DataSourceWriteOptions.OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+      DataSourceWriteOptions.RECORD_MERGE_MODE.key -> "CUSTOM",
+      DataSourceWriteOptions.RECORD_MERGE_STRATEGY_ID.key -> 
CompactionControlledSparkRecordMerger.STRATEGY_ID,
+      DataSourceWriteOptions.RECORD_MERGE_IMPL_CLASSES.key -> 
classOf[CompactionControlledSparkRecordMerger].getName,
+      HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet")
+
+  private def writeUpdatesToSource(sourcePath: String, instantTime: String, 
numRecords: Int): Unit = {
+    spark.read.json(spark.sparkContext.parallelize(
+      recordsToStrings(dataGen.generateUniqueUpdates(instantTime, 
numRecords)).asScala.toList, 1))
+      .write.mode(SaveMode.Append).json(sourcePath)
+  }
+
+  /**
+   * An async compaction running on the sink's write client when a schema 
change makes the sink re-create its client
+   * may fail once that client is closed. Its service retries it with the new 
client, and the query keeps running.
+   */
+  @Test
+  def testStructuredStreamingRetriesAsyncCompactionFailedByWriteClientSwap(): 
Unit = {
+    val (sourcePath, destPath) = initStreamingSourceAndDestPath("source", 
"dest")
+    val opts = compactionControlledOpts + 
(DataSourceWriteOptions.SET_NULL_FOR_MISSING_COLUMNS.key -> "true")
+    val inputDF = 
spark.read.json(spark.sparkContext.parallelize(recordsToStrings(dataGen.generateInsertsForPartition(
+      "000", 20, 
HoodieTestDataGenerator.DEFAULT_FIRST_PARTITION_PATH)).asScala.toList, 1))
+    inputDF.write.mode(SaveMode.Append).json(sourcePath)
+    val query = initWritingStreamingQuery(inputDF.schema, sourcePath, 
destPath, opts)
+    val compactionStarted = new CountDownLatch(1)
+    val releaseCompaction = new CountDownLatch(1)
+    try {
+      query.processAllAvailable()
+      
CompactionControlledSparkRecordMerger.holdNextCompactionThenFail(compactionStarted,
 releaseCompaction)
+      writeUpdatesToSource(sourcePath, "001", 10)
+      query.processAllAvailable()
+      assertTrue(compactionStarted.await(2, TimeUnit.MINUTES), "the async 
compaction must start")
+
+      // evolve the table schema outside of the query, so that the next 
micro-batch re-creates the sink's write client
+      spark.read.json(spark.sparkContext.parallelize(
+        recordsToStrings(dataGen.generateUniqueUpdates("002", 
5)).asScala.toList, 1))
+        .withColumn("new_col", lit("evolved"))
+        
.write.format("org.apache.hudi").options(opts).mode(SaveMode.Append).save(destPath)
+      writeUpdatesToSource(sourcePath, "003", 10)
+      query.processAllAvailable()
+      // the micro-batch that replaced the write client completed without 
waiting for the running compaction
+      assertTrue(CompactionControlledSparkRecordMerger.holdingCompaction,
+        "the micro-batch must not wait for the running async compaction")
+
+      // the compaction held on the previous write client now fails, and is 
retried with the new one
+      releaseCompaction.countDown()
+      waitTillAtleastNCompletedCompactions(destPath, 1)

Review Comment:
   Test improvement: could we capture the instant held by 
CompactionControlledSparkRecordMerger and wait for that specific instant to 
complete? This helper only checks that the table has at least one completed 
compaction, so a different completed instant can satisfy it without proving the 
failed instant was retried. Asserting a second attempt for the captured instant 
on the replacement client would make the end-to-end client-swap guarantee much 
stronger.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to