nsivabalan commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4608320907
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean>
onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ /**
+ * Runs a table service instant with one write client.
+ */
+ @FunctionalInterface
+ protected interface WriteClientRun {
+ void run(BaseHoodieWriteClient client, String instantTime) throws
IOException;
+ }
+
+ /**
+ * Runs the table service for the instant with the write client the service
has when the run starts. That same
+ * client runs the instant and is passed to {@link #isRetryableFailure}, so
a failure is never attributed to a
+ * different client than the one it happened on. On a retryable failure,
waits until the writer has switched the
+ * service to another client, lets the service prepare the retry, and
re-enqueues the instant if it is still pending.
+ *
+ * @param action name of the table service, for logging
+ * @param currentClient the write client the service currently runs with
+ * @param run runs the instant with the given client
+ * @return whether the run completed; false if it failed and was handed back
for a retry or dropped
+ */
+ protected boolean runWithClientHandoff(String action, String instantTime,
Supplier<BaseHoodieWriteClient> currentClient,
+ WriteClientRun run) throws
IOException, InterruptedException {
+ BaseHoodieWriteClient runClient = currentClient.get();
+ try {
+ run.run(runClient, instantTime);
+ return true;
+ } catch (Exception e) {
+ if (!isRetryableFailure(runClient, e)) {
+ throw e;
+ }
+ log.warn("{} for instant {} failed with a write client that was
replaced, retrying it with the new write client",
+ action, instantTime, e);
+ awaitWriteClientReplaced(runClient, currentClient);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ BaseHoodieWriteClient retryClient = currentClient.get();
+ // checked before preparing the retry too: preparing it may wait, which
an instant completed meanwhile doesn't need
+ if (isPendingInstant(retryClient, instantTime)) {
+ prepareRetry(retryClient, instantTime);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ if (isPendingInstant(retryClient, instantTime)) {
+ enqueuePendingAsyncServiceInstant(instantTime);
Review Comment:
Good catch, fixed in 056e55ff. The service now remembers which queued
instants are retries, and checks them again when they come up:
- **When the retry is dequeued**, it's skipped if the instant is no longer
pending, or, with multi-writer compaction, if another writer holds a fresh
heartbeat on it.
- **If the retry then fails** on a client that wasn't retired, the same
check runs again. If the instant was completed or taken over between the
dequeue check and `compact()`'s locked validation, the instant is dropped with
a warning and the service keeps running. Otherwise the failure stops the
service, as before. If the check itself fails, its exception is added as
suppressed to the original failure.
- **Why a fresh heartbeat after a failure means another writer:** a failed
run of our own deletes its heartbeat in `compact()`'s catch block. The
heartbeat left behind by a closed client has already been waited out in
`prepareRetry` before re-enqueueing.
Taken-over instants are dropped, not waited out. If the other writer dies,
the instant stays pending and the next restart re-enqueues it, the same as an
instant another writer takes before our first run.
I kept this to retried instants. An instant's first run fails the same way
it does on master, so this PR doesn't change how first runs are handled.
Regression test:
`TestAsyncTableServiceRetry#testRetrySkippedWhenConcurrentWriterCompletesOrTakesOverTheInstant`.
B is held, and A is re-enqueued behind it. It covers three cases:
- A completes while queued;
- another writer starts A (fresh heartbeat) while it is queued;
- the takeover lands after A is dequeued, and `compact()` rejects it with
the heartbeat `HoodieLockException`.
Each case asserts that A isn't run again and the service stays up.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala:
##########
@@ -805,6 +810,111 @@ class TestStructuredStreaming extends
HoodieSparkClientTestBase {
}
}
+ private def compactionControlledOpts: Map[String, String] =
+ getCompactionOpts(HoodieTableType.MERGE_ON_READ, isAsyncCompaction = true)
++ Map(
+ DataSourceWriteOptions.OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ DataSourceWriteOptions.RECORD_MERGE_MODE.key -> "CUSTOM",
+ DataSourceWriteOptions.RECORD_MERGE_STRATEGY_ID.key ->
CompactionControlledSparkRecordMerger.STRATEGY_ID,
+ DataSourceWriteOptions.RECORD_MERGE_IMPL_CLASSES.key ->
classOf[CompactionControlledSparkRecordMerger].getName,
+ HoodieStorageConfig.LOGFILE_DATA_BLOCK_FORMAT.key -> "parquet")
+
+ private def writeUpdatesToSource(sourcePath: String, instantTime: String,
numRecords: Int): Unit = {
+ spark.read.json(spark.sparkContext.parallelize(
+ recordsToStrings(dataGen.generateUniqueUpdates(instantTime,
numRecords)).asScala.toList, 1))
+ .write.mode(SaveMode.Append).json(sourcePath)
+ }
+
+ /**
+ * An async compaction running on the sink's write client when a schema
change makes the sink re-create its client
+ * may fail once that client is closed. Its service retries it with the new
client, and the query keeps running.
+ */
+ @Test
+ def testStructuredStreamingRetriesAsyncCompactionFailedByWriteClientSwap():
Unit = {
+ val (sourcePath, destPath) = initStreamingSourceAndDestPath("source",
"dest")
+ val opts = compactionControlledOpts +
(DataSourceWriteOptions.SET_NULL_FOR_MISSING_COLUMNS.key -> "true")
+ val inputDF =
spark.read.json(spark.sparkContext.parallelize(recordsToStrings(dataGen.generateInsertsForPartition(
+ "000", 20,
HoodieTestDataGenerator.DEFAULT_FIRST_PARTITION_PATH)).asScala.toList, 1))
+ inputDF.write.mode(SaveMode.Append).json(sourcePath)
+ val query = initWritingStreamingQuery(inputDF.schema, sourcePath,
destPath, opts)
+ val compactionStarted = new CountDownLatch(1)
+ val releaseCompaction = new CountDownLatch(1)
+ try {
+ query.processAllAvailable()
+
CompactionControlledSparkRecordMerger.holdNextCompactionThenFail(compactionStarted,
releaseCompaction)
+ writeUpdatesToSource(sourcePath, "001", 10)
+ query.processAllAvailable()
+ assertTrue(compactionStarted.await(2, TimeUnit.MINUTES), "the async
compaction must start")
+
+ // evolve the table schema outside of the query, so that the next
micro-batch re-creates the sink's write client
+ spark.read.json(spark.sparkContext.parallelize(
+ recordsToStrings(dataGen.generateUniqueUpdates("002",
5)).asScala.toList, 1))
+ .withColumn("new_col", lit("evolved"))
+
.write.format("org.apache.hudi").options(opts).mode(SaveMode.Append).save(destPath)
+ writeUpdatesToSource(sourcePath, "003", 10)
+ query.processAllAvailable()
+ // the micro-batch that replaced the write client completed without
waiting for the running compaction
+ assertTrue(CompactionControlledSparkRecordMerger.holdingCompaction,
+ "the micro-batch must not wait for the running async compaction")
+
+ // the compaction held on the previous write client now fails, and is
retried with the new one
+ releaseCompaction.countDown()
+ waitTillAtleastNCompletedCompactions(destPath, 1)
Review Comment:
Done in 056e55ff. While the run is held, the test reads the one inflight
compaction from the timeline: the service runs one compaction at a time, so the
held run is the only inflight one. It then waits for that specific instant to
complete. The held run always fails, so its completion proves the same instant
was run again, on the replacement client.
`waitTillAtleastNCompletedCompactions` is replaced by
`waitTillCompactionCompleted(tablePath, instant)`.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/HoodieAsyncTableService.java:
##########
@@ -46,4 +53,103 @@ public void start(Function<Boolean, Boolean>
onShutdownCallback) {
}
super.start(onShutdownCallback);
}
+
+ /**
+ * Runs a table service instant with one write client.
+ */
+ @FunctionalInterface
+ protected interface WriteClientRun {
+ void run(BaseHoodieWriteClient client, String instantTime) throws
IOException;
+ }
+
+ /**
+ * Runs the table service for the instant with the write client the service
has when the run starts. That same
+ * client runs the instant and is passed to {@link #isRetryableFailure}, so
a failure is never attributed to a
+ * different client than the one it happened on. On a retryable failure,
waits until the writer has switched the
+ * service to another client, lets the service prepare the retry, and
re-enqueues the instant if it is still pending.
+ *
+ * @param action name of the table service, for logging
+ * @param currentClient the write client the service currently runs with
+ * @param run runs the instant with the given client
+ * @return whether the run completed; false if it failed and was handed back
for a retry or dropped
+ */
+ protected boolean runWithClientHandoff(String action, String instantTime,
Supplier<BaseHoodieWriteClient> currentClient,
+ WriteClientRun run) throws
IOException, InterruptedException {
+ BaseHoodieWriteClient runClient = currentClient.get();
+ try {
+ run.run(runClient, instantTime);
+ return true;
+ } catch (Exception e) {
+ if (!isRetryableFailure(runClient, e)) {
+ throw e;
+ }
+ log.warn("{} for instant {} failed with a write client that was
replaced, retrying it with the new write client",
+ action, instantTime, e);
+ awaitWriteClientReplaced(runClient, currentClient);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ BaseHoodieWriteClient retryClient = currentClient.get();
+ // checked before preparing the retry too: preparing it may wait, which
an instant completed meanwhile doesn't need
+ if (isPendingInstant(retryClient, instantTime)) {
+ prepareRetry(retryClient, instantTime);
+ if (isShutdownRequested()) {
+ return false;
+ }
+ if (isPendingInstant(retryClient, instantTime)) {
+ enqueuePendingAsyncServiceInstant(instantTime);
+ return false;
+ }
+ }
+ log.info("{} for instant {} is no longer pending, not retrying it",
action, instantTime);
+ return false;
+ }
+ }
+
+ /**
+ * Whether a run that failed with the given exception is retried. The run
used {@code runClient}, the client of the
+ * service when the run started. By default no failure is retried and the
service stops with it.
+ */
+ protected boolean isRetryableFailure(BaseHoodieWriteClient runClient,
Exception failure) {
+ return false;
+ }
+
+ /**
+ * Whether the instant is still pending, i.e. a retry of it has work to do.
Only consulted for retryable failures.
+ */
+ protected boolean isPendingInstant(BaseHoodieWriteClient client, String
instantTime) {
+ return true;
+ }
+
+ /**
+ * Prepares the retry of a failed run of the instant with {@code
retryClient}, before it is re-enqueued.
+ */
+ protected void prepareRetry(BaseHoodieWriteClient retryClient, String
instantTime) throws IOException, InterruptedException {
+ }
+
+ /**
+ * Waits up to the given time, returning early when shutdown is requested
(see {@link #onShutdownRequested}) or the
+ * writer switches this service to another client. Callers re-check what
they wait for after it returns.
+ */
+ protected synchronized void waitUnlessShutdown(long millis) throws
InterruptedException {
+ if (!isShutdownRequested()) {
+ wait(millis);
+ }
+ }
+
+ @Override
+ protected synchronized void onShutdownRequested() {
+ notifyAll();
+ }
+
+ /**
+ * Waits until the writer has switched this service away from the given
client. Services notify on this monitor
+ * when they switch clients.
+ */
+ private synchronized void awaitWriteClientReplaced(BaseHoodieWriteClient
client, Supplier<BaseHoodieWriteClient> currentClient)
+ throws InterruptedException {
+ while (!isShutdownRequested() && currentClient.get() == client) {
+ wait(WRITE_CLIENT_POLL_MS);
Review Comment:
Agreed, removed in 056e55ff. All client switches go through
`AsyncCompactService` / `AsyncClusteringService#updateWriteClient`, which
update the client under this monitor and call `notifyAll()`. That includes the
Spark streaming sink and HoodieStreamer. `onShutdownRequested()` notifies too,
and the wait loop re-checks its condition under the monitor, so a switch made
just before the wait starts isn't missed. `WRITE_CLIENT_POLL_MS` is gone.
Local: `TestAsyncTableService*` 23/23 (Retry 17/17, incl. the 3 new cases),
`TestStructuredStreaming` 29/29, checkstyle and scalastyle clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]