nsivabalan commented on code in PR #20445:
URL: https://github.com/apache/hudi/pull/20445#discussion_r4522052515
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,7 +80,23 @@ protected Pair<CompletableFuture, ExecutorService>
startService() {
if (null != instantTime) {
log.info("Starting Compaction for instant {}", instantTime);
- compactor.compact(instantTime);
+ BaseHoodieWriteClient runClient = compactor.compactionClient;
+ try {
+ compactor.compact(instantTime);
Review Comment:
Fixed in 9b228b04. `BaseCompactor` / `BaseClusterer` now take the write
client to run with (`compact(client, instant)` / `cluster(client, instant)`;
the old one-argument methods delegate with the current client). The service
reads its client **once** per run and passes that same reference both to the
run and to `isRetryableFailure`, so a switch between two reads can no longer
happen. No lock is held across the operation.
`testFailureClassifiedAgainstTheClientOfTheRun` switches clients while the run
is in progress and asserts that the client that ran equals the client the
failure was classified against.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncCompactService.java:
##########
@@ -78,7 +80,23 @@ protected Pair<CompletableFuture, ExecutorService>
startService() {
if (null != instantTime) {
log.info("Starting Compaction for instant {}", instantTime);
- compactor.compact(instantTime);
+ BaseHoodieWriteClient runClient = compactor.compactionClient;
+ try {
+ compactor.compact(instantTime);
+ } catch (Exception e) {
+ if (!isRetryableFailure(runClient, e)) {
+ throw e;
+ }
+ log.warn("Compaction for instant {} failed with a write client
that was replaced, retrying it with the new write client",
+ instantTime, e);
+ awaitWriteClientReplaced(runClient);
+ if (isPendingCompaction(instantTime)) {
+ enqueuePendingAsyncServiceInstant(instantTime);
Review Comment:
Good catch, thanks for reproducing it. Fixed in 9b228b04: with a
multi-writer concurrency mode, before re-enqueueing the compaction the service
waits until the instant's heartbeat has expired, using the same expiry as
`HoodieHeartbeatClient#isHeartbeatExpired` (interval × tolerable misses), which
`compact()` checks via `validateHeartBeat`. It re-reads the heartbeat at least
once per interval.
I went with waiting rather than deleting the file, because the retry path
can't prove whose heartbeat it is: a heartbeat orphaned by our closed client
expires within the timeout, and one a concurrent writer is genuinely refreshing
is waited out, after which the instant is usually no longer pending and isn't
retried. The cost is up to one heartbeat timeout of delay, only on this rare
swap-during-compaction path. Clustering doesn't validate a heartbeat at
execution, so it needs no wait.
Test: `testCompactionRetryWaitsForHeartbeatToExpire` (OCC config, a fresh
heartbeat left by the failed run → the retry runs only after expiry). An
end-to-end OCC variant of the streaming client-swap test isn't included yet;
the unit test pins the wait. Happy to add the integration variant if you think
it's worth the runtime.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/async/AsyncClusteringService.java:
##########
@@ -76,7 +78,23 @@ protected Pair<CompletableFuture, ExecutorService>
startService() {
final String instant = fetchNextAsyncServiceInstant();
if (null != instant) {
log.info("Starting clustering for instant {}", instant);
- clusteringClient.cluster(instant);
+ BaseHoodieWriteClient runClient =
clusteringClient.clusteringClient;
+ try {
+ clusteringClient.cluster(instant);
+ } catch (Exception e) {
+ if (!isRetryableFailure(runClient, e)) {
+ throw e;
+ }
+ log.warn("Clustering for instant {} failed with a write client
that was replaced, retrying it with the new write client",
+ instant, e);
+ awaitWriteClientReplaced(runClient);
+ if (isPendingClustering(instant)) {
+ enqueuePendingAsyncServiceInstant(instant);
+ } else {
+ log.info("Clustering for instant {} is no longer pending, not
retrying it", instant);
+ }
+ continue;
Review Comment:
Done in 9b228b04. The shared part (failure classification against the run's
client, waiting for the replacement client, pending check, re-enqueue) is now
`HoodieAsyncTableService#runWithClientHandoff`, with `isRetryableFailure`,
`isPendingInstant` and `prepareRetry` as hooks. Compaction overrides
`prepareRetry` for the heartbeat wait. Execution stays service-specific.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieStreamingSink.scala:
##########
@@ -315,18 +333,20 @@ class HoodieStreamingSink(sqlContext: SQLContext,
}
/**
- * Same as HoodieStreamer on a schema change (StreamSync#reInitWriteClient
and
- * HoodieStreamer#onInitializingWriteClient): the async table services
switch to the new client for their next
- * runs, and the previous client is closed right away.
+ * Closes the previous client right away, then switches the async table
services to the new client for their next
+ * runs; the switch does not wait for a running compaction or clustering. A
run still using the previous client can
+ * fail once it is closed: the previous client is marked as retired first,
so that its service retries such a
+ * failure with the new client instead of stopping.
*/
private def retireWriteClient(previousClient: SparkRDDWriteClient[_],
newClient: SparkRDDWriteClient[_]): Unit = {
Review Comment:
Addressed in 9b228b04: the sink now switches its services to the new client
first, then closes the previous client in a `finally`. That keeps the guarantee
that the previous client is always closed even if a switch throws, which is why
the close had moved up in the first place, while a slow `close()` no longer
delays the handoff.
##########
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/async/SparkStreamingAsyncClusteringService.java:
##########
@@ -31,8 +33,27 @@ public class SparkStreamingAsyncClusteringService extends
AsyncClusteringService
private static final long serialVersionUID = 1L;
+ // whether a write client was retired by the sink, i.e. closed by the sink
when it replaced its client
+ private final transient Predicate<BaseHoodieWriteClient> isRetiredClient;
+
public SparkStreamingAsyncClusteringService(HoodieEngineContext context,
BaseHoodieWriteClient writeClient) {
+ this(context, writeClient, c -> false);
+ }
+
+ public SparkStreamingAsyncClusteringService(HoodieEngineContext context,
BaseHoodieWriteClient writeClient,
+ Predicate<BaseHoodieWriteClient> isRetiredClient) {
super(context, writeClient, true);
+ this.isRetiredClient = isRetiredClient;
+ }
+
+ /**
+ * A clustering run that failed while using a write client the sink has
since retired is retryable: the sink closes
+ * the previous client when it re-creates its client (e.g. on a schema
change), which can fail a run still using it.
+ * Any other failure is not retryable and stops the service, which fails the
next micro-batch with that failure.
+ */
+ @Override
+ protected boolean isRetryableFailure(BaseHoodieWriteClient runClient,
Exception failure) {
+ return isRetiredClient.test(runClient);
Review Comment:
This is intentional, and it's bounded. A retired client has been closed, so
any failure of a run on it is attributable to the close; the exception types
that surface (closed timeline server, closed heartbeat client, closed storage,
a cancelled job) vary too much to match reliably. If the failure was a real
error that happened to coincide with the switch, the retry runs on the new,
non-retired client, fails again, and that failure is fatal with its cause. So
an unrelated error costs at most one extra attempt. With 9b228b04 the
classification uses the exact client the run executed on, and
`testRetryFailingOnNewClientStopsTheService` covers the "fails again on the new
client" path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]