hudi-agent commented on code in PR #19485:
URL: https://github.com/apache/hudi/pull/19485#discussion_r3953944591
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -1744,22 +1744,43 @@ static void deltaStreamerTestRunner(HoodieDeltaStreamer
ds, HoodieDeltaStreamer.
static void deltaStreamerTestRunner(HoodieDeltaStreamer ds,
HoodieDeltaStreamer.Config cfg, Function<Boolean, Boolean> condition, String
jobId) throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
- Future dsFuture = executor.submit(() -> {
+ try {
+ Future dsFuture = executor.submit(() -> {
+ try {
+ ds.sync();
+ } catch (Exception ex) {
+ log.warn("DS continuous job failed, hence not proceeding with
condition check for {}", jobId);
+ throw new RuntimeException(ex.getMessage(), ex);
+ }
+ });
try {
- ds.sync();
- } catch (Exception ex) {
- log.warn("DS continuous job failed, hence not proceeding with
condition check for {}", jobId);
- throw new RuntimeException(ex.getMessage(), ex);
+ TestHelpers.waitTillCondition(condition, dsFuture, 360);
+ } catch (Throwable failure) {
+ // Surefire runs this module with forkCount=1 and reuseForks=true, so
a continuous streamer left
+ // running here reads on into the next test, whose setup deletes
basePath and whose teardown closes
+ // the data generators underneath it. Stop it before letting the
failure out.
+ try {
+ ds.shutdownGracefully();
Review Comment:
🤖 `shutdownGracefully()` goes through `HoodieAsyncService.shutdown(false)`,
which does `executor.awaitTermination(24, HOURS)` — so if the reason the
condition timed out is that the streamer is stuck mid-`ingestOnce()`, this call
blocks and the new diagnostic `AssertionError` is never thrown before CI kills
the job. Would it be worth logging `failure` (at error level) before calling
shutdown, so the report this PR adds survives that case?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -763,34 +767,92 @@ static HoodieInstant
assertCommitMetadataForIncrSource(String expected, String t
return lastInstant;
}
+ /** Bound for {@link #waitFor}; generous, since it only exists to stop a
hung poll running forever. */
+ private static final long WAIT_FOR_TIMEOUT_SECS = 120;
Review Comment:
🤖 nit: would you mind moving `WAIT_FOR_TIMEOUT_SECS` to the top of the
TestHelpers class (with the other statics) rather than sandwiched between two
methods? It's only used by `waitFor` further down, so it's a little surprising
to find it above `waitTillCondition`.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -763,34 +767,92 @@ static HoodieInstant
assertCommitMetadataForIncrSource(String expected, String t
return lastInstant;
}
+ /** Bound for {@link #waitFor}; generous, since it only exists to stop a
hung poll running forever. */
+ private static final long WAIT_FOR_TIMEOUT_SECS = 120;
+
+ /**
+ * Polls {@code condition} until it holds, the deltastreamer future
finishes, or the timeout expires.
+ *
+ * <p>On timeout the last error the condition threw is attached to the
failure, so the report names the
+ * assertion that never held rather than only this method.
+ */
static void waitTillCondition(Function<Boolean, Boolean> condition, Future
dsFuture, long timeoutInSecs) throws Exception {
- Future<Boolean> res = Executors.newSingleThreadExecutor().submit(() -> {
- boolean ret = false;
- while (!ret && !dsFuture.isDone()) {
- try {
- Thread.sleep(2000);
- ret = condition.apply(true);
- log.info("Condition completed successfully");
- } catch (Throwable error) {
- log.debug("Got error waiting for condition", error);
- ret = false;
+ AtomicReference<Throwable> lastError = new AtomicReference<>();
+ AtomicInteger completedEvaluations = new AtomicInteger();
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ try {
+ Future<Boolean> res = executor.submit(() -> {
+ boolean ret = false;
+ // The executor check matters as well as the interrupt flag: the
interrupt from shutdownNow is
+ // delivered once, and a condition that swallows it would otherwise
leave the flag clear and keep
+ // this thread polling for the lifetime of the JVM.
+ while (!ret && !dsFuture.isDone() &&
!Thread.currentThread().isInterrupted() && !executor.isShutdown()) {
+ try {
+ Thread.sleep(2000);
+ ret = condition.apply(true);
+ completedEvaluations.incrementAndGet();
+ if (ret) {
+ log.info("Condition completed successfully");
+ }
+ } catch (InterruptedException interrupted) {
+ // Thread.sleep clears the interrupt flag when it throws, so
catching this with everything
+ // else would re-enter the loop. Restore the flag and stop; this
is not a condition failure,
+ // so it is deliberately not recorded as one.
+ Thread.currentThread().interrupt();
+ break;
+ } catch (Throwable error) {
+ log.debug("Got error waiting for condition", error);
+ lastError.set(error);
+ completedEvaluations.incrementAndGet();
+ ret = false;
+ }
}
+ return ret;
+ });
+ try {
+ res.get(timeoutInSecs, TimeUnit.SECONDS);
+ } catch (TimeoutException e) {
Review Comment:
🤖 nit: could the three-way message building here move into a small helper
like `describeTimeout(lastError, completedEvaluations, timeoutInSecs)`?
waitTillCondition is now ~50 lines mixing the polling loop with report
formatting, and the helper would also be trivially unit-testable in
TestWaitTillCondition.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
