voonhous commented on code in PR #19485:
URL: https://github.com/apache/hudi/pull/19485#discussion_r3955405661
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java:
##########
@@ -1744,22 +1744,43 @@ static void deltaStreamerTestRunner(HoodieDeltaStreamer
ds, HoodieDeltaStreamer.
static void deltaStreamerTestRunner(HoodieDeltaStreamer ds,
HoodieDeltaStreamer.Config cfg, Function<Boolean, Boolean> condition, String
jobId) throws Exception {
ExecutorService executor = Executors.newSingleThreadExecutor();
- Future dsFuture = executor.submit(() -> {
+ try {
+ Future dsFuture = executor.submit(() -> {
+ try {
+ ds.sync();
+ } catch (Exception ex) {
+ log.warn("DS continuous job failed, hence not proceeding with
condition check for {}", jobId);
+ throw new RuntimeException(ex.getMessage(), ex);
+ }
+ });
try {
- ds.sync();
- } catch (Exception ex) {
- log.warn("DS continuous job failed, hence not proceeding with
condition check for {}", jobId);
- throw new RuntimeException(ex.getMessage(), ex);
+ TestHelpers.waitTillCondition(condition, dsFuture, 360);
+ } catch (Throwable failure) {
+ // Surefire runs this module with forkCount=1 and reuseForks=true, so
a continuous streamer left
+ // running here reads on into the next test, whose setup deletes
basePath and whose teardown closes
+ // the data generators underneath it. Stop it before letting the
failure out.
+ try {
+ ds.shutdownGracefully();
Review Comment:
Fixed in `ae6065c`. The stop now runs on its own thread with a 60s bound and
cancels the ingest task if it does not come back, so a wedged streamer can no
longer outlive the failure it is reporting.
##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/JavaTestUtils.java:
##########
@@ -28,7 +28,10 @@ public static boolean checkNestedExceptionContains(Throwable
t, String errorMsg)
Throwable throwable = t;
boolean res = false;
while (throwable != null) {
- if (throwable.getMessage().contains(errorMsg)) {
+ // String.valueOf rather than getMessage().contains: a null message
anywhere in the chain would
+ // otherwise NPE here and lose the failure the caller was trying to
assert on. A TimeoutException
+ // raised before its condition ever threw is one such case, and NPEs in
a chain are another.
+ if (String.valueOf(throwable.getMessage()).contains(errorMsg)) {
Review Comment:
Fixed in `ae6065c`.
`timeoutDistinguishesAConditionThatNeverCompletedAnEvaluation` now also asserts
`assertFalse(checkNestedExceptionContains(error, "no such text"))`, which walks
past the null-message `TimeoutException` this path attaches and NPEs without
the `String.valueOf`.
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamerTestBase.java:
##########
@@ -763,34 +767,92 @@ static HoodieInstant
assertCommitMetadataForIncrSource(String expected, String t
return lastInstant;
}
+ /** Bound for {@link #waitFor}; generous, since it only exists to stop a
hung poll running forever. */
+ private static final long WAIT_FOR_TIMEOUT_SECS = 120;
+
+ /**
+ * Polls {@code condition} until it holds, the deltastreamer future
finishes, or the timeout expires.
+ *
+ * <p>On timeout the last error the condition threw is attached to the
failure, so the report names the
+ * assertion that never held rather than only this method.
+ */
static void waitTillCondition(Function<Boolean, Boolean> condition, Future
dsFuture, long timeoutInSecs) throws Exception {
- Future<Boolean> res = Executors.newSingleThreadExecutor().submit(() -> {
- boolean ret = false;
- while (!ret && !dsFuture.isDone()) {
- try {
- Thread.sleep(2000);
- ret = condition.apply(true);
- log.info("Condition completed successfully");
- } catch (Throwable error) {
- log.debug("Got error waiting for condition", error);
- ret = false;
+ AtomicReference<Throwable> lastError = new AtomicReference<>();
+ AtomicInteger completedEvaluations = new AtomicInteger();
+ ExecutorService executor = Executors.newSingleThreadExecutor();
+ try {
+ Future<Boolean> res = executor.submit(() -> {
+ boolean ret = false;
+ // The executor check matters as well as the interrupt flag: the
interrupt from shutdownNow is
+ // delivered once, and a condition that swallows it would otherwise
leave the flag clear and keep
+ // this thread polling for the lifetime of the JVM.
+ while (!ret && !dsFuture.isDone() &&
!Thread.currentThread().isInterrupted() && !executor.isShutdown()) {
+ try {
+ Thread.sleep(2000);
+ ret = condition.apply(true);
+ completedEvaluations.incrementAndGet();
+ if (ret) {
+ log.info("Condition completed successfully");
+ }
+ } catch (InterruptedException interrupted) {
+ // Thread.sleep clears the interrupt flag when it throws, so
catching this with everything
+ // else would re-enter the loop. Restore the flag and stop; this
is not a condition failure,
+ // so it is deliberately not recorded as one.
+ Thread.currentThread().interrupt();
+ break;
+ } catch (Throwable error) {
+ log.debug("Got error waiting for condition", error);
+ lastError.set(error);
+ completedEvaluations.incrementAndGet();
+ ret = false;
+ }
}
+ return ret;
+ });
+ try {
+ res.get(timeoutInSecs, TimeUnit.SECONDS);
+ } catch (TimeoutException e) {
+ Throwable last = lastError.get();
+ int completed = completedEvaluations.get();
+ String detail;
+ if (completed == 0) {
+ // Distinguishes a condition that is stuck part-way through its
first evaluation - a hung Spark
+ // read, say - from one that simply kept returning false.
+ detail = "No evaluation of the condition completed, so it was
still running or never started.";
+ } else if (last == null) {
+ detail = String.format("%d evaluations completed and returned
false without throwing, "
+ + "so there is no further detail.", completed);
+ } else {
+ detail = String.format("%d evaluations completed; the last failure
reported was: %s", completed, last);
+ }
+ Throwable cause = last == null ? e : last;
+ throw new AssertionError(
+ String.format("Condition was not met within %d seconds. %s",
timeoutInSecs, detail), cause);
}
- return ret;
- });
- res.get(timeoutInSecs, TimeUnit.SECONDS);
+ } finally {
+ // stop the polling thread: this method runs once per continuous-mode
test, so a leak accumulates
+ executor.shutdownNow();
+ }
}
/**
* Waits for booleanSupplier to return true
* @param booleanSupplier Boolean supplier
*/
static void waitFor(BooleanSupplier booleanSupplier) {
+ // Bounded, and the interrupt is restored rather than swallowed: this
runs inside conditions passed to
Review Comment:
Fixed in `ae6065c` by keeping the change and making it testable: a
`waitFor(BooleanSupplier, long)` overload plus `waitForGivesUpAtItsBound`,
which pins the bound in about a second. The default 120s path is unchanged.
Sorry again for the wrong premise in round 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
