[
https://issues.apache.org/jira/browse/FLINK-39998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reopened FLINK-39998:
------------------------------------
Reopening, the test added here failed on master and only one of the three
failures made it into the exception:
https://github.com/apache/flink/actions/runs/35414462164/job/105822294622
(master nightly 2026-09-19, Java 21, module core)
{code}
[ERROR]
org.apache.flink.state.rocksdb.RocksDBStateDownloaderTest.testMultipleDistinctFailuresMergedInMessage
-- Time elapsed: 0.022 s <<< FAILURE!
java.lang.AssertionError: Expected merged error message, got:
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.io.IOException: java.io.IOException: error-B: file not found
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at
org.apache.flink.state.rocksdb.RocksDBStateDownloaderTest.testMultipleDistinctFailuresMergedInMessage(RocksDBStateDownloaderTest.java:349)
{code}
The message is the ExecutionException from {{downloadFuture.get()}}, not the
result of the {{whenComplete}} callback. {{RocksDBStateDownloader}} only passes
that exception to {{buildDownloadException}} when {{rawException}} is still
null in the catch block, so in this run {{get()}} returned before the callback
had stored the raw failure.
> RocksDBStateDownloader loses parallel download failures, surfacing a
> misleading ClosedChannelException instead of the real root cause
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-39998
> URL: https://issues.apache.org/jira/browse/FLINK-39998
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 2.0.0, 1.20.5, 2.1.3
> Reporter: Raj
> Assignee: Raj
> Priority: Major
> Labels: pull-request-available, starter
> Fix For: 2.4.0
>
> Attachments: Screenshot 2026-06-26 at 3.23.42 PM.png
>
>
> When RocksDBStateDownloader.transferAllStateDataToDirectory fails during
> incremental state restore, the exception surfaced to the operator is always
> ClosedChannelException — regardless of the actual root cause. This makes
> diagnosing restore failures extremely difficult.
> *Example scenario:* A checkpoint file is accidentally deleted from S3. On job
> restart, Flink keeps crashing with:
> Caused by: java.io.IOException: java.nio.channels.ClosedChannelException
> at
> org.apache.flink.state.rocksdb.RocksDBStateDownloader.downloadDataForStateHandle
> There is no indication of which file is missing, which checkpoint is
> affected, or that S3 is involved at all.
>
> *Root cause:*
> FutureUtils.completeAll() collects all parallel thread failures as
> suppressed exceptions on the first exception to arrive (CompletionException).
> However, CompletableFuture.get()
> internally calls JDK's reportGet() which strips the CompletionException
> wrapper before throwing ExecutionException:
> // JDK CompletableFuture.reportGet()
> if (x instanceof CompletionException && x.getCause() != null)
> x = cause; // strips CompletionException — suppressed list GONE
> throw new ExecutionException(x);
> By the time the catch block in transferAllStateDataToDirectory runs, all
> thread failures except one are permanently lost. Which failure "wins" is
> non-deterministic — it is whichever
> thread completes first, which is typically the cascade ClosedChannelException
> (a local operation, fast) rather than the real cause (e.g.
> FileNotFoundException from a remote storage call).
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
