[
https://issues.apache.org/jira/browse/FLINK-41064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137207#comment-18137207
]
Purushottam Sinha edited comment on FLINK-41064 at 11/3/26, 12:40 PM:
----------------------------------------------------------------------
I had a look at this. It's a race during cancellation: the async checkpoint
thread can still log after the TM has closed the job's classloader. log4j
2.24.3 trips over the closed classloader while printing the stack trace and
writes an error to stderr, which is what ends up in the .out file. 2.25.3
handles it fine, so master and 2.3 aren't affected (they already got the
upgrade in FLINK-39146).
I've opened a backport of that upgrade for release-2.2:
https://github.com/apache/flink/pull/29767. I could reproduce the failure
locally with 2.24.3, and it goes away with 2.25.3.
was (Author: JIRAUSER310845):
Cause: when the job is cancelled mid-checkpoint, the {{AsyncOperations}} thread
can log at {{AsyncCheckpointRunnable.java:150}} after the TaskManager has
already closed the job classloader. log4j 2.24.3 loads stack-frame classes
through the thread's context classloader, which is that closed classloader, and
doesn't catch the resulting {{IllegalStateException}}
({{ThrowableProxyHelper.java:213}}). The error goes to stderr, so the {{.out}}
check fails. log4j 2.25.3 catches it; release-2.3 got 2.25.3 through
FLINK-39146, and master is on 2.26.1.
Backport to release-2.2: https://github.com/apache/flink/pull/29767. A local
reproduction that forces this ordering with release-2.2's classloader and
TaskManager log4j config fails the {{.out}} check 10/10 on 2.24.3, with the
same trace as the CI run, and passes 10/10 on 2.25.3.
> 'State TTL Heap backend end-to-end test' fails on a "Trying to access closed
> classloader" log4j error in the TaskManager .out files
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41064
> URL: https://issues.apache.org/jira/browse/FLINK-41064
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 2.2.2
> Environment:
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=80344&view=logs&j=fb37c667-81b7-5c22-dd91-846535e99a97
> (release-2.2, nightly 2026-10-21, e2e_2_cron_adaptive_scheduler)
> {code}
> The TTL verification logic failed. See *.out file for more information.
> [FAIL] 'State TTL Heap backend end-to-end test' failed after 1 minutes and 5
> seconds! Test exited with exit code 1
> {code}
> The test fails when any .out file is non-empty. All three TaskManager .out
> files contain only this log4j error, written at 04:42:32.46, about 0.4 s
> after the job was cancelled:
> {code}
> 2026-10-21T04:42:32.460222966Z AsyncOperations-thread-1 ERROR An exception
> occurred processing Appender MainAppender java.lang.IllegalStateException:
> Trying to access closed classloader. Please check if you store classloaders
> directly or indirectly in static fields. ...
> at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:189)
> at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:197)
> ...
> at
> org.apache.logging.log4j.core.impl.ThrowableProxyHelper.toExtendedStackTrace(ThrowableProxyHelper.java:112)
> ...
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:150)
> {code}
> The PrintFailedVerifications sink received 0 records on all three subtasks,
> so no TTL verification failed.
> Reporter: Martijn Visser
> Priority: Major
> Labels: test-stability
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
