[ https://issues.apache.org/jira/browse/FLINK-20886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761591#comment-17761591 ]
Zakelly Lan commented on FLINK-20886: ------------------------------------- Hi [~yunta], I finished my PR for this issue. In current implementation, when checkpoint expires, the JM will notify TM(tasks) to do a thread dump via the mailbox (the same way JM notify tasks about a checkpoint abortion). And the task will log a thread dump {*}only when the checkpoint is currently running{*}. So if the task cannot receive the checkpoint barrier, it have not started the checkpoint yet, thus no thread dump will be logged. I think logging a thread dump when a task has not started the checkpoint is also useful, and I can create another ticket for it. WDYT? Besides, would you like review my PR? Thanks! > Add the option to get a threaddump on checkpoint timeouts > --------------------------------------------------------- > > Key: FLINK-20886 > URL: https://issues.apache.org/jira/browse/FLINK-20886 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing > Reporter: Nico Kruber > Assignee: Zakelly Lan > Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > usability > > For debugging checkpoint timeouts, I was thinking about the following > addition to Flink: > When a checkpoint times out and the async thread is still running, create a > thread dump [1] and either add this to the checkpoint stats, log it, or write > it out. > This may help identifying where the checkpoint is stuck (maybe a lock, could > also be in a third party lib like the FS connectors,...). It would give us > some insights into what the thread is currently doing. > Limiting the scope of the threads would be nice but may not be possible in > the general case since additional threads (spawned by the FS connector lib, > or otherwise connected) may interact with the async thread(s) by e.g. going > through the same locks. Maybe we can reduce the thread dumps to all async > threads of the failed checkpoint + all thready that interact with it, e.g. > via locks? > I'm also not sure whether the ability to have thread dumps or not should be > user-configurable (Could it contain sensitive information from other jobs if > you run a session cluster? Is that even relevant since we don't give > isolation guarantees anyway?). If it is configurable, it should be on by > default. > [1] https://crunchify.com/how-to-generate-java-thread-dump-programmatically/ -- This message was sent by Atlassian Jira (v8.20.10#820010)