[
https://issues.apache.org/jira/browse/FLINK-17468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096430#comment-17096430
]
Congxian Qiu(klion26) commented on FLINK-17468:
-----------------------------------------------
Hi, [~qqibrow], there may so many situations that lead to snapshot timeouts.
but in my experience, all of them will narrow down into two: disk performance
and network performance(include network speed and threads used to upload
state). the disk performance/network bottleneck information we can get from
somewhere other than Apache Flink. I don't think we can add more metrics here.
But for user experience(especially someone wants to find out the root cause of
the timeout), I agree to add some more debug(or trace) log in the async part of
a snapshot, so that the user can easily know what the step of current snapshot
going now.(There are so many steps in the async part of RocksDBStateBackend...)
I proposed to add more debug(or trace log) for such purpose in FLINK-13808, and
created an issue which wants to limit to the max concurrent snapshot in TM
side(FLINK-15236), if we reach agreement on this, I can help to contribute them.
> Provide more detailed metrics why asynchronous part of checkpoint is taking
> long time
> -------------------------------------------------------------------------------------
>
> Key: FLINK-17468
> URL: https://issues.apache.org/jira/browse/FLINK-17468
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Checkpointing, Runtime / Metrics, Runtime /
> State Backends
> Affects Versions: 1.10.0
> Reporter: Piotr Nowojski
> Priority: Major
>
> As [reported by
> users|https://lists.apache.org/thread.html/r0833452796ca7d1c9d5e35c110089c95cfdadee9d81884a13613a4ce%40%3Cuser.flink.apache.org%3E]
> it's not obvious why asynchronous part of checkpoint is taking so long time.
> Maybe we could provide some more detailed metrics/UI/logs about uploading
> files, materializing meta data, or other things that are happening during the
> asynchronous checkpoint process?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
