[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803478#comment-17803478 ]
Piotr Nowojski edited comment on FLINK-33856 at 1/5/24 9:58 AM: ---------------------------------------------------------------- {quote} Maybe a new flip that supports task-level trace reporter can builded ? I’m willing to participate in the development. {quote} Please again check the FLIP-384 discussions. I was highlighting there a couple of difficulties: {quote} However, if we would like to create true distributed traces, with spans reported from many different components, potentially both on JM and TM, the problem is a bit deeper. The issue in that case is how to actually fill out `parrent_id` and `trace_id`? Passing some context entity as a java object would be unfeasible. That would require too many changes in too many places. I think the only realistic way to do it, would be to have a deterministic generator of `parten_id` and `trace_id` values. For example we could create the parent trace/span of the checkpoint on JM, and set those ids to something like: `jobId#attemptId#checkpointId`. Each subtask then could re-generate those ids and subtasks' checkpoint span would have an id of `jobId#attemptId#checkpointId#subTaskId`. Note that this is just an example, as most likely distributed spans for checkpointing do not make sense, as we can generate them much easier on the JM anyway. {quote} https://lists.apache.org/thread/7lql5f5q1np68fw1wc9trq3d9l2ox8f4 At the same time: {quote} I am worried that a large amount of data aggregation to JM may have performance problems. {quote} I wouldn't worry about that too much. This data is already aggregated on the JM from all of the TMs via {{CheckpointMetricsBuilder}} and {{CheckpointMetrics}}. Besides, it's just a single RPC from subtask -> JM per checkpoint. If that becomes a problem, we would have problems in many different areas as well (for example {{notifyCheckpointCompleted}} is a very similar call but the other direction). Also AFAIR there are/were different ideas how to solve this potential bottleneck in a more generic way (having multiple job coordinators in the cluster to spread the load). [~hejufang001] I would suggest that both of yo chat offline about the scope of the changes in [~fanrui]'s FLIP and/or eventual division of work. I'm not sure if [~fanrui] plans to add per task/subtask spans for checkpoints and/or recovery. was (Author: pnowojski): {quote} Maybe a new flip that supports task-level trace reporter can builded ? I’m willing to participate in the development. {quote} Please again check the FLIP-384 discussions. I was highlighting there a couple of difficulties: {quote} However, if we would like to create true distributed traces, with spans reported from many different components, potentially both on JM and TM, the problem is a bit deeper. The issue in that case is how to actually fill out `parrent_id` and `trace_id`? Passing some context entity as a java object would be unfeasible. That would require too many changes in too many places. I think the only realistic way to do it, would be to have a deterministic generator of `parten_id` and `trace_id` values. For example we could create the parent trace/span of the checkpoint on JM, and set those ids to something like: `jobId#attemptId#checkpointId`. Each subtask then could re-generate those ids and subtasks' checkpoint span would have an id of `jobId#attemptId#checkpointId#subTaskId`. Note that this is just an example, as most likely distributed spans for checkpointing do not make sense, as we can generate them much easier on the JM anyway. {quote} https://lists.apache.org/thread/7lql5f5q1np68fw1wc9trq3d9l2ox8f4 At the same time: {quote} I am worried that a large amount of data aggregation to JM may have performance problems. {quote} I wouldn't worry about that too much. This data is already aggregated on the JM from all of the TMs via {{CheckpointMetricsBuilder}} and {{CheckpointMetrics}}. Besides, it's just a single RPC from subtask -> JM per checkpoint. If that becomes a problem, we would have problems in many different areas as well (for example {{notifyCheckpointCompleted}} is a very similar call but the other direction). Also AFAIR there are/were different ideas how to solve this potential bottleneck in a more generic way (having multiple job coordinators in the cluster to spread the load). > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > ------------------------------------------------------------------------------------------------------------------------------- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing > Affects Versions: 1.18.0 > Reporter: Jufang He > Assignee: Jufang He > Priority: Major > Labels: pull-request-available > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)