[
https://issues.apache.org/jira/browse/FLINK-41054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130226#comment-18130226
]
Diljeet Singh commented on FLINK-41054:
---------------------------------------
Created the following PR: https://github.com/apache/flink/pull/29564
> Metrics of finished source splits are never removed from the REST API's
> metric store
> ------------------------------------------------------------------------------------
>
> Key: FLINK-41054
> URL: https://issues.apache.org/jira/browse/FLINK-41054
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics, Runtime / REST
> Affects Versions: 2.3.0, 2.2.1, 2.1.3, 2.3.1
> Reporter: Diljeet Singh
> Priority: Major
> Labels: pull-request-available
>
> InternalSourceSplitMetricGroup registers per-split metrics under
> split.<splitId>.watermark.*, for example currentWatermark,
> pausedTimePerSecond, idleTimePerSecond and activeTimePerSecond. When a split
> finishes, SourceOperator#splitFinished closes the group. That unregisters the
> metrics on the TaskManager, so they no longer appear in its
> MetricQueryService dump.
> The JobManager's MetricStore only merges each fetched dump into its existing
> maps. It removes whole jobs, TaskManagers, subtasks and old attempts, but
> never an individual metric that stops being reported. The finished splits'
> metrics therefore stay in the store with their last values. As a result,
> endpoints such as /jobs/:jobid/vertices/:vertexid/subtasks/metrics (and the
> subtask and attempt metric endpoints) list a growing number of metrics for
> each split ever processed. For sources with many short-lived splits, such as
> file or Kafka sources with partition discovery, this grows without bound. It
> increases JobManager memory use, makes REST responses slower, and clutters
> the metric picker in the web UI with stale entries.
> Steps to reproduce:
> 1. Run a job with a FLIP-27 source that processes many splits over time, such
> as a FileSource in continuous monitoring mode.
> 2. Call GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics repeatedly.
> 3. The returned list keeps growing with split.<id>.watermark.* entries for
> splits that have already finished.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
