[
https://issues.apache.org/jira/browse/FLINK-32957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759463#comment-17759463
]
Hangxiang Yu commented on FLINK-32957:
--
IMO, This metric is not so necessary.
Users cannot get more infos from this metric, they may only know that "the
timer trigger is delayed" which they could get from the backpressure metric
(IIUC, delayed timer trigger usually means backpressure), but they still need
to figure out why (maybe it's not related to timer)
It's a bit different with FLINK-32954 which could help us to know whether heap
timer is the bottleneck (large heap timer size is the most common case from my
experience)
BTW, if we want to know the rocksdb timer size, some metrics in RocksDBProperty
which reports in column family level (e.g. estimate-num-keys,
estimate-live-data-size) could help to estimate.
> Add current timer trigger lag to metrics
>
>
> Key: FLINK-32957
> URL: https://issues.apache.org/jira/browse/FLINK-32957
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / State Backends
>Reporter: Rui Xia
>Priority: Minor
>
> Timer trigger lag denotes the gap between the actual trigger timestamp and
> the expected trigger timestamp (registered timestamp to `TimeService`). This
> metric can aid users to find out whether there is a backlog of timers.
> The backlog of timers may affect downstream data processing. Users customize
> the trigger logic, which may interact with downstream data processing. For
> example, a trigger logic can inject some records to downstream operators. The
> backlog of timers blocks the record injection.
> On the other side, The backlog of timers makes jobs unstable. Timers are used
> by window operators, which leverage a timer to remove the window state of a
> triggered window. The backlog of timers blocks data removal, and the state
> size may grow unexpectedly large. The large state size affects the
> performance of state-backend. In cloud-native environment, a k8s pod is prone
> to reach local disk limit due to large state files (RocksDB SST).
> Currently, users are hard to observe the backlog of timers. As far as I
> known, heap dump is the only way to learn the backlog of timers. Thus, users
> cannot notice the backlog of timers in time. FLINK-32954
> (https://issues.apache.org/jira/browse/FLINK-32954) exposes number of heap
> timers, but is not suitable for RocksDB timer due to performance loss.
> Compare with FLINK-32954, timer trigger lag is much more lightweight for
> RocksDB timer.
> * Reason 1: Timer trigger lag does not affect timer registering.
> * Reason 2: The effect on timer triggering is limited. Timer registering is
> a hot code-path, while timer triggering is much colder. In general, the
> trigger interval is tens of second, and the timer trigger code-path is
> invoked every tens of second. Thus, the addition of timer trigger lag
> calculation has little performance overhead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
