Hi,
While adding metrics for the Python state cache [1], I was wondering
about the story of Gauges in Beam. It seems like we only keep a value at
a time and use a combiner [2] that replaces an old, possibly not
reported gauge result, with a newer gauge result based on their timestamps.
This behavior is an issue because if the SDK reports faster than the
Runner queries, metrics will just be swallowed. Gauges seem important to
get right because often users want to see all the values, e.g. in case
of spikes in the data.
What do you think about keeping all gauge values until they are reported?
Thanks,
Max
[1] https://github.com/apache/beam/pull/9769
[2]
https://github.com/apache/beam/blob/fa74467b82e78962e9f170ad0d95fa6b345add67/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerStepMap.java#L134
