jto opened a new pull request, #40610: URL: https://github.com/apache/beam/pull/40610
Addresses #40607. `ReaderInvocationUtil` pushed Beam metrics to Flink after every call to `Reader#start` and `Reader#advance`. Each push builds the metric results of all steps of the task and filters them down to one step. In a profiled batch job this took 37.6% of the CPU of the read stage, more than reading the records. ## What changes Metrics are pushed to Flink on the first call, at most once per second after that, and whenever the reader has no data. Metrics are still recorded on every call, and the metrics of the pipeline result, which come from the Flink accumulator, are not affected. Only the Flink metric views (e.g. the Flink UI) may lag by up to one second. ## Testing New `ReaderInvocationUtilTest`: Flink metrics are updated at most once per interval, when the reader has no data, and not at all when metrics are disabled. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
