skywalker0618 opened a new pull request, #20564: URL: https://github.com/apache/hudi/pull/20564
### Describe the issue this Pull Request addresses Part 1 of #20531. The stream read metrics (`FlinkStreamReadMetrics`) don't reflect the reader's state: - On the FLIP-27 source (`read.source-v2.enabled=true`), `splitLatestCommit` and `splitLatestCommitDelay` always report 0. `HoodieSourceSplitReader` registers them but never sets them; only the legacy `StreamReadOperator` does. - `issuedInstantDelay` and `splitLatestCommitDelay` are computed once, in the setter, and the gauge returns the stored value. If the reader stalls on a split, or discovery stops finding commits, the delay freezes instead of growing. There is also a registration problem on Source V2. Flink closes an idle split fetcher and creates a new `HoodieSourceSplitReader` when new splits arrive, and each split reader created and registered its own `FlinkStreamReadMetrics`. Flink keeps the first metric registered under a name and ignores later ones, so even once set, the reported values would have come from the first split reader only. ### Summary and Changelog The stream read delays are now computed when the gauges are read, and the Source V2 reader reports `splitLatestCommit` and `splitLatestCommitDelay`. No metric names change, and values are still based on instant time. - `FlinkStreamReadMetrics` stores the issued instant and the split's latest commit, and computes `issuedInstantDelay` and `splitLatestCommitDelay` in the gauges. A new `clearCurrentSplit()` makes `splitLatestCommitDelay` report 0 until the next split starts, so an idle, caught-up reader doesn't look like it's lagging. `splitLatestCommit` keeps reporting the last split's commit. - `HoodieSourceReader` creates and registers one `FlinkStreamReadMetrics` and passes it to every split reader it creates. `HoodieSourceSplitReader` now takes the metrics instead of the table name and reader context. - `HoodieSourceSplitReader` sets the split's latest commit when it opens a split and clears it when the split finishes. - `StreamReadOperator` clears the current split when it finishes one. Without this, the delay computed at read time would keep growing after the legacy reader went idle. - Tests: new `TestFlinkStreamReadMetrics` covering the delays growing with time, the idle reader, the values before anything is set and invalid instants; a new `TestHoodieSourceSplitReader` case checking that the split reader sets and clears the split's commit. ### Impact Metric values change; names don't. - `issuedInstantDelay` now keeps growing while discovery finds no new commits, rather than holding the delay from when the last commit was discovered. On a table that commits rarely it rises between commits; it still resets when the next commit is issued. - On Source V2, `splitLatestCommit` and `splitLatestCommitDelay` now report real values instead of 0. - On both readers, `splitLatestCommitDelay` grows while a split is being read and reports 0 while the reader has no split. `HoodieSourceSplitReader`'s constructor changes, but it's internal to the Flink source. ### Risk Level low. The changes are limited to metric values; reading is unaffected. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
