skywalker0618 opened a new pull request, #20735:
URL: https://github.com/apache/hudi/pull/20735

   ### Describe the issue this Pull Request addresses
   
   Part 2 of #20531. The Source V2 streaming reader has no metric that says how 
far behind the table it is, the way Kafka consumer lag does. This PR adds read 
lag and pending work metrics to the continuous split enumerator, for 
append-only and copy-on-write tables. Merge-on-read is left for future work.
   
   ### Summary and Changelog
   
   The continuous split enumerator now tracks every discovered split until a 
reader reports it finished, and registers these gauges under the table metric 
group:
   
   | Gauge | Meaning |
   |---|---|
   | `readLagSeconds` | Seconds since the oldest pending commit completed, 0 
when nothing is pending |
   | `pendingCommits` | `undiscoveredCommits` + `inFlightCommits` |
   | `undiscoveredCommits` | Completed commits after the issued offset that 
discovery has not turned into splits yet |
   | `inFlightCommits` | Discovered commits that still have queued or 
unfinished splits |
   | `pendingSplits` | Discovered splits not finished yet, queued or being read 
|
   | `pendingRecords` | Expected records still to be emitted by the pending 
splits |
   | `pendingBytes` | Size of the files the pending splits still have to read |
   
   **What a split reads.** At discovery, each split records the commits within 
the batch that wrote its file group, taken from the commit write stats that 
discovery already loads. For copy-on-write and append-only tables, these are 
the only commits whose records the split emits, because the reader filters rows 
by commit time. When they are unknown (the initial snapshot read from earliest, 
or a split restored from a version 1 checkpoint), the split falls back to the 
instants of its exact-match range, or to its latest commit.
   
   **How the metrics are maintained.** All state lives in 
`FlinkStreamReadLagMetrics` on the enumerator. Discovery, restore and splits 
added back by a failed reader call `addSplits`; the finished split ids sent 
with each split request call `finishSplits`. Both are synchronized, so a gauge 
read never sees half an update.
   
   - **`readLagSeconds`**: each split carries the completion time of the oldest 
commit it reads. A sorted map counts the pending splits per completion time; 
finishing a split decrements its entry and removes it at 0. The gauge returns 
now minus the first key, or minus the oldest undiscovered commit when no split 
is pending. Completion times come from the timeline, and the value is computed 
when the gauge is read, so it keeps growing while the readers stall.
   - **`inFlightCommits`**: a map counts the pending splits that read each 
commit; a split reading two commits counts under both. A commit is removed once 
every split reading it is finished. The gauge returns the number of commits in 
the map.
   - **`undiscoveredCommits`**: after each discovery pass, the completed 
commits after the new offset are counted from the timeline that pass already 
loaded. When discovery is paused because of `read.splits.limit`, the timeline 
is reloaded instead. With the default `read.commits.limit` and 
`read.splits.limit` every pass discovers all completed commits, so this is 0; 
it is non-zero for jobs that throttle discovery. Archived commits are not 
counted.
   - **`pendingSplits`**: the number of tracked splits. One split is one base 
file for copy-on-write and append-only tables, and one file slice for 
merge-on-read.
   - **`pendingRecords`**: a running sum of each split's expected records, 
which are the inserts plus updates its write commits wrote to its file group. 
Exact for append-only tables. For upserts it is an estimate: a record written 
by two commits of the same batch counts twice. Reading the file footer would 
not help, since a copy-on-write file holds every row of the file group but the 
split only emits the rows of its commits. 0 for the initial snapshot read.
   - **`pendingBytes`**: a running sum of each split's file size: the size of 
the latest file written for the file group, from the write stats, or the file 
slice size for the initial snapshot read. For merge-on-read this is only the 
last log file, which under-reports.
   
   **Changelog**
   
   - `HoodieSourceSplit`: new fields for the write commits, the oldest commit 
completion time, the estimated records and the estimated bytes, and 
`getReadCommits()`.
   - `HoodieSourceSplitSerializer` and `HoodieEnumeratorStateSerializer` move 
to version 2. Both still read version 1, which is covered by tests; they move 
together because the state serializer passes its version to the split 
serializer.
   - `IncrementalInputSplits.Result` carries the instant completion times and 
per file group read estimates, filled by 
`HoodieContinuousSplitBatch.fromResult`.
   - `HoodieContinuousSplitDiscover.getUndiscoveredCommits` (default empty), 
implemented by `DefaultHoodieSplitDiscover`.
   - `AbstractHoodieSplitEnumerator` gets hooks for finished and added-back 
splits, used by `HoodieContinuousSplitEnumerator`.
   - New `FlinkStreamReadLagMetrics`.
   
   ### Impact
   
   New enumerator metrics for Source V2 streaming reads. No new configs and no 
change to read behavior.
   
   Performance: no new storage requests in the normal case. Undiscovered 
commits reuse the timeline loaded by discovery, and write commits, records and 
bytes come from commit metadata discovery already parses. A paused discovery 
pass reloads the timeline once per check interval, which is still cheaper than 
the pass it replaces. On the enumerator, each split adds a few map updates on 
discovery and on finish, and the JobManager keeps a small list of write commits 
per split. Each checkpointed split grows by roughly 50 to 70 bytes, which 
matters only when many splits are queued (about 6 MB of enumerator state per 
100K queued splits).
   
   ### Risk Level
   
   low. The split and enumerator state formats change, but version 1 
checkpoints still restore, and both formats are covered by tests. The metrics 
only observe the enumerator and don't change split assignment.
   
   Known limitation: splits that the readers restore from their own state are 
not known to the enumerator, so right after a restore up to about one split per 
subtask is not counted until it finishes or is added back.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to