9aman commented on PR #20223: URL: https://github.com/apache/pinot/pull/20223#issuecomment-6818704534
> This comes from a model of upsert thread orders that StarTree runs on upsert PRs. > > **This keeps a late replica out of the state that five of the model's known bad orders start from, and a search of the orders it adds found nothing bad. One rare order can put the other replicas in that state instead.** > > All five begin with a replica past the end offset and end with a wrong partial upsert value or a dedup repeat. The search ran for 65 minutes. It made 4,111 runs over 14 pauseless table shapes. > > **The order that moves.** A late replica now reports at the end time with a low offset. `isWinnerPicked` takes the largest offset reported by the time it picks, so the late one wins when the others report a few seconds later. That needs clock skew or a stalled consumer, so it is rare. > > ```mermaid > sequenceDiagram > participant L as Late replica > participant C as Controller > participant A as Other replica > L->>C: segmentConsumed at offset 100, end time reached > C-->>L: HOLD > Note over A: clock is behind, still consuming at offset 1000 > L->>C: segmentConsumed at offset 100, time to pick is over > C-->>L: COMMIT at offset 100 > L->>C: commit start, end offset 100 goes to ZK > C->>A: CONSUMING to ONLINE with end offset 100 > A->>A: past the end offset, so it can only download > A->>A: waits for the segment to leave COMMITTING > Note over L,A: if the late replica fails before the upload, no replica has a copy > ``` > Model history > ``` > N0 is committed here and holds A > append A, B, A, C at offsets 1 to 4 > N1' consumes offsets 1 to 4 on this replica, which never restarted > commit_remote N1 at offset 1, start only (the late replica wins at its lower offset) > observe_commit N1' returns download and waits while N1 is COMMITTING > commit_remote_end N1 (or time_out_download N1', which leaves N1' in ERROR with no copy) > replace N1': add A at offset 1, revert A from offset 3 back to offset 1, drop B and C > N2' consumes offsets 2 to 4 again and the table ends correct > ``` > > The model follows one replica, so the controller's pick comes from reading the code. It also never times out a catch up, so the 31 second limit in `goOnlineFromConsuming` was not searched. @KKcorps what is the expectation from the author wrt this analysis ? Does it's rare qualify that there is not blocker ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
