KKcorps commented on PR #20223:
URL: https://github.com/apache/pinot/pull/20223#issuecomment-6818404994
This comes from a model of upsert thread orders that StarTree runs on upsert
PRs.
**This keeps a late replica out of the state that five of the model's known
bad orders start from, and a search of the orders it adds found nothing bad.
One rare order can put the other replicas in that state instead.**
All five begin with a replica past the end offset and end with a wrong
partial upsert value or a dedup repeat. The search ran for 65 minutes. It made
4,111 runs over 14 pauseless table shapes.
**The order that moves.** A late replica now reports at the end time with a
low offset. `isWinnerPicked` takes the largest offset reported by the time it
picks, so the late one wins when the others report a few seconds later. That
needs clock skew or a stalled consumer, so it is rare.
```mermaid
sequenceDiagram
participant L as Late replica
participant C as Controller
participant A as Other replica
L->>C: segmentConsumed at offset 100, end time reached
C-->>L: HOLD
Note over A: clock is behind, still consuming at offset 1000
L->>C: segmentConsumed at offset 100, time to pick is over
C-->>L: COMMIT at offset 100
L->>C: commit start, end offset 100 goes to ZK
C->>A: CONSUMING to ONLINE with end offset 100
A->>A: past the end offset, so it can only download
A->>A: waits for the segment to leave COMMITTING
Note over L,A: if the late replica fails before the upload, no replica
has a copy
```
<details><summary>Model history</summary>
```
N0 is committed here and holds A
append A, B, A, C at offsets 1 to 4
N1' consumes offsets 1 to 4 on this replica, which never restarted
commit_remote N1 at offset 1, start only (the late replica wins at its lower
offset)
observe_commit N1' returns download and waits while N1 is COMMITTING
commit_remote_end N1 (or time_out_download N1', which leaves N1' in ERROR
with no copy)
replace N1': add A at offset 1, revert A from offset 3 back to offset 1,
drop B and C
N2' consumes offsets 2 to 4 again and the table ends correct
```
</details>
The model follows one replica, so the controller's pick comes from reading
the code. It also never times out a catch up, so the 31 second limit in
`goOnlineFromConsuming` was not searched.
<!-- upsert-model-check head=feb5a00549eb2f922f5b14bab4e3fd94bb157203 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
