KKcorps commented on PR #19969:
URL: https://github.com/apache/pinot/pull/19969#issuecomment-6517763847
**This closes the restart flip it targets, including a crash before the next
snapshot round. The model found one new order, where a restart serves a tied
key twice.**
I replayed the model's 52 known bad orders and none changed. A 61 minute
search over 8 table shapes that re-push uploads found the consuming tie below,
and a hand walk of the reload found the duplicate.
**A snapshot round that cannot lock N0.** The reload hands key A from N0 to
an upload that already has a validDocIds file, so if the next round cannot lock
N0, both files mark A. It is rare, but a restart with preload in that window
serves A twice, and the extra row stays. Deleting the reloaded segment's file
on tables without TTL would avoid it.
```mermaid
sequenceDiagram
participant C as Controller
participant S as Server
participant R as Snapshot round
participant F as validDocIds files
Note over S,F: N0 holds A after a tie with upload U, and both have files
C->>S: same CRC push of U, then SegmentMetadataRefreshMessage
S->>S: reload U with the newer time, A moves from N0 to U
S->>S: a reload of N0 takes N0's segment lock
R->>F: tryLock(N0) fails, so N0's file still marks A
R->>F: rewrite U's file with A valid
S->>S: restart before the next round
F->>S: preload puts A in both N0 and U
Note over S: queries return two rows for A
```
<details><summary>Model history</summary>
```
('upload', 'U1', 'A', (1,), {'a': 7, 'm': 0})
('step', 'upload', 'U1', 'start_add')
('step', 'upload', 'U1', 'add_row:A')
('step', 'upload', 'U1', 'register')
('assign_consumer', "N0'")
('acquire', "N0'")
('append', 'A', (1,), 0, {'a': 2, 'm': 0}, False)
('consume', "N0'", 0)
('commit_local', "N0'", 0)
('step', 'replace', "N0'", 'release_semaphore_early')
('step', 'replace', "N0'", 'start_replace')
('step', 'replace', "N0'", 'add_committed_row:A@0')
('step', 'replace', "N0'", 'begin_leftover')
('step', 'replace', "N0'", 'untrack_and_offload')
('assign_consumer', "N1'")
('acquire', "N1'")
('repush', 'U1', True)
('check_metadata_message', 'U1')
('start_metadata_reload', 'U1')
('step', 'reload', 'U1', 'start_replace')
('step', 'reload', 'U1', 'add_row:A@-1')
('step', 'reload', 'U1', 'begin_leftover')
('step', 'reload', 'U1', 'untrack')
('reload', 'N0')
('take_snapshot', ['U1'])
('step', 'reload', 'N0', 'start_replace')
('step', 'reload', 'N0', 'add_row:A@0')
('step', 'reload', 'N0', 'begin_leftover')
('step', 'reload', 'N0', 'untrack')
('crash_restart',)
('step', 'online', 'N0', 'register')
('step', 'online', 'U1', 'register')
('acquire', "N1'r1")
```
</details>
**A tie with a consuming segment.** A consumed row wins every tie, but the
reload ranks a consuming segment by creation time, so replicas split on whether
they reload before or after reading the tied row.
```mermaid
sequenceDiagram
participant C as Controller
participant S as Server reload
participant K as Consumer N0
Note over S,K: upload U holds key A
C->>S: same CRC push of U, its time is now newer than N0's
K->>K: read A with the same comparison value, the consumed row wins
S->>S: reload U, the tie with N0 now goes by creation time
Note over S,K: A moves to U, but a replica that reloaded before the read
keeps N0's row
```
<details><summary>Model history</summary>
```
('upload', 'U1', 'A', (1,), {'a': 7, 'm': 0})
('step', 'upload', 'U1', 'start_add')
('step', 'upload', 'U1', 'add_row:A')
('step', 'upload', 'U1', 'register')
('assign_consumer', "N0'")
('acquire', "N0'")
('append', 'A', (1,), 0, {'a': 2, 'm': 0}, False)
('repush', 'U1', True)
('consume', "N0'", 0)
('check_metadata_message', 'U1')
('start_metadata_reload', 'U1')
('step', 'reload', 'U1', 'start_replace')
('step', 'reload', 'U1', 'add_row:A@-1')
('step', 'reload', 'U1', 'begin_leftover')
('step', 'reload', 'U1', 'untrack')
```
</details>
The model covers one replica of one partition and no offline upsert tables.
<!-- upsert-model-check head=900cbcef08819b6e404c1b4645cf96779e13ce1c -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
