9aman opened a new pull request, #20223:
URL: https://github.com/apache/pinot/pull/20223
## Problem
With a time flush threshold, each replica sets the end time of a consuming
segment to the segment creation time in ZK
plus the threshold. A replica that starts to consume less than 10 minutes
before this end time, for example after a
restart, gets a later end time: now + min(threshold, 10 minutes). When the
threshold is 10 minutes or less, every
replica gets its own end time this way.
For a pauseless table, this can cause data loss:
1. The late replica continues to consume after the other replicas stop. It
does not report in time, so the controller
selects the committer and the end offset without it.
2. At commit start, the controller sets the end offset, creates the next
segment and sends CONSUMING to ONLINE. The
late replica is now past the end offset, so it can only download the
committed segment.
3. If the committer fails before the upload, no replica has a copy. Only
re-ingestion from the stream can recover it.
## Change
- `RealtimeSegmentDataManager.setConsumeEndTime`: on a pauseless table, a
replica keeps the end time from ZK while that
time is in the future. All replicas stop at the same time and report
together, and the controller selects the highest
offset as the end offset.
- When the end time has already passed, the minimum consume time (10
minutes) still applies.
- With a threshold of 10 minutes or less, all replicas of a pauseless
segment now use creation time + threshold.
Before, each replica used its own start time + threshold.
- Non-pauseless tables do not change. There, a replica that is past the end
offset downloads a segment that is already
uploaded.
## Pauseless replica that restarts before the end time
The committer does not fail:
- The replica is behind at the end time: it gets CATCH_UP to the end offset
and builds its own copy. If CONSUMING to
ONLINE comes first, it still holds the partition consumer semaphore, so it
catches up for up to 31 seconds on the state
transition thread and then builds the segment. If it cannot catch up, it
downloads the committed copy.
- The replica is at or above the other replicas: the controller selects the
highest offset, so the replica becomes the
committer or stays at the end offset. No replica is past the end offset.
The committer fails after commit start:
- The replica reported with the committer. It is at the end offset, or it
catches up to it before the ONLINE transition
plus 31 seconds, and it builds its own copy. The segment stays available.
Before this change, the replica could be
past the end offset and wait for a download that never completes.
- If the replica cannot catch up in time, it must download and gets no copy,
the same as with a row or size threshold.
The committer fails before commit start:
- If the committer reported, the replica catches up to its offset. After the
commit timeout, the controller selects the
replica as the new committer with the same end offset.
- If the committer did not report, the replica is the only one to report and
commits at its own offset, so the segment
can be smaller. The next segment starts at that offset, so no data is lost.
## Limitations
- The controller selects the end offset 3.3 to 6.6 seconds after the first
report. A replica that reports later (for
example after a long GC pause) can still be past the end offset.
- If the restarted replica completes its first non-empty fetch only after
the end time (for example while it waits for
dedup or partial-upsert readiness), the 1-hour empty-segment extension
applies, so it does not stop at the end time.
This is the same as before the change.
## Testing
`RealtimeSegmentDataManagerTest#testConsumeEndTimeOnLateStart` fails without
the change. A pauseless replica that starts
late keeps the end time from ZK, also with a 5-minute threshold. A
non-pauseless replica, and a pauseless replica after
the end time, consume for at least 10 minutes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
