Vamsi-klu opened a new pull request, #19083:
URL: https://github.com/apache/pinot/pull/19083
## Why
When a server hits fatal consumption errors it marks its replica **OFFLINE**
in IdealState. RSVM already recreates a consuming segment when **all** replicas
are OFFLINE, but **partial** OFFLINE (common after transient stream blips) was
left alone. Queries then fan in to the remaining CONSUMING replicas
indefinitely — imbalance and partial-result risk.
There is a separate flag that only flips OFFLINE→CONSUMING (#11314 path).
This PR implements the issue’s requested approach: **partition-scoped
force-commit** when enough progress is likely, behind an **opt-in** config.
## Impact
- **Heals chronic under-replication** of consuming segments without manual
force-commit.
- **Rebalances** sealed data onto a new CONSUMING generation for all
replicas.
- **Safe by default**: feature flag off; age gate avoids force-committing
brand-new empty segments; once-per-segment tracking avoids RSVM storms; reuses
`validateForceCommitAllowed` (partial upsert / drop-OOO guards).
- New meters: success / failed / skipped auto force-commit.
## How
- Config:
- `controller.realtime.segment.autoForceCommitOnPartialOfflineEnabled`
(default **false**)
- `controller.realtime.segment.autoForceCommitOnPartialOfflineMinAgeMs`
(default 5 minutes)
- In `ensureAllPartitionsConsuming`, when IdealState shows mixed
CONSUMING+OFFLINE for an IN_PROGRESS segment and age gate passes, call existing
partition-scoped `forceCommit`.
- Do not combine with OFFLINE→CONSUMING repair in the same tick when
force-commit fires.
## Test plan
- [x] `PinotLLCRealtimeSegmentManagerTest`: force-commit when partial
OFFLINE + age ok; age gate skip; all-OFFLINE still recreates; paused table
skip; validateForceCommit failure skip.
- [ ] `./mvnw -pl pinot-controller -am
-Dtest=PinotLLCRealtimeSegmentManagerTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- [ ] Staging: enable flag on a non-prod realtime table, mark one replica
OFFLINE after the segment is aged, confirm force-commit + new CONSUMING.
## Related
fixes: #15897
## Reviewers
Suggested: sajjad-moradi (issue author), maintainers of LLC realtime
## Config / release notes
New controller configs (default off) — consider `release-notes` label.
---
##### Was generative AI tooling used to co-author this PR?
- [x] Yes — Grok Build (xAI)
Generated-by: Grok Build (xAI)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
