Jackie-Jiang commented on issue #10147:
URL: https://github.com/apache/pinot/issues/10147#issuecomment-2197768243
We sync'd on this offline, and here are the some conclusions:
- When segment commit happens (end offset of the consuming segment is
determined), controller can create the ZKMetadata for the next consuming
segment, and use commit protocol to ask servers to start consuming the next
segment.
- Controller should try to assign the next consuming segment, and only ask
servers to start consuming when it is assigned to the same servers. This can
avoid duplicate consuming segment during rebalance.
- Note that controller won't change the ideal state at this moment because
we want to avoid having 2 CONSUMING segments on the same partition. If we allow
that, it can cause problem when server crashed restarted because it could skip
one CONSUMING segment.
- Controller updates ideal state the same way as current code when commit
end (change consuming segment to ONLINE and add new consuming segment as
CONSUMING)
- Server creates the new consuming segment when informed by controller via
the commit protocol, and associate it with the last consuming segment for query
purpose
- When server gets the state transition from OFFLINE -> CONSUMING on the new
consuming segment, it moves the new consuming segment as the regular consuming
segment. It can remove the association of new consuming segment and last
consuming segment, and broker should be able to directly query the new
consuming segment as optional (introduced in #11978).
- `RealtimeSegmentValidationManager` should be modified to handle cleaning
up the orphan ZKMetadata when segment commit failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
