wirybeaver commented on issue #14815:
URL: https://github.com/apache/pinot/issues/14815#issuecomment-2594312265

   backfill through minion jobs makes more sense, creating a temporary topic 
would cause more overhead to address the failure scenario, for example: the 
number of topics would blows up if multiple ingestion spike happens. How to 
guarantee the consistency between temporary topics and number offset gaps.
   
   We could add the consuming segments metadata in helix for minion tasks. When 
the backfilling is done, delete the temporarily task info. There should be a GC 
minion task to delete minion task info to reconciliation.
   MinionTask: ConsumingSegments1: startOffset=offset1, endOffset=offset2, 
status=InProgress
   
   and then add the consuming segments metadata as normal for pinot servers.
   ConsumingSegments2: startOffset=offset2, status=InProgress
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to