wirybeaver commented on issue #14815: URL: https://github.com/apache/pinot/issues/14815#issuecomment-2594312265
backfill through minion jobs makes more sense, creating a temporary topic would cause more overhead to address the failure scenario, for example: the number of topics would blows up if multiple ingestion spike happens. How to guarantee the consistency between temporary topics and number offset gaps. We could add the consuming segments metadata in helix for minion tasks. When the backfilling is done, delete the temporarily task info. There should be a GC minion task to delete minion task info to reconciliation. MinionTask: ConsumingSegments1: startOffset=offset1, endOffset=offset2, status=InProgress and then add the consuming segments metadata as normal for pinot servers. ConsumingSegments2: startOffset=offset2, status=InProgress -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
