rohityadav1993 commented on issue #13140: URL: https://github.com/apache/pinot/issues/13140#issuecomment-2136001342
I don't see any inconsistency issues after moving the logic of releasing partition consumer semaphore(`_partitionGroupConsumerSemaphore`) in `closeStreamConsumers()` after replace is complete. There is also an indication that not just downloadAndReplace() flow is affected as we see the inconsistent segments even when there is no peer download and consuming segment is built and replaced. To ensure data accuracy, we will have to ensure to release `_partitionGroupConsumerSemaphore` only after replace is replace is completed. The side effect is stopped ingestion during segment commit. On our test table this has resulted in a jump of ingestion lag from ~5 minutes to ~15 minutes Before:  After:  @Jackie-Jiang , @klsince , do you have a suggestion how we should go about this: 1. For partial upserts, always release semaphore after replace 2. A new table config to release semaphore after replace e.g. `pausePartitionConsumptionDuringReplace: true` `_partitionGroupConsumerSemaphore` acquires in RealtimeSegmentDataManager's constructor, so it would stay stuck in the constructor until it is released after replace. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
