deepthi912 commented on PR #19512: URL: https://github.com/apache/pinot/pull/19512#issuecomment-5628429578
> Trying to understand why this race can even happen. When this race happens, it means the records within the same segment span over TTL. This itself is invalid, and undefined. I don't think we can fix it in any way. Imagine a segment is being added, in the meanwhile a consuming record pushes the watermark higher, and we run into the same problem. Also, why is an uploaded segment moving the watermark? Shouldn't it always be moved by consuming segment? Agree on the condition that is hard to fix across the replicas when uploaded segment comes in during consumption, there are lot more corner cases as I was fixing this. But this PR was mainly focussed on ensuring the watermark is updated after rocksdb updates rather than before during each record addition and segment replacement. The reason why we are updating watermark for uploaded segment could be to make sure the watermark is not too advanced if there is over consumed or under consumed rows. The bug scope is narrow but I feel it is important that we update after the rocksdb updates than before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
