Re: [I] Allowing empty segments with no offset advancing [pinot]
jadami10 commented on issue #12703: URL: https://github.com/apache/pinot/issues/12703#issuecomment-2351186136 One other issue we've seen (QA only so far) from our multi-kafka cluster setup - table is created and creates as segment for partition A on cluster 1 - partition A has no data - cluster 1 is later removed for this topic, so Pinot should no longer get this partition - Pinot keeps trying to consume because the segment is open indefinitely and spews endless errors -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Allowing empty segments with no offset advancing [pinot]
Jackie-Jiang commented on issue #12703: URL: https://github.com/apache/pinot/issues/12703#issuecomment-2159568541 > Is this as straight-forward as removing these lines? I think so. The drawback is that we might end up with a lot of segments that are all empty, but I guess we can make retention manager keep cleaning up empty segments (it might already do) as long as they are not the last completed segment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
Re: [I] Allowing empty segments with no offset advancing [pinot]
tibrewalpratik17 commented on issue #12703: URL: https://github.com/apache/pinot/issues/12703#issuecomment-2154850044 Hey @Jackie-Jiang any updates on this? Is this as straight-forward as removing these lines? https://github.com/apache/pinot/blob/0f92742b45aa9ac57439b68a63e4a9ef8d9d9d8e/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java#L334-L338 Saw this in one of our tables today. The table existed for more than a year and we enabled snapshot on it recently. Now since the last consuming segment (which was like months old) never received any message, the snapshot flow didn't run and removed older records via compaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
