jadami10 opened a new pull request, #17758: URL: https://github.com/apache/pinot/pull/17758
This is a small bug fix for an edge case where freshness never catches up on server restart. The series of events are: - you have a realtime Pinot table consuming from a partition - that partition ingests at least 1 event with freshness > than threshold - that partition ingests more events, but they are all filtered out by filterConfig or other custom stream plugin filtering - the `minimumIngestionLag` never drops below the initial indexed event - the server will wait until the timeout period is reached to start Instead, we now detect when we've conusmed events but they've all been filtered, and we correctly set latest ingestion and min freshness based on current time. I've added 1 unit test and 1 integration test for this. We also deployed this to the main production cluster that was having issues. Previously we would see - our ingestion plugin reporting < 1 minute lag within ~15 minutes - but the server taking ~3 hours to go healthy based on freshness if an unfiltered event came in After this deployment: - the same server was healthy in < 15 minutes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
