jadami10 opened a new issue, #9014: URL: https://github.com/apache/pinot/issues/9014
In this part of the code, https://github.com/apache/pinot/blob/2b82366c369cf668175695605f0b77d7105f4aff/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L301-L304, we extend the segment build time by 1 hour if no rows were consumed. We have some common cases where certain partitions may never have data. We are either filtering out a lot of the data or there's a temporary error in our ingestion plugin. We also use a custom ingestion plugin. But by forever extending the lease like this, whenever we restart the server, the server attempts to consume going back days or weeks and causes high CPU usage and throttling for both the server and kafka. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
