KKcorps commented on PR #20118: URL: https://github.com/apache/pinot/pull/20118#issuecomment-6572067405
This comment comes from a model of upsert thread orders that StarTree runs on upsert PRs. **The model found no new bad thread order. A pause for this quota force commits the consuming segment and holds back the next one, and today's normal rollovers already reach every order that can follow.** I modeled the pause as a commit at the consumer's offset with no next consuming segment, and the resume as the consuming segment that `ensureAllPartitionsConsuming` then creates. A 67 minute search ran 12,976 times with restarts and rebalances during the pause, on pauseless and regular tables. Six of the bad orders it found passed through a pause, and each one breaks the same way on today's code. The javadoc expects TTL to bring the count down, but a metadataTTL table cannot fall back under the cap while paused. TTL measures age against the largest comparison value consumed, which stops moving when consumption stops, and the server only sweeps expired keys when a consuming segment starts. So apart from a restart or a retention delete, only a raised cap resumes the table. The model covers one replica of one partition, so the max across replicas and a partial server response were not checked. <!-- upsert-model-check head=2ef0d5a8798a258a3a00b2552b280b90f1aa1ce4 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
