KKcorps opened a new pull request, #18531: URL: https://github.com/apache/pinot/pull/18531
## Summary - Add a shared per-JVM Kinesis request limiter keyed by stream, shard, and AWS read operation. - Add bounded backoff/retry handling for Kinesis ProvisionedThroughputExceededException from GetRecords and GetShardIterator. - Allow fractional requests_per_second_limit values so deployments can divide shard read budget across Pinot replicas/tables and other consumers. ## Why AWS throttling occurs before Pinot's post-fetch realtime consumption limiters run. When Kinesis rejects GetRecords, Pinot currently logs the throttle and returns an empty batch immediately, which can keep the consumer in a no-progress polling loop. Moving the control into the Kinesis consumer gives Pinot a way to coordinate local read pressure and back off before surfacing a no-progress batch. ## Configuration | Config | Default | Change | Impact | | --- | --- | --- | --- | | requests_per_second_limit | 1.0 | Existing key now accepts decimal values such as 0.25 | Operators can split the AWS shard read budget across replicas, tables, and non-Pinot consumers without changing the config key. | ## Reviewer Notes - GetRecords retries reuse the existing shard iterator instead of repeatedly calling GetShardIterator after a read throttle. - Timeout exhaustion still returns an empty non-EOP batch at the original offset, preserving the existing no-progress contract without falsely ending the shard. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
