Lakshmi Rao created FLINK-11501:
-----------------------------------
Summary: Add a ratelimiting feature to the FlinkKafkaConsumer
Key: FLINK-11501
URL: https://issues.apache.org/jira/browse/FLINK-11501
Project: Flink
Issue Type: Improvement
Components: Kafka Connector
Reporter: Lakshmi Rao
Assignee: Lakshmi Rao
There are instances when a Flink job that reads from Kafka can read at a
significantly high throughput (particularly while processing a backlog) and
degrade the underlying Kafka cluster.
While Kafka quotas are perhaps the best way to enforce this ratelimiting, there
are cases where such a setup is not available or easily enabled.
The approach is essentially to use Guava's rate limiter to ratelimit the bytes
read from Kafka.
More discussion here:
https://lists.apache.org/thread.html/8140b759ba83f33a22d809887fd2d711f5ffe7069c888eb9b1142272@%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
