Request quotas was just added to 0.11. Does that help in your use case? https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas
-hans > On Jun 29, 2017, at 12:55 AM, sukumar.np <sukumar...@zohocorp.com> wrote: > > Hi Team, > > > > We are having a Kafka cluster with multiple Topics in it and shared with > multiple services(clients). Each service will have multiple events source > from where they will be pushing messages to Kafka brokers. Once a service > starts producing message at a high rate, it will affect other > services(clients) because it will fill the disk quickly which will bring the > cluster down. So, we want to throttle Producer request once it crosses the > specified threshold(Which can be set on Topic basis, not for each service). > > > > After checking with Quota feature available in Kafka we found that It allows > pushing data to a queue and will keep responses in delay queue(if it requires > getting throttled). If we apply quota for our use case then below problems > can happen: > > a). Since quota observes message rate for a window and starts to throttle > the producer responses, meanwhile all incoming messages will be added to the > queue. It may fill the disks quickly as there are many producers for the same > Topics and will create an outage for Kafka service. > > b). For sync Producers, because of throttling response will be delayed, which > will result in hang the user-thread or app-servers. > > > > So we don't want to go for applying quota for our use case. Can you please > share some suggestions to handle this use-case in our Kafka broker. Like, > before messages get appended to log, it should validate for throttling and if > it requires being throttled. Throttling mechanism should be either slow down > the request rate up to specified time frame or throw some generic exception > from broker side to clients. > > > > Our Kafka setup like, > > Having 3 brokers in a cluster and each Topic has replication factor 3 and > using Kafka-0.10.0.1. > > > > Looking forward to your suggestions. > > > > Thanks > > Sukumar N > > > > >
