Hello everyone. I send you this mail because I have a question related to quota in Kafka.
I read the following articles there : - https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas - https://kafka.apache.org/documentation/#design_quotas My Kafka cluster has the version 0.10.1 embedded with the distribution HDP 2.6.2. I have Kerberos enabled for Kafka. I set a network bandwidth quota on a specific user to limit its production throughput. /usr/hdp/current/kafka-broker/bin/kafka-configs.sh --zookeeper ${ZK} --alter --add-config 'producer_byte_rate=102400' --entity-type users --entity-name <MY_USER> with ZK the zookeeper quorum With this command, if I understand well, <MY_USER> should be limited at 100 KB/s to produce event in any kafka topics. Then I used the utility kafka-producer-perf-test.sh to perform some tests to check if <MY_USER> is really limited to this bandwidth. So first I take a kerberos ticket for <MY_USER>. And then, here are my tests : /usr/hdp/current/kafka-broker/bin/kafka-producer-perf-test.sh --topic <MY_TOPIC> --broker-list ${BL} --messages 1000 --security-protocol PLAINTEXTSASL /usr/hdp/current/kafka-broker/bin/kafka-producer-perf-test.sh --topic <MY_TOPIC> --broker-list ${BL} --messages 10000 --security-protocol PLAINTEXTSASL /usr/hdp/current/kafka-broker/bin/kafka-producer-perf-test.sh --topic <MY_TOPIC> --broker-list ${BL} --messages 100000 --security-protocol PLAINTEXTSASL With - ${BL} the bootstrap kafka servers with the right ports. - <MY_TOPIC> : a topic in my kafka cluster Here are the results in a table : start.time end.time compression message.size batch.size total.data.sent.in.MB MB.sec total.data.sent.in.nMsg nMsg.sec KB.sec 2020-09-02 10:47:59:455 2020-09-02 10:48:01:035 0 100 200 0,1 0,0604 1000 632,9114 61,8496 2020-09-02 10:49:20:093 2020-09-02 10:49:22:438 0 100 200 0,95 0,4067 10000 4264,392 416,4608 2020-09-02 10:49:49:649 2020-09-02 10:51:10:424 0 100 200 9,54 0,1181 100000 1238,007 120,9344 What I don't understand is the test with 10 000 and 100 000 messages. If I understand well the total bandwidth used is 416 KB/s and 120 KB/s, which is beyond the limit I set for this user. I wanted to know how this is possible ? Why is the quota not respected ? Thanks in advance for your help o/ Best regards. Tallanel
