Hi Igor, thanks for the feedback and sorry for the late response. > 10 Given the goal is to prevent OOMs, do we also need to > limit the number of KafkaPrincipals in use?
None of the Kafka quotas ever limited number of KafkaPrincipals and I don’t really think this is the issue as you just need one misconfigured application to hit the OOM issue. Also limiting number of KafkaPrincipals will cause problems for on-boarding new cases to use the cluster. What is the question behind this question? Is there a use case you have in mind that need this? > 11. How would an operator know or decide to change the configuration > for the number layers – producer.id.quota.cache.layer.count – > e.g. increasing from 4 to 5; and why? > Do we need a new metric to indicate that change could be useful? > > 12. Is producer.id.quota.cache.cleanup.scheduler.interval.ms a > guaranteed interval, or rather simply a delay between cleanups? > How did you decide on the default value of 10ms? I am leaning toward dropping all the caching layer configs and keeping them as constant and we can later make them a config if there is a request for this. The 10min doesn’t really have a reason behind. It was a random number. > 16. LayeredBloomFilter will have a fixed size (right?), but some > users (KafkaPrincipal) might only use a small number of PIDs. > It it worth having a dual strategy, where we simply keep a Set of > PIDs until we reach certain size where it pays off to use > the LayeredBloomFilter? The size is driven from the `producer_id_rate` for each KafkaPrincipal. We might hit this issue for sure. I believe Claude covered this point. > > a. INIT_PRODUCER_ID for idempotent producer request PIDs from > random controller every time so if a client got throttled on > one controller doesn't guarantee it will not go through on next > controller causing OOM at the leader later. > > Is the INIT_PRODUCER_ID request really sent to a "random controller"? > From a quick look at Sender.maybeSendAndPollTransactionalRequest, > for an idempotent producer, targetNode is set to the broker with > fewest outstanding requests. Am I looking at the wrong place? No you are not looking at the wrong place however this still a random comaring with TXN_ID for example or GROUP_ID and not dedicated by any mean as it can change at any point depend on which one has less outstanding requests. However if we want to focus on where the problem happened this will be the leader. We can throttle INIT_PID but this will not limit how many PID can produce to one 1 broker Let’s image this scenario where we have a cluster where each broker can max have 200 PID without hitting OOM. And each KafkaPrincipal can INIT_PID for X number (let say 10) and we have like 20 active KafkaPrincipal. Which in total we can have 200 INIT_PID request. Now on the leadership side of producing we have Broker-1 has 100 PID in memory Broker-2 has 150 PID in memory Broker-3 has 100 PID in memory Broker-4 has 120 PID in memory Broker-5 has 100 PID in memory Broker-6 has 90 PID in memory Now Broker-1 is down and for the sake of this example not all 100 PID will move to one broker but 60 PID will connect to Broker-2 bringing its total PID to 210. Now these PIDs aren’t going to be throttle by INIT_PID as they already has been initialised which will cause OOM now Broker-2 is down. If Broker-1/2 down now the problem is propagated and the cluster will be down as each broker try to take leadership will get the initialised PIDs and hit OOM until someone manually increase the memory of the brokers or if we hit the PID cleanup at some point of all of this mess. Hope this explain my concerns with throttling `INIT_PRODUCER_ID`. Thanks Omnia > On 1 May 2024, at 15:42, Igor Soarez <i...@soarez.me> wrote: > > Hi Omnia, Hi Claude, > > Thanks for putting this KIP together. > This is an important unresolved issue in Kafka, > which I have witnessed several times in production. > > Please see my questions below: > > 10 Given the goal is to prevent OOMs, do we also need to > limit the number of KafkaPrincipals in use? > > 11. How would an operator know or decide to change the configuration > for the number layers – producer.id.quota.cache.layer.count – > e.g. increasing from 4 to 5; and why? > Do we need a new metric to indicate that change could be useful? > > 12. Is producer.id.quota.cache.cleanup.scheduler.interval.ms a > guaranteed interval, or rather simply a delay between cleanups? > How did you decide on the default value of 10ms? > > 13. Under "New ProducerIdQuotaManagerCache", the documentation for > the constructor params for ProducerIDQuotaManagerCache does not > match the constructor signature. > > 14. Under "New ProducerIdQuotaManagerCache": > public boolean track(KafkaPrincipal principal, int producerIdRate, long pid) > How is producerIdRate used? The reference implementation Claude shared > does not use it. > https://github.com/Claudenw/kafka/blob/49b6eb0fb5cfaf19b072fd87986072a683ab976c/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerIDQuotaManager.java > > 15. I could not find a description or definition for > TimestampedBloomFilter, could we add that to the KIP? > > 16. LayeredBloomFilter will have a fixed size (right?), but some > users (KafkaPrincipal) might only use a small number of PIDs. > It it worth having a dual strategy, where we simply keep a Set of > PIDs until we reach certain size where it pays off to use > the LayeredBloomFilter? > > 17. Under "Rejected Alternatives" > "4. Throttle INIT_PRODUCER_ID requests", > the KIP states: > > a. INIT_PRODUCER_ID for idempotent producer request PIDs from > random controller every time so if a client got throttled on > one controller doesn't guarantee it will not go through on next > controller causing OOM at the leader later. > > Is the INIT_PRODUCER_ID request really sent to a "random controller"? > From a quick look at Sender.maybeSendAndPollTransactionalRequest, > for an idempotent producer, targetNode is set to the broker with > fewest outstanding requests. Am I looking at the wrong place? > > 18. Under "Rejected Alternatives" > "4. Throttle INIT_PRODUCER_ID requests", > the KIP states: > > This solution might look simple however throttling the INIT_PRODUCER_ID > doesn't guarantee the OOM wouldn't happened as > (...) > b. The problem happened on the activation of the PID when it > produce and not at the initialisation. Which means Kafka wouldn't > have OOM problem if the producer got assigned PID but crashed before > producing anything. > > Point b. does not seem to support the claim above? > > 19. Under "Rejected Alternatives" > "4. Throttle INIT_PRODUCER_ID requests", > the KIP states: > > c. Throttling producers that crash between initialisation and > producing could slow them down when they recover/fix the > problem that caused them to crash right after initialising PID. > > Doesn't it depend on the back-off time or how quotas are enforced? > I’m not sure this would necessarily be a problem? > > 20. If the allocation of PIDs for idempotent producers was > centralized, or otherwise the the targetNode for that request > was predictable, would that make throttling INIT_PRODUCER_ID > a viable solution? > > > Best, > > -- > Igor > >
