[ https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100946#comment-16100946 ]
Apurva Mehta commented on KAFKA-5621: ------------------------------------- I think there are a few points here: 1. What does 'accumulator time' really mean to an application developer? The accumulator is an internal detail, and we should not have application developers thinking about this. They only need to think about batching, and thus about the {{batch.size}} and {{linger.ms}} parameters. From this point of view, having one {{request.timeout.ms}} which applies to the accumulator queue time as well as the actual request time makes sense: From the developers point of view, the request begins as soon as the {{producer.send}} is called and completes once a response is received. 2. I think the main reason to introduce batch expiry was to free up memory on the client. But we already have a configurable bound on memory, so this reason is less strong: you will never explode the memory utilization on the client just because you lose the brokers. Instead, the {{producer.send}} call will just block when there is no available memory. So the benefits of expiring batches proactively is a bit dubious. 3. Finally, why is it desirable to 'send a failure notification' as soon as possible? The simplest programming model from the developers point of view would be for the Kafka client to retry in as many failure scenarios as possible and only return a hard error when there is no way to proceed. This places less of a burden on the application developer. From this point of view, retrying when we hit a timeout in the queue makes sense: it is a retriable error. Stepping back a bit further, the reason we had a retries configuration in the first place was because producer retries could introduce duplicates. And so we needed to provide developers with an option to control that on their end. With the recent idempotent producer and ongoing work to improve the performance of that feature, we aim to have the idempotent producer turned on by default with infinite retries. This would mean that the application developer would only have to worry about a very small class of errors (basically only authorization or configuration exceptions) and yet enjoy strong semantics without any loss of performance. So in the ideal future default situation, we would never expire a batch, but let it sit in the queue until the cluster recovers to the point when the batch can be sent. With this in mind, it makes sense to retry the batch on expiry right now. Hence the current proposal. > The producer should retry expired batches when retries are enabled > ------------------------------------------------------------------ > > Key: KAFKA-5621 > URL: https://issues.apache.org/jira/browse/KAFKA-5621 > Project: Kafka > Issue Type: Bug > Reporter: Apurva Mehta > Fix For: 1.0.0 > > > Today, when a batch is expired in the accumulator, a {{TimeoutException}} is > raised to the user. > It might be better the producer to retry the expired batch rather up to the > configured number of retries. This is more intuitive from the user's point of > view. > Further the proposed behavior makes it easier for applications like mirror > maker to provide ordering guarantees even when batches expire. Today, they > would resend the expired batch and it would get added to the back of the > queue, causing the output ordering to be different from the input ordering. -- This message was sent by Atlassian JIRA (v6.4.14#64029)