[
https://issues.apache.org/jira/browse/KAFKA-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131507#comment-15131507
]
Jiangjie Qin commented on KAFKA-3197:
-------------------------------------
Hey Jay,
Yes, idempotent producer would solve the problem. And I completely agree that
when people set in flight request to one they are expecting no re-ordering.
I was initially thinking of treating in.flight.request.per.connection=1 as
in.flight.batch.per.partition=1 implicitly needed. This does not need
additional configuration. But there is a subtle difference in terms of
performance. If a producer has a lot partitions to send to the same broker,
theoretically we can allow in flight request > 1 as long as each request
addresses distinct partitions. If we enforce in.flight.request=1, we lose this
parallelism. But given this is what already there, so it is probably fine to
leave it as is.
I'll update the patch to remove the newly added configuration but simply reuse
in.flight.request.per.connection. Otherwise please let me know if you think the
subtle optimization worth the configuration change.
> Producer can send message out of order even when in flight request is set to
> 1.
> -------------------------------------------------------------------------------
>
> Key: KAFKA-3197
> URL: https://issues.apache.org/jira/browse/KAFKA-3197
> Project: Kafka
> Issue Type: Bug
> Components: clients, producer
> Affects Versions: 0.9.0.0
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> The issue we saw is following:
> 1. Producer send message 0 to topic-partition-0 on broker A. The in-flight
> request to broker A is 1.
> 2. The request is somehow lost
> 3. Producer refreshed its topic metadata and found leader of
> topic-partition-0 migrated from broker A to broker B.
> 4. Because there is no in-flight request to broker B. All the subsequent
> messages to topic-partition-0 in the record accumulator are sent to broker B.
> 5. Later on when the request in step (1) times out, message 0 will be retried
> and sent to broker B. At this point, all the later messages has already been
> sent, so we have re-order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
