[ https://issues.apache.org/jira/browse/KAFKA-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343682#comment-17343682 ]
NEERAJ VAIDYA commented on KAFKA-12776: --------------------------------------- Assuming that a Producer application is started, but there is no Cluster available. If at this point the application gets a number of requests to publish messages (all with the same key) to Kafka, then because Kafka Producer does not have any meta-data about the topic, it arranges the messages in batches which then might not necessarily cause the messages to go to the same partition ? When a message is retried, does Kafka Producer NOT try to get metadata again, before creating the batch ? If that is true, then I can understand why the messages are batched in a sequence where the ordering is not preserved because it probably does not have meta-data. > Producer sends messages out-of-order inspite of enabling idempotence > -------------------------------------------------------------------- > > Key: KAFKA-12776 > URL: https://issues.apache.org/jira/browse/KAFKA-12776 > Project: Kafka > Issue Type: Bug > Components: producer > Affects Versions: 2.6.0, 2.7.0 > Environment: Linux RHEL 7.9 and Ubuntu 20.04 > Reporter: NEERAJ VAIDYA > Priority: Major > Attachments: mocker.zip > > > I have an Apache Kafka 2.6 Producer which writes to topic-A (TA). > My application is basically a Spring boot web-application which accepts JSON > payloads via HTTP and then pushes each to a Kafka topic. I also use Spring > Cloud Stream Kafka in the application to create and use a Producer. > For one of my failure handling test cases, I shutdown the Kafka cluster while > my applications are running. (Note : No messages have been published to the > Kafka cluster before I stop the cluster) > When the producer application tries to write messages to TA, it cannot > because the cluster is down and hence (I assume) buffers the messages. Let's > say it receives 4 messages m1,m2,m3,m4 in increasing time order. (i.e. m1 is > first and m4 is last). > When I bring the Kafka cluster back online, the producer sends the buffered > messages to the topic, but they are not in order. I receive for example, m2 > then m3 then m1 and then m4. > Why is that ? Is it because the buffering in the producer is multi-threaded > with each producing to the topic at the same time ? > My project code is attached herewith. > I can confirm that I have enabled idempotence. I have also tried with > ```max.in.flight.requests=1``` -- This message was sent by Atlassian Jira (v8.3.4#803005)