Hi Jun, This issue does not only affect flush(). It is just with flush() the probability is much higher. It will affect the following scenario: 1. Producer started and refreshed metadata. 2. User call producer.send() to send message 1 to topic A, but message A is in accumulator. 3. User call producer.send() to send message 2 to topic B (topic B is a new topic, which does not exist in broker) 4. Message 1 will not get sent out until user try to send message to topic A again.
If a flush() is called at this point, it will block forever. Jiangjie (Becket) Qin On 3/24/15, 3:09 PM, "Jun Rao" <j...@confluent.io> wrote: >Hi, Guozhang, > >The flush() was added to the new producer in trunk, not in 0.8.2, right? > >Thanks, > >Jun > >On Tue, Mar 24, 2015 at 2:42 PM, Guozhang Wang <wangg...@gmail.com> wrote: > >> Hello, >> >> We found a serious bug while testing flush() calls in the new producer, >> which is summarized in KAFKA-2042. >> >> In general, when the producer starts up it will try to refresh metadata >> with empty topic list, and hence get all the topic metadata. When >>sending >> the message with some topic later, it will hence not cause the topic to >>be >> added into metadata's topic list since the metadata is available. When >>the >> data is still sitting in the accumulator and a new topic is created, >>that >> will cause metadata refresh with just this single topic, hence losing >>the >> metadata for any other topics. Under usual scenarios the messages will >>be >> sitting in the accumulator until another send() is triggered with the >>same >> topic, but with flush() as a blocking call the likelihood of this issue >> being exposed that messages gets blocked forever inside flush() could be >> largely increased. >> >> I am writing to ask if people think this problem is severe enough that >> requires another bug-fix release. >> >> -- Guozhang >>
