I had different experience with expanding partition for new producer and its impact. I only tried for non-key message. I would always advice to keep batch size relatively low or plan for expansion with new java producer in advance or since inception otherwise running producer code is impacted.
Here is mail chain: http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3ccaoejijit4cgry97dgzfjkfvaqfduv-o1x1kafefbshgirkm...@mail.gmail.com%3E Thanks, Bhavesh On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shl...@viber.com> wrote: > Hmmm.. > The Java producer example seems to ignore added partitions too... > How can I auto refresh keyed producers to use new partitions as these > partitions are added? > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shl...@viber.com> wrote: > > > One more thing: > > I saw that the Python client is also unaffected by addition of partitions > > to a topic and that it continues to send requests only to the old > > partitions. > > is this also handled appropriately by the Java producer? Will he see the > > change and produce to the new partitions as well? > > Shlomi > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <shl...@viber.com> wrote: > > > >> No I don't see anything like that, the question was aimed at learning if > >> it is worthwhile to make the effort of reimplementing the Python > producer > >> in Java, I so I will not make all the effort just to be disappointed > >> afterwards. > >> understand I have nothing to worry about, so I will try to simulate this > >> situation in small scale... > >> maybe 3 brokers, one topic with one partition and then add partitions. > >> we'll see. > >> thanks for clarifying. > >> Oh, Good luck with Confluent!! > >> :) > >> > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede <neha.narkh...@gmail.com > > > >> wrote: > >> > >>> The producer might get an error code if the leader of the partitions > >>> being > >>> reassigned also changes. However it should retry and succeed. Do you > see > >>> a > >>> behavior that suggests otherwise? > >>> > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan <shl...@viber.com> > wrote: > >>> > >>> > Hi All, > >>> > I recently had an issue producing from python where expanding a > cluster > >>> > from 3 to 5 nodes and reassigning partitions forced me to restart the > >>> > producer b/c of KeyError thrown. > >>> > Is this situation handled by the Java producer automatically or need > I > >>> do > >>> > something to have the java producer refresh itself to see the > >>> reassigned > >>> > partition layout and produce away ? > >>> > Shlomi > >>> > > >>> > >> > >> > > >
