Neha, I understand that the producer kafka.javaapi.producer.Producer shown in examples is old, and that a new producer (org.apache.kafka.clients.producer) is avail? is it available for 0.8.1.1? how does it work? does it have a trigger fired when partitions are added or does the producer refresh some cache every some given time period?
Shlomi On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > How can I auto refresh keyed producers to use new partitions as these > partitions are added? > > Try using the new producer under org.apache.kafka.clients.producer. > > Thanks, > Neha > > On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > I had different experience with expanding partition for new producer and > > its impact. I only tried for non-key message. I would always advice > to > > keep batch size relatively low or plan for expansion with new java > producer > > in advance or since inception otherwise running producer code is > impacted. > > > > Here is mail chain: > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3ccaoejijit4cgry97dgzfjkfvaqfduv-o1x1kafefbshgirkm...@mail.gmail.com%3E > > > > Thanks, > > > > Bhavesh > > > > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shl...@viber.com> wrote: > > > > > Hmmm.. > > > The Java producer example seems to ignore added partitions too... > > > How can I auto refresh keyed producers to use new partitions as these > > > partitions are added? > > > > > > > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shl...@viber.com> > wrote: > > > > > > > One more thing: > > > > I saw that the Python client is also unaffected by addition of > > partitions > > > > to a topic and that it continues to send requests only to the old > > > > partitions. > > > > is this also handled appropriately by the Java producer? Will he see > > the > > > > change and produce to the new partitions as well? > > > > Shlomi > > > > > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <shl...@viber.com> > > wrote: > > > > > > > >> No I don't see anything like that, the question was aimed at > learning > > if > > > >> it is worthwhile to make the effort of reimplementing the Python > > > producer > > > >> in Java, I so I will not make all the effort just to be disappointed > > > >> afterwards. > > > >> understand I have nothing to worry about, so I will try to simulate > > this > > > >> situation in small scale... > > > >> maybe 3 brokers, one topic with one partition and then add > partitions. > > > >> we'll see. > > > >> thanks for clarifying. > > > >> Oh, Good luck with Confluent!! > > > >> :) > > > >> > > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede < > > neha.narkh...@gmail.com > > > > > > > >> wrote: > > > >> > > > >>> The producer might get an error code if the leader of the > partitions > > > >>> being > > > >>> reassigned also changes. However it should retry and succeed. Do > you > > > see > > > >>> a > > > >>> behavior that suggests otherwise? > > > >>> > > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan <shl...@viber.com> > > > wrote: > > > >>> > > > >>> > Hi All, > > > >>> > I recently had an issue producing from python where expanding a > > > cluster > > > >>> > from 3 to 5 nodes and reassigning partitions forced me to restart > > the > > > >>> > producer b/c of KeyError thrown. > > > >>> > Is this situation handled by the Java producer automatically or > > need > > > I > > > >>> do > > > >>> > something to have the java producer refresh itself to see the > > > >>> reassigned > > > >>> > partition layout and produce away ? > > > >>> > Shlomi > > > >>> > > > > >>> > > > >> > > > >> > > > > > > > > > >
