Just to extend what Neha said. The new producer also picks up the new partitions by refreshing the metadata periodically (controlled metadata.max.age.ms). The new producer distributes the data more evenly to all partitions than the old producer.
Thanks, Jun On Tue, Nov 11, 2014 at 11:19 AM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > The new producer is available in 0.8.2-beta (the most recent Kafka > release). The old producer only detects new partitions at an interval > configured by topic.metadata.refresh.interval.ms. This constraint is no > longer true for the new producer and you would likely end up with an even > distribution of data across all partitions. If you want to stay with the > old producer on 0.8.1.1, you can try reducing > topic.metadata.refresh.interval.ms but it may have some performance impact > on the Kafka cluster since it ends up sending topic metadata requests to > the broker at that interval. > > Thanks, > Neha > > On Tue, Nov 11, 2014 at 1:45 AM, Shlomi Hazan <shl...@viber.com> wrote: > > > Neha, I understand that the producer kafka.javaapi.producer.Producer > shown > > in examples is old, > > and that a new producer (org.apache.kafka.clients.producer) is avail? is > it > > available for 0.8.1.1? > > how does it work? does it have a trigger fired when partitions are added > or > > does the producer refresh some cache every some given time period? > > > > Shlomi > > > > > > On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede <neha.narkh...@gmail.com> > > wrote: > > > > > How can I auto refresh keyed producers to use new partitions as these > > > partitions are added? > > > > > > Try using the new producer under org.apache.kafka.clients.producer. > > > > > > Thanks, > > > Neha > > > > > > On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry < > > > mistry.p.bhav...@gmail.com> > > > wrote: > > > > > > > I had different experience with expanding partition for new producer > > and > > > > its impact. I only tried for non-key message. I would always > advice > > > to > > > > keep batch size relatively low or plan for expansion with new java > > > producer > > > > in advance or since inception otherwise running producer code is > > > impacted. > > > > > > > > Here is mail chain: > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3ccaoejijit4cgry97dgzfjkfvaqfduv-o1x1kafefbshgirkm...@mail.gmail.com%3E > > > > > > > > Thanks, > > > > > > > > Bhavesh > > > > > > > > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shl...@viber.com> > > wrote: > > > > > > > > > Hmmm.. > > > > > The Java producer example seems to ignore added partitions too... > > > > > How can I auto refresh keyed producers to use new partitions as > these > > > > > partitions are added? > > > > > > > > > > > > > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shl...@viber.com> > > > wrote: > > > > > > > > > > > One more thing: > > > > > > I saw that the Python client is also unaffected by addition of > > > > partitions > > > > > > to a topic and that it continues to send requests only to the old > > > > > > partitions. > > > > > > is this also handled appropriately by the Java producer? Will he > > see > > > > the > > > > > > change and produce to the new partitions as well? > > > > > > Shlomi > > > > > > > > > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <shl...@viber.com> > > > > wrote: > > > > > > > > > > > >> No I don't see anything like that, the question was aimed at > > > learning > > > > if > > > > > >> it is worthwhile to make the effort of reimplementing the Python > > > > > producer > > > > > >> in Java, I so I will not make all the effort just to be > > disappointed > > > > > >> afterwards. > > > > > >> understand I have nothing to worry about, so I will try to > > simulate > > > > this > > > > > >> situation in small scale... > > > > > >> maybe 3 brokers, one topic with one partition and then add > > > partitions. > > > > > >> we'll see. > > > > > >> thanks for clarifying. > > > > > >> Oh, Good luck with Confluent!! > > > > > >> :) > > > > > >> > > > > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede < > > > > neha.narkh...@gmail.com > > > > > > > > > > > >> wrote: > > > > > >> > > > > > >>> The producer might get an error code if the leader of the > > > partitions > > > > > >>> being > > > > > >>> reassigned also changes. However it should retry and succeed. > Do > > > you > > > > > see > > > > > >>> a > > > > > >>> behavior that suggests otherwise? > > > > > >>> > > > > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan < > shl...@viber.com> > > > > > wrote: > > > > > >>> > > > > > >>> > Hi All, > > > > > >>> > I recently had an issue producing from python where > expanding a > > > > > cluster > > > > > >>> > from 3 to 5 nodes and reassigning partitions forced me to > > restart > > > > the > > > > > >>> > producer b/c of KeyError thrown. > > > > > >>> > Is this situation handled by the Java producer automatically > or > > > > need > > > > > I > > > > > >>> do > > > > > >>> > something to have the java producer refresh itself to see the > > > > > >>> reassigned > > > > > >>> > partition layout and produce away ? > > > > > >>> > Shlomi > > > > > >>> > > > > > > >>> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
