Understood. Thank you guys. On Wed, Nov 12, 2014 at 4:48 AM, Jun Rao <jun...@gmail.com> wrote:
> Just to extend what Neha said. The new producer also picks up the new > partitions by refreshing the metadata periodically (controlled > metadata.max.age.ms). The new producer distributes the data more evenly to > all partitions than the old producer. > > Thanks, > > Jun > > On Tue, Nov 11, 2014 at 11:19 AM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > > > The new producer is available in 0.8.2-beta (the most recent Kafka > > release). The old producer only detects new partitions at an interval > > configured by topic.metadata.refresh.interval.ms. This constraint is no > > longer true for the new producer and you would likely end up with an even > > distribution of data across all partitions. If you want to stay with the > > old producer on 0.8.1.1, you can try reducing > > topic.metadata.refresh.interval.ms but it may have some performance > impact > > on the Kafka cluster since it ends up sending topic metadata requests to > > the broker at that interval. > > > > Thanks, > > Neha > > > > On Tue, Nov 11, 2014 at 1:45 AM, Shlomi Hazan <shl...@viber.com> wrote: > > > > > Neha, I understand that the producer kafka.javaapi.producer.Producer > > shown > > > in examples is old, > > > and that a new producer (org.apache.kafka.clients.producer) is avail? > is > > it > > > available for 0.8.1.1? > > > how does it work? does it have a trigger fired when partitions are > added > > or > > > does the producer refresh some cache every some given time period? > > > > > > Shlomi > > > > > > > > > On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede < > neha.narkh...@gmail.com> > > > wrote: > > > > > > > How can I auto refresh keyed producers to use new partitions as these > > > > partitions are added? > > > > > > > > Try using the new producer under org.apache.kafka.clients.producer. > > > > > > > > Thanks, > > > > Neha > > > > > > > > On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry < > > > > mistry.p.bhav...@gmail.com> > > > > wrote: > > > > > > > > > I had different experience with expanding partition for new > producer > > > and > > > > > its impact. I only tried for non-key message. I would always > > advice > > > > to > > > > > keep batch size relatively low or plan for expansion with new java > > > > producer > > > > > in advance or since inception otherwise running producer code is > > > > impacted. > > > > > > > > > > Here is mail chain: > > > > > > > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3ccaoejijit4cgry97dgzfjkfvaqfduv-o1x1kafefbshgirkm...@mail.gmail.com%3E > > > > > > > > > > Thanks, > > > > > > > > > > Bhavesh > > > > > > > > > > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shl...@viber.com> > > > wrote: > > > > > > > > > > > Hmmm.. > > > > > > The Java producer example seems to ignore added partitions too... > > > > > > How can I auto refresh keyed producers to use new partitions as > > these > > > > > > partitions are added? > > > > > > > > > > > > > > > > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shl...@viber.com > > > > > > wrote: > > > > > > > > > > > > > One more thing: > > > > > > > I saw that the Python client is also unaffected by addition of > > > > > partitions > > > > > > > to a topic and that it continues to send requests only to the > old > > > > > > > partitions. > > > > > > > is this also handled appropriately by the Java producer? Will > he > > > see > > > > > the > > > > > > > change and produce to the new partitions as well? > > > > > > > Shlomi > > > > > > > > > > > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan < > shl...@viber.com> > > > > > wrote: > > > > > > > > > > > > > >> No I don't see anything like that, the question was aimed at > > > > learning > > > > > if > > > > > > >> it is worthwhile to make the effort of reimplementing the > Python > > > > > > producer > > > > > > >> in Java, I so I will not make all the effort just to be > > > disappointed > > > > > > >> afterwards. > > > > > > >> understand I have nothing to worry about, so I will try to > > > simulate > > > > > this > > > > > > >> situation in small scale... > > > > > > >> maybe 3 brokers, one topic with one partition and then add > > > > partitions. > > > > > > >> we'll see. > > > > > > >> thanks for clarifying. > > > > > > >> Oh, Good luck with Confluent!! > > > > > > >> :) > > > > > > >> > > > > > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede < > > > > > neha.narkh...@gmail.com > > > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > >>> The producer might get an error code if the leader of the > > > > partitions > > > > > > >>> being > > > > > > >>> reassigned also changes. However it should retry and succeed. > > Do > > > > you > > > > > > see > > > > > > >>> a > > > > > > >>> behavior that suggests otherwise? > > > > > > >>> > > > > > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan < > > shl...@viber.com> > > > > > > wrote: > > > > > > >>> > > > > > > >>> > Hi All, > > > > > > >>> > I recently had an issue producing from python where > > expanding a > > > > > > cluster > > > > > > >>> > from 3 to 5 nodes and reassigning partitions forced me to > > > restart > > > > > the > > > > > > >>> > producer b/c of KeyError thrown. > > > > > > >>> > Is this situation handled by the Java producer > automatically > > or > > > > > need > > > > > > I > > > > > > >>> do > > > > > > >>> > something to have the java producer refresh itself to see > the > > > > > > >>> reassigned > > > > > > >>> > partition layout and produce away ? > > > > > > >>> > Shlomi > > > > > > >>> > > > > > > > >>> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
