Neha, I understand that the producer kafka.javaapi.producer.Producer shown
in examples is old,
and that a new producer (org.apache.kafka.clients.producer) is avail? is it
available for 0.8.1.1?
how does it work? does it have a trigger fired when partitions are added or
does the producer refresh
The new producer is available in 0.8.2-beta (the most recent Kafka
release). The old producer only detects new partitions at an interval
configured by topic.metadata.refresh.interval.ms. This constraint is no
longer true for the new producer and you would likely end up with an even
distribution of
Just to extend what Neha said. The new producer also picks up the new
partitions by refreshing the metadata periodically (controlled
metadata.max.age.ms). The new producer distributes the data more evenly to
all partitions than the old producer.
Thanks,
Jun
On Tue, Nov 11, 2014 at 11:19 AM,
Understood.
Thank you guys.
On Wed, Nov 12, 2014 at 4:48 AM, Jun Rao jun...@gmail.com wrote:
Just to extend what Neha said. The new producer also picks up the new
partitions by refreshing the metadata periodically (controlled
metadata.max.age.ms). The new producer distributes the data more
One more thing:
I saw that the Python client is also unaffected by addition of partitions
to a topic and that it continues to send requests only to the old
partitions.
is this also handled appropriately by the Java producer? Will he see the
change and produce to the new partitions as well?
Shlomi
Hmmm..
The Java producer example seems to ignore added partitions too...
How can I auto refresh keyed producers to use new partitions as these
partitions are added?
On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan shl...@viber.com wrote:
One more thing:
I saw that the Python client is also
I had different experience with expanding partition for new producer and
its impact. I only tried for non-key message.I would always advice to
keep batch size relatively low or plan for expansion with new java producer
in advance or since inception otherwise running producer code is impacted.
How can I auto refresh keyed producers to use new partitions as these
partitions are added?
Try using the new producer under org.apache.kafka.clients.producer.
Thanks,
Neha
On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry mistry.p.bhav...@gmail.com
wrote:
I had different experience with
The producer might get an error code if the leader of the partitions being
reassigned also changes. However it should retry and succeed. Do you see a
behavior that suggests otherwise?
On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan shl...@viber.com wrote:
Hi All,
I recently had an issue
No I don't see anything like that, the question was aimed at learning if it
is worthwhile to make the effort of reimplementing the Python producer in
Java, I so I will not make all the effort just to be disappointed
afterwards.
understand I have nothing to worry about, so I will try to simulate
Hi All,
I recently had an issue producing from python where expanding a cluster
from 3 to 5 nodes and reassigning partitions forced me to restart the
producer b/c of KeyError thrown.
Is this situation handled by the Java producer automatically or need I do
something to have the java producer
11 matches
Mail list logo
