Understood.
Thank you guys.
On Wed, Nov 12, 2014 at 4:48 AM, Jun Rao wrote:
> Just to extend what Neha said. The new producer also picks up the new
> partitions by refreshing the metadata periodically (controlled
> metadata.max.age.ms). The new producer distributes the data more evenly to
> all
Just to extend what Neha said. The new producer also picks up the new
partitions by refreshing the metadata periodically (controlled
metadata.max.age.ms). The new producer distributes the data more evenly to
all partitions than the old producer.
Thanks,
Jun
On Tue, Nov 11, 2014 at 11:19 AM, Neha
The new producer is available in 0.8.2-beta (the most recent Kafka
release). The old producer only detects new partitions at an interval
configured by topic.metadata.refresh.interval.ms. This constraint is no
longer true for the new producer and you would likely end up with an even
distribution of
Neha, I understand that the producer kafka.javaapi.producer.Producer shown
in examples is old,
and that a new producer (org.apache.kafka.clients.producer) is avail? is it
available for 0.8.1.1?
how does it work? does it have a trigger fired when partitions are added or
does the producer refresh som
How can I auto refresh keyed producers to use new partitions as these
partitions are added?
Try using the new producer under org.apache.kafka.clients.producer.
Thanks,
Neha
On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry
wrote:
> I had different experience with expanding partition for new prod
I had different experience with expanding partition for new producer and
its impact. I only tried for non-key message.I would always advice to
keep batch size relatively low or plan for expansion with new java producer
in advance or since inception otherwise running producer code is impacted.
Hmmm..
The Java producer example seems to ignore added partitions too...
How can I auto refresh keyed producers to use new partitions as these
partitions are added?
On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan wrote:
> One more thing:
> I saw that the Python client is also unaffected by addit
One more thing:
I saw that the Python client is also unaffected by addition of partitions
to a topic and that it continues to send requests only to the old
partitions.
is this also handled appropriately by the Java producer? Will he see the
change and produce to the new partitions as well?
Shlomi
No I don't see anything like that, the question was aimed at learning if it
is worthwhile to make the effort of reimplementing the Python producer in
Java, I so I will not make all the effort just to be disappointed
afterwards.
understand I have nothing to worry about, so I will try to simulate thi
The producer might get an error code if the leader of the partitions being
reassigned also changes. However it should retry and succeed. Do you see a
behavior that suggests otherwise?
On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan wrote:
> Hi All,
> I recently had an issue producing from python w
Hi All,
I recently had an issue producing from python where expanding a cluster
from 3 to 5 nodes and reassigning partitions forced me to restart the
producer b/c of KeyError thrown.
Is this situation handled by the Java producer automatically or need I do
something to have the java producer refres
11 matches
Mail list logo
