Thanks Jason,
That's definitely something I can work with. I
expect this to be very rare scenario.
Thanks for your help
Michael
On Mon, Mar 14, 2016 at 5:16 PM, Jason Gustafson wrote:
> Hey Michael,
>
> I don't think a policy of retrying indefinitely is generally possib
Awesome, thanks! I'll take a look!
On Mon, Mar 14, 2016 at 11:27 AM, Jason Gustafson
wrote:
> Yeah, that's the idea. Here's the JIRA I was thinking of:
> https://issues.apache.org/jira/browse/KAFKA-2273. I'm guessing this will
> need a KIP after 0.10 is out.
>
> -Jason
>
> On Mon, Mar 14, 2016
Yeah, that's the idea. Here's the JIRA I was thinking of:
https://issues.apache.org/jira/browse/KAFKA-2273. I'm guessing this will
need a KIP after 0.10 is out.
-Jason
On Mon, Mar 14, 2016 at 11:21 AM, Christian Posta wrote:
> Jason,
>
> Can you link to the proposal so I can take a look? Would
Jason,
Can you link to the proposal so I can take a look? Would the "sticky"
proposal prefer to keep partitions assigned to consumers who currently have
them and have not failed?
On Mon, Mar 14, 2016 at 10:16 AM, Jason Gustafson
wrote:
> Hey Michael,
>
> I don't think a policy of retrying indef
Hey Michael,
I don't think a policy of retrying indefinitely is generally possible with
the new consumer even if you had a heartbeat API. The problem is that the
consumer itself doesn't control when the group needs to rebalance. If
another consumer joins or leaves the group, then all consumers wil
Thanks Christian,
Sending a heartbeat without having to poll would
also be useful when using a large max.partition.fetch.bytes.
For now I'm just going to shut the consumer down and restart after x period of
time.
Thanks for your insights.
Michael
> On 10 Mar 201
Yah that's a good point. That was brought up in another thread.
The granularity of what poll() needs to be addressed. It tries to do too
many things at once, including heartbeating. Not so sure that's entirely
necessary.
On Thu, Mar 10, 2016 at 1:40 AM, Michael Freeman
wrote:
> Thanks Christian
Thanks Christian,
We would want to retry indefinitely. Or at least
for say x minutes. If we don't poll how do we keep the heart beat alive to
Kafka. We never want to loose this message and only want to commit to Kafka
when the message is in Mongo. That's either as
So can you have to decide how long you're willing to "wait" for the mongo
db to come back, and what you'd like to do with that message. So for
example, do you just retry inserting to Mongo for a predefined period of
time? Do you try forever? If you try forever, are you okay with the
consumer thread
Hey,
My team is new to Kafka and we are using the examples found at.
http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
We process messages from kafka and persist them to Mongo.
If Mongo is unavailable we are wondering how we can re-consume
10 matches
Mail list logo
