Thanks Ian.
Is your consumer multi-threaded? If so can you share how you coordinated
each of the threads so you knew it was 'okay' to commit across all the
threads? I'm stuck on how to do this without really complicating the
consumer.
Thanks,
Chris
On Tue, Jul 9, 2013 at 5:51 PM, Ian Friedman
Hey Chris,
The way I handled this in my application using the High Level Consumer was to
turn off auto-commit and commit manually after finishing a batch of messages
(obviously you could do it after every message, but for my purposes it was
better to have batches)
--
Ian Friedman
On Tuesd
Enhancement submitted: https://issues.apache.org/jira/browse/KAFKA-966
On Tue, Jul 9, 2013 at 3:53 PM, Chris Curtin wrote:
> Thanks. I know I can write a SimpleConsumer to do this, but it feels like
> the High Level consumer is _so_ close to being robust enough tohandle
> what I'd think pe
Thanks. I know I can write a SimpleConsumer to do this, but it feels like
the High Level consumer is _so_ close to being robust enough tohandle
what I'd think people want to do in most applications. I'm going to submit
an enhancement request.
I'm trying to understand the level of data loss in
OK.
It sounds like you're requesting functionality that the high-level consumer
simply doesn't have. As I am sure you know, there is no API call that
supports "handing back a message".
I might be missing something, but if you need this kind of control, I think
you need to code your application di
Hi Philip,
Correct, I don't want to explicitly control the offset committing. The
ConsumerConnector handles that well enough except for when I want to
shutdown and NOT have Kafka think I consumed that last message for a
stream. This isn't the crash case, it is a case where the logic consuming
the
It seems like you're not explicitly controlling the offsets. Is that
correct?
If so, the moment you pull a message from the stream, the client framework
considers it processed. So if your app subsequently crashes before the
message is fully processed, and "auto-commit" updates the offsets in
Zooke
Hi,
I'm working through a production-level High Level Consumer app and have a
couple of error/shutdown questions to understand how the offset storage is
handled.
Test case - simulate an error writing to destination application, for
example a database, offset is 'lost'
Scenario
- write 500 messag
