high level consumer rollback

[Luiz Geovani Vier]

Hello,

I'm using the high level consumer with auto-commit disabled and a
single thread per consumer, in order to consume messages in batches.
In case of failures on the database, I'd like to stop processing,
rollback and restart from the last commited offset.
Is there a way to receive the messages since the last commit again,
without reconnecting? (something like a reset on KafkaStream or
ConsumerIterator)

Thanks,
-Geovani

Re: high level consumer rollback

[Mayuresh Gharat]

As per my knowledge, I don't think we you can do that with an online
stream. You will have to reset the offsets to a particular offset in the
past to start consuming from that. Another way would be start a separate
consumer with different groupId.

In any case you cannot consume from past offset without bouncing the
consumer I think. Once you bounce the consumer it will start consuming from
the last committed offset.

Thanks,

Mayuresh

On Wed, Mar 4, 2015 at 12:04 PM, Luiz Geovani Vier lgv...@gmail.com wrote:

 Hello,

 I'm using the high level consumer with auto-commit disabled and a
 single thread per consumer, in order to consume messages in batches.
 In case of failures on the database, I'd like to stop processing,
 rollback and restart from the last commited offset.
 Is there a way to receive the messages since the last commit again,
 without reconnecting? (something like a reset on KafkaStream or
 ConsumerIterator)

 Thanks,
 -Geovani




-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: high level consumer rollback

[Joel Koshy]

This is not possible with the current high-level consumer without a
restart, but the new consumer (under development) does have support
for this.

On Wed, Mar 04, 2015 at 03:04:57PM -0500, Luiz Geovani Vier wrote:
 Hello,
 
 I'm using the high level consumer with auto-commit disabled and a
 single thread per consumer, in order to consume messages in batches.
 In case of failures on the database, I'd like to stop processing,
 rollback and restart from the last commited offset.
 Is there a way to receive the messages since the last commit again,
 without reconnecting? (something like a reset on KafkaStream or
 ConsumerIterator)
 
 Thanks,
 -Geovani

Re: high level consumer rollback

[Luiz Geovani Vier]

Thanks, Mayuresh and Joel. Reconnecting works just fine, although it's
much more complex than just calling rollback(), so I'm looking forward
to the new version :)

-Geovani


On Wed, Mar 4, 2015 at 4:57 PM, Joel Koshy jjkosh...@gmail.com wrote:
 This is not possible with the current high-level consumer without a
 restart, but the new consumer (under development) does have support
 for this.

 On Wed, Mar 04, 2015 at 03:04:57PM -0500, Luiz Geovani Vier wrote:
 Hello,

 I'm using the high level consumer with auto-commit disabled and a
 single thread per consumer, in order to consume messages in batches.
 In case of failures on the database, I'd like to stop processing,
 rollback and restart from the last commited offset.
 Is there a way to receive the messages since the last commit again,
 without reconnecting? (something like a reset on KafkaStream or
 ConsumerIterator)

 Thanks,
 -Geovani