Assuming one can store the offset in an external location (redis/db
etc), along with the rest of the state that a program requires,
wouldn't it be possible to manage things such that, you use the High
Level API with auto commit turned off and do your custom offset
management followed by the
We are developing a new java consumer that addresses the offset reset issue
better.
Thanks,
Jun
On Thu, Feb 19, 2015 at 11:53 PM, Arunkumar Srambikkal (asrambik)
asram...@cisco.com wrote:
If I may use the same thread to discuss the exact same issue
Assuming one can store the offset in
Joel/All,
The SimpleConsumer constructor requires a specific host and port.
Can this be any broker?
If it needs to be a specific broker, for 0.8.2, should this be the offset
coordinator? For 0.8.1, does it matter?
-Suren
On Thursday, February 19, 2015 10:43 AM, Joel Koshy
We are using the High Level Consumer API to interact with Kafka for our normal
use cases.
However, on consumer restart in the case of consumer failures, we want to be
able to manually
reset offsets in certain situations.
And ideally we'd like to use the same api in 0.8.1 and 0.8.2. :-)
It
Not sure what you mean by using the SimpleConsumer on failure
recovery. Can you elaborate on this?
On Thu, Feb 19, 2015 at 03:04:47PM +, Suren wrote:
Haven't used either one now. Sounds like 0.8.2.1 will help.
We are using the High Level Consumer generally but are thinking to use the
I see - yes, you can use the SimpleConsumer for that. However, your
high-level consumers need to be shutdown while you do that (otherwise
they may auto-commit while you are resetting offsets).
Thanks,
Joel
On Thu, Feb 19, 2015 at 03:29:19PM +, Suren wrote:
We are using the High Level
If I may use the same thread to discuss the exact same issue
Assuming one can store the offset in an external location (redis/db etc), along
with the rest of the state that a program requires, wouldn't it be possible to
manage things such that, you use the High Level API with auto commit
Haven't used either one now. Sounds like 0.8.2.1 will help.
We are using the High Level Consumer generally but are thinking to use the
SimpleConsumer on failure recovery to set the offsets.
Is that the recommended approach for this use case?
Thanks.
-Suren
On Thursday, February 19, 2015
Are you using it from Java or Scala? i.e., are you using the
javaapi.SimpleConsumer or kafka.consumer.SimpleConsumer
In 0.8.2 javaapi we explicitly set version 0 of the
OffsetCommitRequest/OffsetFetchRequest which means it will
commit/fetch to/from ZooKeeper only. If you use the scala API you can
Yeah that is a good point - will do the update as part of the doc
changes in KAFKA-1729
On Thu, Feb 19, 2015 at 09:26:30PM -0500, Evan Huus wrote:
On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy jjkosh...@gmail.com wrote:
If you are using v0 of OffsetCommit/FetchRequest then you can issue
that
If you are using v0 of OffsetCommit/FetchRequest then you can issue
that to any broker. For version 0 you will need to issue it to the
coordinator. You can discover the coordinator by sending a
ConsumerMetadataRequest to any broker.
On Thu, Feb 19, 2015 at 07:55:16PM +, Suren wrote:
On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy jjkosh...@gmail.com wrote:
If you are using v0 of OffsetCommit/FetchRequest then you can issue
that to any broker. For version 0 you will need to issue it to the
coordinator. You can discover the coordinator by sending a
ConsumerMetadataRequest to
Hi Chris,
In 0.8.2, the simple consumer Java API supports committing/fetching
offsets that are stored in ZooKeeper. You don't need to issue any
ConsumerMetadataRequest for this. Unfortunately, the API currently
does not support fetching offsets that are stored in Kafka.
Thanks,
Joel
On Mon,
13 matches
Mail list logo
