1. If you don't have a good reason to store offsets externally, I would not recommend it, but use the client's built-in mechanism. It will be more work (ie, code you need to write) if you store offsets externally.
For some use-cases, it's beneficial to store the offsets in an external system to get exaclty-once delivery semantics. But it depends on the use case if you can apply this pattern. 2. > "If consumer not call another poll() after max.poll.interval.ms, it will be > remove from consumer group." I think it should be "within" not "after" (can you link to the docs? -- I could not find the snipped you quote) > And the only active approach for consumer to detect failures, is to call > consumer.commit() and try catch CommitFailedException Not sure what you mean by this? Can you elaborate? For the blocking poll() issue, this is addressed via https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior (available in version 2.0) -Matthias On 9/18/18 7:58 PM, ???????????? wrote: > Hi, > > I have two questions about Kafka consumer api: > > > 1. I read the section "Storing Offsets Outside Kafka" in jdoc of > KafkaConsumer. Is it safer or more efficient to manage offset with external > storage compared to manage with Kafka(Zookeeper or topic _consumer_offset)? > Or is there any benefit for system safety or robustness, when network issue > occur or consumer crash? > > > 2. I read the section "Detecting Consumer Failures" in jdoc of > KafkaConsumder, and got bellow points: > - If consumer not call another poll() after max.poll.interval.ms, it will be > remove from consumer group. > - And the only active approach for consumer to detect failures, is to call > consumer.commit() and try catch CommitFailedException . > Sometimes I found that consumer hanging at method poll() when network issue > occur or rebalance failure, and no exception throw from poll(). How do we > detect that failure from consumer side? I refer the old mail "[Kafka-users] > How are rebalance failures raised to consumers?" and it seems no approach. > > > Thanks, > Ruiping Li >
signature.asc
Description: OpenPGP digital signature
