Hi team, I'm building an application which uses Kafka Consumer APIs to read messages from topics. I plan to manually assign TopicPartitions to my consumer and seek a certain offset before starting to read. I'll also materialize the last read offset and reuse it when creating the consumer later.
Within my usage, I' curious whether I need to commit offset automatically or manually. While going through the doc, it seems like committing offset is only important to dynamic assignment. Another question around manual assignment is, is it still true that I need to call poll() continuously to keep the consumer in the group described as below? > It is also possible that the consumer could encounter a "livelock" > situation where it is continuing to send heartbeats, but no progress is > being made. To prevent the consumer from holding onto its partitions > indefinitely in this case, we provide a liveness detection mechanism using > the max.poll.interval.ms setting. Basically if you don't call poll at > least as frequently as the configured max interval, then the client will > proactively leave the group so that another consumer can take over its > partitions. When this happens, you may see an offset commit failure (as > indicated by a CommitFailedException > <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html> > thrown > from a call to commitSync() > <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-->). > This is a safety mechanism which guarantees that only active members of the > group are able to commit offsets. So to stay in the group, you must > continue to call poll. What will happen to poll() with manual assignment if the consumer is removed from the group? Thanks for your help! Boyuan
