Re: How does a consumer know the given partition is removed?
Thanks, folks! It seems like partitionsFor() and listTopics() is what I want. Do we have performance estimates on these 2 API calls, e.g., the time cost of waiting for responses? I would invoke these API along a hot path so I want to have a general idea on how bad it could be. Many thanks to your help! On Thu, Jan 7, 2021 at 1:44 AM Bruno Cadonna wrote: > Hi Luke, > > I am afraid the ConsumerRebalanceListener will not work in this case > since Boyuan assigns the partitions manually. The Java docs you linked > state > > If the consumer directly assigns partitions, those partitions will never > be reassigned and this callback is not applicable. > > > Hi Boyuan, > > The consumer has methods partitionsFor() and listTopics(). Probably > there is a better way to get the information you want that I am not > aware of. > > Best, > Bruno > > On 07.01.21 05:09, Luke Chen wrote: > > Hi Boyuan, > > You can create a *ConsumerRebalanceListener* and do something you want > when > > *onPartitionsRevoked. * > > Please check this java doc for more information: > > > https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html > > > > Thanks. > > Luke > > > > On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang wrote: > > > >> Hi team, > >> > >> I'm working on a long run application, which uses the Kafka Consumer > API to > >> poll messages from a given topic and partition. I'm assigning the topic > and > >> partition manually by using consumer.assign() API and polling messages > by > >> using consumer.poll(). > >> > >> One common scenario for my application is that certain partitions could > be > >> removed outside of my application and my application needs to know one > >> partition has been removed to stop processing that partition. My > question > >> is that is there any way to get the removal information when I do > >> consumer.assign() or consumer.poll() or any APIs that I can use? > >> > >> Thanks for your help! > >> > > >
How does a consumer know the given partition is removed?
Hi team, I'm working on a long run application, which uses the Kafka Consumer API to poll messages from a given topic and partition. I'm assigning the topic and partition manually by using consumer.assign() API and polling messages by using consumer.poll(). One common scenario for my application is that certain partitions could be removed outside of my application and my application needs to know one partition has been removed to stop processing that partition. My question is that is there any way to get the removal information when I do consumer.assign() or consumer.poll() or any APIs that I can use? Thanks for your help!
Fwd: Is committing offset required for Consumer
Add dev mailing list as well. Thanks for your help! -- Forwarded message - From: Boyuan Zhang Date: Thu, May 7, 2020 at 3:51 PM Subject: Is committing offset required for Consumer To: Hi team, I'm building an application which uses Kafka Consumer APIs to read messages from topics. I plan to manually assign TopicPartitions to my consumer and seek a certain offset before starting to read. I'll also materialize the last read offset and reuse it when creating the consumer later. Within my usage, I' curious whether I need to commit offset automatically or manually. While going through the doc, it seems like committing offset is only important to dynamic assignment. Another question around manual assignment is, is it still true that I need to call poll() continuously to keep the consumer in the group described as below? > It is also possible that the consumer could encounter a "livelock" > situation where it is continuing to send heartbeats, but no progress is > being made. To prevent the consumer from holding onto its partitions > indefinitely in this case, we provide a liveness detection mechanism using > the max.poll.interval.ms setting. Basically if you don't call poll at > least as frequently as the configured max interval, then the client will > proactively leave the group so that another consumer can take over its > partitions. When this happens, you may see an offset commit failure (as > indicated by a CommitFailedException > <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/CommitFailedException.html> > thrown > from a call to commitSync() > <https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#commitSync-->). > This is a safety mechanism which guarantees that only active members of the > group are able to commit offsets. So to stay in the group, you must > continue to call poll. What will happen to poll() with manual assignment if the consumer is removed from the group? Thanks for your help! Boyuan
