Re: How does a consumer know the given partition is removed?
Hi Bruno, Thanks for the update! Hi Boyuan, For listTopics() method, it'll *always* do a remote call, which will have performance impact for sure. For partitionsFor() method, it'll *check cache first*, if not found in cache, then do a remote call to retrieve the topic partition info. So, I think partitionsFor() should be a better option for you. Thanks. Luke On Fri, Jan 8, 2021 at 2:38 AM Boyuan Zhang wrote: > Thanks, folks! > > It seems like partitionsFor() and listTopics() is what I want. Do we have > performance estimates on these 2 API calls, e.g., the time cost of waiting > for responses? I would invoke these API along a hot path so I want to have > a general idea on how bad it could be. > > Many thanks to your help! > > On Thu, Jan 7, 2021 at 1:44 AM Bruno Cadonna wrote: > > > Hi Luke, > > > > I am afraid the ConsumerRebalanceListener will not work in this case > > since Boyuan assigns the partitions manually. The Java docs you linked > > state > > > > If the consumer directly assigns partitions, those partitions will never > > be reassigned and this callback is not applicable. > > > > > > Hi Boyuan, > > > > The consumer has methods partitionsFor() and listTopics(). Probably > > there is a better way to get the information you want that I am not > > aware of. > > > > Best, > > Bruno > > > > On 07.01.21 05:09, Luke Chen wrote: > > > Hi Boyuan, > > > You can create a *ConsumerRebalanceListener* and do something you want > > when > > > *onPartitionsRevoked. * > > > Please check this java doc for more information: > > > > > > https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html > > > > > > Thanks. > > > Luke > > > > > > On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang > wrote: > > > > > >> Hi team, > > >> > > >> I'm working on a long run application, which uses the Kafka Consumer > > API to > > >> poll messages from a given topic and partition. I'm assigning the > topic > > and > > >> partition manually by using consumer.assign() API and polling messages > > by > > >> using consumer.poll(). > > >> > > >> One common scenario for my application is that certain partitions > could > > be > > >> removed outside of my application and my application needs to know one > > >> partition has been removed to stop processing that partition. My > > question > > >> is that is there any way to get the removal information when I do > > >> consumer.assign() or consumer.poll() or any APIs that I can use? > > >> > > >> Thanks for your help! > > >> > > > > > >
Re: How does a consumer know the given partition is removed?
Thanks, folks! It seems like partitionsFor() and listTopics() is what I want. Do we have performance estimates on these 2 API calls, e.g., the time cost of waiting for responses? I would invoke these API along a hot path so I want to have a general idea on how bad it could be. Many thanks to your help! On Thu, Jan 7, 2021 at 1:44 AM Bruno Cadonna wrote: > Hi Luke, > > I am afraid the ConsumerRebalanceListener will not work in this case > since Boyuan assigns the partitions manually. The Java docs you linked > state > > If the consumer directly assigns partitions, those partitions will never > be reassigned and this callback is not applicable. > > > Hi Boyuan, > > The consumer has methods partitionsFor() and listTopics(). Probably > there is a better way to get the information you want that I am not > aware of. > > Best, > Bruno > > On 07.01.21 05:09, Luke Chen wrote: > > Hi Boyuan, > > You can create a *ConsumerRebalanceListener* and do something you want > when > > *onPartitionsRevoked. * > > Please check this java doc for more information: > > > https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html > > > > Thanks. > > Luke > > > > On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang wrote: > > > >> Hi team, > >> > >> I'm working on a long run application, which uses the Kafka Consumer > API to > >> poll messages from a given topic and partition. I'm assigning the topic > and > >> partition manually by using consumer.assign() API and polling messages > by > >> using consumer.poll(). > >> > >> One common scenario for my application is that certain partitions could > be > >> removed outside of my application and my application needs to know one > >> partition has been removed to stop processing that partition. My > question > >> is that is there any way to get the removal information when I do > >> consumer.assign() or consumer.poll() or any APIs that I can use? > >> > >> Thanks for your help! > >> > > >
Re: How does a consumer know the given partition is removed?
Hi Luke, I am afraid the ConsumerRebalanceListener will not work in this case since Boyuan assigns the partitions manually. The Java docs you linked state If the consumer directly assigns partitions, those partitions will never be reassigned and this callback is not applicable. Hi Boyuan, The consumer has methods partitionsFor() and listTopics(). Probably there is a better way to get the information you want that I am not aware of. Best, Bruno On 07.01.21 05:09, Luke Chen wrote: Hi Boyuan, You can create a *ConsumerRebalanceListener* and do something you want when *onPartitionsRevoked. * Please check this java doc for more information: https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html Thanks. Luke On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang wrote: Hi team, I'm working on a long run application, which uses the Kafka Consumer API to poll messages from a given topic and partition. I'm assigning the topic and partition manually by using consumer.assign() API and polling messages by using consumer.poll(). One common scenario for my application is that certain partitions could be removed outside of my application and my application needs to know one partition has been removed to stop processing that partition. My question is that is there any way to get the removal information when I do consumer.assign() or consumer.poll() or any APIs that I can use? Thanks for your help!
Re: How does a consumer know the given partition is removed?
Hi Boyuan, You can create a *ConsumerRebalanceListener* and do something you want when *onPartitionsRevoked. * Please check this java doc for more information: https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html Thanks. Luke On Thu, Jan 7, 2021 at 8:45 AM Boyuan Zhang wrote: > Hi team, > > I'm working on a long run application, which uses the Kafka Consumer API to > poll messages from a given topic and partition. I'm assigning the topic and > partition manually by using consumer.assign() API and polling messages by > using consumer.poll(). > > One common scenario for my application is that certain partitions could be > removed outside of my application and my application needs to know one > partition has been removed to stop processing that partition. My question > is that is there any way to get the removal information when I do > consumer.assign() or consumer.poll() or any APIs that I can use? > > Thanks for your help! >
How does a consumer know the given partition is removed?
Hi team, I'm working on a long run application, which uses the Kafka Consumer API to poll messages from a given topic and partition. I'm assigning the topic and partition manually by using consumer.assign() API and polling messages by using consumer.poll(). One common scenario for my application is that certain partitions could be removed outside of my application and my application needs to know one partition has been removed to stop processing that partition. My question is that is there any way to get the removal information when I do consumer.assign() or consumer.poll() or any APIs that I can use? Thanks for your help!
