Re: Should the mock consumer call the consumer rebalance listener on rebalance?
I have also created the pull request for the behaviour change. https://github.com/apache/kafka/pull/13455 Thanks again, Dan On Thu, Mar 23, 2023 at 11:40 PM Dan S wrote: > I have created the jira, the PR will follow within the next few days. > https://issues.apache.org/jira/browse/KAFKA-14841 > Thanks again > > On Thu, Mar 23, 2023 at 7:33 PM Philip Nee wrote: > >> Awesome! >> >> On Thu, Mar 23, 2023 at 12:31 PM Dan S wrote: >> >> > I'll pick it up, thanks! >> > >> > On Thu, Mar 23, 2023, 19:27 Philip Nee wrote: >> > >> > > Hey Dan, >> > > >> > > Your analysis looks right. I do see TODO item there to implement the >> > > rebalance callback. Would you like to create a jira issue and work on >> > that? >> > > >> > > Thanks, >> > > P >> > > >> > > On Thu, Mar 23, 2023 at 12:11 PM Dan S wrote: >> > > >> > > > Hi Philip, >> > > > >> > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our >> case >> > > > we're calling the variant of subscribe that takes a custom >> > > > ConsumerRebalanceListener(which among other things logs when it's >> > > called), >> > > > and we're then calling rebalance (to simulate a rebalance) and >> removing >> > > all >> > > > partitions from the consumer, polling a few times, and then adding >> them >> > > > back. We're noticing our custom listener is never called, which was >> > > > unexpected, but based on the code analysis in the original email >> seems >> > to >> > > > be the current implementation. The question is whether this is >> indeed >> > > > desired behavior. >> > > > >> > > > On Thu, Mar 23, 2023, 18:17 Philip Nee wrote: >> > > > >> > > > > Hey Dan, >> > > > > >> > > > > Thanks for looking into this. Are you talking about >> MockConsumer? If >> > > you >> > > > > invoke subscribe(Collection topics), it actually >> registers a >> > > Noop >> > > > > callback. Perhaps this is what you are seeing? >> > > > > >> > > > > P >> > > > > >> > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S >> > wrote: >> > > > > >> > > > > > Hello all, >> > > > > > >> > > > > > It seems to me based on reading the code, that the consumer >> > rebalance >> > > > > > listener that is passed into the mock consumer when subscribing >> to >> > a >> > > > > topic >> > > > > > is not actually called when a rebalance is simulated. My >> > > understanding >> > > > is >> > > > > > that the consumer rebalance listener is called from the consumer >> > > > > > coordinator, which is called by kafka consumer. The mock >> consumer >> > > > doesn't >> > > > > > seem to use the consumer coordinator or use any other mechanism >> to >> > > call >> > > > > the >> > > > > > consumer rebalance listener. Is my understanding correct? Would >> it >> > > make >> > > > > > sense to trigger the consumer rebalance listener when rebalance >> is >> > > > > called? >> > > > > > >> > > > > > I would be willing to try to make the patch if the behavior is >> > indeed >> > > > > > currently incorrect/incomplete. >> > > > > > >> > > > > > Thanks, >> > > > > > >> > > > > > Daniel >> > > > > > >> > > > > >> > > > >> > > >> > >> >
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
I have created the jira, the PR will follow within the next few days. https://issues.apache.org/jira/browse/KAFKA-14841 Thanks again On Thu, Mar 23, 2023 at 7:33 PM Philip Nee wrote: > Awesome! > > On Thu, Mar 23, 2023 at 12:31 PM Dan S wrote: > > > I'll pick it up, thanks! > > > > On Thu, Mar 23, 2023, 19:27 Philip Nee wrote: > > > > > Hey Dan, > > > > > > Your analysis looks right. I do see TODO item there to implement the > > > rebalance callback. Would you like to create a jira issue and work on > > that? > > > > > > Thanks, > > > P > > > > > > On Thu, Mar 23, 2023 at 12:11 PM Dan S wrote: > > > > > > > Hi Philip, > > > > > > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our > case > > > > we're calling the variant of subscribe that takes a custom > > > > ConsumerRebalanceListener(which among other things logs when it's > > > called), > > > > and we're then calling rebalance (to simulate a rebalance) and > removing > > > all > > > > partitions from the consumer, polling a few times, and then adding > them > > > > back. We're noticing our custom listener is never called, which was > > > > unexpected, but based on the code analysis in the original email > seems > > to > > > > be the current implementation. The question is whether this is indeed > > > > desired behavior. > > > > > > > > On Thu, Mar 23, 2023, 18:17 Philip Nee wrote: > > > > > > > > > Hey Dan, > > > > > > > > > > Thanks for looking into this. Are you talking about MockConsumer? > If > > > you > > > > > invoke subscribe(Collection topics), it actually registers > a > > > Noop > > > > > callback. Perhaps this is what you are seeing? > > > > > > > > > > P > > > > > > > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > It seems to me based on reading the code, that the consumer > > rebalance > > > > > > listener that is passed into the mock consumer when subscribing > to > > a > > > > > topic > > > > > > is not actually called when a rebalance is simulated. My > > > understanding > > > > is > > > > > > that the consumer rebalance listener is called from the consumer > > > > > > coordinator, which is called by kafka consumer. The mock consumer > > > > doesn't > > > > > > seem to use the consumer coordinator or use any other mechanism > to > > > call > > > > > the > > > > > > consumer rebalance listener. Is my understanding correct? Would > it > > > make > > > > > > sense to trigger the consumer rebalance listener when rebalance > is > > > > > called? > > > > > > > > > > > > I would be willing to try to make the patch if the behavior is > > indeed > > > > > > currently incorrect/incomplete. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Daniel > > > > > > > > > > > > > > > > > > > > >
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Awesome! On Thu, Mar 23, 2023 at 12:31 PM Dan S wrote: > I'll pick it up, thanks! > > On Thu, Mar 23, 2023, 19:27 Philip Nee wrote: > > > Hey Dan, > > > > Your analysis looks right. I do see TODO item there to implement the > > rebalance callback. Would you like to create a jira issue and work on > that? > > > > Thanks, > > P > > > > On Thu, Mar 23, 2023 at 12:11 PM Dan S wrote: > > > > > Hi Philip, > > > > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case > > > we're calling the variant of subscribe that takes a custom > > > ConsumerRebalanceListener(which among other things logs when it's > > called), > > > and we're then calling rebalance (to simulate a rebalance) and removing > > all > > > partitions from the consumer, polling a few times, and then adding them > > > back. We're noticing our custom listener is never called, which was > > > unexpected, but based on the code analysis in the original email seems > to > > > be the current implementation. The question is whether this is indeed > > > desired behavior. > > > > > > On Thu, Mar 23, 2023, 18:17 Philip Nee wrote: > > > > > > > Hey Dan, > > > > > > > > Thanks for looking into this. Are you talking about MockConsumer? If > > you > > > > invoke subscribe(Collection topics), it actually registers a > > Noop > > > > callback. Perhaps this is what you are seeing? > > > > > > > > P > > > > > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S > wrote: > > > > > > > > > Hello all, > > > > > > > > > > It seems to me based on reading the code, that the consumer > rebalance > > > > > listener that is passed into the mock consumer when subscribing to > a > > > > topic > > > > > is not actually called when a rebalance is simulated. My > > understanding > > > is > > > > > that the consumer rebalance listener is called from the consumer > > > > > coordinator, which is called by kafka consumer. The mock consumer > > > doesn't > > > > > seem to use the consumer coordinator or use any other mechanism to > > call > > > > the > > > > > consumer rebalance listener. Is my understanding correct? Would it > > make > > > > > sense to trigger the consumer rebalance listener when rebalance is > > > > called? > > > > > > > > > > I would be willing to try to make the patch if the behavior is > indeed > > > > > currently incorrect/incomplete. > > > > > > > > > > Thanks, > > > > > > > > > > Daniel > > > > > > > > > > > > > > >
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
I'll pick it up, thanks! On Thu, Mar 23, 2023, 19:27 Philip Nee wrote: > Hey Dan, > > Your analysis looks right. I do see TODO item there to implement the > rebalance callback. Would you like to create a jira issue and work on that? > > Thanks, > P > > On Thu, Mar 23, 2023 at 12:11 PM Dan S wrote: > > > Hi Philip, > > > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case > > we're calling the variant of subscribe that takes a custom > > ConsumerRebalanceListener(which among other things logs when it's > called), > > and we're then calling rebalance (to simulate a rebalance) and removing > all > > partitions from the consumer, polling a few times, and then adding them > > back. We're noticing our custom listener is never called, which was > > unexpected, but based on the code analysis in the original email seems to > > be the current implementation. The question is whether this is indeed > > desired behavior. > > > > On Thu, Mar 23, 2023, 18:17 Philip Nee wrote: > > > > > Hey Dan, > > > > > > Thanks for looking into this. Are you talking about MockConsumer? If > you > > > invoke subscribe(Collection topics), it actually registers a > Noop > > > callback. Perhaps this is what you are seeing? > > > > > > P > > > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S wrote: > > > > > > > Hello all, > > > > > > > > It seems to me based on reading the code, that the consumer rebalance > > > > listener that is passed into the mock consumer when subscribing to a > > > topic > > > > is not actually called when a rebalance is simulated. My > understanding > > is > > > > that the consumer rebalance listener is called from the consumer > > > > coordinator, which is called by kafka consumer. The mock consumer > > doesn't > > > > seem to use the consumer coordinator or use any other mechanism to > call > > > the > > > > consumer rebalance listener. Is my understanding correct? Would it > make > > > > sense to trigger the consumer rebalance listener when rebalance is > > > called? > > > > > > > > I would be willing to try to make the patch if the behavior is indeed > > > > currently incorrect/incomplete. > > > > > > > > Thanks, > > > > > > > > Daniel > > > > > > > > > >
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Hey Dan, Your analysis looks right. I do see TODO item there to implement the rebalance callback. Would you like to create a jira issue and work on that? Thanks, P On Thu, Mar 23, 2023 at 12:11 PM Dan S wrote: > Hi Philip, > > Thanks for the quick reply. Yes, it's the MockConsumer, but in our case > we're calling the variant of subscribe that takes a custom > ConsumerRebalanceListener(which among other things logs when it's called), > and we're then calling rebalance (to simulate a rebalance) and removing all > partitions from the consumer, polling a few times, and then adding them > back. We're noticing our custom listener is never called, which was > unexpected, but based on the code analysis in the original email seems to > be the current implementation. The question is whether this is indeed > desired behavior. > > On Thu, Mar 23, 2023, 18:17 Philip Nee wrote: > > > Hey Dan, > > > > Thanks for looking into this. Are you talking about MockConsumer? If you > > invoke subscribe(Collection topics), it actually registers a Noop > > callback. Perhaps this is what you are seeing? > > > > P > > > > On Thu, Mar 23, 2023 at 11:11 AM Dan S wrote: > > > > > Hello all, > > > > > > It seems to me based on reading the code, that the consumer rebalance > > > listener that is passed into the mock consumer when subscribing to a > > topic > > > is not actually called when a rebalance is simulated. My understanding > is > > > that the consumer rebalance listener is called from the consumer > > > coordinator, which is called by kafka consumer. The mock consumer > doesn't > > > seem to use the consumer coordinator or use any other mechanism to call > > the > > > consumer rebalance listener. Is my understanding correct? Would it make > > > sense to trigger the consumer rebalance listener when rebalance is > > called? > > > > > > I would be willing to try to make the patch if the behavior is indeed > > > currently incorrect/incomplete. > > > > > > Thanks, > > > > > > Daniel > > > > > >
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Hi Philip, Thanks for the quick reply. Yes, it's the MockConsumer, but in our case we're calling the variant of subscribe that takes a custom ConsumerRebalanceListener(which among other things logs when it's called), and we're then calling rebalance (to simulate a rebalance) and removing all partitions from the consumer, polling a few times, and then adding them back. We're noticing our custom listener is never called, which was unexpected, but based on the code analysis in the original email seems to be the current implementation. The question is whether this is indeed desired behavior. On Thu, Mar 23, 2023, 18:17 Philip Nee wrote: > Hey Dan, > > Thanks for looking into this. Are you talking about MockConsumer? If you > invoke subscribe(Collection topics), it actually registers a Noop > callback. Perhaps this is what you are seeing? > > P > > On Thu, Mar 23, 2023 at 11:11 AM Dan S wrote: > > > Hello all, > > > > It seems to me based on reading the code, that the consumer rebalance > > listener that is passed into the mock consumer when subscribing to a > topic > > is not actually called when a rebalance is simulated. My understanding is > > that the consumer rebalance listener is called from the consumer > > coordinator, which is called by kafka consumer. The mock consumer doesn't > > seem to use the consumer coordinator or use any other mechanism to call > the > > consumer rebalance listener. Is my understanding correct? Would it make > > sense to trigger the consumer rebalance listener when rebalance is > called? > > > > I would be willing to try to make the patch if the behavior is indeed > > currently incorrect/incomplete. > > > > Thanks, > > > > Daniel > > >
Re: Should the mock consumer call the consumer rebalance listener on rebalance?
Hey Dan, Thanks for looking into this. Are you talking about MockConsumer? If you invoke subscribe(Collection topics), it actually registers a Noop callback. Perhaps this is what you are seeing? P On Thu, Mar 23, 2023 at 11:11 AM Dan S wrote: > Hello all, > > It seems to me based on reading the code, that the consumer rebalance > listener that is passed into the mock consumer when subscribing to a topic > is not actually called when a rebalance is simulated. My understanding is > that the consumer rebalance listener is called from the consumer > coordinator, which is called by kafka consumer. The mock consumer doesn't > seem to use the consumer coordinator or use any other mechanism to call the > consumer rebalance listener. Is my understanding correct? Would it make > sense to trigger the consumer rebalance listener when rebalance is called? > > I would be willing to try to make the patch if the behavior is indeed > currently incorrect/incomplete. > > Thanks, > > Daniel >
Should the mock consumer call the consumer rebalance listener on rebalance?
Hello all, It seems to me based on reading the code, that the consumer rebalance listener that is passed into the mock consumer when subscribing to a topic is not actually called when a rebalance is simulated. My understanding is that the consumer rebalance listener is called from the consumer coordinator, which is called by kafka consumer. The mock consumer doesn't seem to use the consumer coordinator or use any other mechanism to call the consumer rebalance listener. Is my understanding correct? Would it make sense to trigger the consumer rebalance listener when rebalance is called? I would be willing to try to make the patch if the behavior is indeed currently incorrect/incomplete. Thanks, Daniel