Re: about consumer group name
Thanks Guozhang! Cheers, Churu On Mar 17, 2014, at 3:48 PM, Guozhang Wang wrote: > The short answer to both questions is yes. > > For the second question, the long answer will be: on startup of the > high-level consumer, it will register itself under its ZK group path, so it > knows other members in the group also; and upon rebalance triggered, it > will read its peer number, and also the number of partitions, and make a > decision who-consume-which. This decision is done independently on each > consumer and is supposed to result in the same decision across the > consumers since the decision algorithm is deterministic. > > In the coming Kafka 0.9, this logic will be moved to the brokers so > consumer clients will be thinner: no ZK dependency and no coordination > logic at all. > > Guozhang > > > > > > > On Mon, Mar 17, 2014 at 11:43 AM, Churu Tang wrote: > >> Hi Guozhang, >> >> I really appreciate the detailed explanation! >> Does this mean that the rule "1 partition will be consumed by exactly one >> consumer in the consumer group" will only be checked and ensured at the >> consumer side? Also, does the "consumer group registration in Zookeeper" >> need to be taken care by the consumer group itself? >> I think made a mistake in "there is no consumer group name included in any >> kind of request" because the "clientId acts as the logical grouping". Sorry >> about that. >> >> Thanks! >> >> Churu >> >> On Mar 14, 2014, at 5:21 PM, Guozhang Wang wrote: >> >>> Hi Churu, >>> >>> Brokers are actually not aware of the consumer groups, the consumer >> groups >>> is maintained within the consumers (and registered in ZK) to achieve load >>> balance. After the group has decided who-consume-which-partition, then >> each >>> consumer will do their fetching independently and send fetching requests >> to >>> the brokers. Brokers on the other hand will just blindly respond to the >>> fetch requests. >>> >>> Hope this helps. >>> >>> Guozhang >>> >>> >>> On Fri, Mar 14, 2014 at 4:50 PM, Churu Tang >> wrote: >>> Hi, "Consumers label themselves with a consumer group name", and consumer group name should be global across each Kafka cluster. However, when I check the API, there is no "consumer group name" included in any kind of request(metadata, produce, fetch, offset). Does the broker know about >> the consumer group name? Thanks for your time! Cheers, churu >>> >>> >>> >>> >>> -- >>> -- Guozhang >> >> > > > -- > -- Guozhang
Re: about consumer group name
The short answer to both questions is yes. For the second question, the long answer will be: on startup of the high-level consumer, it will register itself under its ZK group path, so it knows other members in the group also; and upon rebalance triggered, it will read its peer number, and also the number of partitions, and make a decision who-consume-which. This decision is done independently on each consumer and is supposed to result in the same decision across the consumers since the decision algorithm is deterministic. In the coming Kafka 0.9, this logic will be moved to the brokers so consumer clients will be thinner: no ZK dependency and no coordination logic at all. Guozhang On Mon, Mar 17, 2014 at 11:43 AM, Churu Tang wrote: > Hi Guozhang, > > I really appreciate the detailed explanation! > Does this mean that the rule "1 partition will be consumed by exactly one > consumer in the consumer group" will only be checked and ensured at the > consumer side? Also, does the "consumer group registration in Zookeeper" > need to be taken care by the consumer group itself? > I think made a mistake in "there is no consumer group name included in any > kind of request" because the "clientId acts as the logical grouping". Sorry > about that. > > Thanks! > > Churu > > On Mar 14, 2014, at 5:21 PM, Guozhang Wang wrote: > > > Hi Churu, > > > > Brokers are actually not aware of the consumer groups, the consumer > groups > > is maintained within the consumers (and registered in ZK) to achieve load > > balance. After the group has decided who-consume-which-partition, then > each > > consumer will do their fetching independently and send fetching requests > to > > the brokers. Brokers on the other hand will just blindly respond to the > > fetch requests. > > > > Hope this helps. > > > > Guozhang > > > > > > On Fri, Mar 14, 2014 at 4:50 PM, Churu Tang >wrote: > > > >> Hi, > >> > >> "Consumers label themselves with a consumer group name", and consumer > >> group name should be global across each Kafka cluster. However, when I > >> check the API, there is no "consumer group name" included in any kind of > >> request(metadata, produce, fetch, offset). Does the broker know about > the > >> consumer group name? > >> > >> Thanks for your time! > >> > >> Cheers, > >> churu > > > > > > > > > > -- > > -- Guozhang > > -- -- Guozhang
Re: about consumer group name
Hi Guozhang, I really appreciate the detailed explanation! Does this mean that the rule “1 partition will be consumed by exactly one consumer in the consumer group” will only be checked and ensured at the consumer side? Also, does the “consumer group registration in Zookeeper” need to be taken care by the consumer group itself? I think made a mistake in “there is no consumer group name included in any kind of request” because the "clientId acts as the logical grouping”. Sorry about that. Thanks! Churu On Mar 14, 2014, at 5:21 PM, Guozhang Wang wrote: > Hi Churu, > > Brokers are actually not aware of the consumer groups, the consumer groups > is maintained within the consumers (and registered in ZK) to achieve load > balance. After the group has decided who-consume-which-partition, then each > consumer will do their fetching independently and send fetching requests to > the brokers. Brokers on the other hand will just blindly respond to the > fetch requests. > > Hope this helps. > > Guozhang > > > On Fri, Mar 14, 2014 at 4:50 PM, Churu Tang wrote: > >> Hi, >> >> "Consumers label themselves with a consumer group name", and consumer >> group name should be global across each Kafka cluster. However, when I >> check the API, there is no "consumer group name" included in any kind of >> request(metadata, produce, fetch, offset). Does the broker know about the >> consumer group name? >> >> Thanks for your time! >> >> Cheers, >> churu > > > > > -- > -- Guozhang
Re: about consumer group name
Hi Churu, Brokers are actually not aware of the consumer groups, the consumer groups is maintained within the consumers (and registered in ZK) to achieve load balance. After the group has decided who-consume-which-partition, then each consumer will do their fetching independently and send fetching requests to the brokers. Brokers on the other hand will just blindly respond to the fetch requests. Hope this helps. Guozhang On Fri, Mar 14, 2014 at 4:50 PM, Churu Tang wrote: > Hi, > > "Consumers label themselves with a consumer group name", and consumer > group name should be global across each Kafka cluster. However, when I > check the API, there is no "consumer group name" included in any kind of > request(metadata, produce, fetch, offset). Does the broker know about the > consumer group name? > > Thanks for your time! > > Cheers, > churu -- -- Guozhang
about consumer group name
Hi, “Consumers label themselves with a consumer group name”, and consumer group name should be global across each Kafka cluster. However, when I check the API, there is no “consumer group name” included in any kind of request(metadata, produce, fetch, offset). Does the broker know about the consumer group name? Thanks for your time! Cheers, churu
