Thanks for the details Dana! I think this sort of thing could be worked into the new "Protocol Guide" documentation: http://kafka.apache.org/protocol.html
On Tue, Mar 29, 2016 at 11:25 AM, Gwen Shapira <g...@confluent.io> wrote: > Awesome summary, Dana. I'd like to fit this into our docs, but I'm not sure > where does step-by-step-description of the protocol fits. Maybe in "Design" > section? > > Just one more thing: > 8) At any time, the broker can respond to a fetch request with > "Rebalancing" error code, at which point the assignment dance begins from > scratch: consumers needs to send the sync request, leaders need to create > an assignment, etc. > > Gwen > > > > On Tue, Mar 29, 2016 at 9:05 AM, Dana Powers <dana.pow...@gmail.com> > wrote: > > > I also found the documentation difficult to parse when it came time to > > implement group APIs. I ended up just reading the client source code and > > trying api calls until it made sense. > > > > My general description from off the top of my head: > > > > (1) have all consumers submit a shared protocol_name string* and > > bytes-serialized protocol_metadata, which includes the topic(s) for > > subscription. > > (2) All consumers should be prepared to parse a JoinResponse and identify > > whether they have been selected as the group leader. > > (3) The non-leaders go straight to SyncGroupRequest, with > group_assignment > > empty, and will wait get their partition assignments via the > > SyncGroupResponse. > > (4) Unlike the others, the consumer-leader will get the agreed > > protocol_name and all member's metadata in its JoinGroupResponse. > > (5) The consumer-leader must apply the protocol_name assignment strategy > to > > the member list + metadata (plus the cluster metadata), generating a set > of > > member assignments (member_id -> topic partitions). > > (6)The consumer-leader then submits that member_assignment data in its > > SyncGroupRequest. > > (7) The broker will take the group member_assignment data and split it > out > > into each members SyncGroupResponse (including the consumer-leader). > > > > At this point all members have their assignments and can start chugging > > along. > > > > I've tried to encapsulate the group request / response protocol info > > cleanly into kafka-python, if that helps you: > > > > https://github.com/dpkp/kafka-python/blob/1.0.2/kafka/protocol/group.py > > > > I'd be willing to help improve the docs if someone points me at the > "source > > of truth". Or feel free to ping on irc #apache-kafka > > > > -Dana > > > > *for java client these are 'range' or 'roundrobin', but are opaque and > can > > be anything if you aren't interested in joining groups that have > > heterogeneous clients. related, the protocol_metadata and member_metadata > > are also opaque and technically can be anything you like. But sticking > with > > the documented metadata specs should in theory allow heterogenous clients > > to cooperate. > > > > On Tue, Mar 29, 2016 at 8:21 AM, Heath Ivie <hi...@autoanything.com> > > wrote: > > > > > Does anyone have better documentation around the group membership APIs? > > > > > > The information about the APIs are great at the beginning but get > > > progressively sparse towards then end. > > > > > > I am not finding enough information about the values of the request > > fields > > > to join / sync the group. > > > > > > Can anyone help me or send me the some additional documentation? > > > > > > Heath Ivie > > > Solutions Architect > > > > > > > > > Warning: This e-mail may contain information proprietary to > AutoAnything > > > Inc. and is intended only for the use of the intended recipient(s). If > > the > > > reader of this message is not the intended recipient(s), you have > > received > > > this message in error and any review, dissemination, distribution or > > > copying of this message is strictly prohibited. If you have received > this > > > message in error, please notify the sender immediately and delete all > > > copies. > > > > > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
