Thanks for the feedback. I have added a concrete example to the document that I think illustrates the benefit relatively well.
The observation about scaling the workload of individual consumers is certainly valid. I had not really considered this. Our primary concern is being able to gradually roll out consumption configuration changes in a minimally disruptive fashion, including load-balancing. If the round robin strategy can be enhanced to adequately handle that use case, we would be happy. Is there a Jira open for the "flaw" that you mentioned? On 2/26/16, 7:22 PM, "Joel Koshy" <jjkosh...@gmail.com> wrote: >Hi Andrew, > >Thanks for the wiki. Just a couple of comments: > > - The disruptive config change issue that you mentioned is pretty much a > non-issue in the new consumer due to central assignment. > - Optional: but it may be helpful to add a concrete example. > - More of an orthogonal observation than a comment: with heavily skewed > subscriptions fairness is sort of moot. i.e., people would generally scale > up or down subscription counts with the express purpose of > reducing/increasing load on those instances. > - WRT roundrobin we later realized a significant flaw in the way we lay > out partitions: we originally wanted to randomize the partition layout to > reduce the likelihood of most partitions of the same topic from ending up > on a given consumer which is important if you have a few very large topics. > Unfortunately we used hashCode - which does a splendid job of clumping > partitions from the same topic together :( We can probably just "fix" that > in the new consumer's roundrobin assignor. > >Thanks, > >Joel > > >On Fri, Feb 26, 2016 at 2:32 PM, Olson,Andrew <aols...@cerner.com> wrote: > >> Here is a proposal for a new partition assignment strategy, >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy >> >> This KIP corresponds to these two pending pull requests, >> https://github.com/apache/kafka/pull/146 >> https://github.com/apache/kafka/pull/979 >> >> thanks, >> Andrew >> >> CONFIDENTIALITY NOTICE This message and any included attachments are from >> Cerner Corporation and are intended only for the addressee. The information >> contained in this message is confidential and may constitute inside or >> non-public information under international, federal, or state securities >> laws. Unauthorized forwarding, printing, copying, distribution, or use of >> such information is strictly prohibited and may be unlawful. If you are not >> the addressee, please promptly delete this message and notify the sender of >> the delivery error by e-mail or you may call Cerner's corporate offices in >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024. >>
