Hi Andrew, Thanks for the wiki. Just a couple of comments:
- The disruptive config change issue that you mentioned is pretty much a non-issue in the new consumer due to central assignment. - Optional: but it may be helpful to add a concrete example. - More of an orthogonal observation than a comment: with heavily skewed subscriptions fairness is sort of moot. i.e., people would generally scale up or down subscription counts with the express purpose of reducing/increasing load on those instances. - WRT roundrobin we later realized a significant flaw in the way we lay out partitions: we originally wanted to randomize the partition layout to reduce the likelihood of most partitions of the same topic from ending up on a given consumer which is important if you have a few very large topics. Unfortunately we used hashCode - which does a splendid job of clumping partitions from the same topic together :( We can probably just "fix" that in the new consumer's roundrobin assignor. Thanks, Joel On Fri, Feb 26, 2016 at 2:32 PM, Olson,Andrew <aols...@cerner.com> wrote: > Here is a proposal for a new partition assignment strategy, > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy > > This KIP corresponds to these two pending pull requests, > https://github.com/apache/kafka/pull/146 > https://github.com/apache/kafka/pull/979 > > thanks, > Andrew > > CONFIDENTIALITY NOTICE This message and any included attachments are from > Cerner Corporation and are intended only for the addressee. The information > contained in this message is confidential and may constitute inside or > non-public information under international, federal, or state securities > laws. Unauthorized forwarding, printing, copying, distribution, or use of > such information is strictly prohibited and may be unlawful. If you are not > the addressee, please promptly delete this message and notify the sender of > the delivery error by e-mail or you may call Cerner's corporate offices in > Kansas City, Missouri, U.S.A at (+1) (816)221-1024. >
