Hi Team, Since there is a fairly busy release coming up in 2 weeks, and since partition-assignors are pluggable and don't need to be part of an Apache Kafka release in order to be useful, can we delay this KIP to release 0.10.1 or 0.11.0 (whichever is earlier)?
This will give the community a chance to focus on features that are critical for the next release. Gwen On Mon, Mar 7, 2016 at 10:30 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > Hey Andrew, > > Thanks for adding the example. No I don't think we have a jira open for > that issue. I'm not sure if we need to fix it in roundrobin (now that it's > already out there and some may be using it) vs. just going with your new > "fair" strategy and maybe add a new explicit roundrobinv2. > > Thanks > > Joel, > > On Mon, Feb 29, 2016 at 7:53 AM, Olson,Andrew <aols...@cerner.com> wrote: > >> Thanks for the feedback. I have added a concrete example to the document >> that I think illustrates the benefit relatively well. >> >> The observation about scaling the workload of individual consumers is >> certainly valid. I had not really considered this. Our primary concern is >> being able to gradually roll out consumption configuration changes in a >> minimally disruptive fashion, including load-balancing. If the round robin >> strategy can be enhanced to adequately handle that use case, we would be >> happy. Is there a Jira open for the "flaw" that you mentioned? >> >> >> >> >> On 2/26/16, 7:22 PM, "Joel Koshy" <jjkosh...@gmail.com> wrote: >> >> >Hi Andrew, >> > >> >Thanks for the wiki. Just a couple of comments: >> > >> > - The disruptive config change issue that you mentioned is pretty much >> a >> > non-issue in the new consumer due to central assignment. >> > - Optional: but it may be helpful to add a concrete example. >> > - More of an orthogonal observation than a comment: with heavily skewed >> > subscriptions fairness is sort of moot. i.e., people would generally >> scale >> > up or down subscription counts with the express purpose of >> > reducing/increasing load on those instances. >> > - WRT roundrobin we later realized a significant flaw in the way we lay >> > out partitions: we originally wanted to randomize the partition layout >> to >> > reduce the likelihood of most partitions of the same topic from ending >> up >> > on a given consumer which is important if you have a few very large >> topics. >> > Unfortunately we used hashCode - which does a splendid job of clumping >> > partitions from the same topic together :( We can probably just "fix" >> that >> > in the new consumer's roundrobin assignor. >> > >> >Thanks, >> > >> >Joel >> > >> > >> >On Fri, Feb 26, 2016 at 2:32 PM, Olson,Andrew <aols...@cerner.com> wrote: >> > >> >> Here is a proposal for a new partition assignment strategy, >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy >> >> >> >> This KIP corresponds to these two pending pull requests, >> >> https://github.com/apache/kafka/pull/146 >> >> https://github.com/apache/kafka/pull/979 >> >> >> >> thanks, >> >> Andrew >> >> >> >> CONFIDENTIALITY NOTICE This message and any included attachments are >> from >> >> Cerner Corporation and are intended only for the addressee. The >> information >> >> contained in this message is confidential and may constitute inside or >> >> non-public information under international, federal, or state securities >> >> laws. Unauthorized forwarding, printing, copying, distribution, or use >> of >> >> such information is strictly prohibited and may be unlawful. If you are >> not >> >> the addressee, please promptly delete this message and notify the >> sender of >> >> the delivery error by e-mail or you may call Cerner's corporate offices >> in >> >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024. >> >> >>
