Hey Gwen, Could we get the actual changes in that KIP? I.e. changes to metadata request, changes to UpdateMetadataRequest, new configs and what will their valid values be, etc. This kind of says that those things will change but doesn't say what they will change to...
-Jay On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > I created a KIP for the multi-port broker change. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs > > I'm not re-opening the discussion, since it was agreed on over a month > ago and implementation is close to complete (I hope!). Lets consider > this voted and accepted? > > Gwen > > On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > Great! Sounds like everyone is on the same page > > > > - I created a template page to make things easier. If you do > Tools->Copy > > on this page you can just fill in the italic portions with your > details. > > - I retrofitted KIP-1 to match this formatting > > - I added the metadata section people asked for (a link to the > > discussion, the JIRA, and the current status). Let's make sure we > remember > > to update the current status as things are figured out. > > - Let's keep the discussion on the mailing list rather than on the > wiki > > pages. It makes sense to do one or the other so all the comments are > in one > > place and I think prior experience is that the wiki comments are the > worse > > way. > > > > I think it would be great do KIPs for some of the in-flight items folks > > mentioned. > > > > -Jay > > > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > >> +1 > >> > >> Will be happy to provide a KIP for the multiple-listeners patch. > >> > >> Gwen > >> > >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <joe.st...@stealth.ly> > wrote: > >> > +1 to everything we have been saying and where this (has settled > to)/(is > >> > settling to). > >> > > >> > I am sure other folks have some more feedback and think we should try > to > >> > keep this discussion going if need be. I am also a firm believer of > form > >> > following function so kicking the tires some to flesh out the details > of > >> > this and have some organic growth with the process will be healthy and > >> > beneficial to the community. > >> > > >> > For my part, what I will do is open a few KIP based on some of the > work I > >> > have been involved with for 0.8.3. Off the top of my head this would > >> > include 1) changes to re-assignment of partitions 2) kafka cli 3) > global > >> > configs 4) security white list black list by ip 5) SSL 6) We probably > >> will > >> > have lots of Security related KIPs and should treat them all > individually > >> > when the time is appropriate 7) Kafka on Mesos. > >> > > >> > If someone else wants to jump in to start getting some of the security > >> KIP > >> > that we are going to have in 0.8.3 I think that would be great (e.g. > >> > Multiple Listeners for Kafka Brokers). There are also a few other > >> tickets I > >> > can think of that are important to have in the code in 0.8.3 that > should > >> > have KIP also that I haven't really been involved in. I will take a > few > >> > minutes and go through JIRA (one I can think of like auto assign id > that > >> is > >> > already committed I think) and ask for a KIP if appropriate or if I > feel > >> > that I can write it up (both from a time and understanding > perspective) > >> do > >> > so. > >> > > >> > long story short, I encourage folks to start moving ahead with the KIP > >> for > >> > 0.8.3 as how we operate. any objections? > >> > > >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <wangg...@gmail.com> > >> wrote: > >> > > >> >> +1 on the idea, and we could mutually link the KIP wiki page with the > >> the > >> >> created JIRA ticket (i.e. include the JIRA number on the page and the > >> KIP > >> >> url on the ticket description). > >> >> > >> >> Regarding the KIP process, probably we do not need two phase > >> communication > >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be > >> clear > >> >> while people discuss about that. > >> >> > >> >> About who should trigger the process, I think the only involved > people > >> >> would be 1) when the patch is submitted / or even the ticket is > created, > >> >> the assignee could choose to start the KIP process if she thought it > is > >> >> necessary; 2) the reviewer of the patch can also suggest starting KIP > >> >> discussions. > >> >> > >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira < > gshap...@cloudera.com> > >> >> wrote: > >> >> > >> >> > +1 to Ewen's suggestions: Deprecation, status and version. > >> >> > > >> >> > Perhaps add the JIRA where the KIP was implemented to the metadata. > >> >> > This will help tie things together. > >> >> > > >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava > >> >> > <e...@confluent.io> wrote: > >> >> > > I think adding a section about deprecation would be helpful. A > good > >> >> > > fraction of the time I would expect the goal of a KIP is to fix > or > >> >> > replace > >> >> > > older functionality that needs continued support for > compatibility, > >> but > >> >> > > should eventually be phased out. This helps Kafka devs understand > >> how > >> >> > long > >> >> > > they'll end up supporting multiple versions of features and helps > >> users > >> >> > > understand when they're going to have to make updates to their > >> >> > applications. > >> >> > > > >> >> > > Less important but useful -- having a bit of standard metadata > like > >> >> PEPs > >> >> > > do. Two I think are important are status (if someone lands on the > >> KIP > >> >> > page, > >> >> > > can they tell whether this KIP was ever completed?) and/or the > >> version > >> >> > the > >> >> > > KIP was first released in. > >> >> > > > >> >> > > > >> >> > > > >> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy <jjkosh...@gmail.com > > > >> >> wrote: > >> >> > > > >> >> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we are > >> already > >> >> > >> doing this in one form or the other. However, IMO it is fairly > ad > >> hoc > >> >> > >> - i.e., a combination of DISCUSS threads, jira comments, RB and > >> code > >> >> > >> comments, wikis and html documentation. In the past I have had > to > >> dig > >> >> > >> into a bunch of these to try and figure out why something was > >> >> > >> implemented a certain way. I think KIPs can help a lot here > first > >> by > >> >> > >> providing guidelines on what to think about (compatibility, new > >> APIs, > >> >> > >> etc.) when working through a major feature; and second by > becoming > >> a > >> >> > >> crisp source of truth documentation for new releases. E.g., for > >> >> > >> feature X: see relevant KIPs: a, b, c, etc. > >> >> > >> > >> >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote: > >> >> > >> > Hey Joe, > >> >> > >> > > >> >> > >> > Yeah I guess the question is what is the definition of major? > I > >> >> agree > >> >> > we > >> >> > >> > definitely don't want to generate a bunch of paperwork. We > have > >> >> enough > >> >> > >> > problems just getting all the contributions reviewed and > checked > >> in > >> >> > in a > >> >> > >> > timely fashion... > >> >> > >> > > >> >> > >> > So obviously bug fixes would not apply here. > >> >> > >> > > >> >> > >> > I think it is also pretty clear that big features should get > >> >> reviewed > >> >> > and > >> >> > >> > discussed. To pick on myself, for example, the log compaction > >> work > >> >> was > >> >> > >> done > >> >> > >> > without enough public discussion about how it worked and why > >> >> (imho). I > >> >> > >> > hope/claim that enough rigour in thinking about use-cases and > >> >> > operations > >> >> > >> > and so on was done that it turned out well, but the discussion > >> was > >> >> > just > >> >> > >> > between a few people with no real public output. This kind of > >> >> feature > >> >> > is > >> >> > >> > clearly a big change and something we should discuss. > >> >> > >> > > >> >> > >> > If we limit ourselves to just the public contracts the KIP > >> >> introduces > >> >> > the > >> >> > >> > discussion would just be on the new configs and monitoring > >> without > >> >> > >> really a > >> >> > >> > discussion of the design and how it works which is obviously > >> closely > >> >> > >> > related. > >> >> > >> > > >> >> > >> > I don't think this should be more work because in practice we > are > >> >> > making > >> >> > >> > wiki pages for any big thing anyway. So this would just be a > >> >> > consistent > >> >> > >> way > >> >> > >> > of numbering and structuring these pages. This would also > give a > >> >> clear > >> >> > >> call > >> >> > >> > to action: "hey kafka people, come read my wiki and think this > >> >> > through". > >> >> > >> > > >> >> > >> > I actually thinking the voting aspect is less important. I > think > >> it > >> >> is > >> >> > >> > generally clear when there is agreement on something and not. > So > >> >> from > >> >> > my > >> >> > >> > point of view we could actually just eliminate that part if > that > >> is > >> >> > too > >> >> > >> > formal, it just seemed like a good way to formally adopt > >> something. > >> >> > >> > > >> >> > >> > To address some of your comments from the wiki: > >> >> > >> > > >> >> > >> > 1. This doesn't inhibit someone coming along and putting up a > >> patch. > >> >> > It > >> >> > >> is > >> >> > >> > just that when they do if it is a big thing introducing new > >> >> > functionality > >> >> > >> > we would ask for a little discussion on the basic > >> feature/contracts > >> >> > prior > >> >> > >> > to code review. > >> >> > >> > > >> >> > >> > 2. We definitely definitely don't want people generating a > lot of > >> >> > these > >> >> > >> > things every time they have an idea that they aren't going to > >> >> > implement. > >> >> > >> So > >> >> > >> > this is only applicable to things you absolutely will check in > >> code > >> >> > for. > >> >> > >> We > >> >> > >> > also don't want to be making proposals before things are > thought > >> >> > through, > >> >> > >> > which often requires writing the code. So I think the right > time > >> >> for a > >> >> > >> KIP > >> >> > >> > is when you are far enough along that you know the issues and > >> >> > tradeoffs > >> >> > >> but > >> >> > >> > not so far along that you are going to be totally opposed to > any > >> >> > change. > >> >> > >> > Sometimes that is prior to writing any code and sometimes not > >> until > >> >> > you > >> >> > >> are > >> >> > >> > practically done. > >> >> > >> > > >> >> > >> > The key problem I see this fixing is that there is enough new > >> >> > development > >> >> > >> > happening that it is pretty hard for everyone to review every > >> line > >> >> of > >> >> > >> every > >> >> > >> > iteration of every patch. But all of us should be fully aware > of > >> new > >> >> > >> > features, the ramifications, the new public interfaces, etc. > If > >> we > >> >> > aren't > >> >> > >> > aware of that we can't really build a holistic system that is > >> >> > beautiful > >> >> > >> and > >> >> > >> > consistent across. So the idea is that if you fully review the > >> KIPs > >> >> > you > >> >> > >> can > >> >> > >> > be sure that even if you don't know every new line of code, > you > >> know > >> >> > the > >> >> > >> > major changes coming in. > >> >> > >> > > >> >> > >> > -Jay > >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein < > >> joe.st...@stealth.ly> > >> >> > >> wrote: > >> >> > >> > > >> >> > >> > > Thanks Jay for kicking this off! I think the confluence page > >> you > >> >> > wrote > >> >> > >> up > >> >> > >> > > is a great start. > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > The KIP makes sense to me (at a minimum) if there is going > to > >> be > >> >> any > >> >> > >> > > "breaking change". This way Kafka can continue to grow and > >> blossom > >> >> > and > >> >> > >> we > >> >> > >> > > have a process in place if we are going to release a thorn > ... > >> and > >> >> > >> when we > >> >> > >> > > do it is *CLEAR* about what and why that is. We can easily > >> >> document > >> >> > >> which > >> >> > >> > > KIPs where involved with this release (which I think should > get > >> >> > >> committed > >> >> > >> > > afterwards somewhere so no chance of edit after release). > This > >> >> > >> approach I > >> >> > >> > > had been thinking about also allows changes to occur as > they do > >> >> now > >> >> > as > >> >> > >> long > >> >> > >> > > as they are backwards compatible. Hopefully we never need a > >> KIP > >> >> but > >> >> > >> when > >> >> > >> > > we do the PMC can vote on it and folks can read the release > >> notes > >> >> > with > >> >> > >> > > *CLEAR* understanding what is going to break their existing > >> >> > setup... at > >> >> > >> > > least that is how I have been thinking about it. > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > Let me know what you think about this base minimum > approach... > >> I > >> >> > hadn't > >> >> > >> > > really thought of the KIP for *ANY* "major change" and I > have > >> to > >> >> > think > >> >> > >> more > >> >> > >> > > about that. I have some other comments for minor items in > the > >> >> > >> confluence > >> >> > >> > > page I will make once I think more about how I feel having a > >> KIP > >> >> for > >> >> > >> more > >> >> > >> > > than what I was thinking about already. > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > I do think we should have "major changes" go through > >> confluence, > >> >> > >> mailing > >> >> > >> > > list discuss and JIRA but kind of feel we have been doing > that > >> >> > already > >> >> > >> ... > >> >> > >> > > if there are cases where that isn't the case we should > >> highlight > >> >> and > >> >> > >> learn > >> >> > >> > > from them and formalize that more if need be. > >> >> > >> > > > >> >> > >> > > > >> >> > >> > > /******************************************* > >> >> > >> > > Joe Stein > >> >> > >> > > Founder, Principal Consultant > >> >> > >> > > Big Data Open Source Security LLC > >> >> > >> > > http://www.stealth.ly > >> >> > >> > > Twitter: @allthingshadoop < > >> >> http://www.twitter.com/allthingshadoop> > >> >> > >> > > ********************************************/ > >> >> > >> > > > >> >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps < > >> jay.kr...@gmail.com> > >> >> > >> wrote: > >> >> > >> > > > >> >> > >> > > > The idea of KIPs came up in a previous discussion but > there > >> was > >> >> no > >> >> > >> real > >> >> > >> > > > crisp definition of what they were. Here is an attempt at > >> >> > defining a > >> >> > >> > > > process: > >> >> > >> > > > > >> >> > >> > > > > >> >> > >> > > > >> >> > >> > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > >> >> > >> > > > > >> >> > >> > > > The trick here is to have something light-weight enough > that > >> it > >> >> > >> isn't a > >> >> > >> > > > hassle for small changes, but enough so that changes get > the > >> >> > >> eyeballs of > >> >> > >> > > > the committers and heavy users. > >> >> > >> > > > > >> >> > >> > > > Thoughts? > >> >> > >> > > > > >> >> > >> > > > -Jay > >> >> > >> > > > > >> >> > >> > > > >> >> > >> > >> >> > >> > >> >> > > > >> >> > > > >> >> > > -- > >> >> > > Thanks, > >> >> > > Ewen > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> -- Guozhang > >> >> > >> >
