I created a KIP for the multi-port broker change. https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
I'm not re-opening the discussion, since it was agreed on over a month ago and implementation is close to complete (I hope!). Lets consider this voted and accepted? Gwen On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > Great! Sounds like everyone is on the same page > > - I created a template page to make things easier. If you do Tools->Copy > on this page you can just fill in the italic portions with your details. > - I retrofitted KIP-1 to match this formatting > - I added the metadata section people asked for (a link to the > discussion, the JIRA, and the current status). Let's make sure we remember > to update the current status as things are figured out. > - Let's keep the discussion on the mailing list rather than on the wiki > pages. It makes sense to do one or the other so all the comments are in one > place and I think prior experience is that the wiki comments are the worse > way. > > I think it would be great do KIPs for some of the in-flight items folks > mentioned. > > -Jay > > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> +1 >> >> Will be happy to provide a KIP for the multiple-listeners patch. >> >> Gwen >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <joe.st...@stealth.ly> wrote: >> > +1 to everything we have been saying and where this (has settled to)/(is >> > settling to). >> > >> > I am sure other folks have some more feedback and think we should try to >> > keep this discussion going if need be. I am also a firm believer of form >> > following function so kicking the tires some to flesh out the details of >> > this and have some organic growth with the process will be healthy and >> > beneficial to the community. >> > >> > For my part, what I will do is open a few KIP based on some of the work I >> > have been involved with for 0.8.3. Off the top of my head this would >> > include 1) changes to re-assignment of partitions 2) kafka cli 3) global >> > configs 4) security white list black list by ip 5) SSL 6) We probably >> will >> > have lots of Security related KIPs and should treat them all individually >> > when the time is appropriate 7) Kafka on Mesos. >> > >> > If someone else wants to jump in to start getting some of the security >> KIP >> > that we are going to have in 0.8.3 I think that would be great (e.g. >> > Multiple Listeners for Kafka Brokers). There are also a few other >> tickets I >> > can think of that are important to have in the code in 0.8.3 that should >> > have KIP also that I haven't really been involved in. I will take a few >> > minutes and go through JIRA (one I can think of like auto assign id that >> is >> > already committed I think) and ask for a KIP if appropriate or if I feel >> > that I can write it up (both from a time and understanding perspective) >> do >> > so. >> > >> > long story short, I encourage folks to start moving ahead with the KIP >> for >> > 0.8.3 as how we operate. any objections? >> > >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <wangg...@gmail.com> >> wrote: >> > >> >> +1 on the idea, and we could mutually link the KIP wiki page with the >> the >> >> created JIRA ticket (i.e. include the JIRA number on the page and the >> KIP >> >> url on the ticket description). >> >> >> >> Regarding the KIP process, probably we do not need two phase >> communication >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should be >> clear >> >> while people discuss about that. >> >> >> >> About who should trigger the process, I think the only involved people >> >> would be 1) when the patch is submitted / or even the ticket is created, >> >> the assignee could choose to start the KIP process if she thought it is >> >> necessary; 2) the reviewer of the patch can also suggest starting KIP >> >> discussions. >> >> >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira <gshap...@cloudera.com> >> >> wrote: >> >> >> >> > +1 to Ewen's suggestions: Deprecation, status and version. >> >> > >> >> > Perhaps add the JIRA where the KIP was implemented to the metadata. >> >> > This will help tie things together. >> >> > >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava >> >> > <e...@confluent.io> wrote: >> >> > > I think adding a section about deprecation would be helpful. A good >> >> > > fraction of the time I would expect the goal of a KIP is to fix or >> >> > replace >> >> > > older functionality that needs continued support for compatibility, >> but >> >> > > should eventually be phased out. This helps Kafka devs understand >> how >> >> > long >> >> > > they'll end up supporting multiple versions of features and helps >> users >> >> > > understand when they're going to have to make updates to their >> >> > applications. >> >> > > >> >> > > Less important but useful -- having a bit of standard metadata like >> >> PEPs >> >> > > do. Two I think are important are status (if someone lands on the >> KIP >> >> > page, >> >> > > can they tell whether this KIP was ever completed?) and/or the >> version >> >> > the >> >> > > KIP was first released in. >> >> > > >> >> > > >> >> > > >> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy <jjkosh...@gmail.com> >> >> wrote: >> >> > > >> >> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we are >> already >> >> > >> doing this in one form or the other. However, IMO it is fairly ad >> hoc >> >> > >> - i.e., a combination of DISCUSS threads, jira comments, RB and >> code >> >> > >> comments, wikis and html documentation. In the past I have had to >> dig >> >> > >> into a bunch of these to try and figure out why something was >> >> > >> implemented a certain way. I think KIPs can help a lot here first >> by >> >> > >> providing guidelines on what to think about (compatibility, new >> APIs, >> >> > >> etc.) when working through a major feature; and second by becoming >> a >> >> > >> crisp source of truth documentation for new releases. E.g., for >> >> > >> feature X: see relevant KIPs: a, b, c, etc. >> >> > >> >> >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote: >> >> > >> > Hey Joe, >> >> > >> > >> >> > >> > Yeah I guess the question is what is the definition of major? I >> >> agree >> >> > we >> >> > >> > definitely don't want to generate a bunch of paperwork. We have >> >> enough >> >> > >> > problems just getting all the contributions reviewed and checked >> in >> >> > in a >> >> > >> > timely fashion... >> >> > >> > >> >> > >> > So obviously bug fixes would not apply here. >> >> > >> > >> >> > >> > I think it is also pretty clear that big features should get >> >> reviewed >> >> > and >> >> > >> > discussed. To pick on myself, for example, the log compaction >> work >> >> was >> >> > >> done >> >> > >> > without enough public discussion about how it worked and why >> >> (imho). I >> >> > >> > hope/claim that enough rigour in thinking about use-cases and >> >> > operations >> >> > >> > and so on was done that it turned out well, but the discussion >> was >> >> > just >> >> > >> > between a few people with no real public output. This kind of >> >> feature >> >> > is >> >> > >> > clearly a big change and something we should discuss. >> >> > >> > >> >> > >> > If we limit ourselves to just the public contracts the KIP >> >> introduces >> >> > the >> >> > >> > discussion would just be on the new configs and monitoring >> without >> >> > >> really a >> >> > >> > discussion of the design and how it works which is obviously >> closely >> >> > >> > related. >> >> > >> > >> >> > >> > I don't think this should be more work because in practice we are >> >> > making >> >> > >> > wiki pages for any big thing anyway. So this would just be a >> >> > consistent >> >> > >> way >> >> > >> > of numbering and structuring these pages. This would also give a >> >> clear >> >> > >> call >> >> > >> > to action: "hey kafka people, come read my wiki and think this >> >> > through". >> >> > >> > >> >> > >> > I actually thinking the voting aspect is less important. I think >> it >> >> is >> >> > >> > generally clear when there is agreement on something and not. So >> >> from >> >> > my >> >> > >> > point of view we could actually just eliminate that part if that >> is >> >> > too >> >> > >> > formal, it just seemed like a good way to formally adopt >> something. >> >> > >> > >> >> > >> > To address some of your comments from the wiki: >> >> > >> > >> >> > >> > 1. This doesn't inhibit someone coming along and putting up a >> patch. >> >> > It >> >> > >> is >> >> > >> > just that when they do if it is a big thing introducing new >> >> > functionality >> >> > >> > we would ask for a little discussion on the basic >> feature/contracts >> >> > prior >> >> > >> > to code review. >> >> > >> > >> >> > >> > 2. We definitely definitely don't want people generating a lot of >> >> > these >> >> > >> > things every time they have an idea that they aren't going to >> >> > implement. >> >> > >> So >> >> > >> > this is only applicable to things you absolutely will check in >> code >> >> > for. >> >> > >> We >> >> > >> > also don't want to be making proposals before things are thought >> >> > through, >> >> > >> > which often requires writing the code. So I think the right time >> >> for a >> >> > >> KIP >> >> > >> > is when you are far enough along that you know the issues and >> >> > tradeoffs >> >> > >> but >> >> > >> > not so far along that you are going to be totally opposed to any >> >> > change. >> >> > >> > Sometimes that is prior to writing any code and sometimes not >> until >> >> > you >> >> > >> are >> >> > >> > practically done. >> >> > >> > >> >> > >> > The key problem I see this fixing is that there is enough new >> >> > development >> >> > >> > happening that it is pretty hard for everyone to review every >> line >> >> of >> >> > >> every >> >> > >> > iteration of every patch. But all of us should be fully aware of >> new >> >> > >> > features, the ramifications, the new public interfaces, etc. If >> we >> >> > aren't >> >> > >> > aware of that we can't really build a holistic system that is >> >> > beautiful >> >> > >> and >> >> > >> > consistent across. So the idea is that if you fully review the >> KIPs >> >> > you >> >> > >> can >> >> > >> > be sure that even if you don't know every new line of code, you >> know >> >> > the >> >> > >> > major changes coming in. >> >> > >> > >> >> > >> > -Jay >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein < >> joe.st...@stealth.ly> >> >> > >> wrote: >> >> > >> > >> >> > >> > > Thanks Jay for kicking this off! I think the confluence page >> you >> >> > wrote >> >> > >> up >> >> > >> > > is a great start. >> >> > >> > > >> >> > >> > > >> >> > >> > > The KIP makes sense to me (at a minimum) if there is going to >> be >> >> any >> >> > >> > > "breaking change". This way Kafka can continue to grow and >> blossom >> >> > and >> >> > >> we >> >> > >> > > have a process in place if we are going to release a thorn ... >> and >> >> > >> when we >> >> > >> > > do it is *CLEAR* about what and why that is. We can easily >> >> document >> >> > >> which >> >> > >> > > KIPs where involved with this release (which I think should get >> >> > >> committed >> >> > >> > > afterwards somewhere so no chance of edit after release). This >> >> > >> approach I >> >> > >> > > had been thinking about also allows changes to occur as they do >> >> now >> >> > as >> >> > >> long >> >> > >> > > as they are backwards compatible. Hopefully we never need a >> KIP >> >> but >> >> > >> when >> >> > >> > > we do the PMC can vote on it and folks can read the release >> notes >> >> > with >> >> > >> > > *CLEAR* understanding what is going to break their existing >> >> > setup... at >> >> > >> > > least that is how I have been thinking about it. >> >> > >> > > >> >> > >> > > >> >> > >> > > Let me know what you think about this base minimum approach... >> I >> >> > hadn't >> >> > >> > > really thought of the KIP for *ANY* "major change" and I have >> to >> >> > think >> >> > >> more >> >> > >> > > about that. I have some other comments for minor items in the >> >> > >> confluence >> >> > >> > > page I will make once I think more about how I feel having a >> KIP >> >> for >> >> > >> more >> >> > >> > > than what I was thinking about already. >> >> > >> > > >> >> > >> > > >> >> > >> > > I do think we should have "major changes" go through >> confluence, >> >> > >> mailing >> >> > >> > > list discuss and JIRA but kind of feel we have been doing that >> >> > already >> >> > >> ... >> >> > >> > > if there are cases where that isn't the case we should >> highlight >> >> and >> >> > >> learn >> >> > >> > > from them and formalize that more if need be. >> >> > >> > > >> >> > >> > > >> >> > >> > > /******************************************* >> >> > >> > > Joe Stein >> >> > >> > > Founder, Principal Consultant >> >> > >> > > Big Data Open Source Security LLC >> >> > >> > > http://www.stealth.ly >> >> > >> > > Twitter: @allthingshadoop < >> >> http://www.twitter.com/allthingshadoop> >> >> > >> > > ********************************************/ >> >> > >> > > >> >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps < >> jay.kr...@gmail.com> >> >> > >> wrote: >> >> > >> > > >> >> > >> > > > The idea of KIPs came up in a previous discussion but there >> was >> >> no >> >> > >> real >> >> > >> > > > crisp definition of what they were. Here is an attempt at >> >> > defining a >> >> > >> > > > process: >> >> > >> > > > >> >> > >> > > > >> >> > >> > > >> >> > >> >> >> > >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >> >> > >> > > > >> >> > >> > > > The trick here is to have something light-weight enough that >> it >> >> > >> isn't a >> >> > >> > > > hassle for small changes, but enough so that changes get the >> >> > >> eyeballs of >> >> > >> > > > the committers and heavy users. >> >> > >> > > > >> >> > >> > > > Thoughts? >> >> > >> > > > >> >> > >> > > > -Jay >> >> > >> > > > >> >> > >> > > >> >> > >> >> >> > >> >> >> > > >> >> > > >> >> > > -- >> >> > > Thanks, >> >> > > Ewen >> >> > >> >> >> >> >> >> >> >> -- >> >> -- Guozhang >> >> >>
