I think what you described was the original design, so no wonder you are confused :)
Following suggestions from Jun, I changed it a bit. The current model is: - Clients (producers and consumers) need to know about the broker ports in advance. They don't need to know about all brokers, but they need to know at least one host:port pair that speaks the protocol they want to use. The change is that all host:port pairs in broker.list must be of the same protocol and match the security.protocol configuration parameter. - Client uses security.protocol configuration parameter to open a connection to one of the brokers and sends the good old MetadataRequest. The broker knows which port it got the connection on, therefore it knows which security protocol is expected (it needs to use the same protocol to accept the connection and respond), and therefore it can send a response that contains only the host:port pairs that are relevant to that protocol. - From the client side the MetadataResponse did not change - it contains a list of brokerId,host,port that the client can connect to. The fact that all those broker endpoints were chosen out of a larger collection to match the right protocol is irrelevant for the client. I really like the new design since it preserves a lot of the same configurations and APIs. Thoughts? Gwen On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > I think I am still confused. In addition to the UpdateMetadataRequest don't > we have to change the MetadataResponse so that it's possible for clients to > discover the new ports? Or is that a second phase? I was imagining it > worked by basically allowing the brokers to advertise multiple ports, one > per security type, and then in the client you configure a protocol which > will implicitly choose the port from the options returned in metadata to > you... > > Likewise in the ConsumerMetadataResponse we are currently giving back full > broker information. I think we would have two options here: either change > the broker information included in that response to match the > metadataresponse or else remove the broker information entirely and just > return the node id (since in order to use that request you would already > have to have the cluster metadata). The second option may be cleaner since > it means we won't have to continue evolving those two in lockstep... > > -Jay > > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> Good point :) >> >> I added the specifics of the new UpdateMetadataRequest, which is the >> only protocol bump in this change. >> >> Highlighted the broker and producer/consumer configuration changes, >> added some example values and added the new zookeeper json. >> >> Hope this makes things clearer. >> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps <jay.kr...@gmail.com> wrote: >> > Hey Gwen, >> > >> > Could we get the actual changes in that KIP? I.e. changes to metadata >> > request, changes to UpdateMetadataRequest, new configs and what will >> their >> > valid values be, etc. This kind of says that those things will change but >> > doesn't say what they will change to... >> > >> > -Jay >> > >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> > >> >> I created a KIP for the multi-port broker change. >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs >> >> >> >> I'm not re-opening the discussion, since it was agreed on over a month >> >> ago and implementation is close to complete (I hope!). Lets consider >> >> this voted and accepted? >> >> >> >> Gwen >> >> >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps <jay.kr...@gmail.com> >> wrote: >> >> > Great! Sounds like everyone is on the same page >> >> > >> >> > - I created a template page to make things easier. If you do >> >> Tools->Copy >> >> > on this page you can just fill in the italic portions with your >> >> details. >> >> > - I retrofitted KIP-1 to match this formatting >> >> > - I added the metadata section people asked for (a link to the >> >> > discussion, the JIRA, and the current status). Let's make sure we >> >> remember >> >> > to update the current status as things are figured out. >> >> > - Let's keep the discussion on the mailing list rather than on the >> >> wiki >> >> > pages. It makes sense to do one or the other so all the comments >> are >> >> in one >> >> > place and I think prior experience is that the wiki comments are >> the >> >> worse >> >> > way. >> >> > >> >> > I think it would be great do KIPs for some of the in-flight items >> folks >> >> > mentioned. >> >> > >> >> > -Jay >> >> > >> >> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <gshap...@cloudera.com> >> >> wrote: >> >> > >> >> >> +1 >> >> >> >> >> >> Will be happy to provide a KIP for the multiple-listeners patch. >> >> >> >> >> >> Gwen >> >> >> >> >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <joe.st...@stealth.ly> >> >> wrote: >> >> >> > +1 to everything we have been saying and where this (has settled >> >> to)/(is >> >> >> > settling to). >> >> >> > >> >> >> > I am sure other folks have some more feedback and think we should >> try >> >> to >> >> >> > keep this discussion going if need be. I am also a firm believer of >> >> form >> >> >> > following function so kicking the tires some to flesh out the >> details >> >> of >> >> >> > this and have some organic growth with the process will be healthy >> and >> >> >> > beneficial to the community. >> >> >> > >> >> >> > For my part, what I will do is open a few KIP based on some of the >> >> work I >> >> >> > have been involved with for 0.8.3. Off the top of my head this >> would >> >> >> > include 1) changes to re-assignment of partitions 2) kafka cli 3) >> >> global >> >> >> > configs 4) security white list black list by ip 5) SSL 6) We >> probably >> >> >> will >> >> >> > have lots of Security related KIPs and should treat them all >> >> individually >> >> >> > when the time is appropriate 7) Kafka on Mesos. >> >> >> > >> >> >> > If someone else wants to jump in to start getting some of the >> security >> >> >> KIP >> >> >> > that we are going to have in 0.8.3 I think that would be great >> (e.g. >> >> >> > Multiple Listeners for Kafka Brokers). There are also a few other >> >> >> tickets I >> >> >> > can think of that are important to have in the code in 0.8.3 that >> >> should >> >> >> > have KIP also that I haven't really been involved in. I will take a >> >> few >> >> >> > minutes and go through JIRA (one I can think of like auto assign id >> >> that >> >> >> is >> >> >> > already committed I think) and ask for a KIP if appropriate or if I >> >> feel >> >> >> > that I can write it up (both from a time and understanding >> >> perspective) >> >> >> do >> >> >> > so. >> >> >> > >> >> >> > long story short, I encourage folks to start moving ahead with the >> KIP >> >> >> for >> >> >> > 0.8.3 as how we operate. any objections? >> >> >> > >> >> >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <wangg...@gmail.com >> > >> >> >> wrote: >> >> >> > >> >> >> >> +1 on the idea, and we could mutually link the KIP wiki page with >> the >> >> >> the >> >> >> >> created JIRA ticket (i.e. include the JIRA number on the page and >> the >> >> >> KIP >> >> >> >> url on the ticket description). >> >> >> >> >> >> >> >> Regarding the KIP process, probably we do not need two phase >> >> >> communication >> >> >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting should >> be >> >> >> clear >> >> >> >> while people discuss about that. >> >> >> >> >> >> >> >> About who should trigger the process, I think the only involved >> >> people >> >> >> >> would be 1) when the patch is submitted / or even the ticket is >> >> created, >> >> >> >> the assignee could choose to start the KIP process if she thought >> it >> >> is >> >> >> >> necessary; 2) the reviewer of the patch can also suggest starting >> KIP >> >> >> >> discussions. >> >> >> >> >> >> >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira < >> >> gshap...@cloudera.com> >> >> >> >> wrote: >> >> >> >> >> >> >> >> > +1 to Ewen's suggestions: Deprecation, status and version. >> >> >> >> > >> >> >> >> > Perhaps add the JIRA where the KIP was implemented to the >> metadata. >> >> >> >> > This will help tie things together. >> >> >> >> > >> >> >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava >> >> >> >> > <e...@confluent.io> wrote: >> >> >> >> > > I think adding a section about deprecation would be helpful. A >> >> good >> >> >> >> > > fraction of the time I would expect the goal of a KIP is to >> fix >> >> or >> >> >> >> > replace >> >> >> >> > > older functionality that needs continued support for >> >> compatibility, >> >> >> but >> >> >> >> > > should eventually be phased out. This helps Kafka devs >> understand >> >> >> how >> >> >> >> > long >> >> >> >> > > they'll end up supporting multiple versions of features and >> helps >> >> >> users >> >> >> >> > > understand when they're going to have to make updates to their >> >> >> >> > applications. >> >> >> >> > > >> >> >> >> > > Less important but useful -- having a bit of standard metadata >> >> like >> >> >> >> PEPs >> >> >> >> > > do. Two I think are important are status (if someone lands on >> the >> >> >> KIP >> >> >> >> > page, >> >> >> >> > > can they tell whether this KIP was ever completed?) and/or the >> >> >> version >> >> >> >> > the >> >> >> >> > > KIP was first released in. >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy < >> jjkosh...@gmail.com >> >> > >> >> >> >> wrote: >> >> >> >> > > >> >> >> >> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we >> are >> >> >> already >> >> >> >> > >> doing this in one form or the other. However, IMO it is >> fairly >> >> ad >> >> >> hoc >> >> >> >> > >> - i.e., a combination of DISCUSS threads, jira comments, RB >> and >> >> >> code >> >> >> >> > >> comments, wikis and html documentation. In the past I have >> had >> >> to >> >> >> dig >> >> >> >> > >> into a bunch of these to try and figure out why something was >> >> >> >> > >> implemented a certain way. I think KIPs can help a lot here >> >> first >> >> >> by >> >> >> >> > >> providing guidelines on what to think about (compatibility, >> new >> >> >> APIs, >> >> >> >> > >> etc.) when working through a major feature; and second by >> >> becoming >> >> >> a >> >> >> >> > >> crisp source of truth documentation for new releases. E.g., >> for >> >> >> >> > >> feature X: see relevant KIPs: a, b, c, etc. >> >> >> >> > >> >> >> >> >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote: >> >> >> >> > >> > Hey Joe, >> >> >> >> > >> > >> >> >> >> > >> > Yeah I guess the question is what is the definition of >> major? >> >> I >> >> >> >> agree >> >> >> >> > we >> >> >> >> > >> > definitely don't want to generate a bunch of paperwork. We >> >> have >> >> >> >> enough >> >> >> >> > >> > problems just getting all the contributions reviewed and >> >> checked >> >> >> in >> >> >> >> > in a >> >> >> >> > >> > timely fashion... >> >> >> >> > >> > >> >> >> >> > >> > So obviously bug fixes would not apply here. >> >> >> >> > >> > >> >> >> >> > >> > I think it is also pretty clear that big features should >> get >> >> >> >> reviewed >> >> >> >> > and >> >> >> >> > >> > discussed. To pick on myself, for example, the log >> compaction >> >> >> work >> >> >> >> was >> >> >> >> > >> done >> >> >> >> > >> > without enough public discussion about how it worked and >> why >> >> >> >> (imho). I >> >> >> >> > >> > hope/claim that enough rigour in thinking about use-cases >> and >> >> >> >> > operations >> >> >> >> > >> > and so on was done that it turned out well, but the >> discussion >> >> >> was >> >> >> >> > just >> >> >> >> > >> > between a few people with no real public output. This kind >> of >> >> >> >> feature >> >> >> >> > is >> >> >> >> > >> > clearly a big change and something we should discuss. >> >> >> >> > >> > >> >> >> >> > >> > If we limit ourselves to just the public contracts the KIP >> >> >> >> introduces >> >> >> >> > the >> >> >> >> > >> > discussion would just be on the new configs and monitoring >> >> >> without >> >> >> >> > >> really a >> >> >> >> > >> > discussion of the design and how it works which is >> obviously >> >> >> closely >> >> >> >> > >> > related. >> >> >> >> > >> > >> >> >> >> > >> > I don't think this should be more work because in practice >> we >> >> are >> >> >> >> > making >> >> >> >> > >> > wiki pages for any big thing anyway. So this would just be >> a >> >> >> >> > consistent >> >> >> >> > >> way >> >> >> >> > >> > of numbering and structuring these pages. This would also >> >> give a >> >> >> >> clear >> >> >> >> > >> call >> >> >> >> > >> > to action: "hey kafka people, come read my wiki and think >> this >> >> >> >> > through". >> >> >> >> > >> > >> >> >> >> > >> > I actually thinking the voting aspect is less important. I >> >> think >> >> >> it >> >> >> >> is >> >> >> >> > >> > generally clear when there is agreement on something and >> not. >> >> So >> >> >> >> from >> >> >> >> > my >> >> >> >> > >> > point of view we could actually just eliminate that part if >> >> that >> >> >> is >> >> >> >> > too >> >> >> >> > >> > formal, it just seemed like a good way to formally adopt >> >> >> something. >> >> >> >> > >> > >> >> >> >> > >> > To address some of your comments from the wiki: >> >> >> >> > >> > >> >> >> >> > >> > 1. This doesn't inhibit someone coming along and putting >> up a >> >> >> patch. >> >> >> >> > It >> >> >> >> > >> is >> >> >> >> > >> > just that when they do if it is a big thing introducing new >> >> >> >> > functionality >> >> >> >> > >> > we would ask for a little discussion on the basic >> >> >> feature/contracts >> >> >> >> > prior >> >> >> >> > >> > to code review. >> >> >> >> > >> > >> >> >> >> > >> > 2. We definitely definitely don't want people generating a >> >> lot of >> >> >> >> > these >> >> >> >> > >> > things every time they have an idea that they aren't going >> to >> >> >> >> > implement. >> >> >> >> > >> So >> >> >> >> > >> > this is only applicable to things you absolutely will >> check in >> >> >> code >> >> >> >> > for. >> >> >> >> > >> We >> >> >> >> > >> > also don't want to be making proposals before things are >> >> thought >> >> >> >> > through, >> >> >> >> > >> > which often requires writing the code. So I think the right >> >> time >> >> >> >> for a >> >> >> >> > >> KIP >> >> >> >> > >> > is when you are far enough along that you know the issues >> and >> >> >> >> > tradeoffs >> >> >> >> > >> but >> >> >> >> > >> > not so far along that you are going to be totally opposed >> to >> >> any >> >> >> >> > change. >> >> >> >> > >> > Sometimes that is prior to writing any code and sometimes >> not >> >> >> until >> >> >> >> > you >> >> >> >> > >> are >> >> >> >> > >> > practically done. >> >> >> >> > >> > >> >> >> >> > >> > The key problem I see this fixing is that there is enough >> new >> >> >> >> > development >> >> >> >> > >> > happening that it is pretty hard for everyone to review >> every >> >> >> line >> >> >> >> of >> >> >> >> > >> every >> >> >> >> > >> > iteration of every patch. But all of us should be fully >> aware >> >> of >> >> >> new >> >> >> >> > >> > features, the ramifications, the new public interfaces, >> etc. >> >> If >> >> >> we >> >> >> >> > aren't >> >> >> >> > >> > aware of that we can't really build a holistic system that >> is >> >> >> >> > beautiful >> >> >> >> > >> and >> >> >> >> > >> > consistent across. So the idea is that if you fully review >> the >> >> >> KIPs >> >> >> >> > you >> >> >> >> > >> can >> >> >> >> > >> > be sure that even if you don't know every new line of code, >> >> you >> >> >> know >> >> >> >> > the >> >> >> >> > >> > major changes coming in. >> >> >> >> > >> > >> >> >> >> > >> > -Jay >> >> >> >> > >> > >> >> >> >> > >> > >> >> >> >> > >> > >> >> >> >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein < >> >> >> joe.st...@stealth.ly> >> >> >> >> > >> wrote: >> >> >> >> > >> > >> >> >> >> > >> > > Thanks Jay for kicking this off! I think the confluence >> page >> >> >> you >> >> >> >> > wrote >> >> >> >> > >> up >> >> >> >> > >> > > is a great start. >> >> >> >> > >> > > >> >> >> >> > >> > > >> >> >> >> > >> > > The KIP makes sense to me (at a minimum) if there is >> going >> >> to >> >> >> be >> >> >> >> any >> >> >> >> > >> > > "breaking change". This way Kafka can continue to grow >> and >> >> >> blossom >> >> >> >> > and >> >> >> >> > >> we >> >> >> >> > >> > > have a process in place if we are going to release a >> thorn >> >> ... >> >> >> and >> >> >> >> > >> when we >> >> >> >> > >> > > do it is *CLEAR* about what and why that is. We can >> easily >> >> >> >> document >> >> >> >> > >> which >> >> >> >> > >> > > KIPs where involved with this release (which I think >> should >> >> get >> >> >> >> > >> committed >> >> >> >> > >> > > afterwards somewhere so no chance of edit after release). >> >> This >> >> >> >> > >> approach I >> >> >> >> > >> > > had been thinking about also allows changes to occur as >> >> they do >> >> >> >> now >> >> >> >> > as >> >> >> >> > >> long >> >> >> >> > >> > > as they are backwards compatible. Hopefully we never >> need a >> >> >> KIP >> >> >> >> but >> >> >> >> > >> when >> >> >> >> > >> > > we do the PMC can vote on it and folks can read the >> release >> >> >> notes >> >> >> >> > with >> >> >> >> > >> > > *CLEAR* understanding what is going to break their >> existing >> >> >> >> > setup... at >> >> >> >> > >> > > least that is how I have been thinking about it. >> >> >> >> > >> > > >> >> >> >> > >> > > >> >> >> >> > >> > > Let me know what you think about this base minimum >> >> approach... >> >> >> I >> >> >> >> > hadn't >> >> >> >> > >> > > really thought of the KIP for *ANY* "major change" and I >> >> have >> >> >> to >> >> >> >> > think >> >> >> >> > >> more >> >> >> >> > >> > > about that. I have some other comments for minor items in >> >> the >> >> >> >> > >> confluence >> >> >> >> > >> > > page I will make once I think more about how I feel >> having a >> >> >> KIP >> >> >> >> for >> >> >> >> > >> more >> >> >> >> > >> > > than what I was thinking about already. >> >> >> >> > >> > > >> >> >> >> > >> > > >> >> >> >> > >> > > I do think we should have "major changes" go through >> >> >> confluence, >> >> >> >> > >> mailing >> >> >> >> > >> > > list discuss and JIRA but kind of feel we have been doing >> >> that >> >> >> >> > already >> >> >> >> > >> ... >> >> >> >> > >> > > if there are cases where that isn't the case we should >> >> >> highlight >> >> >> >> and >> >> >> >> > >> learn >> >> >> >> > >> > > from them and formalize that more if need be. >> >> >> >> > >> > > >> >> >> >> > >> > > >> >> >> >> > >> > > /******************************************* >> >> >> >> > >> > > Joe Stein >> >> >> >> > >> > > Founder, Principal Consultant >> >> >> >> > >> > > Big Data Open Source Security LLC >> >> >> >> > >> > > http://www.stealth.ly >> >> >> >> > >> > > Twitter: @allthingshadoop < >> >> >> >> http://www.twitter.com/allthingshadoop> >> >> >> >> > >> > > ********************************************/ >> >> >> >> > >> > > >> >> >> >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps < >> >> >> jay.kr...@gmail.com> >> >> >> >> > >> wrote: >> >> >> >> > >> > > >> >> >> >> > >> > > > The idea of KIPs came up in a previous discussion but >> >> there >> >> >> was >> >> >> >> no >> >> >> >> > >> real >> >> >> >> > >> > > > crisp definition of what they were. Here is an attempt >> at >> >> >> >> > defining a >> >> >> >> > >> > > > process: >> >> >> >> > >> > > > >> >> >> >> > >> > > > >> >> >> >> > >> > > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >> >> >> >> > >> > > > >> >> >> >> > >> > > > The trick here is to have something light-weight enough >> >> that >> >> >> it >> >> >> >> > >> isn't a >> >> >> >> > >> > > > hassle for small changes, but enough so that changes >> get >> >> the >> >> >> >> > >> eyeballs of >> >> >> >> > >> > > > the committers and heavy users. >> >> >> >> > >> > > > >> >> >> >> > >> > > > Thoughts? >> >> >> >> > >> > > > >> >> >> >> > >> > > > -Jay >> >> >> >> > >> > > > >> >> >> >> > >> > > >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > -- >> >> >> >> > > Thanks, >> >> >> >> > > Ewen >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> -- Guozhang >> >> >> >> >> >> >> >> >> >>
