Oh yeah I think that is better, I hadn't thought of that approach! Any way you could describe the usage in the KIP, just for completeness?
-Jay On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > I think what you described was the original design, so no wonder you > are confused :) > > Following suggestions from Jun, I changed it a bit. The current model is: > > - Clients (producers and consumers) need to know about the broker > ports in advance. They don't need to know about all brokers, but they > need to know at least one host:port pair that speaks the protocol they > want to use. The change is that all host:port pairs in broker.list > must be of the same protocol and match the security.protocol > configuration parameter. > > - Client uses security.protocol configuration parameter to open a > connection to one of the brokers and sends the good old > MetadataRequest. The broker knows which port it got the connection on, > therefore it knows which security protocol is expected (it needs to > use the same protocol to accept the connection and respond), and > therefore it can send a response that contains only the host:port > pairs that are relevant to that protocol. > > - From the client side the MetadataResponse did not change - it > contains a list of brokerId,host,port that the client can connect to. > The fact that all those broker endpoints were chosen out of a larger > collection to match the right protocol is irrelevant for the client. > > I really like the new design since it preserves a lot of the same > configurations and APIs. > > Thoughts? > > Gwen > > On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > I think I am still confused. In addition to the UpdateMetadataRequest > don't > > we have to change the MetadataResponse so that it's possible for clients > to > > discover the new ports? Or is that a second phase? I was imagining it > > worked by basically allowing the brokers to advertise multiple ports, one > > per security type, and then in the client you configure a protocol which > > will implicitly choose the port from the options returned in metadata to > > you... > > > > Likewise in the ConsumerMetadataResponse we are currently giving back > full > > broker information. I think we would have two options here: either change > > the broker information included in that response to match the > > metadataresponse or else remove the broker information entirely and just > > return the node id (since in order to use that request you would already > > have to have the cluster metadata). The second option may be cleaner > since > > it means we won't have to continue evolving those two in lockstep... > > > > -Jay > > > > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > >> Good point :) > >> > >> I added the specifics of the new UpdateMetadataRequest, which is the > >> only protocol bump in this change. > >> > >> Highlighted the broker and producer/consumer configuration changes, > >> added some example values and added the new zookeeper json. > >> > >> Hope this makes things clearer. > >> > >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> > Hey Gwen, > >> > > >> > Could we get the actual changes in that KIP? I.e. changes to metadata > >> > request, changes to UpdateMetadataRequest, new configs and what will > >> their > >> > valid values be, etc. This kind of says that those things will change > but > >> > doesn't say what they will change to... > >> > > >> > -Jay > >> > > >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira <gshap...@cloudera.com> > >> wrote: > >> > > >> >> I created a KIP for the multi-port broker change. > >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs > >> >> > >> >> I'm not re-opening the discussion, since it was agreed on over a > month > >> >> ago and implementation is close to complete (I hope!). Lets consider > >> >> this voted and accepted? > >> >> > >> >> Gwen > >> >> > >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps <jay.kr...@gmail.com> > >> wrote: > >> >> > Great! Sounds like everyone is on the same page > >> >> > > >> >> > - I created a template page to make things easier. If you do > >> >> Tools->Copy > >> >> > on this page you can just fill in the italic portions with your > >> >> details. > >> >> > - I retrofitted KIP-1 to match this formatting > >> >> > - I added the metadata section people asked for (a link to the > >> >> > discussion, the JIRA, and the current status). Let's make sure > we > >> >> remember > >> >> > to update the current status as things are figured out. > >> >> > - Let's keep the discussion on the mailing list rather than on > the > >> >> wiki > >> >> > pages. It makes sense to do one or the other so all the comments > >> are > >> >> in one > >> >> > place and I think prior experience is that the wiki comments are > >> the > >> >> worse > >> >> > way. > >> >> > > >> >> > I think it would be great do KIPs for some of the in-flight items > >> folks > >> >> > mentioned. > >> >> > > >> >> > -Jay > >> >> > > >> >> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira < > gshap...@cloudera.com> > >> >> wrote: > >> >> > > >> >> >> +1 > >> >> >> > >> >> >> Will be happy to provide a KIP for the multiple-listeners patch. > >> >> >> > >> >> >> Gwen > >> >> >> > >> >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <joe.st...@stealth.ly> > >> >> wrote: > >> >> >> > +1 to everything we have been saying and where this (has settled > >> >> to)/(is > >> >> >> > settling to). > >> >> >> > > >> >> >> > I am sure other folks have some more feedback and think we > should > >> try > >> >> to > >> >> >> > keep this discussion going if need be. I am also a firm > believer of > >> >> form > >> >> >> > following function so kicking the tires some to flesh out the > >> details > >> >> of > >> >> >> > this and have some organic growth with the process will be > healthy > >> and > >> >> >> > beneficial to the community. > >> >> >> > > >> >> >> > For my part, what I will do is open a few KIP based on some of > the > >> >> work I > >> >> >> > have been involved with for 0.8.3. Off the top of my head this > >> would > >> >> >> > include 1) changes to re-assignment of partitions 2) kafka cli > 3) > >> >> global > >> >> >> > configs 4) security white list black list by ip 5) SSL 6) We > >> probably > >> >> >> will > >> >> >> > have lots of Security related KIPs and should treat them all > >> >> individually > >> >> >> > when the time is appropriate 7) Kafka on Mesos. > >> >> >> > > >> >> >> > If someone else wants to jump in to start getting some of the > >> security > >> >> >> KIP > >> >> >> > that we are going to have in 0.8.3 I think that would be great > >> (e.g. > >> >> >> > Multiple Listeners for Kafka Brokers). There are also a few > other > >> >> >> tickets I > >> >> >> > can think of that are important to have in the code in 0.8.3 > that > >> >> should > >> >> >> > have KIP also that I haven't really been involved in. I will > take a > >> >> few > >> >> >> > minutes and go through JIRA (one I can think of like auto > assign id > >> >> that > >> >> >> is > >> >> >> > already committed I think) and ask for a KIP if appropriate or > if I > >> >> feel > >> >> >> > that I can write it up (both from a time and understanding > >> >> perspective) > >> >> >> do > >> >> >> > so. > >> >> >> > > >> >> >> > long story short, I encourage folks to start moving ahead with > the > >> KIP > >> >> >> for > >> >> >> > 0.8.3 as how we operate. any objections? > >> >> >> > > >> >> >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang < > wangg...@gmail.com > >> > > >> >> >> wrote: > >> >> >> > > >> >> >> >> +1 on the idea, and we could mutually link the KIP wiki page > with > >> the > >> >> >> the > >> >> >> >> created JIRA ticket (i.e. include the JIRA number on the page > and > >> the > >> >> >> KIP > >> >> >> >> url on the ticket description). > >> >> >> >> > >> >> >> >> Regarding the KIP process, probably we do not need two phase > >> >> >> communication > >> >> >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting > should > >> be > >> >> >> clear > >> >> >> >> while people discuss about that. > >> >> >> >> > >> >> >> >> About who should trigger the process, I think the only involved > >> >> people > >> >> >> >> would be 1) when the patch is submitted / or even the ticket is > >> >> created, > >> >> >> >> the assignee could choose to start the KIP process if she > thought > >> it > >> >> is > >> >> >> >> necessary; 2) the reviewer of the patch can also suggest > starting > >> KIP > >> >> >> >> discussions. > >> >> >> >> > >> >> >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira < > >> >> gshap...@cloudera.com> > >> >> >> >> wrote: > >> >> >> >> > >> >> >> >> > +1 to Ewen's suggestions: Deprecation, status and version. > >> >> >> >> > > >> >> >> >> > Perhaps add the JIRA where the KIP was implemented to the > >> metadata. > >> >> >> >> > This will help tie things together. > >> >> >> >> > > >> >> >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava > >> >> >> >> > <e...@confluent.io> wrote: > >> >> >> >> > > I think adding a section about deprecation would be > helpful. A > >> >> good > >> >> >> >> > > fraction of the time I would expect the goal of a KIP is to > >> fix > >> >> or > >> >> >> >> > replace > >> >> >> >> > > older functionality that needs continued support for > >> >> compatibility, > >> >> >> but > >> >> >> >> > > should eventually be phased out. This helps Kafka devs > >> understand > >> >> >> how > >> >> >> >> > long > >> >> >> >> > > they'll end up supporting multiple versions of features and > >> helps > >> >> >> users > >> >> >> >> > > understand when they're going to have to make updates to > their > >> >> >> >> > applications. > >> >> >> >> > > > >> >> >> >> > > Less important but useful -- having a bit of standard > metadata > >> >> like > >> >> >> >> PEPs > >> >> >> >> > > do. Two I think are important are status (if someone lands > on > >> the > >> >> >> KIP > >> >> >> >> > page, > >> >> >> >> > > can they tell whether this KIP was ever completed?) and/or > the > >> >> >> version > >> >> >> >> > the > >> >> >> >> > > KIP was first released in. > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy < > >> jjkosh...@gmail.com > >> >> > > >> >> >> >> wrote: > >> >> >> >> > > > >> >> >> >> > >> I'm definitely +1 on the KIP concept. As Joe mentioned, we > >> are > >> >> >> already > >> >> >> >> > >> doing this in one form or the other. However, IMO it is > >> fairly > >> >> ad > >> >> >> hoc > >> >> >> >> > >> - i.e., a combination of DISCUSS threads, jira comments, > RB > >> and > >> >> >> code > >> >> >> >> > >> comments, wikis and html documentation. In the past I have > >> had > >> >> to > >> >> >> dig > >> >> >> >> > >> into a bunch of these to try and figure out why something > was > >> >> >> >> > >> implemented a certain way. I think KIPs can help a lot > here > >> >> first > >> >> >> by > >> >> >> >> > >> providing guidelines on what to think about > (compatibility, > >> new > >> >> >> APIs, > >> >> >> >> > >> etc.) when working through a major feature; and second by > >> >> becoming > >> >> >> a > >> >> >> >> > >> crisp source of truth documentation for new releases. > E.g., > >> for > >> >> >> >> > >> feature X: see relevant KIPs: a, b, c, etc. > >> >> >> >> > >> > >> >> >> >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps wrote: > >> >> >> >> > >> > Hey Joe, > >> >> >> >> > >> > > >> >> >> >> > >> > Yeah I guess the question is what is the definition of > >> major? > >> >> I > >> >> >> >> agree > >> >> >> >> > we > >> >> >> >> > >> > definitely don't want to generate a bunch of paperwork. > We > >> >> have > >> >> >> >> enough > >> >> >> >> > >> > problems just getting all the contributions reviewed and > >> >> checked > >> >> >> in > >> >> >> >> > in a > >> >> >> >> > >> > timely fashion... > >> >> >> >> > >> > > >> >> >> >> > >> > So obviously bug fixes would not apply here. > >> >> >> >> > >> > > >> >> >> >> > >> > I think it is also pretty clear that big features should > >> get > >> >> >> >> reviewed > >> >> >> >> > and > >> >> >> >> > >> > discussed. To pick on myself, for example, the log > >> compaction > >> >> >> work > >> >> >> >> was > >> >> >> >> > >> done > >> >> >> >> > >> > without enough public discussion about how it worked and > >> why > >> >> >> >> (imho). I > >> >> >> >> > >> > hope/claim that enough rigour in thinking about > use-cases > >> and > >> >> >> >> > operations > >> >> >> >> > >> > and so on was done that it turned out well, but the > >> discussion > >> >> >> was > >> >> >> >> > just > >> >> >> >> > >> > between a few people with no real public output. This > kind > >> of > >> >> >> >> feature > >> >> >> >> > is > >> >> >> >> > >> > clearly a big change and something we should discuss. > >> >> >> >> > >> > > >> >> >> >> > >> > If we limit ourselves to just the public contracts the > KIP > >> >> >> >> introduces > >> >> >> >> > the > >> >> >> >> > >> > discussion would just be on the new configs and > monitoring > >> >> >> without > >> >> >> >> > >> really a > >> >> >> >> > >> > discussion of the design and how it works which is > >> obviously > >> >> >> closely > >> >> >> >> > >> > related. > >> >> >> >> > >> > > >> >> >> >> > >> > I don't think this should be more work because in > practice > >> we > >> >> are > >> >> >> >> > making > >> >> >> >> > >> > wiki pages for any big thing anyway. So this would just > be > >> a > >> >> >> >> > consistent > >> >> >> >> > >> way > >> >> >> >> > >> > of numbering and structuring these pages. This would > also > >> >> give a > >> >> >> >> clear > >> >> >> >> > >> call > >> >> >> >> > >> > to action: "hey kafka people, come read my wiki and > think > >> this > >> >> >> >> > through". > >> >> >> >> > >> > > >> >> >> >> > >> > I actually thinking the voting aspect is less > important. I > >> >> think > >> >> >> it > >> >> >> >> is > >> >> >> >> > >> > generally clear when there is agreement on something and > >> not. > >> >> So > >> >> >> >> from > >> >> >> >> > my > >> >> >> >> > >> > point of view we could actually just eliminate that > part if > >> >> that > >> >> >> is > >> >> >> >> > too > >> >> >> >> > >> > formal, it just seemed like a good way to formally adopt > >> >> >> something. > >> >> >> >> > >> > > >> >> >> >> > >> > To address some of your comments from the wiki: > >> >> >> >> > >> > > >> >> >> >> > >> > 1. This doesn't inhibit someone coming along and putting > >> up a > >> >> >> patch. > >> >> >> >> > It > >> >> >> >> > >> is > >> >> >> >> > >> > just that when they do if it is a big thing introducing > new > >> >> >> >> > functionality > >> >> >> >> > >> > we would ask for a little discussion on the basic > >> >> >> feature/contracts > >> >> >> >> > prior > >> >> >> >> > >> > to code review. > >> >> >> >> > >> > > >> >> >> >> > >> > 2. We definitely definitely don't want people > generating a > >> >> lot of > >> >> >> >> > these > >> >> >> >> > >> > things every time they have an idea that they aren't > going > >> to > >> >> >> >> > implement. > >> >> >> >> > >> So > >> >> >> >> > >> > this is only applicable to things you absolutely will > >> check in > >> >> >> code > >> >> >> >> > for. > >> >> >> >> > >> We > >> >> >> >> > >> > also don't want to be making proposals before things are > >> >> thought > >> >> >> >> > through, > >> >> >> >> > >> > which often requires writing the code. So I think the > right > >> >> time > >> >> >> >> for a > >> >> >> >> > >> KIP > >> >> >> >> > >> > is when you are far enough along that you know the > issues > >> and > >> >> >> >> > tradeoffs > >> >> >> >> > >> but > >> >> >> >> > >> > not so far along that you are going to be totally > opposed > >> to > >> >> any > >> >> >> >> > change. > >> >> >> >> > >> > Sometimes that is prior to writing any code and > sometimes > >> not > >> >> >> until > >> >> >> >> > you > >> >> >> >> > >> are > >> >> >> >> > >> > practically done. > >> >> >> >> > >> > > >> >> >> >> > >> > The key problem I see this fixing is that there is > enough > >> new > >> >> >> >> > development > >> >> >> >> > >> > happening that it is pretty hard for everyone to review > >> every > >> >> >> line > >> >> >> >> of > >> >> >> >> > >> every > >> >> >> >> > >> > iteration of every patch. But all of us should be fully > >> aware > >> >> of > >> >> >> new > >> >> >> >> > >> > features, the ramifications, the new public interfaces, > >> etc. > >> >> If > >> >> >> we > >> >> >> >> > aren't > >> >> >> >> > >> > aware of that we can't really build a holistic system > that > >> is > >> >> >> >> > beautiful > >> >> >> >> > >> and > >> >> >> >> > >> > consistent across. So the idea is that if you fully > review > >> the > >> >> >> KIPs > >> >> >> >> > you > >> >> >> >> > >> can > >> >> >> >> > >> > be sure that even if you don't know every new line of > code, > >> >> you > >> >> >> know > >> >> >> >> > the > >> >> >> >> > >> > major changes coming in. > >> >> >> >> > >> > > >> >> >> >> > >> > -Jay > >> >> >> >> > >> > > >> >> >> >> > >> > > >> >> >> >> > >> > > >> >> >> >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein < > >> >> >> joe.st...@stealth.ly> > >> >> >> >> > >> wrote: > >> >> >> >> > >> > > >> >> >> >> > >> > > Thanks Jay for kicking this off! I think the > confluence > >> page > >> >> >> you > >> >> >> >> > wrote > >> >> >> >> > >> up > >> >> >> >> > >> > > is a great start. > >> >> >> >> > >> > > > >> >> >> >> > >> > > > >> >> >> >> > >> > > The KIP makes sense to me (at a minimum) if there is > >> going > >> >> to > >> >> >> be > >> >> >> >> any > >> >> >> >> > >> > > "breaking change". This way Kafka can continue to grow > >> and > >> >> >> blossom > >> >> >> >> > and > >> >> >> >> > >> we > >> >> >> >> > >> > > have a process in place if we are going to release a > >> thorn > >> >> ... > >> >> >> and > >> >> >> >> > >> when we > >> >> >> >> > >> > > do it is *CLEAR* about what and why that is. We can > >> easily > >> >> >> >> document > >> >> >> >> > >> which > >> >> >> >> > >> > > KIPs where involved with this release (which I think > >> should > >> >> get > >> >> >> >> > >> committed > >> >> >> >> > >> > > afterwards somewhere so no chance of edit after > release). > >> >> This > >> >> >> >> > >> approach I > >> >> >> >> > >> > > had been thinking about also allows changes to occur > as > >> >> they do > >> >> >> >> now > >> >> >> >> > as > >> >> >> >> > >> long > >> >> >> >> > >> > > as they are backwards compatible. Hopefully we never > >> need a > >> >> >> KIP > >> >> >> >> but > >> >> >> >> > >> when > >> >> >> >> > >> > > we do the PMC can vote on it and folks can read the > >> release > >> >> >> notes > >> >> >> >> > with > >> >> >> >> > >> > > *CLEAR* understanding what is going to break their > >> existing > >> >> >> >> > setup... at > >> >> >> >> > >> > > least that is how I have been thinking about it. > >> >> >> >> > >> > > > >> >> >> >> > >> > > > >> >> >> >> > >> > > Let me know what you think about this base minimum > >> >> approach... > >> >> >> I > >> >> >> >> > hadn't > >> >> >> >> > >> > > really thought of the KIP for *ANY* "major change" > and I > >> >> have > >> >> >> to > >> >> >> >> > think > >> >> >> >> > >> more > >> >> >> >> > >> > > about that. I have some other comments for minor > items in > >> >> the > >> >> >> >> > >> confluence > >> >> >> >> > >> > > page I will make once I think more about how I feel > >> having a > >> >> >> KIP > >> >> >> >> for > >> >> >> >> > >> more > >> >> >> >> > >> > > than what I was thinking about already. > >> >> >> >> > >> > > > >> >> >> >> > >> > > > >> >> >> >> > >> > > I do think we should have "major changes" go through > >> >> >> confluence, > >> >> >> >> > >> mailing > >> >> >> >> > >> > > list discuss and JIRA but kind of feel we have been > doing > >> >> that > >> >> >> >> > already > >> >> >> >> > >> ... > >> >> >> >> > >> > > if there are cases where that isn't the case we should > >> >> >> highlight > >> >> >> >> and > >> >> >> >> > >> learn > >> >> >> >> > >> > > from them and formalize that more if need be. > >> >> >> >> > >> > > > >> >> >> >> > >> > > > >> >> >> >> > >> > > /******************************************* > >> >> >> >> > >> > > Joe Stein > >> >> >> >> > >> > > Founder, Principal Consultant > >> >> >> >> > >> > > Big Data Open Source Security LLC > >> >> >> >> > >> > > http://www.stealth.ly > >> >> >> >> > >> > > Twitter: @allthingshadoop < > >> >> >> >> http://www.twitter.com/allthingshadoop> > >> >> >> >> > >> > > ********************************************/ > >> >> >> >> > >> > > > >> >> >> >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps < > >> >> >> jay.kr...@gmail.com> > >> >> >> >> > >> wrote: > >> >> >> >> > >> > > > >> >> >> >> > >> > > > The idea of KIPs came up in a previous discussion > but > >> >> there > >> >> >> was > >> >> >> >> no > >> >> >> >> > >> real > >> >> >> >> > >> > > > crisp definition of what they were. Here is an > attempt > >> at > >> >> >> >> > defining a > >> >> >> >> > >> > > > process: > >> >> >> >> > >> > > > > >> >> >> >> > >> > > > > >> >> >> >> > >> > > > >> >> >> >> > >> > >> >> >> >> > > >> >> >> >> > >> >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > >> >> >> >> > >> > > > > >> >> >> >> > >> > > > The trick here is to have something light-weight > enough > >> >> that > >> >> >> it > >> >> >> >> > >> isn't a > >> >> >> >> > >> > > > hassle for small changes, but enough so that changes > >> get > >> >> the > >> >> >> >> > >> eyeballs of > >> >> >> >> > >> > > > the committers and heavy users. > >> >> >> >> > >> > > > > >> >> >> >> > >> > > > Thoughts? > >> >> >> >> > >> > > > > >> >> >> >> > >> > > > -Jay > >> >> >> >> > >> > > > > >> >> >> >> > >> > > > >> >> >> >> > >> > >> >> >> >> > >> > >> >> >> >> > > > >> >> >> >> > > > >> >> >> >> > > -- > >> >> >> >> > > Thanks, > >> >> >> >> > > Ewen > >> >> >> >> > > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> -- > >> >> >> >> -- Guozhang > >> >> >> >> > >> >> >> > >> >> > >> >
