Thanks. Just to add more detail as to why I think it's a good idea to be able to segregate traffic like that...
One reason would just be to separate out the intra-cluster communication to a separate port. This can allow you to run it over a different interface (for example, you could have dedicated links for the brokers to do replication), though you could do that with a wildcard bind and multiple interfaces with a little care on the broker config. It also allows for firewalling off clients from a cluster while leaving the broker communication intact. We've run into this situation a couple times where it was advantageous to do this during recovery to prevent things like runaway file descriptor allocation. It also gives the ability to use QOS tools to work with networking guarantees for broker traffic. Maybe it's enough to be able to specify a special protocol name to do this. Meaning you could configure a port with protocol "broker" (or something like that) which could be used by the brokers if it exists. Otherwise they would default back to something else. -Todd -TOdd On Tue, Dec 2, 2014 at 3:23 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > The good news is that I'm not using a map to represent protocol list. > > The bad news is that as mentioned in the wiki: producers, consumers > and broker configuration specify "security protocol", so we'll know > which host/port pair to use for communication. This implicitly assumes > there will be only one host/port per protocol. > > I'll think a bit on how this assumption can be relaxed. > > Gwen > > On Tue, Dec 2, 2014 at 3:14 PM, Todd Palino <tpal...@gmail.com> wrote: > > Gwen - just my reading of what we could expect from what you had > described > > so far. Without having gone into implementation details, there didn't > seem > > to be anything that would block the ability to run two ports with the > same > > protocol configuration, at least from the way you proposed to represent > it > > in Zookeeper. I'd just like to not go down the path of using something > like > > a map for representing the protocol list that would eliminate this > > possibility, unless there's a pretty good reason to do so. > > > > -Todd > > > > On Tue, Dec 2, 2014 at 3:00 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > >> Hey Todd, > >> > >> You say "lose the ability" - you mean this ability actually exist now? > >> Or is this something you hope the new patch will support? > >> > >> Gwen > >> > >> On Tue, Dec 2, 2014 at 2:08 PM, Todd Palino <tpal...@gmail.com> wrote: > >> > Leaving aside the rest of this, on #1, while I consider being able to > >> > advertise the ports a good idea, I don't want to lose the ability for > >> > maintaining multiple ports with the same protocol. For example, being > >> able > >> > to have 2 plaintext ports, one that only brokers communicate over, and > >> one > >> > that general clients use. The ability to segregate this traffic is > useful > >> > in a number of situations, over and above other controls like quotas, > and > >> > is relatively easy to do once we support multiple ports. > >> > > >> > -Todd > >> > > >> > > >> > On Tue, Dec 2, 2014 at 1:58 PM, Jun Rao <jun...@gmail.com> wrote: > >> > > >> >> Hi, Gwen, > >> >> > >> >> Thanks for writing up the wiki. Some comments below. > >> >> > >> >> 1. To make it more general, should we support a binding and an > >> advertised > >> >> host for each protocol (e.g. plaintext, ssl, etc)? We will also need > to > >> >> figure out how to specify the wildcard binding host. > >> >> > >> >> 2. Broker format change in ZK > >> >> The broker registration in ZK needs to store the host/port for all > >> >> protocols. We will need to bump up the version of the broker > >> registration > >> >> data. Since this is an intra-cluster protocol change, we need an > extra > >> >> config for rolling upgrades. So, in the first step, each broker is > >> upgraded > >> >> and is ready to parse brokers registered in the new format, but not > >> >> registering using the new format yet. In the second step, when that > new > >> >> config is enabled, the broker will register using the new format. > >> >> > >> >> 3. Wire protocol changes. Currently, the broker info is used in the > >> >> following requests/responses: TopicMetadataResponse , > >> >> ConsumerMetadataResponse, LeaderAndIsrRequest and > >> UpdateMetadataRequest. > >> >> 3.1 TopicMetadataResponse and ConsumerMetadataResponse: > >> >> These two are used between the clients and the broker. I am not sure > >> that > >> >> we need to make a wire protocol change for them. Currently, the > protocol > >> >> includes a single host/port pair in those responses. Based on the > type > >> of > >> >> the port on which the request is sent, it seems that we can just pick > >> the > >> >> corresponding host and port to include in the response. > >> >> 3.2 UpdateMetadataRequest: > >> >> This is used between the controller and the broker. Since each broker > >> needs > >> >> to cache the host/port of all protocols, we need to make a wire > protocol > >> >> change. We also need to change the broker format in MetadataCache > >> >> accordingly. This is also an intra-cluster protocol change. So the > >> upgrade > >> >> path will need to follow that in item 2. > >> >> 3.3 LeaderAndIsrRequest: > >> >> This is also used between the controller and the broker. The > receiving > >> >> broker uses the host/port of the leader replica to send the fetch > >> request. > >> >> I am not sure if we need a wire protocol change in this case. I was > >> >> imagining that we will just add a new broker config, sth like > >> >> replication.socket.protocol. Base on this config, the controller will > >> pick > >> >> the right host/port to include in the request. > >> >> > >> >> 4. Should we plan to support security just on the new java clients? > >> >> Supporting security in both the old and the new clients adds more > work > >> and > >> >> gives people less incentive to migrate off the old clients. > >> >> > >> >> Thanks, > >> >> > >> >> Jun > >> >> > >> >> > >> >> On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira < > gshap...@cloudera.com> > >> >> wrote: > >> >> > >> >> > Hi Everyone, > >> >> > > >> >> > One of the pre-requisites we have for supporting multiple security > >> >> > protocols (SSL, Kerberos) is to support them on separate ports. > >> >> > > >> >> > This is done in KAFKA-1684 (The SSL Patch), but that patch > addresses > >> >> > several different issues - Multiple ports, enriching the channels, > SSL > >> >> > implementation - which makes it more challenging to review and to > >> test. > >> >> > > >> >> > I'd like to split this into 3 separate patches: multi-port brokers, > >> >> > enriching SocketChannel, and the actual security implementations. > >> >> > > >> >> > Since even just adding support for multiple listeners per broker is > >> >> > somewhat involved and touches multiple components, I wrote a short > >> design > >> >> > document that covers the necessary changes and the upgrade process: > >> >> > > >> >> > > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers > >> >> > > >> >> > Comments are more than welcome :) > >> >> > > >> >> > If this is acceptable, hope to have a patch ready in few days. > >> >> > > >> >> > Gwen Shapira > >> >> > > >> >> > >> >
