The good news is that I'm not using a map to represent protocol list. The bad news is that as mentioned in the wiki: producers, consumers and broker configuration specify "security protocol", so we'll know which host/port pair to use for communication. This implicitly assumes there will be only one host/port per protocol.
I'll think a bit on how this assumption can be relaxed. Gwen On Tue, Dec 2, 2014 at 3:14 PM, Todd Palino <tpal...@gmail.com> wrote: > Gwen - just my reading of what we could expect from what you had described > so far. Without having gone into implementation details, there didn't seem > to be anything that would block the ability to run two ports with the same > protocol configuration, at least from the way you proposed to represent it > in Zookeeper. I'd just like to not go down the path of using something like > a map for representing the protocol list that would eliminate this > possibility, unless there's a pretty good reason to do so. > > -Todd > > On Tue, Dec 2, 2014 at 3:00 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > >> Hey Todd, >> >> You say "lose the ability" - you mean this ability actually exist now? >> Or is this something you hope the new patch will support? >> >> Gwen >> >> On Tue, Dec 2, 2014 at 2:08 PM, Todd Palino <tpal...@gmail.com> wrote: >> > Leaving aside the rest of this, on #1, while I consider being able to >> > advertise the ports a good idea, I don't want to lose the ability for >> > maintaining multiple ports with the same protocol. For example, being >> able >> > to have 2 plaintext ports, one that only brokers communicate over, and >> one >> > that general clients use. The ability to segregate this traffic is useful >> > in a number of situations, over and above other controls like quotas, and >> > is relatively easy to do once we support multiple ports. >> > >> > -Todd >> > >> > >> > On Tue, Dec 2, 2014 at 1:58 PM, Jun Rao <jun...@gmail.com> wrote: >> > >> >> Hi, Gwen, >> >> >> >> Thanks for writing up the wiki. Some comments below. >> >> >> >> 1. To make it more general, should we support a binding and an >> advertised >> >> host for each protocol (e.g. plaintext, ssl, etc)? We will also need to >> >> figure out how to specify the wildcard binding host. >> >> >> >> 2. Broker format change in ZK >> >> The broker registration in ZK needs to store the host/port for all >> >> protocols. We will need to bump up the version of the broker >> registration >> >> data. Since this is an intra-cluster protocol change, we need an extra >> >> config for rolling upgrades. So, in the first step, each broker is >> upgraded >> >> and is ready to parse brokers registered in the new format, but not >> >> registering using the new format yet. In the second step, when that new >> >> config is enabled, the broker will register using the new format. >> >> >> >> 3. Wire protocol changes. Currently, the broker info is used in the >> >> following requests/responses: TopicMetadataResponse , >> >> ConsumerMetadataResponse, LeaderAndIsrRequest and >> UpdateMetadataRequest. >> >> 3.1 TopicMetadataResponse and ConsumerMetadataResponse: >> >> These two are used between the clients and the broker. I am not sure >> that >> >> we need to make a wire protocol change for them. Currently, the protocol >> >> includes a single host/port pair in those responses. Based on the type >> of >> >> the port on which the request is sent, it seems that we can just pick >> the >> >> corresponding host and port to include in the response. >> >> 3.2 UpdateMetadataRequest: >> >> This is used between the controller and the broker. Since each broker >> needs >> >> to cache the host/port of all protocols, we need to make a wire protocol >> >> change. We also need to change the broker format in MetadataCache >> >> accordingly. This is also an intra-cluster protocol change. So the >> upgrade >> >> path will need to follow that in item 2. >> >> 3.3 LeaderAndIsrRequest: >> >> This is also used between the controller and the broker. The receiving >> >> broker uses the host/port of the leader replica to send the fetch >> request. >> >> I am not sure if we need a wire protocol change in this case. I was >> >> imagining that we will just add a new broker config, sth like >> >> replication.socket.protocol. Base on this config, the controller will >> pick >> >> the right host/port to include in the request. >> >> >> >> 4. Should we plan to support security just on the new java clients? >> >> Supporting security in both the old and the new clients adds more work >> and >> >> gives people less incentive to migrate off the old clients. >> >> >> >> Thanks, >> >> >> >> Jun >> >> >> >> >> >> On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira <gshap...@cloudera.com> >> >> wrote: >> >> >> >> > Hi Everyone, >> >> > >> >> > One of the pre-requisites we have for supporting multiple security >> >> > protocols (SSL, Kerberos) is to support them on separate ports. >> >> > >> >> > This is done in KAFKA-1684 (The SSL Patch), but that patch addresses >> >> > several different issues - Multiple ports, enriching the channels, SSL >> >> > implementation - which makes it more challenging to review and to >> test. >> >> > >> >> > I'd like to split this into 3 separate patches: multi-port brokers, >> >> > enriching SocketChannel, and the actual security implementations. >> >> > >> >> > Since even just adding support for multiple listeners per broker is >> >> > somewhat involved and touches multiple components, I wrote a short >> design >> >> > document that covers the necessary changes and the upgrade process: >> >> > >> >> > >> >> > >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers >> >> > >> >> > Comments are more than welcome :) >> >> > >> >> > If this is acceptable, hope to have a patch ready in few days. >> >> > >> >> > Gwen Shapira >> >> > >> >> >>
