Hey Todd, You say "lose the ability" - you mean this ability actually exist now? Or is this something you hope the new patch will support?
Gwen On Tue, Dec 2, 2014 at 2:08 PM, Todd Palino <tpal...@gmail.com> wrote: > Leaving aside the rest of this, on #1, while I consider being able to > advertise the ports a good idea, I don't want to lose the ability for > maintaining multiple ports with the same protocol. For example, being able > to have 2 plaintext ports, one that only brokers communicate over, and one > that general clients use. The ability to segregate this traffic is useful > in a number of situations, over and above other controls like quotas, and > is relatively easy to do once we support multiple ports. > > -Todd > > > On Tue, Dec 2, 2014 at 1:58 PM, Jun Rao <jun...@gmail.com> wrote: > >> Hi, Gwen, >> >> Thanks for writing up the wiki. Some comments below. >> >> 1. To make it more general, should we support a binding and an advertised >> host for each protocol (e.g. plaintext, ssl, etc)? We will also need to >> figure out how to specify the wildcard binding host. >> >> 2. Broker format change in ZK >> The broker registration in ZK needs to store the host/port for all >> protocols. We will need to bump up the version of the broker registration >> data. Since this is an intra-cluster protocol change, we need an extra >> config for rolling upgrades. So, in the first step, each broker is upgraded >> and is ready to parse brokers registered in the new format, but not >> registering using the new format yet. In the second step, when that new >> config is enabled, the broker will register using the new format. >> >> 3. Wire protocol changes. Currently, the broker info is used in the >> following requests/responses: TopicMetadataResponse , >> ConsumerMetadataResponse, LeaderAndIsrRequest and UpdateMetadataRequest. >> 3.1 TopicMetadataResponse and ConsumerMetadataResponse: >> These two are used between the clients and the broker. I am not sure that >> we need to make a wire protocol change for them. Currently, the protocol >> includes a single host/port pair in those responses. Based on the type of >> the port on which the request is sent, it seems that we can just pick the >> corresponding host and port to include in the response. >> 3.2 UpdateMetadataRequest: >> This is used between the controller and the broker. Since each broker needs >> to cache the host/port of all protocols, we need to make a wire protocol >> change. We also need to change the broker format in MetadataCache >> accordingly. This is also an intra-cluster protocol change. So the upgrade >> path will need to follow that in item 2. >> 3.3 LeaderAndIsrRequest: >> This is also used between the controller and the broker. The receiving >> broker uses the host/port of the leader replica to send the fetch request. >> I am not sure if we need a wire protocol change in this case. I was >> imagining that we will just add a new broker config, sth like >> replication.socket.protocol. Base on this config, the controller will pick >> the right host/port to include in the request. >> >> 4. Should we plan to support security just on the new java clients? >> Supporting security in both the old and the new clients adds more work and >> gives people less incentive to migrate off the old clients. >> >> Thanks, >> >> Jun >> >> >> On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> >> > Hi Everyone, >> > >> > One of the pre-requisites we have for supporting multiple security >> > protocols (SSL, Kerberos) is to support them on separate ports. >> > >> > This is done in KAFKA-1684 (The SSL Patch), but that patch addresses >> > several different issues - Multiple ports, enriching the channels, SSL >> > implementation - which makes it more challenging to review and to test. >> > >> > I'd like to split this into 3 separate patches: multi-port brokers, >> > enriching SocketChannel, and the actual security implementations. >> > >> > Since even just adding support for multiple listeners per broker is >> > somewhat involved and touches multiple components, I wrote a short design >> > document that covers the necessary changes and the upgrade process: >> > >> > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers >> > >> > Comments are more than welcome :) >> > >> > If this is acceptable, hope to have a patch ready in few days. >> > >> > Gwen Shapira >> > >>
