Leaving aside the rest of this, on #1, while I consider being able to advertise the ports a good idea, I don't want to lose the ability for maintaining multiple ports with the same protocol. For example, being able to have 2 plaintext ports, one that only brokers communicate over, and one that general clients use. The ability to segregate this traffic is useful in a number of situations, over and above other controls like quotas, and is relatively easy to do once we support multiple ports.
-Todd On Tue, Dec 2, 2014 at 1:58 PM, Jun Rao <jun...@gmail.com> wrote: > Hi, Gwen, > > Thanks for writing up the wiki. Some comments below. > > 1. To make it more general, should we support a binding and an advertised > host for each protocol (e.g. plaintext, ssl, etc)? We will also need to > figure out how to specify the wildcard binding host. > > 2. Broker format change in ZK > The broker registration in ZK needs to store the host/port for all > protocols. We will need to bump up the version of the broker registration > data. Since this is an intra-cluster protocol change, we need an extra > config for rolling upgrades. So, in the first step, each broker is upgraded > and is ready to parse brokers registered in the new format, but not > registering using the new format yet. In the second step, when that new > config is enabled, the broker will register using the new format. > > 3. Wire protocol changes. Currently, the broker info is used in the > following requests/responses: TopicMetadataResponse , > ConsumerMetadataResponse, LeaderAndIsrRequest and UpdateMetadataRequest. > 3.1 TopicMetadataResponse and ConsumerMetadataResponse: > These two are used between the clients and the broker. I am not sure that > we need to make a wire protocol change for them. Currently, the protocol > includes a single host/port pair in those responses. Based on the type of > the port on which the request is sent, it seems that we can just pick the > corresponding host and port to include in the response. > 3.2 UpdateMetadataRequest: > This is used between the controller and the broker. Since each broker needs > to cache the host/port of all protocols, we need to make a wire protocol > change. We also need to change the broker format in MetadataCache > accordingly. This is also an intra-cluster protocol change. So the upgrade > path will need to follow that in item 2. > 3.3 LeaderAndIsrRequest: > This is also used between the controller and the broker. The receiving > broker uses the host/port of the leader replica to send the fetch request. > I am not sure if we need a wire protocol change in this case. I was > imagining that we will just add a new broker config, sth like > replication.socket.protocol. Base on this config, the controller will pick > the right host/port to include in the request. > > 4. Should we plan to support security just on the new java clients? > Supporting security in both the old and the new clients adds more work and > gives people less incentive to migrate off the old clients. > > Thanks, > > Jun > > > On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > Hi Everyone, > > > > One of the pre-requisites we have for supporting multiple security > > protocols (SSL, Kerberos) is to support them on separate ports. > > > > This is done in KAFKA-1684 (The SSL Patch), but that patch addresses > > several different issues - Multiple ports, enriching the channels, SSL > > implementation - which makes it more challenging to review and to test. > > > > I'd like to split this into 3 separate patches: multi-port brokers, > > enriching SocketChannel, and the actual security implementations. > > > > Since even just adding support for multiple listeners per broker is > > somewhat involved and touches multiple components, I wrote a short design > > document that covers the necessary changes and the upgrade process: > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers > > > > Comments are more than welcome :) > > > > If this is acceptable, hope to have a patch ready in few days. > > > > Gwen Shapira > > >
