The good news is that I'm not using a map to represent protocol list.

The bad news is that as mentioned in the wiki: producers, consumers
and broker configuration specify "security protocol", so we'll know
which host/port pair to use for communication. This implicitly assumes
there will be only one host/port per protocol.

I'll think a bit on how this assumption can be relaxed.

Gwen

On Tue, Dec 2, 2014 at 3:14 PM, Todd Palino <tpal...@gmail.com> wrote:
> Gwen - just my reading of what we could expect from what you had described
> so far. Without having gone into implementation details, there didn't seem
> to be anything that would block the ability to run two ports with the same
> protocol configuration, at least from the way you proposed to represent it
> in Zookeeper. I'd just like to not go down the path of using something like
> a map for representing the protocol list that would eliminate this
> possibility, unless there's a pretty good reason to do so.
>
> -Todd
>
> On Tue, Dec 2, 2014 at 3:00 PM, Gwen Shapira <gshap...@cloudera.com> wrote:
>
>> Hey Todd,
>>
>> You say "lose the ability" - you mean this ability actually exist now?
>> Or is this something you hope the new patch will support?
>>
>> Gwen
>>
>> On Tue, Dec 2, 2014 at 2:08 PM, Todd Palino <tpal...@gmail.com> wrote:
>> > Leaving aside the rest of this, on #1, while I consider being able to
>> > advertise the ports a good idea, I don't want to lose the ability for
>> > maintaining multiple ports with the same protocol. For example, being
>> able
>> > to have 2 plaintext ports, one that only brokers communicate over, and
>> one
>> > that general clients use. The ability to segregate this traffic is useful
>> > in a number of situations, over and above other controls like quotas, and
>> > is relatively easy to do once we support multiple ports.
>> >
>> > -Todd
>> >
>> >
>> > On Tue, Dec 2, 2014 at 1:58 PM, Jun Rao <jun...@gmail.com> wrote:
>> >
>> >> Hi, Gwen,
>> >>
>> >> Thanks for writing up the wiki. Some comments below.
>> >>
>> >> 1. To make it more general, should we support a binding and an
>> advertised
>> >> host for each protocol (e.g. plaintext, ssl, etc)? We will also need to
>> >> figure out how to specify the wildcard binding host.
>> >>
>> >> 2. Broker format change in ZK
>> >> The broker registration in ZK needs to store the host/port for all
>> >> protocols. We will need to bump up the version of the broker
>> registration
>> >> data. Since this is an intra-cluster protocol change, we need an extra
>> >> config for rolling upgrades. So, in the first step, each broker is
>> upgraded
>> >> and is ready to parse brokers registered in the new format, but not
>> >> registering using the new format yet. In the second step, when that new
>> >> config is enabled, the broker will register using the new format.
>> >>
>> >> 3. Wire protocol changes. Currently, the broker info is used in the
>> >> following requests/responses: TopicMetadataResponse ,
>> >> ConsumerMetadataResponse, LeaderAndIsrRequest  and
>> UpdateMetadataRequest.
>> >> 3.1 TopicMetadataResponse and ConsumerMetadataResponse:
>> >> These two are used between the clients and the broker. I am not sure
>> that
>> >> we need to make a wire protocol change for them. Currently, the protocol
>> >> includes a single host/port pair in those responses. Based on the type
>> of
>> >> the port on which the request is sent, it seems that we can just pick
>> the
>> >> corresponding host and port to include in the response.
>> >> 3.2 UpdateMetadataRequest:
>> >> This is used between the controller and the broker. Since each broker
>> needs
>> >> to cache the host/port of all protocols, we need to make a wire protocol
>> >> change. We also need to change the broker format in MetadataCache
>> >> accordingly. This is also an intra-cluster protocol change. So the
>> upgrade
>> >> path will need to follow that in item 2.
>> >> 3.3 LeaderAndIsrRequest:
>> >> This is also used between the controller and the broker. The receiving
>> >> broker uses the host/port of the leader replica to send the fetch
>> request.
>> >> I am not sure if we need a wire protocol change in this case. I was
>> >> imagining that we will just add a new broker config, sth like
>> >> replication.socket.protocol. Base on this config, the controller will
>> pick
>> >> the right host/port to include in the request.
>> >>
>> >> 4. Should we plan to support security just on the new java clients?
>> >> Supporting security in both the old and the new clients adds more work
>> and
>> >> gives people less incentive to migrate off the old clients.
>> >>
>> >> Thanks,
>> >>
>> >> Jun
>> >>
>> >>
>> >> On Tue, Nov 25, 2014 at 11:13 AM, Gwen Shapira <gshap...@cloudera.com>
>> >> wrote:
>> >>
>> >> > Hi Everyone,
>> >> >
>> >> > One of the pre-requisites we have for supporting multiple security
>> >> > protocols (SSL, Kerberos) is to support them on separate ports.
>> >> >
>> >> > This is done in KAFKA-1684 (The SSL Patch), but that patch addresses
>> >> > several different issues - Multiple ports, enriching the channels, SSL
>> >> > implementation - which makes it more challenging to review and to
>> test.
>> >> >
>> >> > I'd like to split this into 3 separate patches: multi-port brokers,
>> >> > enriching SocketChannel, and  the actual security implementations.
>> >> >
>> >> > Since even just adding support for multiple listeners per broker is
>> >> > somewhat involved and touches multiple components, I wrote a short
>> design
>> >> > document that covers the necessary changes and the upgrade process:
>> >> >
>> >> >
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers
>> >> >
>> >> > Comments are more than welcome :)
>> >> >
>> >> > If this is acceptable, hope to have a patch ready in few days.
>> >> >
>> >> > Gwen Shapira
>> >> >
>> >>
>>

Reply via email to