*Awesome* explanation Rajini - thank you!
Just to confirm: the SASL/PLAIN configs would only be for the interbroker communication, correct? Meaning, beyond your recommended changes to server.properties, and the addition of the new jaas.conf file, the producers (Ruby clients) wouldn't need to authenticate, correct? Thanks again for all the great help so far, you've already helped me more than you know! Zac ________________________________ From: Rajini Sivaram <rajinisiva...@googlemail.com> Sent: Monday, November 21, 2016 3:53:47 AM To: users@kafka.apache.org Subject: Re: Can Kafka/SSL be terminated at a load balancer? Zac, *advertised.listeners* is used to make client connections from producers/consumers as well as for client-side connections for inter-broker communication. In your scenario, setting it to *PLAINTEXT://mykafka01* would work for inter-broker, bypassing the load balancer, but clients would also then attempt to connect directly to *mykafka01*. Setting it to *SSL://mybalancer01* would work for producers/consumers, but brokers would try to connect to *mybalancer01* using PLAINTEXT. Unfortunately neither works for both. You need two endpoints, one for inter-broker that bypasses *mybalancer01* and another for clients that uses *mybalancer01*. With the current Kafka configuration, you would require two security protocols to enable two endpoints. You could enable SSL in Kafka (using self-signed certificates if you need) for one of the two endpoints to overcome this limitation. But presumably you have a secure internal network running Kafka and want to avoid the cost of encryption in Kafka. The simplest solution I can think of is to use SASL_PLAINTEXT using SASL/PLAIN for inter-broker as a workaround. The configuration options in server.properties would look like: listeners=PLAINTEXT://:9093,SASL_PLAINTEXT://:9092 advertised.listeners=PLAINTEXT://mybalancer01.example.com:9093 ,SASL_PLAINTEXT://mykafka01.example.com:9092 security.inter.broker.protocol=SASL_PLAINTEXT sasl.enabled.mechanisms=PLAIN sasl.mechanism.inter.broker.protocol=PLAIN You also need a JAAS configuration file configured for the broker JVM ( *KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/jaas.conf"*) . See https://kafka.apache.org/documentation#security_sasl for configuring SASL.* jaas.conf* would look something like: KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka" user_kafka="kafka-password" password="kafka-password"; }; Hope that helps. On Fri, Nov 18, 2016 at 6:39 PM, Zac Harvey <zac.har...@welltok.com> wrote: > Thanks again Rajini! > > > One last followup question, if you don't mind. You said that my > server.properties file should look something like this: > > > listeners=SSL://:9093 > advertised.listeners=SSL://mybalancer01.example.com:9093 > security.inter.broker.protocol=SSL > > However, please remember that I'm looking for the load balancer to > terminate SSL, meaning that (my desired) communication between the load > balancer and Kafka would be over plaintext (not SSL). In other words: > > Ruby Producers/Clients <----SSL:9093----> Load Balancer <---- > Plaintext:9092 ----> Kafka > > So producers/client connect to the load balancer over SSL and port 9093, > but then the load balancer communicates with Kafka over plaintext and port > 9092. > > I also don't need inter broker communication to be SSL; it can be > plaintext. > > If this is the case, do I still need to change server.properties, or can I > leave it like so: > > listeners=plaintext://:9092 > advertised.listeners=plaintext://mybalancer01.example.com:9092 > > Or could it just be: > > listeners=plaintext://:9092 > advertised.listeners=plaintext://mykafka01.example.com:9092 > > Thanks again! > Zac > > > > > > ________________________________ > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Friday, November 18, 2016 9:57:22 AM > To: users@kafka.apache.org > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > You should set advertised.listeners rather than the older > advertised.host.name property in server.properties: > > > - listeners=SSL://:9093 > - advertised.listeners=SSL://mybalancer01.example.com:9093 > - security.inter.broker.protocol=SSL > > > If your listeners are on particular interfaces, you can set address in the > 'listeners' property too. > > > If you want inter-broker communication to bypass the SSL proxy, you would > need another security protocol that can be used for inter-broker > communication (PLAINTEXT in the example below). > > > > - listeners=SSL://:9093,PLAINTEXT://:9092 > - advertised.listeners=SSL://mybalancer01.example.com:9093,PLAINTEXT:// > mykafka01.example.com:9092 > - security.inter.broker.protocol=PLAINTEXT > > I haven't used the Ruby clients, so I am not sure about client > configuration. With Java clients, if you don't specify truststore, the > default trust stores are used, so with trusted CA-signed certificates, no > additional client configuration is required. You can test your installation > using the console producer and consumer that are shipped with Kafka to make > sure it is working before you run with Ruby clients. > > > > On Fri, Nov 18, 2016 at 1:23 PM, Zac Harvey <zac.har...@welltok.com> > wrote: > > > > > Thanks Rajini, > > > > > > So currently one of our Kafka nodes is 'mykafka01.example.com', and in > > its server.properties file, I have advertised.host.name=mykafka01 > > .example.com. Our load balancer lives at mybalancer01.example.com, and > > this what producers will connect to (over SSL) to send messages to Kafka. > > > > > > It sounds like you're saying I need to change my Kafka node's > > server.properties to have advertised.host.name=mybalancer01.example.com, > > yes? If not, can you perhaps provide a quick snippet of the changes I > would > > need to make to server.properties? > > > > > > Again, the cert served by the balancer will be a highly-trusted (root > > CA-signed) certificate that all clients will natively trust. > Interestingly > > enough, most (if not all) the Kafka producers/clients will be written in > > Ruby (using the zendesk Kafka-Ruby gem<https://github.com/ > > zendesk/ruby-kafka>), so there wont be any JKS configuration options > > available for those Ruby clients. > > > > > > Besides making the change to server.properties that I mentioned above, > are > > there any other client-side configs that will need to be made for the > Ruby > > clients to connect over SSL? > > > > > > Thank you enormously here! > > > > > > Best, > > > > Zac > > > > > > ________________________________ > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > Sent: Friday, November 18, 2016 5:15:13 AM > > To: users@kafka.apache.org > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > Zac, > > > > Kafka has its own built-in load-balancing mechanism based on partition > > assignment. Requests are processed by partition leaders, distributing > load > > across the brokers in the cluster. If you want to put a proxy like > HAProxy > > with SSL termination in front of your brokers for added security, you can > > do that. You can have completely independent trust chain between > > clients->proxy and proxy->broker. You need to configure Kafka brokers > with > > the proxy host as the host in the advertised listeners for the security > > protocol used by clients. > > > > On Thu, Nov 17, 2016 at 9:44 PM, Zac Harvey <zac.har...@welltok.com> > > wrote: > > > > > We have two Kafka nodes and for reasons outside of this question, would > > > like to set up a load balancer to terminate SSL with producers > (clients). > > > The SSL cert hosted by the load balancer will be signed by trusted/root > > CA > > > that clients should natively trust. > > > > > > > > > Is this possible to do, or does Kafka somehow require SSL to be setup > > > directly on the Kafka servers themselves? > > > > > > > > > Thanks! > > > > > > > > > > > -- > > Regards, > > > > Rajini > > > > > > -- > Regards, > > Rajini > -- Regards, Rajini
