I doubt the LB solution will work for Kafka. Client needs to connect to the leader of a partition to produce/consume messages. If we put a LB in front of all brokers which means all brokers share the same LB how does the LB figure out the leader? On Mon, Nov 21, 2016 at 10:26 PM Martin Gainty <mgai...@hotmail.com> wrote:
> > > > > ________________________________ > From: Zac Harvey <zac.har...@welltok.com> > Sent: Monday, November 21, 2016 8:59 AM > To: users@kafka.apache.org > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > Thanks again Rajini, > > > Using these configs, would clients connect to the load balancer over > SSL/9093? And then would I configure the load balancer to forward traffic > from SSL/9093 to plaintext/9093? > > MG>Zach > > MG>i could be wrong but SSL port != plaintext port ..but consider: > > MG>consider recent testcase where all traffic around a certain location > gets bogged with DOS attacks > > MG>what are the legitimate role(s) of the LB when SSL Traffic and HTTP1.1 > Traffic and FTP Traffic are ALL blocked? > > MG>LB should never be stripping SSL headers to redirect to PlainText > because you are not rerouting to a faster route > > MG>most net engineers worth their salt will configure their routers to > static routes to loop around bogged-down routers > > MG>WDYT? > > Thanks again, just still a little uncertain about the traffic/ports coming > into the load balancer! > > > Best, > > Zac > > ________________________________ > From: Rajini Sivaram <rajinisiva...@googlemail.com> > Sent: Monday, November 21, 2016 8:48:41 AM > To: users@kafka.apache.org > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > Zac, > > Yes, that is correct. Ruby clients will not be authenticated by Kafka. They > talk SSL to the load balancer and the load balancer uses PLAINTEXT without > authentication to talk to Kafka. > > On Mon, Nov 21, 2016 at 1:29 PM, Zac Harvey <zac.har...@welltok.com> > wrote: > > > *Awesome* explanation Rajini - thank you! > > > > > > Just to confirm: the SASL/PLAIN configs would only be for the interbroker > > communication, correct? Meaning, beyond your recommended changes to > > server.properties, and the addition of the new jaas.conf file, the > > producers (Ruby clients) wouldn't need to authenticate, correct? > > > > > > Thanks again for all the great help so far, you've already helped me more > > than you know! > > > > > > Zac > > > > ________________________________ > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > Sent: Monday, November 21, 2016 3:53:47 AM > > To: users@kafka.apache.org > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > Zac, > > > > *advertised.listeners* is used to make client connections from > > producers/consumers as well as for client-side connections for > inter-broker > > communication. In your scenario, setting it to *PLAINTEXT://mykafka01* > > would work for inter-broker, bypassing the load balancer, but clients > would > > also then attempt to connect directly to *mykafka01*. Setting it to > > *SSL://mybalancer01* would work for producers/consumers, but brokers > would > > try to connect to *mybalancer01* using PLAINTEXT. Unfortunately neither > > works for both. You need two endpoints, one for inter-broker that > bypasses > > *mybalancer01* and another for clients that uses *mybalancer01*. With the > > current Kafka configuration, you would require two security protocols to > > enable two endpoints. > > > > You could enable SSL in Kafka (using self-signed certificates if you > need) > > for one of the two endpoints to overcome this limitation. But presumably > > you have a secure internal network running Kafka and want to avoid the > cost > > of encryption in Kafka. The simplest solution I can think of is to use > > SASL_PLAINTEXT using SASL/PLAIN for inter-broker as a workaround. The > > configuration options in server.properties would look like: > > > > listeners=PLAINTEXT://:9093,SASL_PLAINTEXT://:9092 > > > > advertised.listeners=PLAINTEXT://mybalancer01.example.com:9093 > > ,SASL_PLAINTEXT://mykafka01.example.com:9092 > > > > security.inter.broker.protocol=SASL_PLAINTEXT > > > > sasl.enabled.mechanisms=PLAIN > > > > sasl.mechanism.inter.broker.protocol=PLAIN > > > > > > You also need a JAAS configuration file configured for the broker JVM ( > > *KAFKA_OPTS="-Djava.security.auth.login.config=/kafka/jaas.conf"*) . See > > https://kafka.apache.org/documentation#security_sasl for configuring > > SASL.* > > jaas.conf* would look something like: > > > > KafkaServer { > > > > org.apache.kafka.common.security.plain.PlainLoginModule required > > > > username="kafka" > > > > user_kafka="kafka-password" > > > > password="kafka-password"; > > > > }; > > > > > > Hope that helps. > > > > > > On Fri, Nov 18, 2016 at 6:39 PM, Zac Harvey <zac.har...@welltok.com> > > wrote: > > > > > Thanks again Rajini! > > > > > > > > > One last followup question, if you don't mind. You said that my > > > server.properties file should look something like this: > > > > > > > > > listeners=SSL://:9093 > > > advertised.listeners=SSL://mybalancer01.example.com:9093 > > > security.inter.broker.protocol=SSL > > > > > > However, please remember that I'm looking for the load balancer to > > > terminate SSL, meaning that (my desired) communication between the load > > > balancer and Kafka would be over plaintext (not SSL). In other words: > > > > > > Ruby Producers/Clients <----SSL:9093----> Load Balancer <---- > > > Plaintext:9092 ----> Kafka > > > > > > So producers/client connect to the load balancer over SSL and port > 9093, > > > but then the load balancer communicates with Kafka over plaintext and > > port > > > 9092. > > > > > > I also don't need inter broker communication to be SSL; it can be > > > plaintext. > > > > > > If this is the case, do I still need to change server.properties, or > can > > I > > > leave it like so: > > > > > > listeners=plaintext://:9092 > > > advertised.listeners=plaintext://mybalancer01.example.com:9092 > > > > > > Or could it just be: > > > > > > listeners=plaintext://:9092 > > > advertised.listeners=plaintext://mykafka01.example.com:9092 > > > > > > Thanks again! > > > Zac > > > > > > > > > > > > > > > > > > ________________________________ > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > > Sent: Friday, November 18, 2016 9:57:22 AM > > > To: users@kafka.apache.org > > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > > > You should set advertised.listeners rather than the older > > > advertised.host.name property in server.properties: > > > > > > > > > - listeners=SSL://:9093 > > > - advertised.listeners=SSL://mybalancer01.example.com:9093 > > > - security.inter.broker.protocol=SSL > > > > > > > > > If your listeners are on particular interfaces, you can set address in > > the > > > 'listeners' property too. > > > > > > > > > If you want inter-broker communication to bypass the SSL proxy, you > would > > > need another security protocol that can be used for inter-broker > > > communication (PLAINTEXT in the example below). > > > > > > > > > > > > - listeners=SSL://:9093,PLAINTEXT://:9092 > > > - advertised.listeners=SSL://mybalancer01.example.com:9093, > > PLAINTEXT:// > > > mykafka01.example.com:9092 > > > - security.inter.broker.protocol=PLAINTEXT > > > > > > I haven't used the Ruby clients, so I am not sure about client > > > configuration. With Java clients, if you don't specify truststore, the > > > default trust stores are used, so with trusted CA-signed certificates, > no > > > additional client configuration is required. You can test your > > installation > > > using the console producer and consumer that are shipped with Kafka to > > make > > > sure it is working before you run with Ruby clients. > > > > > > > > > > > > On Fri, Nov 18, 2016 at 1:23 PM, Zac Harvey <zac.har...@welltok.com> > > > wrote: > > > > > > > > > > > Thanks Rajini, > > > > > > > > > > > > So currently one of our Kafka nodes is 'mykafka01.example.com', and > in > > > > its server.properties file, I have advertised.host.name=mykafka01 > > > > .example.com. Our load balancer lives at mybalancer01.example.com, > and > > > > this what producers will connect to (over SSL) to send messages to > > Kafka. > > > > > > > > > > > > It sounds like you're saying I need to change my Kafka node's > > > > server.properties to have advertised.host.name=mybalance > > r01.example.com, > > > > yes? If not, can you perhaps provide a quick snippet of the changes I > > > would > > > > need to make to server.properties? > > > > > > > > > > > > Again, the cert served by the balancer will be a highly-trusted (root > > > > CA-signed) certificate that all clients will natively trust. > > > Interestingly > > > > enough, most (if not all) the Kafka producers/clients will be written > > in > > > > Ruby (using the zendesk Kafka-Ruby gem<https://github.com/ > > > > zendesk/ruby-kafka>), so there wont be any JKS configuration options > > > > available for those Ruby clients. > > > > > > > > > > > > Besides making the change to server.properties that I mentioned > above, > > > are > > > > there any other client-side configs that will need to be made for the > > > Ruby > > > > clients to connect over SSL? > > > > > > > > > > > > Thank you enormously here! > > > > > > > > > > > > Best, > > > > > > > > Zac > > > > > > > > > > > > ________________________________ > > > > From: Rajini Sivaram <rajinisiva...@googlemail.com> > > > > Sent: Friday, November 18, 2016 5:15:13 AM > > > > To: users@kafka.apache.org > > > > Subject: Re: Can Kafka/SSL be terminated at a load balancer? > > > > > > > > Zac, > > > > > > > > Kafka has its own built-in load-balancing mechanism based on > partition > > > > assignment. Requests are processed by partition leaders, distributing > > > load > > > > across the brokers in the cluster. If you want to put a proxy like > > > HAProxy > > > > with SSL termination in front of your brokers for added security, you > > can > > > > do that. You can have completely independent trust chain between > > > > clients->proxy and proxy->broker. You need to configure Kafka brokers > > > with > > > > the proxy host as the host in the advertised listeners for the > security > > > > protocol used by clients. > > > > > > > > On Thu, Nov 17, 2016 at 9:44 PM, Zac Harvey <zac.har...@welltok.com> > > > > wrote: > > > > > > > > > We have two Kafka nodes and for reasons outside of this question, > > would > > > > > like to set up a load balancer to terminate SSL with producers > > > (clients). > > > > > The SSL cert hosted by the load balancer will be signed by > > trusted/root > > > > CA > > > > > that clients should natively trust. > > > > > > > > > > > > > > > Is this possible to do, or does Kafka somehow require SSL to be > setup > > > > > directly on the Kafka servers themselves? > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Rajini > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Rajini > > > > > > > > > > > -- > > Regards, > > > > Rajini > > > > > > -- > Regards, > > Rajini >
