Re: Add Nginx in front of Kafka cluster?

I have 2 dedicated ports on Nginx that accepts filebeat messages in SSL format, it then forward those messages to those 2 Kafka brokers in PLAINTEXT. The Nginx server does accept traffic on other ports, but those traffic are never forwarded to Kafka brokers. And the 2 Kafka brokers only listen

Re: Add Nginx in front of Kafka cluster?

parties = ports * On Thu, Sep 14, 2017 at 8:04 PM, Ali Akhtar wrote: > I would try to put the SSL on different ports than what you're sending > kafka to. Make sure the kafka ports don't do anything except communicate in > plaintext, put all 3rd parties on different

Re: Add Nginx in front of Kafka cluster?

I would try to put the SSL on different ports than what you're sending kafka to. Make sure the kafka ports don't do anything except communicate in plaintext, put all 3rd parties on different parties. On Thu, Sep 14, 2017 at 7:23 PM, Yongtao You wrote: > Does the

Re: Add Nginx in front of Kafka cluster?

Does the following message mean broker 6 is having trouble talking to broker 7? Broker 6's advertised listener is "PLAINTEXT://nginx:9906" and Broker 7's advertised listener is "PLAINTEXT://nginx:9907". However, on nginx server, port 9906 and 9907 are both SSL ports because that's what

Re: Add Nginx in front of Kafka cluster?

You are correct, that error message was a result of my misconfiguration. I've corrected that. Although filebeat still can't send messages to Kafka. In the Nginx log, I see the following: 2017/09/14 21:35:09 [info] 4030#4030: *60056 SSL_do_handshake() failed (SSL: error:140760FC:SSL

Re: Add Nginx in front of Kafka cluster?

If you ssh to the server where you got this error, are you able to ping the ip of node 7 on the port its trying to reach? On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You wrote: > I'm getting a lot of these in the server.log: > > > [2017-09-14 20:18:32,753] WARN Connection to

Re: Add Nginx in front of Kafka cluster?

I'm getting a lot of these in the server.log: [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) where node 7 is another broker in the cluster. Thanks. -Yongtao On Thursday, September 14,

Re: Add Nginx in front of Kafka cluster?

I got errors saying the other brokers are not reachable, or something like that. Let me dig up the exact error messages. I am guessing the problem was that the advertised listeners are of PLAINTEXT format, but the Nginx requires SSL. But I could be wrong. Thanks! -Yongtao On Thursday,

Re: Add Nginx in front of Kafka cluster?

Hi! I ask: Wouldn't it be more advisable that you send metrics through logtash sending directly to kafka brokers without going through Nginx and mounting a virtual ip (corosync/pacemaker) in the kafka cluster? Regards! 2017-09-14 13:32 GMT+02:00 Yongtao You : >

Add Nginx in front of Kafka cluster?

Hi, I would like to know the right way to setup a Kafka cluster with Nginx in front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on 2 different hosts; and an Nginx server running on another host. Nginx will listen on 2 different ports, and each will forward to one Kafka