Re: KTable-KTable Join Semantics on NULL Key
Perhaps a clarification to what Damian said: It is shown in the (HTML) table at the link you shared [1] what happens when you get null values for a key. We also have slightly better join documentation at [2], the content/text of which we are currently migrating over to the official Apache Kafka documentation for the Streams API (under kafka.apache.org/documentation/streams). [1] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Join+Semantics#KafkaStreamsJoinSemantics-KTable-KTableJoin [2] https://docs.confluent.io/current/streams/developer-guide.html#kstream-kstream-join On Fri, Sep 8, 2017 at 3:19 PM, Damian Guy wrote: > It is shown in the table what happens when you get null values for a key. > > On Fri, 8 Sep 2017 at 12:31 Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Hi Kafka Users, > > > > KTable-KTable Join Semantics is explained in detailed [here][1]. But, > > it's not clear when the input record is , some times the output > > records are generated and in some cases it's not. > > > > It will be helpful, if someone explain on how the output records are > > generated for all the 3 types of joins on receiving a record with NULL > > value. > > > > [1]: https://cwiki.apache.org/confluence/display/KAFKA/ > > Kafka+Streams+Join+Semantics#KafkaStreamsJoinSemantics-KTable-KTableJoin > > > > -- Kamal > > >
Add Nginx in front of Kafka cluster?
Hi, I would like to know the right way to setup a Kafka cluster with Nginx in front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on 2 different hosts; and an Nginx server running on another host. Nginx will listen on 2 different ports, and each will forward to one Kafka broker. Producers will connect to one of the 2 ports on the Nginx host. Nginx-Host: listens on 9000 ssl (forward to :9092 in plain text); 9001 ssl (forward to :9092 in plain text); Kafka-Host-0: listeners=PLAINTEXT://:9092; advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: listeners=PLAINTEXT://:9092; advertised.listeners=PLAINTEXT://:9001 Ports on Nginx will have SSL enabled so that messages sent from producers to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain text since it's on the internal network. Why have producers go through Nginx? The main reason is that producers will only need to open their firewall to a single IP so that even later on when I add another Kafka broker, I don't need to modify the firewall of all the producers. My problem is that I can't make the above setup work. Brokers are unable to talk to one another. :( So, what's the right way to do this? Anyone has experience setting up something similar? Or any recommendations for a different setup that will not require changes on the producer's side when new Kafka brokers are added? Thanks!Yongtao PS. The producers in question are Filebeats (https://www.elastic.co/products/beats/filebeat).
Re: Add Nginx in front of Kafka cluster?
Hi! I ask: Wouldn't it be more advisable that you send metrics through logtash sending directly to kafka brokers without going through Nginx and mounting a virtual ip (corosync/pacemaker) in the kafka cluster? Regards! 2017-09-14 13:32 GMT+02:00 Yongtao You : > Hi, > I would like to know the right way to setup a Kafka cluster with Nginx in > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on > 2 different hosts; and an Nginx server running on another host. Nginx will > listen on 2 different ports, and each will forward to one Kafka broker. > Producers will connect to one of the 2 ports on the Nginx host. > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > text); 9001 ssl (forward to :9092 in plain text); > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > listeners=PLAINTEXT://:9092; advertised.listeners= > PLAINTEXT://:9001 > Ports on Nginx will have SSL enabled so that messages sent from producers > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > text since it's on the internal network. > Why have producers go through Nginx? The main reason is that producers > will only need to open their firewall to a single IP so that even later on > when I add another Kafka broker, I don't need to modify the firewall of all > the producers. > My problem is that I can't make the above setup work. Brokers are unable > to talk to one another. :( > So, what's the right way to do this? Anyone has experience setting up > something similar? Or any recommendations for a different setup that will > not require changes on the producer's side when new Kafka brokers are added? > > Thanks!Yongtao > PS. The producers in question are Filebeats (https://www.elastic.co/ > products/beats/filebeat). >
Re: Add Nginx in front of Kafka cluster?
How do you know that the brokers don't talk to each other? On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You wrote: > Hi, > I would like to know the right way to setup a Kafka cluster with Nginx in > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on > 2 different hosts; and an Nginx server running on another host. Nginx will > listen on 2 different ports, and each will forward to one Kafka broker. > Producers will connect to one of the 2 ports on the Nginx host. > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > text); 9001 ssl (forward to :9092 in plain text); > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > listeners=PLAINTEXT://:9092; advertised.listeners= > PLAINTEXT://:9001 > Ports on Nginx will have SSL enabled so that messages sent from producers > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > text since it's on the internal network. > Why have producers go through Nginx? The main reason is that producers > will only need to open their firewall to a single IP so that even later on > when I add another Kafka broker, I don't need to modify the firewall of all > the producers. > My problem is that I can't make the above setup work. Brokers are unable > to talk to one another. :( > So, what's the right way to do this? Anyone has experience setting up > something similar? Or any recommendations for a different setup that will > not require changes on the producer's side when new Kafka brokers are added? > > Thanks!Yongtao > PS. The producers in question are Filebeats (https://www.elastic.co/ > products/beats/filebeat). >
Re: Add Nginx in front of Kafka cluster?
That sounds like a viable option. Thanks! -Yongtao On Thursday, September 14, 2017, 7:47:14 PM GMT+8, Jorge Pérez wrote: Hi! I ask: Wouldn't it be more advisable that you send metrics through logtash sending directly to kafka brokers without going through Nginx and mounting a virtual ip (corosync/pacemaker) in the kafka cluster? Regards! 2017-09-14 13:32 GMT+02:00 Yongtao You : > Hi, > I would like to know the right way to setup a Kafka cluster with Nginx in > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on > 2 different hosts; and an Nginx server running on another host. Nginx will > listen on 2 different ports, and each will forward to one Kafka broker. > Producers will connect to one of the 2 ports on the Nginx host. > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > text); 9001 ssl (forward to :9092 in plain text); > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > listeners=PLAINTEXT://:9092; advertised.listeners= > PLAINTEXT://:9001 > Ports on Nginx will have SSL enabled so that messages sent from producers > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > text since it's on the internal network. > Why have producers go through Nginx? The main reason is that producers > will only need to open their firewall to a single IP so that even later on > when I add another Kafka broker, I don't need to modify the firewall of all > the producers. > My problem is that I can't make the above setup work. Brokers are unable > to talk to one another. :( > So, what's the right way to do this? Anyone has experience setting up > something similar? Or any recommendations for a different setup that will > not require changes on the producer's side when new Kafka brokers are added? > > Thanks!Yongtao > PS. The producers in question are Filebeats (https://www.elastic.co/ > products/beats/filebeat). >
Re: Add Nginx in front of Kafka cluster?
I got errors saying the other brokers are not reachable, or something like that. Let me dig up the exact error messages. I am guessing the problem was that the advertised listeners are of PLAINTEXT format, but the Nginx requires SSL. But I could be wrong. Thanks! -Yongtao On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar wrote: How do you know that the brokers don't talk to each other? On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You wrote: > Hi, > I would like to know the right way to setup a Kafka cluster with Nginx in > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on > 2 different hosts; and an Nginx server running on another host. Nginx will > listen on 2 different ports, and each will forward to one Kafka broker. > Producers will connect to one of the 2 ports on the Nginx host. > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > text); 9001 ssl (forward to :9092 in plain text); > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > listeners=PLAINTEXT://:9092; advertised.listeners= > PLAINTEXT://:9001 > Ports on Nginx will have SSL enabled so that messages sent from producers > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > text since it's on the internal network. > Why have producers go through Nginx? The main reason is that producers > will only need to open their firewall to a single IP so that even later on > when I add another Kafka broker, I don't need to modify the firewall of all > the producers. > My problem is that I can't make the above setup work. Brokers are unable > to talk to one another. :( > So, what's the right way to do this? Anyone has experience setting up > something similar? Or any recommendations for a different setup that will > not require changes on the producer's side when new Kafka brokers are added? > > Thanks!Yongtao > PS. The producers in question are Filebeats (https://www.elastic.co/ > products/beats/filebeat). >
Re: Kafka 11 | Stream Application crashed the brokers
;Ok. I will inspect this further and keep everyone posted on this. -Sameer. On Thu, Sep 14, 2017 at 1:46 AM, Guozhang Wang wrote: > When exactly_once is turned on the transactional id would be set > automatically by the Streams client. > > What I'd inspect is the healthiness of the brokers since the " > *TimeoutException*", if you have metrics on the broker servers regarding > request handler thread idleness / request queue length / request rate etc, > you can monitor that and see what could be the possible causes of the > broker unavailability. > > > Guozhang > > > On Wed, Sep 13, 2017 at 8:26 AM, Sameer Kumar > wrote: > > > Adding more info:- > > > > Hi Guozhang, > > > > I was using exactly_once processing here, I can see this in the client > > logs, however I am not setting transaction id though. > > > > application.id = c-7-e6 > > application.server = > > bootstrap.servers = [172.29.65.190:9092, 172.29.65.192:9092, > > 172.29.65.193:9092] > > buffered.records.per.partition = 1 > > cache.max.bytes.buffering = 2097152000 > > client.id = > > commit.interval.ms = 5000 > > connections.max.idle.ms = 54 > > default.key.serde = class > > org.apache.kafka.common.serialization.Serdes$ByteArraySerde > > default.timestamp.extractor = class > > org.apache.kafka.streams.processor.FailOnInvalidTimestamp > > default.value.serde = class > > org.apache.kafka.common.serialization.Serdes$ByteArraySerde > > key.serde = class org.apache.kafka.common.serialization.Serdes$ > StringSerde > > metadata.max.age.ms = 6 > > metric.reporters = [] > > metrics.num.samples = 2 > > metrics.recording.level = INFO > > metrics.sample.window.ms = 3 > > num.standby.replicas = 0 > > num.stream.threads = 15 > > partition.grouper = class > > org.apache.kafka.streams.processor.DefaultPartitionGrouper > > poll.ms = 100 > > processing.guarantee = exactly_once > > receive.buffer.bytes = 32768 > > reconnect.backoff.max.ms = 1000 > > reconnect.backoff.ms = 50 > > replication.factor = 1 > > request.timeout.ms = 4 > > retry.backoff.ms = 100 > > rocksdb.config.setter = null > > security.protocol = PLAINTEXT > > send.buffer.bytes = 131072 > > state.cleanup.delay.ms = 4611686018427386903 > > state.dir = /data/streampoc/ > > timestamp.extractor = class > > org.apache.kafka.streams.processor.WallclockTimestampExtractor > > value.serde = class org.apache.kafka.common.serialization.Serdes$ > > StringSerde > > windowstore.changelog.additional.retention.ms = 8640 > > zookeeper.connect = > > > > > > On Wed, Sep 13, 2017 at 12:16 PM, Sameer Kumar > > wrote: > > > > > Hi Guozhang, > > > > > > The producer sending data to this topic is not running concurrently > with > > > the stream processing. I had first ingested the data from another > cluster > > > and then have the stream processing ran on it. The producer code is > > written > > > by me and it doesnt have transactions on by default. > > > > > > I will double check if someone else has transaction turned on, but this > > is > > > quite unlikely. Is there someway to verify it through logs. > > > > > > All of this behavior works fine when brokers are run on Kafka 10, this > > > might be because transactions are only available on Kafka11. I am > > > suspecting would there be a case that too much processing is causing > one > > of > > > the brokers to crash. The timeouts are indicating that it is taking > time > > to > > > send data > > > > > > I have tried this behavior also on a another cluster which I > exclusively > > > use it for myself and found the same behavior there as well. > > > > > > What do you think should be our next step so that we can get to the > root > > > of the issue. > > > > > > -Sameer. > > > > > > On Wed, Sep 13, 2017 at 6:14 AM, Guozhang Wang > > wrote: > > > > > >> Hi Sameer, > > >> > > >> If no clients has transactions turned on the `__transaction_state` > > >> internal > > >> topic would not be created at all. So I still suspect that some of > your > > >> clients (maybe not your Streams client, but your Producer client that > is > > >> sending data to the source topic?) has transactions turned on. > > >> > > >> BTW from your logs I saw lots of the following errors on client side: > > >> > > >> 2017-09-11 12:42:34 ERROR RecordCollectorImpl:113 - task [0_6] Error > > >> sending record to topic c-7-e6-KSTREAM-BRANCHCHILD-000 > > >> 007-repartition. > > >> No more offsets will be recorded for this task and the exception will > > >> eventually be thrown > > >> > > >> org.apache.kafka.common.errors.*TimeoutException*: Expiring 13 > > record(s) > > >> for c-7-e6-KSTREAM-BRANCHCHILD-07-repartition-3: 31174 ms has > > >> passed since last append > > >> > > >> 2017-09-11 12:42:36 WARN Sender:511 - Got error produce response with > > >> correlation id 82862 on topic-partition > > >> c-7-e6-KSTREAM-JOINTHIS-18-store-changelog-22, retrying > > >> (2147483646 > > >> attempts left). *Error: NETWORK_EXCEPTION* > > >> > > >> 2017-09-11 12:42:36 ERROR R
Re: Add Nginx in front of Kafka cluster?
I'm getting a lot of these in the server.log: [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) where node 7 is another broker in the cluster. Thanks. -Yongtao On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You wrote: I got errors saying the other brokers are not reachable, or something like that. Let me dig up the exact error messages. I am guessing the problem was that the advertised listeners are of PLAINTEXT format, but the Nginx requires SSL. But I could be wrong. Thanks! -Yongtao On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar wrote: How do you know that the brokers don't talk to each other? On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You wrote: > Hi, > I would like to know the right way to setup a Kafka cluster with Nginx in > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on > 2 different hosts; and an Nginx server running on another host. Nginx will > listen on 2 different ports, and each will forward to one Kafka broker. > Producers will connect to one of the 2 ports on the Nginx host. > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > text); 9001 ssl (forward to :9092 in plain text); > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > listeners=PLAINTEXT://:9092; advertised.listeners= > PLAINTEXT://:9001 > Ports on Nginx will have SSL enabled so that messages sent from producers > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > text since it's on the internal network. > Why have producers go through Nginx? The main reason is that producers > will only need to open their firewall to a single IP so that even later on > when I add another Kafka broker, I don't need to modify the firewall of all > the producers. > My problem is that I can't make the above setup work. Brokers are unable > to talk to one another. :( > So, what's the right way to do this? Anyone has experience setting up > something similar? Or any recommendations for a different setup that will > not require changes on the producer's side when new Kafka brokers are added? > > Thanks!Yongtao > PS. The producers in question are Filebeats (https://www.elastic.co/ > products/beats/filebeat). >
Re: Add Nginx in front of Kafka cluster?
If you ssh to the server where you got this error, are you able to ping the ip of node 7 on the port its trying to reach? On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You wrote: > I'm getting a lot of these in the server.log: > > > [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be > established. Broker may not be available. (org.apache.kafka.clients. > NetworkClient) > > > where node 7 is another broker in the cluster. > > > Thanks. > > -Yongtao > > > On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You < > yongtao_...@yahoo.com> wrote: > > > I got errors saying the other brokers are not reachable, or something like > that. Let me dig up the exact error messages. I am guessing the problem was > that the advertised listeners are of PLAINTEXT format, but the Nginx > requires SSL. But I could be wrong. > > > Thanks! > > -Yongtao > > > On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar < > ali.rac...@gmail.com> wrote: > > > How do you know that the brokers don't talk to each other? > > On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You < > yongtao_...@yahoo.com.invalid> > wrote: > > > Hi, > > I would like to know the right way to setup a Kafka cluster with Nginx in > > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running > on > > 2 different hosts; and an Nginx server running on another host. Nginx > will > > listen on 2 different ports, and each will forward to one Kafka broker. > > Producers will connect to one of the 2 ports on the Nginx host. > > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > > text); 9001 ssl (forward to :9092 in plain text); > > > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > > listeners=PLAINTEXT://:9092; advertised.listeners= > > PLAINTEXT://:9001 > > Ports on Nginx will have SSL enabled so that messages sent from producers > > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > > text since it's on the internal network. > > Why have producers go through Nginx? The main reason is that producers > > will only need to open their firewall to a single IP so that even later > on > > when I add another Kafka broker, I don't need to modify the firewall of > all > > the producers. > > My problem is that I can't make the above setup work. Brokers are unable > > to talk to one another. :( > > So, what's the right way to do this? Anyone has experience setting up > > something similar? Or any recommendations for a different setup that will > > not require changes on the producer's side when new Kafka brokers are > added? > > > > Thanks!Yongtao > > PS. The producers in question are Filebeats (https://www.elastic.co/ > > products/beats/filebeat). > > >
Re: Add Nginx in front of Kafka cluster?
You are correct, that error message was a result of my misconfiguration. I've corrected that. Although filebeat still can't send messages to Kafka. In the Nginx log, I see the following: 2017/09/14 21:35:09 [info] 4030#4030: *60056 SSL_do_handshake() failed (SSL: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol) while SSL handshaking, client: 172.16.16.101, server: 0.0.0.0:9907 where 172.16.16.101 is the host where one of the two Kafka brokers is running. Looks like it tries to connect to port 9907 which is where the other Kafka broker listens on. It's an [info] message so I'm not sure how serious it is, but I don't see messages sent from filebeat in Kafka. :( Thanks! -Yongtao On Thursday, September 14, 2017, 8:31:31 PM GMT+8, Ali Akhtar wrote: If you ssh to the server where you got this error, are you able to ping the ip of node 7 on the port its trying to reach? On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You wrote: > I'm getting a lot of these in the server.log: > > > [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be > established. Broker may not be available. (org.apache.kafka.clients. > NetworkClient) > > > where node 7 is another broker in the cluster. > > > Thanks. > > -Yongtao > > > On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You < > yongtao_...@yahoo.com> wrote: > > > I got errors saying the other brokers are not reachable, or something like > that. Let me dig up the exact error messages. I am guessing the problem was > that the advertised listeners are of PLAINTEXT format, but the Nginx > requires SSL. But I could be wrong. > > > Thanks! > > -Yongtao > > > On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar < > ali.rac...@gmail.com> wrote: > > > How do you know that the brokers don't talk to each other? > > On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You < > yongtao_...@yahoo.com.invalid> > wrote: > > > Hi, > > I would like to know the right way to setup a Kafka cluster with Nginx in > > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running > on > > 2 different hosts; and an Nginx server running on another host. Nginx > will > > listen on 2 different ports, and each will forward to one Kafka broker. > > Producers will connect to one of the 2 ports on the Nginx host. > > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain > > text); 9001 ssl (forward to :9092 in plain text); > > > > Kafka-Host-0: listeners=PLAINTEXT://:9092; > > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1: > > listeners=PLAINTEXT://:9092; advertised.listeners= > > PLAINTEXT://:9001 > > Ports on Nginx will have SSL enabled so that messages sent from producers > > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain > > text since it's on the internal network. > > Why have producers go through Nginx? The main reason is that producers > > will only need to open their firewall to a single IP so that even later > on > > when I add another Kafka broker, I don't need to modify the firewall of > all > > the producers. > > My problem is that I can't make the above setup work. Brokers are unable > > to talk to one another. :( > > So, what's the right way to do this? Anyone has experience setting up > > something similar? Or any recommendations for a different setup that will > > not require changes on the producer's side when new Kafka brokers are > added? > > > > Thanks!Yongtao > > PS. The producers in question are Filebeats (https://www.elastic.co/ > > products/beats/filebeat). > > >
Re: Add Nginx in front of Kafka cluster?
Does the following message mean broker 6 is having trouble talking to broker
7? Broker 6's advertised listener is "PLAINTEXT://nginx:9906" and Broker 7's
advertised listener is "PLAINTEXT://nginx:9907". However, on nginx server, port
9906 and 9907 are both SSL ports because that's what producers (filebeat) send
data to and they need to be encrypted.
[2017-09-14 21:59:32,543] WARN [Controller-6-to-broker-7-send-thread]:
Controller 6 epoch 1 fails to send request (type: UpdateMetadataRequest=,
controllerId=6, controllerEpoch=1, partitionStates={}, liveBrokers=(id=6,
endPoints=(host=nginx, port=9906, listenerName=ListenerName(PLAINTEXT),
securityProtocol=PLAINTEXT), rack=null), (id=7, endPoints=(host=nginx,
port=9907, listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT),
rack=null)) to broker nginx:9907 (id: 7 rack: null). Reconnecting to broker.
(kafka.controller.RequestSendThread)
java.io.IOException: Connection to 7 was disconnected before the response was
read
at
org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:93)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:225)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
On Thursday, September 14, 2017, 9:42:58 PM GMT+8, Yongtao You
wrote:
You are correct, that error message was a result of my misconfiguration. I've
corrected that. Although filebeat still can't send messages to Kafka. In the
Nginx log, I see the following:
2017/09/14 21:35:09 [info] 4030#4030: *60056 SSL_do_handshake() failed (SSL:
error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol) while SSL
handshaking, client: 172.16.16.101, server: 0.0.0.0:9907
where 172.16.16.101 is the host where one of the two Kafka brokers is running.
Looks like it tries to connect to port 9907 which is where the other Kafka
broker listens on. It's an [info] message so I'm not sure how serious it is,
but I don't see messages sent from filebeat in Kafka. :(
Thanks!
-Yongtao
On Thursday, September 14, 2017, 8:31:31 PM GMT+8, Ali Akhtar
wrote:
If you ssh to the server where you got this error, are you able to ping the
ip of node 7 on the port its trying to reach?
On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You wrote:
> I'm getting a lot of these in the server.log:
>
>
> [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be
> established. Broker may not be available. (org.apache.kafka.clients.
> NetworkClient)
>
>
> where node 7 is another broker in the cluster.
>
>
> Thanks.
>
> -Yongtao
>
>
> On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You <
> yongtao_...@yahoo.com> wrote:
>
>
> I got errors saying the other brokers are not reachable, or something like
> that. Let me dig up the exact error messages. I am guessing the problem was
> that the advertised listeners are of PLAINTEXT format, but the Nginx
> requires SSL. But I could be wrong.
>
>
> Thanks!
>
> -Yongtao
>
>
> On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar <
> ali.rac...@gmail.com> wrote:
>
>
> How do you know that the brokers don't talk to each other?
>
> On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You <
> yongtao_...@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > I would like to know the right way to setup a Kafka cluster with Nginx in
> > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running
> on
> > 2 different hosts; and an Nginx server running on another host. Nginx
> will
> > listen on 2 different ports, and each will forward to one Kafka broker.
> > Producers will connect to one of the 2 ports on the Nginx host.
> > Nginx-Host: listens on 9000 ssl (forward to :9092 in plain
> > text); 9001 ssl (forward to :9092 in plain text);
> >
> > Kafka-Host-0: listeners=PLAINTEXT://:9092;
> > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1:
> > listeners=PLAINTEXT://:9092; advertised.listeners=
> > PLAINTEXT://:9001
> > Ports on Nginx will have SSL enabled so that messages sent from producers
> > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain
> > text since it's on the internal network.
> > Why have producers go through Nginx? The main reason is that producers
> > will only need to open their firewall to a single IP so that even later
> on
> > when I add another Kafka broker, I don't need to modify the firewall of
> all
> > the producers.
> > My problem is that I can't make the above setup work. Brokers are unable
> > to talk to one another. :(
> > So, what's the right way to do this? Anyone has experience setting up
> > something similar? Or any recommendations for a different setup that will
> > not require changes on the producer's side when new Kafka brokers are
> added?
> >
> > Thanks!Yongtao
> > PS. The producers in question are Filebeats (https://www.elastic.co/
> > products/beats/filebeat).
> >
>
Re: Add Nginx in front of Kafka cluster?
I would try to put the SSL on different ports than what you're sending
kafka to. Make sure the kafka ports don't do anything except communicate in
plaintext, put all 3rd parties on different parties.
On Thu, Sep 14, 2017 at 7:23 PM, Yongtao You wrote:
> Does the following message mean broker 6 is having trouble talking to
> broker 7? Broker 6's advertised listener is "PLAINTEXT://nginx:9906" and
> Broker 7's advertised listener is "PLAINTEXT://nginx:9907". However, on
> nginx server, port 9906 and 9907 are both SSL ports because that's what
> producers (filebeat) send data to and they need to be encrypted.
>
>
> [2017-09-14 21:59:32,543] WARN [Controller-6-to-broker-7-send-thread]:
> Controller 6 epoch 1 fails to send request (type: UpdateMetadataRequest=,
> controllerId=6, controllerEpoch=1, partitionStates={}, liveBrokers=(id=6,
> endPoints=(host=nginx, port=9906, listenerName=ListenerName(PLAINTEXT),
> securityProtocol=PLAINTEXT), rack=null), (id=7, endPoints=(host=nginx,
> port=9907, listenerName=ListenerName(PLAINTEXT),
> securityProtocol=PLAINTEXT), rack=null)) to broker nginx:9907 (id: 7 rack:
> null). Reconnecting to broker. (kafka.controller.RequestSendThread)
> java.io.IOException: Connection to 7 was disconnected before the response
> was read
> at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(
> NetworkClientUtils.java:93)
> at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.
> scala:225)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
>
>
>
>
> On Thursday, September 14, 2017, 9:42:58 PM GMT+8, Yongtao You
> wrote:
>
>
> You are correct, that error message was a result of my misconfiguration.
> I've corrected that. Although filebeat still can't send messages to Kafka.
> In the Nginx log, I see the following:
>
> 2017/09/14 21:35:09 [info] 4030#4030: *60056 SSL_do_handshake() failed
> (SSL: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
> protocol) while SSL handshaking, client: 172.16.16.101, server:
> 0.0.0.0:9907
>
>
> where 172.16.16.101 is the host where one of the two Kafka brokers is
> running. Looks like it tries to connect to port 9907 which is where the
> other Kafka broker listens on. It's an [info] message so I'm not sure how
> serious it is, but I don't see messages sent from filebeat in Kafka. :(
>
> Thanks!
> -Yongtao
>
> On Thursday, September 14, 2017, 8:31:31 PM GMT+8, Ali Akhtar <
> ali.rac...@gmail.com> wrote:
>
> If you ssh to the server where you got this error, are you able to ping the
> ip of node 7 on the port its trying to reach?
>
> On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You
> wrote:
>
> > I'm getting a lot of these in the server.log:
> >
> >
> > [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be
> > established. Broker may not be available. (org.apache.kafka.clients.
> > NetworkClient)
> >
> >
> > where node 7 is another broker in the cluster.
> >
> >
> > Thanks.
> >
> > -Yongtao
> >
> >
> > On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You <
> > yongtao_...@yahoo.com> wrote:
> >
> >
> > I got errors saying the other brokers are not reachable, or something
> like
> > that. Let me dig up the exact error messages. I am guessing the problem
> was
> > that the advertised listeners are of PLAINTEXT format, but the Nginx
> > requires SSL. But I could be wrong.
> >
> >
> > Thanks!
> >
> > -Yongtao
> >
> >
> > On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar <
> > ali.rac...@gmail.com> wrote:
> >
> >
> > How do you know that the brokers don't talk to each other?
> >
> > On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You <
> > yongtao_...@yahoo.com.invalid>
> > wrote:
> >
> > > Hi,
> > > I would like to know the right way to setup a Kafka cluster with Nginx
> in
> > > front of it as a reverse proxy. Let's say I have 2 Kafka brokers
> running
> > on
> > > 2 different hosts; and an Nginx server running on another host. Nginx
> > will
> > > listen on 2 different ports, and each will forward to one Kafka broker.
> > > Producers will connect to one of the 2 ports on the Nginx host.
> > > Nginx-Host: listens on 9000 ssl (forward to :9092 in
> plain
> > > text); 9001 ssl (forward to :9092 in plain text);
> > >
> > > Kafka-Host-0: listeners=PLAINTEXT://:9092;
> > > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1:
> > > listeners=PLAINTEXT://:9092; advertised.listeners=
> > > PLAINTEXT://:9001
> > > Ports on Nginx will have SSL enabled so that messages sent from
> producers
> > > to Nginx will be encrypted; Traffic between Nginx and Kafka are in
> plain
> > > text since it's on the internal network.
> > > Why have producers go through Nginx? The main reason is that producers
> > > will only need to open their firewall to a single IP so that even later
> > on
> > > when I add another Kafka broker, I don't need to modify the firewall of
> > all
> > > the producers.
> > > My problem is that I can't make the above setup work. Brokers are
> unable
> > > to talk to o
Re: Add Nginx in front of Kafka cluster?
parties = ports *
On Thu, Sep 14, 2017 at 8:04 PM, Ali Akhtar wrote:
> I would try to put the SSL on different ports than what you're sending
> kafka to. Make sure the kafka ports don't do anything except communicate in
> plaintext, put all 3rd parties on different parties.
>
>
> On Thu, Sep 14, 2017 at 7:23 PM, Yongtao You
> wrote:
>
>> Does the following message mean broker 6 is having trouble talking to
>> broker 7? Broker 6's advertised listener is "PLAINTEXT://nginx:9906" and
>> Broker 7's advertised listener is "PLAINTEXT://nginx:9907". However, on
>> nginx server, port 9906 and 9907 are both SSL ports because that's what
>> producers (filebeat) send data to and they need to be encrypted.
>>
>>
>> [2017-09-14 21:59:32,543] WARN [Controller-6-to-broker-7-send-thread]:
>> Controller 6 epoch 1 fails to send request (type: UpdateMetadataRequest=,
>> controllerId=6, controllerEpoch=1, partitionStates={}, liveBrokers=(id=6,
>> endPoints=(host=nginx, port=9906, listenerName=ListenerName(PLAINTEXT),
>> securityProtocol=PLAINTEXT), rack=null), (id=7, endPoints=(host=nginx,
>> port=9907, listenerName=ListenerName(PLAINTEXT),
>> securityProtocol=PLAINTEXT), rack=null)) to broker nginx:9907 (id: 7 rack:
>> null). Reconnecting to broker. (kafka.controller.RequestSendThread)
>> java.io.IOException: Connection to 7 was disconnected before the response
>> was read
>> at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(N
>> etworkClientUtils.java:93)
>> at kafka.controller.RequestSendThread.doWork(ControllerChannelM
>> anager.scala:225)
>> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
>>
>>
>>
>>
>> On Thursday, September 14, 2017, 9:42:58 PM GMT+8, Yongtao You
>> wrote:
>>
>>
>> You are correct, that error message was a result of my misconfiguration.
>> I've corrected that. Although filebeat still can't send messages to Kafka.
>> In the Nginx log, I see the following:
>>
>> 2017/09/14 21:35:09 [info] 4030#4030: *60056 SSL_do_handshake() failed
>> (SSL: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
>> protocol) while SSL handshaking, client: 172.16.16.101, server:
>> 0.0.0.0:9907
>>
>>
>> where 172.16.16.101 is the host where one of the two Kafka brokers is
>> running. Looks like it tries to connect to port 9907 which is where the
>> other Kafka broker listens on. It's an [info] message so I'm not sure how
>> serious it is, but I don't see messages sent from filebeat in Kafka. :(
>>
>> Thanks!
>> -Yongtao
>>
>> On Thursday, September 14, 2017, 8:31:31 PM GMT+8, Ali Akhtar <
>> ali.rac...@gmail.com> wrote:
>>
>> If you ssh to the server where you got this error, are you able to ping
>> the
>> ip of node 7 on the port its trying to reach?
>>
>> On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You
>> wrote:
>>
>> > I'm getting a lot of these in the server.log:
>> >
>> >
>> > [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be
>> > established. Broker may not be available. (org.apache.kafka.clients.
>> > NetworkClient)
>> >
>> >
>> > where node 7 is another broker in the cluster.
>> >
>> >
>> > Thanks.
>> >
>> > -Yongtao
>> >
>> >
>> > On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You <
>> > yongtao_...@yahoo.com> wrote:
>> >
>> >
>> > I got errors saying the other brokers are not reachable, or something
>> like
>> > that. Let me dig up the exact error messages. I am guessing the problem
>> was
>> > that the advertised listeners are of PLAINTEXT format, but the Nginx
>> > requires SSL. But I could be wrong.
>> >
>> >
>> > Thanks!
>> >
>> > -Yongtao
>> >
>> >
>> > On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar <
>> > ali.rac...@gmail.com> wrote:
>> >
>> >
>> > How do you know that the brokers don't talk to each other?
>> >
>> > On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You <
>> > yongtao_...@yahoo.com.invalid>
>> > wrote:
>> >
>> > > Hi,
>> > > I would like to know the right way to setup a Kafka cluster with
>> Nginx in
>> > > front of it as a reverse proxy. Let's say I have 2 Kafka brokers
>> running
>> > on
>> > > 2 different hosts; and an Nginx server running on another host. Nginx
>> > will
>> > > listen on 2 different ports, and each will forward to one Kafka
>> broker.
>> > > Producers will connect to one of the 2 ports on the Nginx host.
>> > > Nginx-Host: listens on 9000 ssl (forward to :9092 in
>> plain
>> > > text); 9001 ssl (forward to :9092 in plain text);
>> > >
>> > > Kafka-Host-0: listeners=PLAINTEXT://:9092;
>> > > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1:
>> > > listeners=PLAINTEXT://:9092; advertised.listeners=
>> > > PLAINTEXT://:9001
>> > > Ports on Nginx will have SSL enabled so that messages sent from
>> producers
>> > > to Nginx will be encrypted; Traffic between Nginx and Kafka are in
>> plain
>> > > text since it's on the internal network.
>> > > Why have producers go through Nginx? The main reason is that producers
>> > > will only need to open their firewall to a single IP so that even
>>
Re: Add Nginx in front of Kafka cluster?
I have 2 dedicated ports on Nginx that accepts filebeat messages in SSL format,
it then forward those messages to those 2 Kafka brokers in PLAINTEXT. The Nginx
server does accept traffic on other ports, but those traffic are never
forwarded to Kafka brokers. And the 2 Kafka brokers only listen on 1 port each,
in PLAINTEXT.
Thanks!
-Yongtao
On Thursday, September 14, 2017, 11:05:13 PM GMT+8, Ali Akhtar
wrote:
parties = ports *
On Thu, Sep 14, 2017 at 8:04 PM, Ali Akhtar wrote:
> I would try to put the SSL on different ports than what you're sending
> kafka to. Make sure the kafka ports don't do anything except communicate in
> plaintext, put all 3rd parties on different parties.
>
>
> On Thu, Sep 14, 2017 at 7:23 PM, Yongtao You
> wrote:
>
>> Does the following message mean broker 6 is having trouble talking to
>> broker 7? Broker 6's advertised listener is "PLAINTEXT://nginx:9906" and
>> Broker 7's advertised listener is "PLAINTEXT://nginx:9907". However, on
>> nginx server, port 9906 and 9907 are both SSL ports because that's what
>> producers (filebeat) send data to and they need to be encrypted.
>>
>>
>> [2017-09-14 21:59:32,543] WARN [Controller-6-to-broker-7-send-thread]:
>> Controller 6 epoch 1 fails to send request (type: UpdateMetadataRequest=,
>> controllerId=6, controllerEpoch=1, partitionStates={}, liveBrokers=(id=6,
>> endPoints=(host=nginx, port=9906, listenerName=ListenerName(PLAINTEXT),
>> securityProtocol=PLAINTEXT), rack=null), (id=7, endPoints=(host=nginx,
>> port=9907, listenerName=ListenerName(PLAINTEXT),
>> securityProtocol=PLAINTEXT), rack=null)) to broker nginx:9907 (id: 7 rack:
>> null). Reconnecting to broker. (kafka.controller.RequestSendThread)
>> java.io.IOException: Connection to 7 was disconnected before the response
>> was read
>> at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(N
>> etworkClientUtils.java:93)
>> at kafka.controller.RequestSendThread.doWork(ControllerChannelM
>> anager.scala:225)
>> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
>>
>>
>>
>>
>> On Thursday, September 14, 2017, 9:42:58 PM GMT+8, Yongtao You
>> wrote:
>>
>>
>> You are correct, that error message was a result of my misconfiguration.
>> I've corrected that. Although filebeat still can't send messages to Kafka.
>> In the Nginx log, I see the following:
>>
>> 2017/09/14 21:35:09 [info] 4030#4030: *60056 SSL_do_handshake() failed
>> (SSL: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown
>> protocol) while SSL handshaking, client: 172.16.16.101, server:
>> 0.0.0.0:9907
>>
>>
>> where 172.16.16.101 is the host where one of the two Kafka brokers is
>> running. Looks like it tries to connect to port 9907 which is where the
>> other Kafka broker listens on. It's an [info] message so I'm not sure how
>> serious it is, but I don't see messages sent from filebeat in Kafka. :(
>>
>> Thanks!
>> -Yongtao
>>
>> On Thursday, September 14, 2017, 8:31:31 PM GMT+8, Ali Akhtar <
>> ali.rac...@gmail.com> wrote:
>>
>> If you ssh to the server where you got this error, are you able to ping
>> the
>> ip of node 7 on the port its trying to reach?
>>
>> On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You
>> wrote:
>>
>> > I'm getting a lot of these in the server.log:
>> >
>> >
>> > [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be
>> > established. Broker may not be available. (org.apache.kafka.clients.
>> > NetworkClient)
>> >
>> >
>> > where node 7 is another broker in the cluster.
>> >
>> >
>> > Thanks.
>> >
>> > -Yongtao
>> >
>> >
>> > On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You <
>> > yongtao_...@yahoo.com> wrote:
>> >
>> >
>> > I got errors saying the other brokers are not reachable, or something
>> like
>> > that. Let me dig up the exact error messages. I am guessing the problem
>> was
>> > that the advertised listeners are of PLAINTEXT format, but the Nginx
>> > requires SSL. But I could be wrong.
>> >
>> >
>> > Thanks!
>> >
>> > -Yongtao
>> >
>> >
>> > On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar <
>> > ali.rac...@gmail.com> wrote:
>> >
>> >
>> > How do you know that the brokers don't talk to each other?
>> >
>> > On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You <
>> > yongtao_...@yahoo.com.invalid>
>> > wrote:
>> >
>> > > Hi,
>> > > I would like to know the right way to setup a Kafka cluster with
>> Nginx in
>> > > front of it as a reverse proxy. Let's say I have 2 Kafka brokers
>> running
>> > on
>> > > 2 different hosts; and an Nginx server running on another host. Nginx
>> > will
>> > > listen on 2 different ports, and each will forward to one Kafka
>> broker.
>> > > Producers will connect to one of the 2 ports on the Nginx host.
>> > > Nginx-Host: listens on 9000 ssl (forward to :9092 in
>> plain
>> > > text); 9001 ssl (forward to :9092 in plain text);
>> > >
>> > > Kafka-Host-0: listeners=PLAINTEXT://:9092;
>> > > advertised.listeners=PLAINTEXT://:9000Kafka-Host-1:
>> > > listeners=PLAINTEX
Kafka MirrorMaker - exit on send failure?
I am reading the current MirrorMaker code and am trying to understand if
MirrorMaker has any chance at losing messages. With the usage of the Max
value for ProducerConfig.MAX_BLOCK_MS_CONFIG and ProducerConfig.RETRIES_CONFIG
settings, it appears that the producer.flush() call in
maybeFlushAndCommitOffsets would block forever if there were any long-term
connectivity issues. The callback is called only on success in that case.
Is that correct?
Thanks
Vu
def maybeFlushAndCommitOffsets() {
>
> val commitRequested = mirrorMakerConsumer.commitRequested()
>
> if (commitRequested || System.currentTimeMillis() -
> lastOffsetCommitMs > offsetCommitIntervalMs) {
>
> debug("Committing MirrorMaker state.")
>
> producer.flush()
>
> commitOffsets(mirrorMakerConsumer)
>
> lastOffsetCommitMs = System.currentTimeMillis()
>
> if (commitRequested)
>
> mirrorMakerConsumer.notifyCommit()
>
> }
>
> }
>
>
Re: Kafka MirrorMaker - exit on send failure?
>From MirrorMaker.scala :
// Defaults to no data loss settings.
maybeSetDefaultProperty(producerProps, ProducerConfig.RETRIES_CONFIG,
Int.MaxValue.toString)
maybeSetDefaultProperty(producerProps,
ProducerConfig.MAX_BLOCK_MS_CONFIG, Long.MaxValue.toString)
I think the settings would prevent data loss (see KAFKA-2452).
On Thu, Sep 14, 2017 at 2:47 PM, Vu Nguyen wrote:
> I am reading the current MirrorMaker code and am trying to understand if
> MirrorMaker has any chance at losing messages. With the usage of the Max
> value for ProducerConfig.MAX_BLOCK_MS_CONFIG and
> ProducerConfig.RETRIES_CONFIG
> settings, it appears that the producer.flush() call in
> maybeFlushAndCommitOffsets would block forever if there were any long-term
> connectivity issues. The callback is called only on success in that case.
>
> Is that correct?
>
> Thanks
>
> Vu
>
>
> def maybeFlushAndCommitOffsets() {
> >
> > val commitRequested = mirrorMakerConsumer.commitRequested()
> >
> > if (commitRequested || System.currentTimeMillis() -
> > lastOffsetCommitMs > offsetCommitIntervalMs) {
> >
> > debug("Committing MirrorMaker state.")
> >
> > producer.flush()
> >
> > commitOffsets(mirrorMakerConsumer)
> >
> > lastOffsetCommitMs = System.currentTimeMillis()
> >
> > if (commitRequested)
> >
> > mirrorMakerConsumer.notifyCommit()
> >
> > }
> >
> > }
> >
> >
>
Kafka MirrorMaker - target or source datacenter deployment
Many of the descriptions and diagrams online describe deploying Kafka MirrorMaker into the target data center (near the target Kafka cluster). Since MirrorMaker is supposed to not lose messages, does it matter which data center MirrorMaker is deployed in--source or target data center (with any Kafka MirrorMaker version 0.10.1+)? It might be easier to collect and observe metrics in the source data center if MirrorMaker is also in the source data center (near the source Kafka cluster), especially if I can't aggregate the metrics from the 2 data centers. Is there anything else that would influence me to deploy in MirrorMaker in either data center? Thanks Vu
Re: Kafka MirrorMaker - target or source datacenter deployment
Always in the target datacenter. While you can set up mirror maker for no data loss operation, it’s still a good idea to put the connection more likely to fail (remote) on the consumer side. Additionally, there are significant performance problems with setting it up for remote produce as you must run with one in flight batch in order to maintain message ordering. -Todd On Thu, Sep 14, 2017 at 9:46 PM, Vu Nguyen wrote: > Many of the descriptions and diagrams online describe deploying Kafka > MirrorMaker into the target data center (near the target Kafka cluster). > Since MirrorMaker is supposed to not lose messages, does it matter which > data center MirrorMaker is deployed in--source or target data center (with > any Kafka MirrorMaker version 0.10.1+)? > > It might be easier to collect and observe metrics in the source data center > if MirrorMaker is also in the source data center (near the source Kafka > cluster), especially if I can't aggregate the metrics from the 2 data > centers. Is there anything else that would influence me to deploy in > MirrorMaker in either data center? > > Thanks > > Vu > -- *Todd Palino* Senior Staff Engineer, Site Reliability Data Infrastructure Streaming linkedin.com/in/toddpalino
Re: Kafka MirrorMaker - target or source datacenter deployment
Wouldn't KAFKA-5494 make remote produce more reliable? Original message From: Todd Palino Date: 9/14/17 6:53 PM (GMT-08:00) To: users@kafka.apache.org Subject: Re: Kafka MirrorMaker - target or source datacenter deployment Always in the target datacenter. While you can set up mirror maker for no data loss operation, it’s still a good idea to put the connection more likely to fail (remote) on the consumer side. Additionally, there are significant performance problems with setting it up for remote produce as you must run with one in flight batch in order to maintain message ordering. -Todd On Thu, Sep 14, 2017 at 9:46 PM, Vu Nguyen wrote: > Many of the descriptions and diagrams online describe deploying Kafka > MirrorMaker into the target data center (near the target Kafka cluster). > Since MirrorMaker is supposed to not lose messages, does it matter which > data center MirrorMaker is deployed in--source or target data center (with > any Kafka MirrorMaker version 0.10.1+)? > > It might be easier to collect and observe metrics in the source data center > if MirrorMaker is also in the source data center (near the source Kafka > cluster), especially if I can't aggregate the metrics from the 2 data > centers. Is there anything else that would influence me to deploy in > MirrorMaker in either data center? > > Thanks > > Vu > -- *Todd Palino* Senior Staff Engineer, Site Reliability Data Infrastructure Streaming linkedin.com/in/toddpalino
