Thanks Martin, I will look at those links.
But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper? ________________________________ From: Martin Gainty <mgai...@hotmail.com> Sent: Friday, November 25, 2016 1:46:28 PM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Friday, November 25, 2016 6:17 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Hi Martin, My server.properties looks like this: listeners=PLAINTEXT://0.0.0.0:9092 advertised.host.name=<SERVER_PUBLIC_DNS> broker.id=2 port=9092 num.partitions=4 zookeeper.connect=zkA:2181,zkB:2181,zkC:2181 MG>can you check status for each ZK Node in the quorum? sh>$ZOOKEEPER_HOME/bin/zkServer.sh status http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html ZooKeeper problems and solutions - IBM<http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html> www.ibm.com<http://www.ibm.com> Use these solutions to resolve the problems that you might encounter with Apache ZooKeeper. MG>*greetings from Plimoth Mass* MG>M num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 log.dirs=/tmp/kafka-logs num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connection.timeout.ms=6000 delete.topic.enable=true auto.leader.rebalance.enable=true Above, 'zkA', 'zkB' and 'zkC' are defined in /etc/hosts and are valid ZK servers, and <SERVER_PUBLIC_DNS> is the public DNS of the EC2 (AWS) node that this Kafka is running on. Anything look incorrect to you? And yes, yesterday was a holiday, but there is only work! I'll celebrate one big, long holiday when I'm dead! Thanks for any-and-all input here! Best, Zac ________________________________ From: Martin Gainty <mgai...@hotmail.com> Sent: Thursday, November 24, 2016 9:03:33 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Hi Zach there is a rumour that today thursday is a holiday? in server.properties how are you configuring your server? specifically what are these attributes? num.network.threads= num.io.threads= socket.send.buffer.bytes= socket.receive.buffer.bytes= socket.request.max.bytes= num.partitions= num.recovery.threads.per.data.dir= ? Martin ______________________________________________ ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Thursday, November 24, 2016 7:05 AM To: users@kafka.apache.org Subject: Re: Messages intermittently get lost Anybody?!? This is very disconcerting! ________________________________ From: Zac Harvey <zac.har...@welltok.com> Sent: Wednesday, November 23, 2016 5:07:45 AM To: users@kafka.apache.org Subject: Messages intermittently get lost I am playing around with Kafka and have a simple setup: * 1-node Kafka (Ubuntu) server * 3-node ZK cluster (each on their own Ubuntu server) I have a consumer written in Scala and am using the kafka-console-producer (v0.10) that ships with the distribution. I'd say about 20% of the messages I send via the producer never get consumed by the Scala process (which is running continuously). No errors on either side (producer or consumer): the producer sends, and, nothing... Any ideas as to what might be going on here, or how I could start troubleshooting? Thanks!
