I checked the logs on the brokers, it seems that the zookeeper or the kafka server process is not running on this broker...Thank you guys. I will see if it happens again.
On Tue, Jul 14, 2015 at 4:53 PM, JIEFU GONG <jg...@berkeley.edu> wrote: > Hmm..yeah some error logs would be nice like Gwen pointed out. Do any of > your brokers fall out of the ISR when sending messages? It seems like your > setup should be fine, so I'm not entirely sure. > > On Tue, Jul 14, 2015 at 1:31 PM, Yuheng Du <yuheng.du.h...@gmail.com> > wrote: > > > Jiefu, > > > > I am performing these tests on a 6 nodes cluster in cloudlab (a > > infrastructure built for scientific research). I use 2 nodes as > producers, > > 2 as brokers only, and 2 as consumers. I have tested for each individual > > machines and they work well. I did not use AWS. Thank you! > > > > On Tue, Jul 14, 2015 at 4:20 PM, JIEFU GONG <jg...@berkeley.edu> wrote: > > > > > Yuheng, are you performing these tests locally or using a service such > as > > > AWS? I'd try using each separate machine individually first, connecting > > to > > > the ZK/Kafka servers and ensuring that each is able to first log and > > > consume messages independently. > > > > > > On Tue, Jul 14, 2015 at 1:17 PM, Gwen Shapira <gshap...@cloudera.com> > > > wrote: > > > > > > > Are there any errors on the broker logs? > > > > > > > > On Tue, Jul 14, 2015 at 11:57 AM, Yuheng Du < > yuheng.du.h...@gmail.com> > > > > wrote: > > > > > Jiefu, > > > > > > > > > > Thank you. The three producers can run at the same time. I mean > > should > > > > they > > > > > be started at exactly the same time? (I have three consoles from > each > > > of > > > > > the three machines and I just start the console command manually > one > > by > > > > > one) Or a small variation of the starting time won't matter? > > > > > > > > > > Gwen and Jiefu, > > > > > > > > > > I have started the three producers at three machines, after a > while, > > > all > > > > of > > > > > them gives a java.net.ConnectException: > > > > > > > > > > [2015-07-14 12:56:46,352] WARN Error in I/O with producer0-link-0/ > > > > > 192.168.1.1 (org.apache.kafka.common.network.Selector) > > > > > > > > > > java.net.ConnectException: Connection refused...... > > > > > > > > > > [2015-07-14 12:56:48,056] WARN Error in I/O with producer1-link-0/ > > > > > 192.168.1.2 (org.apache.kafka.common.network.Selector) > > > > > > > > > > java.net.ConnectException: Connection refused..... > > > > > > > > > > What could be the cause? > > > > > > > > > > Thank you guys! > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jul 14, 2015 at 2:47 PM, JIEFU GONG <jg...@berkeley.edu> > > > wrote: > > > > > > > > > >> Yuheng, > > > > >> > > > > >> Yes, if you read the blog post it specifies that he's using three > > > > separate > > > > >> machines. There's no reason the producers cannot be started at the > > > same > > > > >> time, I believe. > > > > >> > > > > >> On Tue, Jul 14, 2015 at 11:42 AM, Yuheng Du < > > yuheng.du.h...@gmail.com > > > > > > > > >> wrote: > > > > >> > > > > >> > Hi, > > > > >> > > > > > >> > I am running the performance test for kafka. > > > > >> > https://gist.github.com/jkreps > > > > >> > /c7ddb4041ef62a900e6c > > > > >> > > > > > >> > For the "Three Producers, 3x async replication" scenario, the > > > command > > > > is > > > > >> > the same as one producer: > > > > >> > > > > > >> > bin/kafka-run-class.sh > > > > org.apache.kafka.clients.tools.ProducerPerformance > > > > >> > test 50000000 100 -1 acks=1 > > > > >> > bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 > > > > >> > buffer.memory=67108864 batch.size=8196 > > > > >> > > > > > >> > So How to I run the test for three producers? Do I just run them > > on > > > > three > > > > >> > separate servers at the same time? Will there be some error in > > this > > > > way > > > > >> > since the three producers can't be started at the same time? > > > > >> > > > > > >> > Thanks. > > > > >> > > > > > >> > best, > > > > >> > Yuheng > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Jiefu Gong > > > > >> University of California, Berkeley | Class of 2017 > > > > >> B.A Computer Science | College of Letters and Sciences > > > > >> > > > > >> jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427 > > > > >> > > > > > > > > > > > > > > > > -- > > > > > > Jiefu Gong > > > University of California, Berkeley | Class of 2017 > > > B.A Computer Science | College of Letters and Sciences > > > > > > jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427 > > > > > > > > > -- > > Jiefu Gong > University of California, Berkeley | Class of 2017 > B.A Computer Science | College of Letters and Sciences > > jg...@berkeley.edu <elise...@berkeley.edu> | (925) 400-3427 >
