Hi, Never mind. I have solved this problem by referring to an earlier question, where Neha had suggested to use a higher value for request-timeout-ms
Regards, prabcs On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj <prabhbha...@gmail.com> wrote: > Hello Folks, > > I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along > the bundle > > I have setup a 5 machine cluster and on the same 5 machines, I'm also > running zookeeper as well > > I am trying to see what is the maximum produce throughput I can get on > this 5 node cluster > > I have created only 1 topic - tops1 in the cluster:- > > root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper > localhost:2182 > > Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: > > Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 > > Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 > > Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 > > Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 > > > When I run this command in parallel from 2 different machines, I get > SocketTimeoutException:- > > time kafka-producer-perf-test.sh --broker-list > x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages > 1000000 --message-size 500 --topics tops1 --show-detailed-stats --threads > 5 --request-num-acks -1 --batch-size 1000 > > > My server.properties has these details:- > > > broker.id=0 > > port=9092 > > num.network.threads=3 > > num.io.threads=8 > > socket.send.buffer.bytes=1048576 > > socket.receive.buffer.bytes=1048576 > > socket.request.max.bytes=104857600 > > log.dirs=/kafka-logs > > num.partitions=1 > > num.recovery.threads.per.data.dir=1 > > log.retention.hours=168 > > log.segment.bytes=1073741824 > > log.retention.check.interval.ms=300000 > > log.cleaner.enable=false > > zookeeper.connect=localhost:2181 > > zookeeper.connection.timeout.ms=6000 > > num.replica.fetchers=4 > > > 2 major changes in the server.properties, which gave a performance boost > on this test were:- > > num.replica.fetchers=4 > > socket.send.buffer.bytes=1048576 > > socket.receive.buffer.bytes=1048576 > > > I think I'm giving enough requests on the network, but not sure how I > should correct it > > I am particularly interested in knowing how much max throughput per topic > can I get, with the highest level of durability (i.e. request-num-acks = -1) > > > Request you to share your thoughts on this. > > Thanks, > > Prabhjot > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
