This link already had the answer I was searching for: http://grokbase.com/t/kafka/users/145vmz70cb/java-net-sockettimeoutexception-in-broker
On Fri, Aug 21, 2015 at 3:03 PM, Prabhjot Bharaj <prabhbha...@gmail.com> wrote: > Hi, > > Never mind. I have solved this problem by referring to an earlier > question, where Neha had suggested to use a higher value for > request-timeout-ms > > Regards, > prabcs > > On Fri, Aug 21, 2015 at 12:22 PM, Prabhjot Bharaj <prabhbha...@gmail.com> > wrote: > >> Hello Folks, >> >> I'm using Kafka 0.8.2.1 with the default zookeeper build that comes along >> the bundle >> >> I have setup a 5 machine cluster and on the same 5 machines, I'm also >> running zookeeper as well >> >> I am trying to see what is the maximum produce throughput I can get on >> this 5 node cluster >> >> I have created only 1 topic - tops1 in the cluster:- >> >> root@x.x.x.x:~# kafka-topics.sh --describe --topic tops1 --zookeeper >> localhost:2182 >> >> Topic:tops1 PartitionCount:4 ReplicationFactor:3 Configs: >> >> Topic: tops1 Partition: 0 Leader: 4 Replicas: 4,1,2 Isr: 4,1,2 >> >> Topic: tops1 Partition: 1 Leader: 5 Replicas: 5,2,3 Isr: 5,3,2 >> >> Topic: tops1 Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3 >> >> Topic: tops1 Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,2,5 >> >> >> When I run this command in parallel from 2 different machines, I get >> SocketTimeoutException:- >> >> time kafka-producer-perf-test.sh --broker-list >> x.x.x.x:9092,x.x.x.y:9092,x.x.x.z:9092,x.x.x.a:9092,x.x.x.b:9092 --messages >> 1000000 --message-size 500 --topics tops1 --show-detailed-stats --threads >> 5 --request-num-acks -1 --batch-size 1000 >> >> >> My server.properties has these details:- >> >> >> broker.id=0 >> >> port=9092 >> >> num.network.threads=3 >> >> num.io.threads=8 >> >> socket.send.buffer.bytes=1048576 >> >> socket.receive.buffer.bytes=1048576 >> >> socket.request.max.bytes=104857600 >> >> log.dirs=/kafka-logs >> >> num.partitions=1 >> >> num.recovery.threads.per.data.dir=1 >> >> log.retention.hours=168 >> >> log.segment.bytes=1073741824 >> >> log.retention.check.interval.ms=300000 >> >> log.cleaner.enable=false >> >> zookeeper.connect=localhost:2181 >> >> zookeeper.connection.timeout.ms=6000 >> >> num.replica.fetchers=4 >> >> >> 2 major changes in the server.properties, which gave a performance boost >> on this test were:- >> >> num.replica.fetchers=4 >> >> socket.send.buffer.bytes=1048576 >> >> socket.receive.buffer.bytes=1048576 >> >> >> I think I'm giving enough requests on the network, but not sure how I >> should correct it >> >> I am particularly interested in knowing how much max throughput per topic >> can I get, with the highest level of durability (i.e. request-num-acks = -1) >> >> >> Request you to share your thoughts on this. >> >> Thanks, >> >> Prabhjot >> > > > > -- > --------------------------------------------------------- > "There are only 10 types of people in the world: Those who understand > binary, and those who don't" > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
