The issue is clearly that you're running out of resources, so I would add more brokers and/or larger instances.
You're also using Node which is not the best for performance. A compiled language such as Java would give you the best performance. Here's a case study that should help: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Good luck, let us know how it goes On Thu, Sep 15, 2016 at 1:42 PM, kant kodali <kanth...@gmail.com> wrote: > yeah.. > I tried it with 10 messages with single broker and only one partiton that > looked > instantaneous and ~5K messages/sec for the data size of 1KBI tried it with > 1000 > messages that looked instantaneous as well ~5K messages/sec for the data > size of > 1KBI tried it with 10K messages with single broker and only one > partiton things > started to go down ~1K messages/sec for the data size of 1KB > having only one partition on a single broker is a bad? My goal is to run > some > basic benchmarks on NATS & NSQ & KAFKA > I have the same environment for all three (NATS & NSQ & KAFKA) > a broker on Machine 1producer on Machine 2Consumer on Machine 3 > with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws > instance. > I have pushed 300K messages with NATS and it was able to handle easily and > receive throughput was 5K messages/secI have pushed 300K messages and NSQ > and > receive throughput was 2K messages/secI am unable to push 300K messages > with > Kafka with the above configuration and environment so at this point my > biggest > question is what is the fair setup for Kafka so its comparable with NATS > and > NSQ? > kant > > > > > > > On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com > wrote: > Lower the workload gradually, start from 10 messages, increase to 100, then > > 1000, and so on. See if it slows down as the workload increases. If so, you > > need more brokers + partitions to handle the workload. > > > > > On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth...@gmail.com> wrote: > > > > > > m4.xlarge > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com > > > > > > wrote: > > > What's the instance size that you're using? With 300k messages your > single > > > > > > broker might not be able to handle it. > > > > > > > > > > > > > > > On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> > wrote: > > > > > > > > > > > > > > > My goal is to test the throughput (#messages per second) given my setup > and > > >> > > > > > > with a data size of 1KB. if you guys already have some idea on these > > >> > > > > > > numbers > > >> > > > > > > that would be helpful as well. > > >> > > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com > > >> > > > > > > wrote: > > >> > > > > > > 172.* is all private ip's for my machine I double checked it.I have not > > >> > > > > > > changed > > >> > > > > > > any default settingsI dont know how to use kafka-consumer.sh > > >> > > > > > > or kafka-producer.sh because it looks like they want me to specify a > group > > >> > > > > > > and I > > >> > > > > > > didn't create any consumer group because I am using single producer and > > >> > > > > > > consumer. is there a default group?Also, I am receiving message but very > > >> > > > > > > late. I > > >> > > > > > > send about 300K messages using the node.js client and I am receiving at a > > >> > > > > > > very > > >> > > > > > > low rate. really not sure what is going on? > > >> > > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com > > >> > > > > > > wrote: > > >> > > > > > > Your code seems to be using the public ip of the servers. If all 3 > machines > > >> > > > > > > > > >> > > > are in the same availability zone on AWS, try using the private ip, and > > >> > > > > > > > > >> > > > then they might communicate over the local network. > > >> > > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > Did you change any default settings? > > >> > > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > Do you get the same results if you run kafka-consumer.sh and > > >> > > > > > > > > >> > > > kafka-producer.sh instead of the Node code? > > >> > > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> > wrote: > > >> > > > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > They are hosted on AWS and I dont think there are any network issues > > >> > > > > > > > > >> > > > > because I > > >> > > > > > > > > >> > > > > tried testing other Queuing systems with no issues however I am using a > > >> > > > > > > > > >> > > > > node.js > > >> > > > > > > > > >> > > > > client with the following code. I am not sure if there are any errors > or > > >> > > > > > > > > >> > > > > anything I didn't set in the following code? > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > //producer var kafka = require('kafka-node'); var > > >> > > > > > > > > >> > > > > Producer = kafka.Producer; var Client = kafka.Client; var client = > > >> > > > > > > > > >> > > > > new Client('172.31.21.175:2181'); var argv = > > >> > > > > > > > > >> > > > > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var > > >> > > > > > > > > >> > > > > p = argv.p || 0; var a = argv.a || 0; var producer = new > > >> > > > > > > > > >> > > > > Producer(client, { requireAcks: 1}); var num = 300005; > > >> > > > > > > > > >> > > > > producer.on('ready', function () { var message = 'Hello World'; > > >> > > > > > > > > >> > > > > for (var i=0; i<num; i++) { producer.send([ { topic: > > >> > > > > > > > > >> > > > > topic, partition: p, messages: message, attributes: a } ], function > > >> > > > > > > > > >> > > > > (err, result) { console.log(err || result); > > >> > > > > > > > > >> > > > > //process.exit(); }); } }); producer.on('error', > > >> > > > > > > > > >> > > > > function (err) { console.log('error', err); process.exit(); > > >> > > > > > > > > >> > > > > }); //Consumer var kafka = require('kafka-node'); var Consumer = > > >> > > > > > > > > >> > > > > kafka.Consumer; var Offset = kafka.Offset; var Client = > > >> > > > > > > > > >> > > > > kafka.Client; var argv = require('optimist').argv; var topic = > > >> > > > > > > > > >> > > > > argv.topic || 'kafka_test'; var client = new > > >> > > > > > > > > >> > > > > Client('172.31.21.175:2181'); var topics = [ {topic: topic, > > >> > > > > > > > > >> > > > > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs: > > >> > > > > > > > > >> > > > > 1000 }; var consumer = new Consumer(client, topics, options); var > > >> > > > > > > > > >> > > > > offset = new Offset(client); var start; var received = 0; var > > >> > > > > > > > > >> > > > > target = 200000; var hash = 1000; consumer.on('message', function > > >> > > > > > > > > >> > > > > (message) { console.log(message); received += 1; if > > >> > > > > > > > > >> > > > > (received === 1) { start = new Date(); } if (received === target) { > > >> > > > > > > > > >> > > > > var stop = new Date(); console.log('\nDone test'); > > >> > > > > > > > > >> > > > > var mps = parseInt(target/((stop-start)/1000)); > > >> > > > > > > > > >> > > > > console.log('Received at ' + mps + ' msgs/sec'); process.exit(); > > >> > > > > > > > > >> > > > > } else if (received % hash === 0){ > > >> > > > > > > > > >> > > > > process.stdout.write(received + '\n'); } }); > > >> > > > > > > > > >> > > > > consumer.on('error', function (err) { console.log('error', err); }); > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Not using Mixmax yet? > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com > > >> > > > > > > > > >> > > > > wrote: > > >> > > > > > > > > >> > > > > It sounds like a network issue. Where are the 3 servers located / > hosted? > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Hi, > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > I have the following setup. > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Machine 2 > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Single Kafka Consumer on Machine 3 > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > When a producer client sends a message to the Kafka broker by pointing > at > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > the > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Zookeeper Server the consumer doesn't seem to get the message right > away > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > instead > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > it gets after a minute or something (pretty late). I am not sure what > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > settings I > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > need to change. any ideas? > > >> > > > > > > > > >> > > > >> > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > Thanks,kant > > >> > > > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > > >
