I'd post to the mailing list again with a new subject and ask that. On Thu, Sep 15, 2016 at 1:52 PM, kant kodali <kanth...@gmail.com> wrote:
> I used node.js client libraries for all three and yes I want to make sure > I am > comparing apples to apples so I make it as equivalent as possible. > Again the big question is What is the right setup for Kafka to be > comparable > with the other I mentioned in my previous email? > > > > > > > On Thu, Sep 15, 2016 1:47 AM, Ali Akhtar ali.rac...@gmail.com > wrote: > The issue is clearly that you're running out of resources, so I would add > > more brokers and/or larger instances. > > > > > You're also using Node which is not the best for performance. A compiled > > language such as Java would give you the best performance. > > > > > Here's a case study that should help: > > https://engineering.linkedin.com/kafka/benchmarking-apache-k > afka-2-million-writes-second-three-cheap-machines > > > > > Good luck, let us know how it goes > > > > > On Thu, Sep 15, 2016 at 1:42 PM, kant kodali <kanth...@gmail.com> wrote: > > > > > yeah.. >> > > I tried it with 10 messages with single broker and only one partiton that >> > > looked >> > > instantaneous and ~5K messages/sec for the data size of 1KBI tried it with >> > > 1000 >> > > messages that looked instantaneous as well ~5K messages/sec for the data >> > > size of >> > > 1KBI tried it with 10K messages with single broker and only one >> > > partiton things >> > > started to go down ~1K messages/sec for the data size of 1KB >> > > having only one partition on a single broker is a bad? My goal is to run >> > > some >> > > basic benchmarks on NATS & NSQ & KAFKA >> > > I have the same environment for all three (NATS & NSQ & KAFKA) >> > > a broker on Machine 1producer on Machine 2Consumer on Machine 3 >> > > with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws >> > > instance. >> > > I have pushed 300K messages with NATS and it was able to handle easily and >> > > receive throughput was 5K messages/secI have pushed 300K messages and NSQ >> > > and >> > > receive throughput was 2K messages/secI am unable to push 300K messages >> > > with >> > > Kafka with the above configuration and environment so at this point my >> > > biggest >> > > question is what is the fair setup for Kafka so its comparable with NATS >> > > and >> > > NSQ? >> > > kant >> > > >> > >> > >> > >> > >> > >> > On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com >> > > wrote: >> > > Lower the workload gradually, start from 10 messages, increase to 100, then >> > > >> > 1000, and so on. See if it slows down as the workload increases. If so, you >> > > >> > need more brokers + partitions to handle the workload. >> > > >> > >> > >> > >> > On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth...@gmail.com> wrote: >> > > >> > >> > >> > >> > > m4.xlarge >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com >> > > >> > > >> > > >> > > wrote: >> > > >> > > What's the instance size that you're using? With 300k messages your >> > > single >> > > >> > > >> > > >> > > broker might not be able to handle it. >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> >> > > wrote: >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > My goal is to test the throughput (#messages per second) given my setup >> > > and >> > > >> > >> >> > > >> > > >> > > >> > > with a data size of 1KB. if you guys already have some idea on these >> > > >> > >> >> > > >> > > >> > > >> > > numbers >> > > >> > >> >> > > >> > > >> > > >> > > that would be helpful as well. >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com >> > > >> > >> >> > > >> > > >> > > >> > > wrote: >> > > >> > >> >> > > >> > > >> > > >> > > 172.* is all private ip's for my machine I double checked it.I have not >> > > >> > >> >> > > >> > > >> > > >> > > changed >> > > >> > >> >> > > >> > > >> > > >> > > any default settingsI dont know how to use kafka-consumer.sh >> > > >> > >> >> > > >> > > >> > > >> > > or kafka-producer.sh because it looks like they want me to specify a >> > > group >> > > >> > >> >> > > >> > > >> > > >> > > and I >> > > >> > >> >> > > >> > > >> > > >> > > didn't create any consumer group because I am using single producer and >> > > >> > >> >> > > >> > > >> > > >> > > consumer. is there a default group?Also, I am receiving message but very >> > > >> > >> >> > > >> > > >> > > >> > > late. I >> > > >> > >> >> > > >> > > >> > > >> > > send about 300K messages using the node.js client and I am receiving at a >> > > >> > >> >> > > >> > > >> > > >> > > very >> > > >> > >> >> > > >> > > >> > > >> > > low rate. really not sure what is going on? >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com >> > > >> > >> >> > > >> > > >> > > >> > > wrote: >> > > >> > >> >> > > >> > > >> > > >> > > Your code seems to be using the public ip of the servers. If all 3 >> > > machines >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > are in the same availability zone on AWS, try using the private ip, and >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > then they might communicate over the local network. >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > Did you change any default settings? >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > Do you get the same results if you run kafka-consumer.sh and >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > kafka-producer.sh instead of the Node code? >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> >> > > wrote: >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > >> > > >> > >> >> > > >> > > > They are hosted on AWS and I dont think there are any network issues >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > because I >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > tried testing other Queuing systems with no issues however I am using a >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > node.js >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > client with the following code. I am not sure if there are any errors >> > > or >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > anything I didn't set in the following code? >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > //producer var kafka = require('kafka-node'); var >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > Producer = kafka.Producer; var Client = kafka.Client; var client = >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > new Client('172.31.21.175:2181'); var argv = >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > p = argv.p || 0; var a = argv.a || 0; var producer = new >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > Producer(client, { requireAcks: 1}); var num = 300005; >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > producer.on('ready', function () { var message = 'Hello World'; >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > for (var i=0; i<num; i++) { producer.send([ { topic: >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > topic, partition: p, messages: message, attributes: a } ], function >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > (err, result) { console.log(err || result); >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > //process.exit(); }); } }); producer.on('error', >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > function (err) { console.log('error', err); process.exit(); >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > }); //Consumer var kafka = require('kafka-node'); var Consumer = >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > kafka.Consumer; var Offset = kafka.Offset; var Client = >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > kafka.Client; var argv = require('optimist').argv; var topic = >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > argv.topic || 'kafka_test'; var client = new >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > Client('172.31.21.175:2181'); var topics = [ {topic: topic, >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs: >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > 1000 }; var consumer = new Consumer(client, topics, options); var >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > offset = new Offset(client); var start; var received = 0; var >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > target = 200000; var hash = 1000; consumer.on('message', function >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > (message) { console.log(message); received += 1; if >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > (received === 1) { start = new Date(); } if (received === target) { >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > var stop = new Date(); console.log('\nDone test'); >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > var mps = parseInt(target/((stop-start)/1000)); >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > console.log('Received at ' + mps + ' msgs/sec'); process.exit(); >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > } else if (received % hash === 0){ >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > >> >> > > >> > > > process.stdout.write(received + '\n'); } }); >> > > >> > >> > >
