I used node.js client libraries for all three and yes I want to make sure I am
comparing apples to apples so I make it as equivalent as possible.
Again the big question is What is the right setup for Kafka to be comparable
with the other I mentioned in my previous email?
On Thu, Sep 15, 2016 1:47 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
The issue is clearly that you're running out of resources, so I would add
more brokers and/or larger instances.
You're also using Node which is not the best for performance. A compiled
language such as Java would give you the best performance.
Here's a case study that should help:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Good luck, let us know how it goes
On Thu, Sep 15, 2016 at 1:42 PM, kant kodali <kanth...@gmail.com> wrote:
yeah..
I tried it with 10 messages with single broker and only one partiton that
looked
instantaneous and ~5K messages/sec for the data size of 1KBI tried it with
1000
messages that looked instantaneous as well ~5K messages/sec for the data
size of
1KBI tried it with 10K messages with single broker and only one
partiton things
started to go down ~1K messages/sec for the data size of 1KB
having only one partition on a single broker is a bad? My goal is to run
some
basic benchmarks on NATS & NSQ & KAFKA
I have the same environment for all three (NATS & NSQ & KAFKA)
a broker on Machine 1producer on Machine 2Consumer on Machine 3
with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
instance.
I have pushed 300K messages with NATS and it was able to handle easily and
receive throughput was 5K messages/secI have pushed 300K messages and NSQ
and
receive throughput was 2K messages/secI am unable to push 300K messages
with
Kafka with the above configuration and environment so at this point my
biggest
question is what is the fair setup for Kafka so its comparable with NATS
and
NSQ?
kant
On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Lower the workload gradually, start from 10 messages, increase to 100, then
1000, and so on. See if it slows down as the workload increases. If so, you
need more brokers + partitions to handle the workload.
On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth...@gmail.com> wrote:
> m4.xlarge
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
>
> wrote:
> What's the instance size that you're using? With 300k messages your
single
>
> broker might not be able to handle it.
>
>
>
>
> On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com>
wrote:
>
>
>
>
> My goal is to test the throughput (#messages per second) given my setup
and
>>
>
> with a data size of 1KB. if you guys already have some idea on these
>>
>
> numbers
>>
>
> that would be helpful as well.
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
>>
>
> wrote:
>>
>
> 172.* is all private ip's for my machine I double checked it.I have not
>>
>
> changed
>>
>
> any default settingsI dont know how to use kafka-consumer.sh
>>
>
> or kafka-producer.sh because it looks like they want me to specify a
group
>>
>
> and I
>>
>
> didn't create any consumer group because I am using single producer and
>>
>
> consumer. is there a default group?Also, I am receiving message but very
>>
>
> late. I
>>
>
> send about 300K messages using the node.js client and I am receiving at a
>>
>
> very
>>
>
> low rate. really not sure what is going on?
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
>>
>
> wrote:
>>
>
> Your code seems to be using the public ip of the servers. If all 3
machines
>>
>
>
>>
> are in the same availability zone on AWS, try using the private ip, and
>>
>
>
>>
> then they might communicate over the local network.
>>
>
>
>>
>
>>
>
>>
>
>>
> Did you change any default settings?
>>
>
>
>>
>
>>
>
>>
>
>>
> Do you get the same results if you run kafka-consumer.sh and
>>
>
>
>>
> kafka-producer.sh instead of the Node code?
>>
>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com>
wrote:
>>
>
>
>>
>
>>
>
>>
>
>>
> > They are hosted on AWS and I dont think there are any network issues
>>
>
>
>>
> > because I
>>
>
>
>>
> > tried testing other Queuing systems with no issues however I am using a
>>
>
>
>>
> > node.js
>>
>
>
>>
> > client with the following code. I am not sure if there are any errors
or
>>
>
>
>>
> > anything I didn't set in the following code?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > //producer var kafka = require('kafka-node'); var
>>
>
>
>>
> > Producer = kafka.Producer; var Client = kafka.Client; var client =
>>
>
>
>>
> > new Client('172.31.21.175:2181'); var argv =
>>
>
>
>>
> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>>
>
>
>>
> > p = argv.p || 0; var a = argv.a || 0; var producer = new
>>
>
>
>>
> > Producer(client, { requireAcks: 1}); var num = 300005;
>>
>
>
>>
> > producer.on('ready', function () { var message = 'Hello World';
>>
>
>
>>
> > for (var i=0; i<num; i++) { producer.send([ { topic:
>>
>
>
>>
> > topic, partition: p, messages: message, attributes: a } ], function
>>
>
>
>>
> > (err, result) { console.log(err || result);
>>
>
>
>>
> > //process.exit(); }); } }); producer.on('error',
>>
>
>
>>
> > function (err) { console.log('error', err); process.exit();
>>
>
>
>>
> > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>>
>
>
>>
> > kafka.Consumer; var Offset = kafka.Offset; var Client =
>>
>
>
>>
> > kafka.Client; var argv = require('optimist').argv; var topic =
>>
>
>
>>
> > argv.topic || 'kafka_test'; var client = new
>>
>
>
>>
> > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>>
>
>
>>
> > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>>
>
>
>>
> > 1000 }; var consumer = new Consumer(client, topics, options); var
>>
>
>
>>
> > offset = new Offset(client); var start; var received = 0; var
>>
>
>
>>
> > target = 200000; var hash = 1000; consumer.on('message', function
>>
>
>
>>
> > (message) { console.log(message); received += 1; if
>>
>
>
>>
> > (received === 1) { start = new Date(); } if (received === target) {
>>
>
>
>>
> > var stop = new Date(); console.log('\nDone test');
>>
>
>
>>
> > var mps = parseInt(target/((stop-start)/1000));
>>
>
>
>>
> > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>>
>
>
>>
> > } else if (received % hash === 0){
>>
>
>
>>
> > process.stdout.write(received + '\n'); } });
>>
>
>
>>
> > consumer.on('error', function (err) { console.log('error', err); });
>>
>
>
>>
> >
>>
>
>
>>
> > Not using Mixmax yet?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
>>
>
>
>>
> > wrote:
>>
>
>
>>
> > It sounds like a network issue. Where are the 3 servers located /
hosted?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com>
>>
>
> wrote:
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > Hi,
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > I have the following setup.
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Machine 2
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Single Kafka Consumer on Machine 3
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > When a producer client sends a message to the Kafka broker by pointing
at
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > the
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Zookeeper Server the consumer doesn't seem to get the message right
away
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > instead
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > it gets after a minute or something (pretty late). I am not sure what
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > settings I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > need to change. any ideas?
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Thanks,kant
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
