Lower the workload gradually, start from 10 messages, increase to 100, then
1000, and so on. See if it slows down as the workload increases. If so, you
need more brokers + partitions to handle the workload.
On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth...@gmail.com> wrote:
> m4.xlarge
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
>
> wrote:
> What's the instance size that you're using? With 300k messages your single
>
> broker might not be able to handle it.
>
>
>
>
> On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> wrote:
>
>
>
>
> My goal is to test the throughput (#messages per second) given my setup and
>>
>
> with a data size of 1KB. if you guys already have some idea on these
>>
>
> numbers
>>
>
> that would be helpful as well.
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
>>
>
> wrote:
>>
>
> 172.* is all private ip's for my machine I double checked it.I have not
>>
>
> changed
>>
>
> any default settingsI dont know how to use kafka-consumer.sh
>>
>
> or kafka-producer.sh because it looks like they want me to specify a group
>>
>
> and I
>>
>
> didn't create any consumer group because I am using single producer and
>>
>
> consumer. is there a default group?Also, I am receiving message but very
>>
>
> late. I
>>
>
> send about 300K messages using the node.js client and I am receiving at a
>>
>
> very
>>
>
> low rate. really not sure what is going on?
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
>>
>
> wrote:
>>
>
> Your code seems to be using the public ip of the servers. If all 3 machines
>>
>
>
>>
> are in the same availability zone on AWS, try using the private ip, and
>>
>
>
>>
> then they might communicate over the local network.
>>
>
>
>>
>
>>
>
>>
>
>>
> Did you change any default settings?
>>
>
>
>>
>
>>
>
>>
>
>>
> Do you get the same results if you run kafka-consumer.sh and
>>
>
>
>>
> kafka-producer.sh instead of the Node code?
>>
>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> wrote:
>>
>
>
>>
>
>>
>
>>
>
>>
> > They are hosted on AWS and I dont think there are any network issues
>>
>
>
>>
> > because I
>>
>
>
>>
> > tried testing other Queuing systems with no issues however I am using a
>>
>
>
>>
> > node.js
>>
>
>
>>
> > client with the following code. I am not sure if there are any errors or
>>
>
>
>>
> > anything I didn't set in the following code?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > //producer var kafka = require('kafka-node'); var
>>
>
>
>>
> > Producer = kafka.Producer; var Client = kafka.Client; var client =
>>
>
>
>>
> > new Client('172.31.21.175:2181'); var argv =
>>
>
>
>>
> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>>
>
>
>>
> > p = argv.p || 0; var a = argv.a || 0; var producer = new
>>
>
>
>>
> > Producer(client, { requireAcks: 1}); var num = 300005;
>>
>
>
>>
> > producer.on('ready', function () { var message = 'Hello World';
>>
>
>
>>
> > for (var i=0; i<num; i++) { producer.send([ { topic:
>>
>
>
>>
> > topic, partition: p, messages: message, attributes: a } ], function
>>
>
>
>>
> > (err, result) { console.log(err || result);
>>
>
>
>>
> > //process.exit(); }); } }); producer.on('error',
>>
>
>
>>
> > function (err) { console.log('error', err); process.exit();
>>
>
>
>>
> > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>>
>
>
>>
> > kafka.Consumer; var Offset = kafka.Offset; var Client =
>>
>
>
>>
> > kafka.Client; var argv = require('optimist').argv; var topic =
>>
>
>
>>
> > argv.topic || 'kafka_test'; var client = new
>>
>
>
>>
> > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>>
>
>
>>
> > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>>
>
>
>>
> > 1000 }; var consumer = new Consumer(client, topics, options); var
>>
>
>
>>
> > offset = new Offset(client); var start; var received = 0; var
>>
>
>
>>
> > target = 200000; var hash = 1000; consumer.on('message', function
>>
>
>
>>
> > (message) { console.log(message); received += 1; if
>>
>
>
>>
> > (received === 1) { start = new Date(); } if (received === target) {
>>
>
>
>>
> > var stop = new Date(); console.log('\nDone test');
>>
>
>
>>
> > var mps = parseInt(target/((stop-start)/1000));
>>
>
>
>>
> > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>>
>
>
>>
> > } else if (received % hash === 0){
>>
>
>
>>
> > process.stdout.write(received + '\n'); } });
>>
>
>
>>
> > consumer.on('error', function (err) { console.log('error', err); });
>>
>
>
>>
> >
>>
>
>
>>
> > Not using Mixmax yet?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
>>
>
>
>>
> > wrote:
>>
>
>
>>
> > It sounds like a network issue. Where are the 3 servers located / hosted?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com>
>>
>
> wrote:
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > Hi,
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > I have the following setup.
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Machine 2
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Single Kafka Consumer on Machine 3
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > When a producer client sends a message to the Kafka broker by pointing at
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > the
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Zookeeper Server the consumer doesn't seem to get the message right away
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > instead
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > it gets after a minute or something (pretty late). I am not sure what
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > settings I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > need to change. any ideas?
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > Thanks,kant
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
