What's the instance size that you're using? With 300k messages your single
broker might not be able to handle it.
On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> wrote:
> My goal is to test the throughput (#messages per second) given my setup and
> with a data size of 1KB. if you guys already have some idea on these
> numbers
> that would be helpful as well.
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
> wrote:
> 172.* is all private ip's for my machine I double checked it.I have not
> changed
> any default settingsI dont know how to use kafka-consumer.sh
> or kafka-producer.sh because it looks like they want me to specify a group
> and I
> didn't create any consumer group because I am using single producer and
> consumer. is there a default group?Also, I am receiving message but very
> late. I
> send about 300K messages using the node.js client and I am receiving at a
> very
> low rate. really not sure what is going on?
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
> wrote:
> Your code seems to be using the public ip of the servers. If all 3 machines
>
> are in the same availability zone on AWS, try using the private ip, and
>
> then they might communicate over the local network.
>
>
>
>
> Did you change any default settings?
>
>
>
>
> Do you get the same results if you run kafka-consumer.sh and
>
> kafka-producer.sh instead of the Node code?
>
>
>
>
> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> wrote:
>
>
>
>
> > They are hosted on AWS and I dont think there are any network issues
>
> > because I
>
> > tried testing other Queuing systems with no issues however I am using a
>
> > node.js
>
> > client with the following code. I am not sure if there are any errors or
>
> > anything I didn't set in the following code?
>
> >
>
> >
>
> > //producer var kafka = require('kafka-node'); var
>
> > Producer = kafka.Producer; var Client = kafka.Client; var client =
>
> > new Client('172.31.21.175:2181'); var argv =
>
> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>
> > p = argv.p || 0; var a = argv.a || 0; var producer = new
>
> > Producer(client, { requireAcks: 1}); var num = 300005;
>
> > producer.on('ready', function () { var message = 'Hello World';
>
> > for (var i=0; i<num; i++) { producer.send([ { topic:
>
> > topic, partition: p, messages: message, attributes: a } ], function
>
> > (err, result) { console.log(err || result);
>
> > //process.exit(); }); } }); producer.on('error',
>
> > function (err) { console.log('error', err); process.exit();
>
> > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>
> > kafka.Consumer; var Offset = kafka.Offset; var Client =
>
> > kafka.Client; var argv = require('optimist').argv; var topic =
>
> > argv.topic || 'kafka_test'; var client = new
>
> > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>
> > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>
> > 1000 }; var consumer = new Consumer(client, topics, options); var
>
> > offset = new Offset(client); var start; var received = 0; var
>
> > target = 200000; var hash = 1000; consumer.on('message', function
>
> > (message) { console.log(message); received += 1; if
>
> > (received === 1) { start = new Date(); } if (received === target) {
>
> > var stop = new Date(); console.log('\nDone test');
>
> > var mps = parseInt(target/((stop-start)/1000));
>
> > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>
> > } else if (received % hash === 0){
>
> > process.stdout.write(received + '\n'); } });
>
> > consumer.on('error', function (err) { console.log('error', err); });
>
> >
>
> > Not using Mixmax yet?
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
>
> > wrote:
>
> > It sounds like a network issue. Where are the 3 servers located / hosted?
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com>
> wrote:
>
> >
>
> >
>
> >
>
> >
>
> > Hi,
>
> >>
>
> >
>
> > I have the following setup.
>
> >>
>
> >
>
> > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on
>
> >>
>
> >
>
> > Machine 2
>
> >>
>
> >
>
> > Single Kafka Consumer on Machine 3
>
> >>
>
> >
>
> > When a producer client sends a message to the Kafka broker by pointing at
>
> >>
>
> >
>
> > the
>
> >>
>
> >
>
> > Zookeeper Server the consumer doesn't seem to get the message right away
>
> >>
>
> >
>
> > instead
>
> >>
>
> >
>
> > it gets after a minute or something (pretty late). I am not sure what
>
> >>
>
> >
>
> > settings I
>
> >>
>
> >
>
> > need to change. any ideas?
>
> >>
>
> >
>
> > Thanks,kant
>
> >
>
> >
>
