Re: hi

[kant kodali]

They are hosted on AWS and I dont think there are any network issues because I
tried testing other Queuing systems with no issues however I am using a node.js
client with the following code. I am not sure if there are any errors or
anything I didn't set in the following code?


 //producervar kafka = require('kafka-node');var Producer 
= kafka.Producer;var Client = kafka.Client;var client = new 
Client('172.31.21.175:2181');var argv = require('optimist').argv;var topic 
= argv.topic || 'kafka_test';var p = argv.p || 0;var a = argv.a || 0;
var producer = new Producer(client, { requireAcks: 1});var num = 35;
producer.on('ready', function () {  var message = 'Hello World';  for 
(var i=0; i   


Not using Mixmax yet?







On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
wrote:
It sounds like a network issue. Where are the 3 servers located / hosted?




On Thu, Sep 15, 2016 at 11:51 AM, kant kodali  wrote:





Hi,



I have the following setup.



Single Kafka broker and Zookeeper on Machine 1single Kafka producer on



Machine 2



Single Kafka Consumer on Machine 3



When a producer client sends a message to the Kafka broker by pointing at



the



Zookeeper Server the consumer doesn't seem to get the message right away



instead



it gets after a minute or something (pretty late). I am not sure what



settings I



need to change. any ideas?



Thanks,kant

Re: hi

[Ali Akhtar]

Your code seems to be using the public ip of the servers. If all 3 machines
are in the same availability zone on AWS, try using the private ip, and
then they might communicate over the local network.

Did you change any default settings?

Do you get the same results if you run kafka-consumer.sh and
kafka-producer.sh instead of the Node code?

On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:

> They are hosted on AWS and I dont think there are any network issues
> because I
> tried testing other Queuing systems with no issues however I am using a
> node.js
> client with the following code. I am not sure if there are any errors or
> anything I didn't set in the following code?
>
>
>  //producervar kafka = require('kafka-node');var
> Producer = kafka.Producer;var Client = kafka.Client;var client =
> new Client('172.31.21.175:2181');var argv =
> require('optimist').argv;var topic = argv.topic || 'kafka_test';var
> p = argv.p || 0;var a = argv.a || 0;var producer = new
> Producer(client, { requireAcks: 1});var num = 35;
> producer.on('ready', function () {  var message = 'Hello World';
>   for (var i=0; i topic, partition: p, messages: message, attributes: a }  ], function
> (err, result) {   console.log(err || result);
>  //process.exit(); });  }});producer.on('error',
> function (err) {  console.log('error', err);  process.exit();
> });//Consumervar kafka = require('kafka-node');var Consumer =
> kafka.Consumer;var Offset = kafka.Offset;var Client =
> kafka.Client;var argv = require('optimist').argv;var topic =
> argv.topic || 'kafka_test';var client = new
> Client('172.31.21.175:2181');var topics = [{topic: topic,
> partition: 0}];var options = { autoCommit: false, fetchMaxWaitMs:
> 1000 };var consumer = new Consumer(client, topics, options);var
> offset = new Offset(client);var start;var received = 0;var
> target = 20;var hash = 1000;consumer.on('message', function
> (message) { console.log(message);received += 1;if
> (received === 1) { start = new Date(); }if (received === target) {
> var stop = new Date();  console.log('\nDone test');
>   var mps = parseInt(target/((stop-start)/1000));
> console.log('Received at ' + mps + ' msgs/sec');  process.exit();
>   } else if (received % hash === 0){
> process.stdout.write(received + '\n');}});
> consumer.on('error', function (err) {  console.log('error', err);});
>
> Not using Mixmax yet?
>
>
>
>
>
>
>
> On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
> wrote:
> It sounds like a network issue. Where are the 3 servers located / hosted?
>
>
>
>
> On Thu, Sep 15, 2016 at 11:51 AM, kant kodali  wrote:
>
>
>
>
> Hi,
>>
>
> I have the following setup.
>>
>
> Single Kafka broker and Zookeeper on Machine 1single Kafka producer on
>>
>
> Machine 2
>>
>
> Single Kafka Consumer on Machine 3
>>
>
> When a producer client sends a message to the Kafka broker by pointing at
>>
>
> the
>>
>
> Zookeeper Server the consumer doesn't seem to get the message right away
>>
>
> instead
>>
>
> it gets after a minute or something (pretty late). I am not sure what
>>
>
> settings I
>>
>
> need to change. any ideas?
>>
>
> Thanks,kant
>
>

Re: hi

[kant kodali]

172.* is all private ip's for my machine I double checked it.I have not changed
any default settingsI dont know how to use  kafka-consumer.sh
or kafka-producer.sh because it looks like they want me to specify a group and I
didn't create any  consumer group because I am using single producer and
consumer. is there a default group?Also, I am receiving message but very late. I
send about 300K messages using the node.js client and I am receiving at a very
low rate. really not sure what is going on?
 





On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Your code seems to be using the public ip of the servers. If all 3 machines

are in the same availability zone on AWS, try using the private ip, and

then they might communicate over the local network.




Did you change any default settings?




Do you get the same results if you run kafka-consumer.sh and

kafka-producer.sh instead of the Node code?




On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:




> They are hosted on AWS and I dont think there are any network issues

> because I

> tried testing other Queuing systems with no issues however I am using a

> node.js

> client with the following code. I am not sure if there are any errors or

> anything I didn't set in the following code?

>

>

> //producer var kafka = require('kafka-node'); var

> Producer = kafka.Producer; var Client = kafka.Client; var client =

> new Client('172.31.21.175:2181'); var argv =

> require('optimist').argv; var topic = argv.topic || 'kafka_test'; var

> p = argv.p || 0; var a = argv.a || 0; var producer = new

> Producer(client, { requireAcks: 1}); var num = 35;

> producer.on('ready', function () { var message = 'Hello World';

> for (var i=0; i topic, partition: p, messages: message, attributes: a } ], function

> (err, result) { console.log(err || result);

> //process.exit(); }); } }); producer.on('error',

> function (err) { console.log('error', err); process.exit();

> }); //Consumer var kafka = require('kafka-node'); var Consumer =

> kafka.Consumer; var Offset = kafka.Offset; var Client =

> kafka.Client; var argv = require('optimist').argv; var topic =

> argv.topic || 'kafka_test'; var client = new

> Client('172.31.21.175:2181'); var topics = [ {topic: topic,

> partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:

> 1000 }; var consumer = new Consumer(client, topics, options); var

> offset = new Offset(client); var start; var received = 0; var

> target = 20; var hash = 1000; consumer.on('message', function

> (message) { console.log(message); received += 1; if

> (received === 1) { start = new Date(); } if (received === target) {

> var stop = new Date(); console.log('\nDone test');

> var mps = parseInt(target/((stop-start)/1000));

> console.log('Received at ' + mps + ' msgs/sec'); process.exit();

> } else if (received % hash === 0){

> process.stdout.write(received + '\n'); } });

> consumer.on('error', function (err) { console.log('error', err); });

>

> Not using Mixmax yet?

>

>

>

>

>

>

>

> On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com

> wrote:

> It sounds like a network issue. Where are the 3 servers located / hosted?

>

>

>

>

> On Thu, Sep 15, 2016 at 11:51 AM, kant kodali  wrote:

>

>

>

>

> Hi,

>>

>

> I have the following setup.

>>

>

> Single Kafka broker and Zookeeper on Machine 1single Kafka producer on

>>

>

> Machine 2

>>

>

> Single Kafka Consumer on Machine 3

>>

>

> When a producer client sends a message to the Kafka broker by pointing at

>>

>

> the

>>

>

> Zookeeper Server the consumer doesn't seem to get the message right away

>>

>

> instead

>>

>

> it gets after a minute or something (pretty late). I am not sure what

>>

>

> settings I

>>

>

> need to change. any ideas?

>>

>

> Thanks,kant

>

>

Re: hi

[kant kodali]

My goal is to test the throughput (#messages per second) given my setup and
with a data size of 1KB. if you guys already have some idea on these numbers
that would be helpful as well.
 





On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
wrote:
172.* is all private ip's for my machine I double checked it.I have not changed
any default settingsI dont know how to use  kafka-consumer.sh
or kafka-producer.sh because it looks like they want me to specify a group and I
didn't create any  consumer group because I am using single producer and
consumer. is there a default group?Also, I am receiving message but very late. I
send about 300K messages using the node.js client and I am receiving at a very
low rate. really not sure what is going on?
 





On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Your code seems to be using the public ip of the servers. If all 3 machines

are in the same availability zone on AWS, try using the private ip, and

then they might communicate over the local network.




Did you change any default settings?




Do you get the same results if you run kafka-consumer.sh and

kafka-producer.sh instead of the Node code?




On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:




> They are hosted on AWS and I dont think there are any network issues

> because I

> tried testing other Queuing systems with no issues however I am using a

> node.js

> client with the following code. I am not sure if there are any errors or

> anything I didn't set in the following code?

>

>

> //producer var kafka = require('kafka-node'); var

> Producer = kafka.Producer; var Client = kafka.Client; var client =

> new Client('172.31.21.175:2181'); var argv =

> require('optimist').argv; var topic = argv.topic || 'kafka_test'; var

> p = argv.p || 0; var a = argv.a || 0; var producer = new

> Producer(client, { requireAcks: 1}); var num = 35;

> producer.on('ready', function () { var message = 'Hello World';

> for (var i=0; i topic, partition: p, messages: message, attributes: a } ], function

> (err, result) { console.log(err || result);

> //process.exit(); }); } }); producer.on('error',

> function (err) { console.log('error', err); process.exit();

> }); //Consumer var kafka = require('kafka-node'); var Consumer =

> kafka.Consumer; var Offset = kafka.Offset; var Client =

> kafka.Client; var argv = require('optimist').argv; var topic =

> argv.topic || 'kafka_test'; var client = new

> Client('172.31.21.175:2181'); var topics = [ {topic: topic,

> partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:

> 1000 }; var consumer = new Consumer(client, topics, options); var

> offset = new Offset(client); var start; var received = 0; var

> target = 20; var hash = 1000; consumer.on('message', function

> (message) { console.log(message); received += 1; if

> (received === 1) { start = new Date(); } if (received === target) {

> var stop = new Date(); console.log('\nDone test');

> var mps = parseInt(target/((stop-start)/1000));

> console.log('Received at ' + mps + ' msgs/sec'); process.exit();

> } else if (received % hash === 0){

> process.stdout.write(received + '\n'); } });

> consumer.on('error', function (err) { console.log('error', err); });

>

> Not using Mixmax yet?

>

>

>

>

>

>

>

> On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com

> wrote:

> It sounds like a network issue. Where are the 3 servers located / hosted?

>

>

>

>

> On Thu, Sep 15, 2016 at 11:51 AM, kant kodali  wrote:

>

>

>

>

> Hi,

>>

>

> I have the following setup.

>>

>

> Single Kafka broker and Zookeeper on Machine 1single Kafka producer on

>>

>

> Machine 2

>>

>

> Single Kafka Consumer on Machine 3

>>

>

> When a producer client sends a message to the Kafka broker by pointing at

>>

>

> the

>>

>

> Zookeeper Server the consumer doesn't seem to get the message right away

>>

>

> instead

>>

>

> it gets after a minute or something (pretty late). I am not sure what

>>

>

> settings I

>>

>

> need to change. any ideas?

>>

>

> Thanks,kant

>

>

Re: hi

[Ali Akhtar]

What's the instance size that you're using? With 300k messages your single
broker might not be able to handle it.

On Thu, Sep 15, 2016 at 12:30 PM, kant kodali  wrote:

> My goal is to test the throughput (#messages per second) given my setup and
> with a data size of 1KB. if you guys already have some idea on these
> numbers
> that would be helpful as well.
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
> wrote:
> 172.* is all private ip's for my machine I double checked it.I have not
> changed
> any default settingsI dont know how to use  kafka-consumer.sh
> or kafka-producer.sh because it looks like they want me to specify a group
> and I
> didn't create any  consumer group because I am using single producer and
> consumer. is there a default group?Also, I am receiving message but very
> late. I
> send about 300K messages using the node.js client and I am receiving at a
> very
> low rate. really not sure what is going on?
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
> wrote:
> Your code seems to be using the public ip of the servers. If all 3 machines
>
> are in the same availability zone on AWS, try using the private ip, and
>
> then they might communicate over the local network.
>
>
>
>
> Did you change any default settings?
>
>
>
>
> Do you get the same results if you run kafka-consumer.sh and
>
> kafka-producer.sh instead of the Node code?
>
>
>
>
> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:
>
>
>
>
> > They are hosted on AWS and I dont think there are any network issues
>
> > because I
>
> > tried testing other Queuing systems with no issues however I am using a
>
> > node.js
>
> > client with the following code. I am not sure if there are any errors or
>
> > anything I didn't set in the following code?
>
> >
>
> >
>
> > //producer var kafka = require('kafka-node'); var
>
> > Producer = kafka.Producer; var Client = kafka.Client; var client =
>
> > new Client('172.31.21.175:2181'); var argv =
>
> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>
> > p = argv.p || 0; var a = argv.a || 0; var producer = new
>
> > Producer(client, { requireAcks: 1}); var num = 35;
>
> > producer.on('ready', function () { var message = 'Hello World';
>
> > for (var i=0; i
> > topic, partition: p, messages: message, attributes: a } ], function
>
> > (err, result) { console.log(err || result);
>
> > //process.exit(); }); } }); producer.on('error',
>
> > function (err) { console.log('error', err); process.exit();
>
> > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>
> > kafka.Consumer; var Offset = kafka.Offset; var Client =
>
> > kafka.Client; var argv = require('optimist').argv; var topic =
>
> > argv.topic || 'kafka_test'; var client = new
>
> > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>
> > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>
> > 1000 }; var consumer = new Consumer(client, topics, options); var
>
> > offset = new Offset(client); var start; var received = 0; var
>
> > target = 20; var hash = 1000; consumer.on('message', function
>
> > (message) { console.log(message); received += 1; if
>
> > (received === 1) { start = new Date(); } if (received === target) {
>
> > var stop = new Date(); console.log('\nDone test');
>
> > var mps = parseInt(target/((stop-start)/1000));
>
> > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>
> > } else if (received % hash === 0){
>
> > process.stdout.write(received + '\n'); } });
>
> > consumer.on('error', function (err) { console.log('error', err); });
>
> >
>
> > Not using Mixmax yet?
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
>
> > wrote:
>
> > It sounds like a network issue. Where are the 3 servers located / hosted?
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali 
> wrote:
>
> >
>
> >
>
> >
>
> >
>
> > Hi,
>
> >>
>
> >
>
> > I have the following setup.
>
> >>
>
> >
>
> > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on
>
> >>
>
> >
>
> > Machine 2
>
> >>
>
> >
>
> > Single Kafka Consumer on Machine 3
>
> >>
>
> >
>
> > When a producer client sends a message to the Kafka broker by pointing at
>
> >>
>
> >
>
> > the
>
> >>
>
> >
>
> > Zookeeper Server the consumer doesn't seem to get the message right away
>
> >>
>
> >
>
> > instead
>
> >>
>
> >
>
> > it gets after a minute or something (pretty late). I am not sure what
>
> >>
>
> >
>
> > settings I
>
> >>
>
> >
>
> > need to change. any ideas?
>
> >>
>
> >
>
> > Thanks,kant
>
> >
>
> >
>

Re: hi

[kant kodali]

m4.xlarge






On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
What's the instance size that you're using? With 300k messages your single

broker might not be able to handle it.




On Thu, Sep 15, 2016 at 12:30 PM, kant kodali  wrote:





My goal is to test the throughput (#messages per second) given my setup and



with a data size of 1KB. if you guys already have some idea on these



numbers



that would be helpful as well.



























On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com



wrote:



172.* is all private ip's for my machine I double checked it.I have not



changed



any default settingsI dont know how to use kafka-consumer.sh



or kafka-producer.sh because it looks like they want me to specify a group



and I



didn't create any consumer group because I am using single producer and



consumer. is there a default group?Also, I am receiving message but very



late. I



send about 300K messages using the node.js client and I am receiving at a



very



low rate. really not sure what is going on?



























On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com



wrote:



Your code seems to be using the public ip of the servers. If all 3 machines







are in the same availability zone on AWS, try using the private ip, and







then they might communicate over the local network.



















Did you change any default settings?



















Do you get the same results if you run kafka-consumer.sh and







kafka-producer.sh instead of the Node code?



















On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:



















> They are hosted on AWS and I dont think there are any network issues







> because I







> tried testing other Queuing systems with no issues however I am using a







> node.js







> client with the following code. I am not sure if there are any errors or







> anything I didn't set in the following code?







>







>







> //producer var kafka = require('kafka-node'); var







> Producer = kafka.Producer; var Client = kafka.Client; var client =







> new Client('172.31.21.175:2181'); var argv =







> require('optimist').argv; var topic = argv.topic || 'kafka_test'; var







> p = argv.p || 0; var a = argv.a || 0; var producer = new







> Producer(client, { requireAcks: 1}); var num = 35;







> producer.on('ready', function () { var message = 'Hello World';







> for (var i=0; i






> topic, partition: p, messages: message, attributes: a } ], function







> (err, result) { console.log(err || result);







> //process.exit(); }); } }); producer.on('error',







> function (err) { console.log('error', err); process.exit();







> }); //Consumer var kafka = require('kafka-node'); var Consumer =







> kafka.Consumer; var Offset = kafka.Offset; var Client =







> kafka.Client; var argv = require('optimist').argv; var topic =







> argv.topic || 'kafka_test'; var client = new







> Client('172.31.21.175:2181'); var topics = [ {topic: topic,







> partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:







> 1000 }; var consumer = new Consumer(client, topics, options); var







> offset = new Offset(client); var start; var received = 0; var







> target = 20; var hash = 1000; consumer.on('message', function







> (message) { console.log(message); received += 1; if







> (received === 1) { start = new Date(); } if (received === target) {







> var stop = new Date(); console.log('\nDone test');







> var mps = parseInt(target/((stop-start)/1000));







> console.log('Received at ' + mps + ' msgs/sec'); process.exit();







> } else if (received % hash === 0){







> process.stdout.write(received + '\n'); } });







> consumer.on('error', function (err) { console.log('error', err); });







>







> Not using Mixmax yet?







>







>







>







>







>







>







>







> On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com







> wrote:







> It sounds like a network issue. Where are the 3 servers located / hosted?







>







>







>







>







> On Thu, Sep 15, 2016 at 11:51 AM, kant kodali 



wrote:







>







>







>







>







> Hi,







>>







>







> I have the following setup.







>>







>







> Single Kafka broker and Zookeeper on Machine 1single Kafka producer on







>>







>







> Machine 2







>>







>







> Single Kafka Consumer on Machine 3







>>







>







> When a producer client sends a message to the Kafka broker by pointing at







>>







>







> the







>>







>







> Zookeeper Server the consumer doesn't seem to get the message right away







>>







>







> instead







>>







>







> it gets after a minute or something (pretty late). I am not sure what







>>







>







> settin

Re: hi

[Ali Akhtar]

Lower the workload gradually, start from 10 messages, increase to 100, then
1000, and so on. See if it slows down as the workload increases. If so, you
need more brokers + partitions to handle the workload.

On Thu, Sep 15, 2016 at 12:42 PM, kant kodali  wrote:

> m4.xlarge
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
>
> wrote:
> What's the instance size that you're using? With 300k messages your single
>
> broker might not be able to handle it.
>
>
>
>
> On Thu, Sep 15, 2016 at 12:30 PM, kant kodali  wrote:
>
>
>
>
> My goal is to test the throughput (#messages per second) given my setup and
>>
>
> with a data size of 1KB. if you guys already have some idea on these
>>
>
> numbers
>>
>
> that would be helpful as well.
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
>>
>
> wrote:
>>
>
> 172.* is all private ip's for my machine I double checked it.I have not
>>
>
> changed
>>
>
> any default settingsI dont know how to use kafka-consumer.sh
>>
>
> or kafka-producer.sh because it looks like they want me to specify a group
>>
>
> and I
>>
>
> didn't create any consumer group because I am using single producer and
>>
>
> consumer. is there a default group?Also, I am receiving message but very
>>
>
> late. I
>>
>
> send about 300K messages using the node.js client and I am receiving at a
>>
>
> very
>>
>
> low rate. really not sure what is going on?
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
>>
>
> wrote:
>>
>
> Your code seems to be using the public ip of the servers. If all 3 machines
>>
>
>
>>
> are in the same availability zone on AWS, try using the private ip, and
>>
>
>
>>
> then they might communicate over the local network.
>>
>
>
>>
>
>>
>
>>
>
>>
> Did you change any default settings?
>>
>
>
>>
>
>>
>
>>
>
>>
> Do you get the same results if you run kafka-consumer.sh and
>>
>
>
>>
> kafka-producer.sh instead of the Node code?
>>
>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:
>>
>
>
>>
>
>>
>
>>
>
>>
> > They are hosted on AWS and I dont think there are any network issues
>>
>
>
>>
> > because I
>>
>
>
>>
> > tried testing other Queuing systems with no issues however I am using a
>>
>
>
>>
> > node.js
>>
>
>
>>
> > client with the following code. I am not sure if there are any errors or
>>
>
>
>>
> > anything I didn't set in the following code?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > //producer var kafka = require('kafka-node'); var
>>
>
>
>>
> > Producer = kafka.Producer; var Client = kafka.Client; var client =
>>
>
>
>>
> > new Client('172.31.21.175:2181'); var argv =
>>
>
>
>>
> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var
>>
>
>
>>
> > p = argv.p || 0; var a = argv.a || 0; var producer = new
>>
>
>
>>
> > Producer(client, { requireAcks: 1}); var num = 35;
>>
>
>
>>
> > producer.on('ready', function () { var message = 'Hello World';
>>
>
>
>>
> > for (var i=0; i>
>
>
>>
> > topic, partition: p, messages: message, attributes: a } ], function
>>
>
>
>>
> > (err, result) { console.log(err || result);
>>
>
>
>>
> > //process.exit(); }); } }); producer.on('error',
>>
>
>
>>
> > function (err) { console.log('error', err); process.exit();
>>
>
>
>>
> > }); //Consumer var kafka = require('kafka-node'); var Consumer =
>>
>
>
>>
> > kafka.Consumer; var Offset = kafka.Offset; var Client =
>>
>
>
>>
> > kafka.Client; var argv = require('optimist').argv; var topic =
>>
>
>
>>
> > argv.topic || 'kafka_test'; var client = new
>>
>
>
>>
> > Client('172.31.21.175:2181'); var topics = [ {topic: topic,
>>
>
>
>>
> > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:
>>
>
>
>>
> > 1000 }; var consumer = new Consumer(client, topics, options); var
>>
>
>
>>
> > offset = new Offset(client); var start; var received = 0; var
>>
>
>
>>
> > target = 20; var hash = 1000; consumer.on('message', function
>>
>
>
>>
> > (message) { console.log(message); received += 1; if
>>
>
>
>>
> > (received === 1) { start = new Date(); } if (received === target) {
>>
>
>
>>
> > var stop = new Date(); console.log('\nDone test');
>>
>
>
>>
> > var mps = parseInt(target/((stop-start)/1000));
>>
>
>
>>
> > console.log('Received at ' + mps + ' msgs/sec'); process.exit();
>>
>
>
>>
> > } else if (received % hash === 0){
>>
>
>
>>
> > process.stdout.write(received + '\n'); } });
>>
>
>
>>
> > consumer.on('error', function (err) { console.log('error', err); });
>>
>
>
>>
> >
>>
>
>
>>
> > Not using Mixmax yet?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
>>
>
>
>>
> > wrote:
>>
>
>
>>
> > It sounds like a network issue. Where are the 3 servers located / hosted?
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali 
>>
>
> wrote:
>>
>
>
>>
> >

v0.10 MirrorMaker producer cannot send v0.8 message from v0.10 broker

[Samuel Zhou]

Hi,

I have a pipeline that publish message with v0.8 client, the message goes
to v0.10 broker first then mirror maker will consume it and publish it to
another v0.10 brokers. But I got the following message from MM log:

java.lang.IllegalArgumentException: Invalid timestamp -1

at
org.apache.kafka.clients.producer.ProducerRecord.(ProducerRecord.java:60)

at
kafka.tools.MirrorMaker$defaultMirrorMakerMessageHandler$.handle(MirrorMaker.scala:678)

at
kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:414)

Above error makes MM dead. I am not sure if KAFKA-3188 covers the test. Or
are there any parameters for MM that can fix above error?

Thanks!

Samuel

Re: hi

[kant kodali]

yeah..
I tried it with 10 messages with single broker and only one partiton that looked
instantaneous and ~5K messages/sec for the data size of 1KBI tried it with 1000
messages that looked instantaneous as well ~5K messages/sec for the data size of
1KBI tried it with 10K messages with single broker and only one partiton  things
started to go down ~1K messages/sec for the data size of 1KB
having only one partition on a single broker is a bad?  My goal is to run some
basic benchmarks on NATS & NSQ & KAFKA
I have the same environment for all three (NATS & NSQ & KAFKA)
a broker  on Machine 1producer on Machine 2Consumer on Machine 3
with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws instance.
I have pushed 300K messages with NATS and it was able to handle easily and
receive throughput was 5K messages/secI have pushed 300K messages and NSQ and
receive throughput was 2K messages/secI am unable to push 300K messages with
Kafka with the above configuration and environment so at this point my biggest
question is what is the fair setup for Kafka so its comparable with NATS and
NSQ?
kant
 





On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Lower the workload gradually, start from 10 messages, increase to 100, then

1000, and so on. See if it slows down as the workload increases. If so, you

need more brokers + partitions to handle the workload.




On Thu, Sep 15, 2016 at 12:42 PM, kant kodali  wrote:




> m4.xlarge

>

>

>

>

>

>

> On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com

>

> wrote:

> What's the instance size that you're using? With 300k messages your single

>

> broker might not be able to handle it.

>

>

>

>

> On Thu, Sep 15, 2016 at 12:30 PM, kant kodali  wrote:

>

>

>

>

> My goal is to test the throughput (#messages per second) given my setup and

>>

>

> with a data size of 1KB. if you guys already have some idea on these

>>

>

> numbers

>>

>

> that would be helpful as well.

>>

>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com

>>

>

> wrote:

>>

>

> 172.* is all private ip's for my machine I double checked it.I have not

>>

>

> changed

>>

>

> any default settingsI dont know how to use kafka-consumer.sh

>>

>

> or kafka-producer.sh because it looks like they want me to specify a group

>>

>

> and I

>>

>

> didn't create any consumer group because I am using single producer and

>>

>

> consumer. is there a default group?Also, I am receiving message but very

>>

>

> late. I

>>

>

> send about 300K messages using the node.js client and I am receiving at a

>>

>

> very

>>

>

> low rate. really not sure what is going on?

>>

>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com

>>

>

> wrote:

>>

>

> Your code seems to be using the public ip of the servers. If all 3 machines

>>

>

>

>>

> are in the same availability zone on AWS, try using the private ip, and

>>

>

>

>>

> then they might communicate over the local network.

>>

>

>

>>

>

>>

>

>>

>

>>

> Did you change any default settings?

>>

>

>

>>

>

>>

>

>>

>

>>

> Do you get the same results if you run kafka-consumer.sh and

>>

>

>

>>

> kafka-producer.sh instead of the Node code?

>>

>

>

>>

>

>>

>

>>

>

>>

> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali  wrote:

>>

>

>

>>

>

>>

>

>>

>

>>

> > They are hosted on AWS and I dont think there are any network issues

>>

>

>

>>

> > because I

>>

>

>

>>

> > tried testing other Queuing systems with no issues however I am using a

>>

>

>

>>

> > node.js

>>

>

>

>>

> > client with the following code. I am not sure if there are any errors or

>>

>

>

>>

> > anything I didn't set in the following code?

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> > //producer var kafka = require('kafka-node'); var

>>

>

>

>>

> > Producer = kafka.Producer; var Client = kafka.Client; var client =

>>

>

>

>>

> > new Client('172.31.21.175:2181'); var argv =

>>

>

>

>>

> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var

>>

>

>

>>

> > p = argv.p || 0; var a = argv.a || 0; var producer = new

>>

>

>

>>

> > Producer(client, { requireAcks: 1}); var num = 35;

>>

>

>

>>

> > producer.on('ready', function () { var message = 'Hello World';

>>

>

>

>>

> > for (var i=0; i>

>

>

>>

> > topic, partition: p, messages: message, attributes: a } ], function

>>

>

>

>>

> > (err, result) { console.log(err || result);

>>

>

>

>>

> > //process.exit(); }); } }); producer.on('error',

>>

>

>

>>

> > function (err) { console.log('error', err); process.exit();

>>

>

>

>>

> > }); //Consumer var kafka = require('kafka-node'); var Consumer =

>>

>

>

>>

> > kafka.Consumer; var Offset = kafka.Offset; var Client =

>>

>

>

>>

> > kafka.Client; var argv = require('optimist').argv; var topic =

>>

>

Publish to 1 topic, consume from N

[Luiz Cordeiro]

Hello,

We’re considering migrating an AMQ-based platform to Kafka. However our 
application logic needs an AMQ feature called Dynamic Binding, that is, on AMQ 
one publishes messages to an Exchange, which can be dynamically configured to 
deliver a copy of the message to several queues, based on binding rules. So 
when a new client comes alive, it may create its binding rules to specify a set 
of topics to listen to, and receive all the messages from these topics on a 
private queue.

I understand that Kafka neither provides this nor will, as it is not its 
objective, but I was wondering if there’s another component, an overlay to 
Kafka, that could provide this feature while using Kafka behind the scenes for 
the persistence, something like this:

Publisher --> Mapping Service --> Kafka <-- Consumers
 ^  |
 |   Binding rules  |
 \--/

Are you aware of such a component? Otherwise, how would you solve this issue of 
publish to 1 place and have it replicated on N topics.

Best Regards,
Luiz

Re: hi

[Ali Akhtar]

The issue is clearly that you're running out of resources, so I would add
more brokers and/or larger instances.

You're also using Node which is not the best for performance. A compiled
language such as Java would give you the best performance.

Here's a case study that should help:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Good luck, let us know how it goes

On Thu, Sep 15, 2016 at 1:42 PM, kant kodali  wrote:

> yeah..
> I tried it with 10 messages with single broker and only one partiton that
> looked
> instantaneous and ~5K messages/sec for the data size of 1KBI tried it with
> 1000
> messages that looked instantaneous as well ~5K messages/sec for the data
> size of
> 1KBI tried it with 10K messages with single broker and only one
> partiton  things
> started to go down ~1K messages/sec for the data size of 1KB
> having only one partition on a single broker is a bad?  My goal is to run
> some
> basic benchmarks on NATS & NSQ & KAFKA
> I have the same environment for all three (NATS & NSQ & KAFKA)
> a broker  on Machine 1producer on Machine 2Consumer on Machine 3
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
> instance.
> I have pushed 300K messages with NATS and it was able to handle easily and
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
> and
> receive throughput was 2K messages/secI am unable to push 300K messages
> with
> Kafka with the above configuration and environment so at this point my
> biggest
> question is what is the fair setup for Kafka so its comparable with NATS
> and
> NSQ?
> kant
>
>
>
>
>
>
> On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com
> wrote:
> Lower the workload gradually, start from 10 messages, increase to 100, then
>
> 1000, and so on. See if it slows down as the workload increases. If so, you
>
> need more brokers + partitions to handle the workload.
>
>
>
>
> On Thu, Sep 15, 2016 at 12:42 PM, kant kodali  wrote:
>
>
>
>
> > m4.xlarge
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
>
> >
>
> > wrote:
>
> > What's the instance size that you're using? With 300k messages your
> single
>
> >
>
> > broker might not be able to handle it.
>
> >
>
> >
>
> >
>
> >
>
> > On Thu, Sep 15, 2016 at 12:30 PM, kant kodali 
> wrote:
>
> >
>
> >
>
> >
>
> >
>
> > My goal is to test the throughput (#messages per second) given my setup
> and
>
> >>
>
> >
>
> > with a data size of 1KB. if you guys already have some idea on these
>
> >>
>
> >
>
> > numbers
>
> >>
>
> >
>
> > that would be helpful as well.
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
>
> >>
>
> >
>
> > wrote:
>
> >>
>
> >
>
> > 172.* is all private ip's for my machine I double checked it.I have not
>
> >>
>
> >
>
> > changed
>
> >>
>
> >
>
> > any default settingsI dont know how to use kafka-consumer.sh
>
> >>
>
> >
>
> > or kafka-producer.sh because it looks like they want me to specify a
> group
>
> >>
>
> >
>
> > and I
>
> >>
>
> >
>
> > didn't create any consumer group because I am using single producer and
>
> >>
>
> >
>
> > consumer. is there a default group?Also, I am receiving message but very
>
> >>
>
> >
>
> > late. I
>
> >>
>
> >
>
> > send about 300K messages using the node.js client and I am receiving at a
>
> >>
>
> >
>
> > very
>
> >>
>
> >
>
> > low rate. really not sure what is going on?
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com
>
> >>
>
> >
>
> > wrote:
>
> >>
>
> >
>
> > Your code seems to be using the public ip of the servers. If all 3
> machines
>
> >>
>
> >
>
> >
>
> >>
>
> > are in the same availability zone on AWS, try using the private ip, and
>
> >>
>
> >
>
> >
>
> >>
>
> > then they might communicate over the local network.
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > Did you change any default settings?
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > Do you get the same results if you run kafka-consumer.sh and
>
> >>
>
> >
>
> >
>
> >>
>
> > kafka-producer.sh instead of the Node code?
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > On Thu, Sep 15, 2016 at 12:01 PM, kant kodali 
> wrote:
>
> >>
>
> >
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> >
>
> >>
>
> > > They are hosted on AWS and I dont think there are any network issues
>
> >>
>
> >
>
> >
>
> >>
>
> > > because I
>
> >>
>
> >
>
> >
>
> >>
>
> > > tried testing other Queuing systems with no issues however I am using a
>
> >>
>
> >
>
> >
>
> >>
>
> > > node.js
>
> >>
>
> >
>
> >
>
> >>
>
> > > client with the following code. I am not sure if there are any errors
> or
>
> >>
>
> >
>
> >
>
> >>
>
> > > anything I didn't set in the following code?
>
> >>
>
> 

Re: hi

[kant kodali]

I used node.js client libraries for all three and yes I want to make sure I am
comparing apples to apples so I make it as equivalent as possible.
Again the big question is What is the right setup for Kafka to be comparable
with the other I mentioned in my previous email?






On Thu, Sep 15, 2016 1:47 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
The issue is clearly that you're running out of resources, so I would add

more brokers and/or larger instances.




You're also using Node which is not the best for performance. A compiled

language such as Java would give you the best performance.




Here's a case study that should help:

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines




Good luck, let us know how it goes




On Thu, Sep 15, 2016 at 1:42 PM, kant kodali  wrote:





yeah..



I tried it with 10 messages with single broker and only one partiton that



looked



instantaneous and ~5K messages/sec for the data size of 1KBI tried it with



1000



messages that looked instantaneous as well ~5K messages/sec for the data



size of



1KBI tried it with 10K messages with single broker and only one



partiton things



started to go down ~1K messages/sec for the data size of 1KB



having only one partition on a single broker is a bad? My goal is to run



some



basic benchmarks on NATS & NSQ & KAFKA



I have the same environment for all three (NATS & NSQ & KAFKA)



a broker on Machine 1producer on Machine 2Consumer on Machine 3



with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws



instance.



I have pushed 300K messages with NATS and it was able to handle easily and



receive throughput was 5K messages/secI have pushed 300K messages and NSQ



and



receive throughput was 2K messages/secI am unable to push 300K messages



with



Kafka with the above configuration and environment so at this point my



biggest



question is what is the fair setup for Kafka so its comparable with NATS



and



NSQ?



kant



























On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com



wrote:



Lower the workload gradually, start from 10 messages, increase to 100, then







1000, and so on. See if it slows down as the workload increases. If so, you







need more brokers + partitions to handle the workload.



















On Thu, Sep 15, 2016 at 12:42 PM, kant kodali  wrote:



















> m4.xlarge







>







>







>







>







>







>







> On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com







>







> wrote:







> What's the instance size that you're using? With 300k messages your



single







>







> broker might not be able to handle it.







>







>







>







>







> On Thu, Sep 15, 2016 at 12:30 PM, kant kodali 



wrote:







>







>







>







>







> My goal is to test the throughput (#messages per second) given my setup



and







>>







>







> with a data size of 1KB. if you guys already have some idea on these







>>







>







> numbers







>>







>







> that would be helpful as well.







>>







>







>







>>







>







>>







>







>>







>







>>







>







>>







>







>>







> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com







>>







>







> wrote:







>>







>







> 172.* is all private ip's for my machine I double checked it.I have not







>>







>







> changed







>>







>







> any default settingsI dont know how to use kafka-consumer.sh







>>







>







> or kafka-producer.sh because it looks like they want me to specify a



group







>>







>







> and I







>>







>







> didn't create any consumer group because I am using single producer and







>>







>







> consumer. is there a default group?Also, I am receiving message but very







>>







>







> late. I







>>







>







> send about 300K messages using the node.js client and I am receiving at a







>>







>







> very







>>







>







> low rate. really not sure what is going on?







>>







>







>







>>







>







>>







>







>>







>







>>







>







>>







>







>>







> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com







>>







>







> wrote:







>>







>







> Your code seems to be using the public ip of the servers. If all 3



machines







>>







>







>







>>







> are in the same availability zone on AWS, try using the private ip, and







>>







>







>







>>







> then they might communicate over the local network.







>>







>







>







>>







>







>>







>







>>







>







>>







> Did you change any default settings?







>>







>







>







>>







>







>>

Re: hi

[Ali Akhtar]

I'd post to the mailing list again with a new subject and ask that.

On Thu, Sep 15, 2016 at 1:52 PM, kant kodali  wrote:

> I used node.js client libraries for all three and yes I want to make sure
> I am
> comparing apples to apples so I make it as equivalent as possible.
> Again the big question is What is the right setup for Kafka to be
> comparable
> with the other I mentioned in my previous email?
>
>
>
>
>
>
> On Thu, Sep 15, 2016 1:47 AM, Ali Akhtar ali.rac...@gmail.com
> wrote:
> The issue is clearly that you're running out of resources, so I would add
>
> more brokers and/or larger instances.
>
>
>
>
> You're also using Node which is not the best for performance. A compiled
>
> language such as Java would give you the best performance.
>
>
>
>
> Here's a case study that should help:
>
> https://engineering.linkedin.com/kafka/benchmarking-apache-k
> afka-2-million-writes-second-three-cheap-machines
>
>
>
>
> Good luck, let us know how it goes
>
>
>
>
> On Thu, Sep 15, 2016 at 1:42 PM, kant kodali  wrote:
>
>
>
>
> yeah..
>>
>
> I tried it with 10 messages with single broker and only one partiton that
>>
>
> looked
>>
>
> instantaneous and ~5K messages/sec for the data size of 1KBI tried it with
>>
>
> 1000
>>
>
> messages that looked instantaneous as well ~5K messages/sec for the data
>>
>
> size of
>>
>
> 1KBI tried it with 10K messages with single broker and only one
>>
>
> partiton things
>>
>
> started to go down ~1K messages/sec for the data size of 1KB
>>
>
> having only one partition on a single broker is a bad? My goal is to run
>>
>
> some
>>
>
> basic benchmarks on NATS & NSQ & KAFKA
>>
>
> I have the same environment for all three (NATS & NSQ & KAFKA)
>>
>
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
>>
>
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
>>
>
> instance.
>>
>
> I have pushed 300K messages with NATS and it was able to handle easily and
>>
>
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
>>
>
> and
>>
>
> receive throughput was 2K messages/secI am unable to push 300K messages
>>
>
> with
>>
>
> Kafka with the above configuration and environment so at this point my
>>
>
> biggest
>>
>
> question is what is the fair setup for Kafka so its comparable with NATS
>>
>
> and
>>
>
> NSQ?
>>
>
> kant
>>
>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com
>>
>
> wrote:
>>
>
> Lower the workload gradually, start from 10 messages, increase to 100, then
>>
>
>
>>
> 1000, and so on. See if it slows down as the workload increases. If so, you
>>
>
>
>>
> need more brokers + partitions to handle the workload.
>>
>
>
>>
>
>>
>
>>
>
>>
> On Thu, Sep 15, 2016 at 12:42 PM, kant kodali  wrote:
>>
>
>
>>
>
>>
>
>>
>
>>
> > m4.xlarge
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
>>
>
>
>>
> >
>>
>
>
>>
> > wrote:
>>
>
>
>>
> > What's the instance size that you're using? With 300k messages your
>>
>
> single
>>
>
>
>>
> >
>>
>
>
>>
> > broker might not be able to handle it.
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > On Thu, Sep 15, 2016 at 12:30 PM, kant kodali 
>>
>
> wrote:
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> > My goal is to test the throughput (#messages per second) given my setup
>>
>
> and
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > with a data size of 1KB. if you guys already have some idea on these
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > numbers
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > that would be helpful as well.
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> >>
>>
>
>
>>
> > On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > wrote:
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > 172.* is all private ip's for my machine I double checked it.I have not
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > changed
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > any default settingsI dont know how to use kafka-consumer.sh
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > or kafka-producer.sh because it looks like they want me to specify a
>>
>
> group
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > and I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > didn't create any consumer group because I am using single producer and
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > consumer. is there a default group?Also, I am receiving message but very
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > late. I
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > send about 300K messages using the node.js client and I am receiving at a
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > very
>>
>
>
>>
> >>
>>
>
>
>>
> >
>>
>
>
>>
> > low rate. really not sure what is going on?
>>
>

What is the fair setup of Kafka to be comparable with NATS or NSQ?

[kant kodali]

with Kafka I tried it with 10 messages with single broker and only one partiton
that looked instantaneous and ~5K messages/sec for the data size of 1KB
I tried it with 1000 messages that looked instantaneous as well ~5K messages/sec
for the data size of 1KBI tried it with 10K messages with single broker and only
one partiton  things started to go down ~1K messages/sec for the data size of
1KB
having only one partition on a single broker is a bad?  My goal is to run some
basic benchmarks on NATS & NSQ & KAFKA
I have the same environment for all three (NATS & NSQ & KAFKA)
a broker  on Machine 1producer on Machine 2Consumer on Machine 3
with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws instance.
I have pushed 300K messages with NATS and it was able to handle easily and
receive throughput was 5K messages/secI have pushed 300K messages and NSQ and
receive throughput was 2K messages/secI am unable to push 300K messages with
Kafka with the above configuration and environmentso at this point my biggest
question is what is the fair setup for Kafka so its comparable with NATS and
NSQ?
kant

Re: Publish to 1 topic, consume from N

[Ali Akhtar]

It sounds like you can implement the 'mapping service'  component yourself
using Kafka.

Have all of your messages go to one kafka topic. Have one consumer group
listening to this 'everything goes here' topic. This consumer group acts as
your mapping service. It looks at each message, and based on your rules, it
sends that message to a different topic for those specific rules.

Then you have your consumers listening to the specific topics that they
need to. Your mapping service does the job of redirecting messages from the
'everything' topic to the specific topics based on your rules.

On Thu, Sep 15, 2016 at 1:43 PM, Luiz Cordeiro <
luiz.corde...@mobilityhouse.com> wrote:

> Hello,
>
> We’re considering migrating an AMQ-based platform to Kafka. However our
> application logic needs an AMQ feature called Dynamic Binding, that is, on
> AMQ one publishes messages to an Exchange, which can be dynamically
> configured to deliver a copy of the message to several queues, based on
> binding rules. So when a new client comes alive, it may create its binding
> rules to specify a set of topics to listen to, and receive all the messages
> from these topics on a private queue.
>
> I understand that Kafka neither provides this nor will, as it is not its
> objective, but I was wondering if there’s another component, an overlay to
> Kafka, that could provide this feature while using Kafka behind the scenes
> for the persistence, something like this:
>
> Publisher --> Mapping Service --> Kafka <-- Consumers
>  ^  |
>  |   Binding rules  |
>  \--/
>
> Are you aware of such a component? Otherwise, how would you solve this
> issue of publish to 1 place and have it replicated on N topics.
>
> Best Regards,
> Luiz
>

Re: What is the fair setup of Kafka to be comparable with NATS or NSQ?

[Ben Davison]

Hi Kant,

I was following the other thread, can you try using a different
benchmarking client for a test.

https://grey-boundary.io/load-testing-apache-kafka-on-aws/

Ben

On Thursday, 15 September 2016, kant kodali  wrote:

> with Kafka I tried it with 10 messages with single broker and only one
> partiton
> that looked instantaneous and ~5K messages/sec for the data size of 1KB
> I tried it with 1000 messages that looked instantaneous as well ~5K
> messages/sec
> for the data size of 1KBI tried it with 10K messages with single broker
> and only
> one partiton  things started to go down ~1K messages/sec for the data size
> of
> 1KB
> having only one partition on a single broker is a bad?  My goal is to run
> some
> basic benchmarks on NATS & NSQ & KAFKA
> I have the same environment for all three (NATS & NSQ & KAFKA)
> a broker  on Machine 1producer on Machine 2Consumer on Machine 3
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
> instance.
> I have pushed 300K messages with NATS and it was able to handle easily and
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
> and
> receive throughput was 2K messages/secI am unable to push 300K messages
> with
> Kafka with the above configuration and environmentso at this point my
> biggest
> question is what is the fair setup for Kafka so its comparable with NATS
> and
> NSQ?
> kant

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Re: What is the fair setup of Kafka to be comparable with NATS or NSQ?

[kant kodali]

Hi Ben,
I can give that a try but can you tell me the suspicion or motivation behind it?
other words you think single partition and single broker should be comparable to
the setup I had with NATS and NSQ except you suspect the client library or
something?
Thanks,Kant






On Thu, Sep 15, 2016 2:16 AM, Ben Davison ben.davi...@7digital.com
wrote:
Hi Kant,




I was following the other thread, can you try using a different

benchmarking client for a test.




https://grey-boundary.io/load-testing-apache-kafka-on-aws/




Ben




On Thursday, 15 September 2016, kant kodali  wrote:





with Kafka I tried it with 10 messages with single broker and only one



partiton



that looked instantaneous and ~5K messages/sec for the data size of 1KB



I tried it with 1000 messages that looked instantaneous as well ~5K



messages/sec



for the data size of 1KBI tried it with 10K messages with single broker



and only



one partiton things started to go down ~1K messages/sec for the data size



of



1KB



having only one partition on a single broker is a bad? My goal is to run



some



basic benchmarks on NATS & NSQ & KAFKA



I have the same environment for all three (NATS & NSQ & KAFKA)



a broker on Machine 1producer on Machine 2Consumer on Machine 3



with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws



instance.



I have pushed 300K messages with NATS and it was able to handle easily and



receive throughput was 5K messages/secI have pushed 300K messages and NSQ



and



receive throughput was 2K messages/secI am unable to push 300K messages



with



Kafka with the above configuration and environmentso at this point my



biggest



question is what is the fair setup for Kafka so its comparable with NATS



and



NSQ?



kant





--







This email, including attachments, is private and confidential. If you have 

received this email in error please notify the sender and delete it from 

your system. Emails are not secure and may contain viruses. No liability 

can be accepted for viruses that might be transferred by this email or any 

attachment. Any unauthorised copying of this message or unauthorised 

distribution and publication of the information contained herein are 


prohibited.




7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.

Registered in England and Wales. Registered No. 04843573.

Re: What is the fair setup of Kafka to be comparable with NATS or NSQ?

[Ben Davison]

Yup, I suspect the client library.

On Thu, Sep 15, 2016 at 10:28 AM, kant kodali  wrote:

> Hi Ben,
> I can give that a try but can you tell me the suspicion or motivation
> behind it?
> other words you think single partition and single broker should be
> comparable to
> the setup I had with NATS and NSQ except you suspect the client library or
> something?
> Thanks,Kant
>
>
>
>
>
>
> On Thu, Sep 15, 2016 2:16 AM, Ben Davison ben.davi...@7digital.com
> wrote:
>
> Hi Kant,
>
>
>
>
> I was following the other thread, can you try using a different
>
> benchmarking client for a test.
>
>
>
>
> https://grey-boundary.io/load-testing-apache-kafka-on-aws/
>
>
>
>
> Ben
>
>
>
>
> On Thursday, 15 September 2016, kant kodali  wrote:
>
>
>
>
> with Kafka I tried it with 10 messages with single broker and only one
>>
>
> partiton
>>
>
> that looked instantaneous and ~5K messages/sec for the data size of 1KB
>>
>
> I tried it with 1000 messages that looked instantaneous as well ~5K
>>
>
> messages/sec
>>
>
> for the data size of 1KBI tried it with 10K messages with single broker
>>
>
> and only
>>
>
> one partiton things started to go down ~1K messages/sec for the data size
>>
>
> of
>>
>
> 1KB
>>
>
> having only one partition on a single broker is a bad? My goal is to run
>>
>
> some
>>
>
> basic benchmarks on NATS & NSQ & KAFKA
>>
>
> I have the same environment for all three (NATS & NSQ & KAFKA)
>>
>
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
>>
>
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
>>
>
> instance.
>>
>
> I have pushed 300K messages with NATS and it was able to handle easily and
>>
>
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
>>
>
> and
>>
>
> receive throughput was 2K messages/secI am unable to push 300K messages
>>
>
> with
>>
>
> Kafka with the above configuration and environmentso at this point my
>>
>
> biggest
>>
>
> question is what is the fair setup for Kafka so its comparable with NATS
>>
>
> and
>>
>
> NSQ?
>>
>
> kant
>>
>
>
>
>
> --
>
>
>
>
>
>
>
> This email, including attachments, is private and confidential. If you
> have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
>
>
>
> 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
>
> Registered in England and Wales. Registered No. 04843573.

-- 


This email, including attachments, is private and confidential. If you have 
received this email in error please notify the sender and delete it from 
your system. Emails are not secure and may contain viruses. No liability 
can be accepted for viruses that might be transferred by this email or any 
attachment. Any unauthorised copying of this message or unauthorised 
distribution and publication of the information contained herein are 
prohibited.

7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.

Re: What is the fair setup of Kafka to be comparable with NATS or NSQ?

[kant kodali]

Hi Ben,
Also the article that you pointed out clearly shows the setup had multiple
partitions and multiple workers and so on..that's whyat this point my biggest
question is what is the fair setup for Kafka so its comparable with NATS and
NSQ? and since you suspect the client Library I can give that a go..but can you
please confirm that one partition on one broker should be able to handle 300K
messages of 1KB data size for each message?
Thanks,kant






On Thu, Sep 15, 2016 2:28 AM, kant kodali kanth...@gmail.com
wrote:
Hi Ben,
I can give that a try but can you tell me the suspicion or motivation behind it?
other words you think single partition and single broker should be comparable to
the setup I had with NATS and NSQ except you suspect the client library or
something?
Thanks,Kant






On Thu, Sep 15, 2016 2:16 AM, Ben Davison ben.davi...@7digital.com
wrote:
Hi Kant,




I was following the other thread, can you try using a different

benchmarking client for a test.




https://grey-boundary.io/load-testing-apache-kafka-on-aws/




Ben




On Thursday, 15 September 2016, kant kodali  wrote:





with Kafka I tried it with 10 messages with single broker and only one



partiton



that looked instantaneous and ~5K messages/sec for the data size of 1KB



I tried it with 1000 messages that looked instantaneous as well ~5K



messages/sec



for the data size of 1KBI tried it with 10K messages with single broker



and only



one partiton things started to go down ~1K messages/sec for the data size



of



1KB



having only one partition on a single broker is a bad? My goal is to run



some



basic benchmarks on NATS & NSQ & KAFKA



I have the same environment for all three (NATS & NSQ & KAFKA)



a broker on Machine 1producer on Machine 2Consumer on Machine 3



with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws



instance.



I have pushed 300K messages with NATS and it was able to handle easily and



receive throughput was 5K messages/secI have pushed 300K messages and NSQ



and



receive throughput was 2K messages/secI am unable to push 300K messages



with



Kafka with the above configuration and environmentso at this point my



biggest



question is what is the fair setup for Kafka so its comparable with NATS



and



NSQ?



kant





--







This email, including attachments, is private and confidential. If you have 

received this email in error please notify the sender and delete it from 

your system. Emails are not secure and may contain viruses. No liability 

can be accepted for viruses that might be transferred by this email or any 

attachment. Any unauthorised copying of this message or unauthorised 

distribution and publication of the information contained herein are 


prohibited.




7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.

Registered in England and Wales. Registered No. 04843573.

Re: no kafka controllers in zookeeper

[Francesco laTorre]

Hi,

*get* /controller

is the way to go, thought there should have been children.

Cheers,
Francesco

On 13 September 2016 at 15:29, Francesco laTorre <
francesco.lato...@openbet.com> wrote:

> Hi,
>
> Anyone else finding this weird ?
>
> Cheers,
> Francesco
>
> On 12 September 2016 at 15:53, Francesco laTorre <
> francesco.lato...@openbet.com> wrote:
>
>> Hi guys,
>>
>> Pretty new to kafka and having sporadic issues with replication.
>> Running Kafka *0.8.2.1 *:
>>
>> $> find ./libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*'
>> kafka_2.11-0.8.2.1-scaladoc.jar
>>
>> on a cluster of *3 brokers* :
>>
>> $> ./zookeeper-shell.sh localhost:19374
>> Connecting to localhost:19374
>>
>>
>> Welcome to ZooKeeper!
>>
>>
>> JLine support is disabled
>>
>>
>>
>>
>>
>> WATCHER::
>>
>>
>>
>>
>>
>> WatchedEvent state:SyncConnected type:None path:null
>>
>>
>> *ls /brokers/ids
>>
>>  *
>> *[1, 2, 3]  *
>>
>> is it normal that zookeeper reports no controllers :
>>
>> [...]
>>
>> *ls /controller *
>>
>>
>> *[] *
>>
>> Or am I missing anything in the configuration ?
>>
>> Cheers,
>> Francesco
>>
>> --
>>
>>
>>
>
>
> --
>  Francesco laTorre
> Senior Developer
> T: +44 208 742 1600
> +44 203 249 8394
>
> E: francesco.lato...@openbet.com
> W: www.openbet.com
> OpenBet Ltd
> Chiswick Park Building 9
> 566 Chiswick High Rd
> London
> W4 5XT
> 
> This message is confidential and intended only for the addressee. If you
> have received this message in error, please immediately notify the
> postmas...@openbet.com and delete it from your system as well as any
> copies. The content of e-mails as well as traffic data may be monitored by
> OpenBet for employment and security purposes. To protect the environment
> please do not print this e-mail unless necessary. OpenBet Ltd. Registered
> Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
> United Kingdom. A company registered in England and Wales. Registered no.
> 3134634. VAT no. GB927523612
>
>


-- 
 Francesco laTorre
Senior Developer
T: +44 208 742 1600
+44 203 249 8394

E: francesco.lato...@openbet.com
W: www.openbet.com
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT

This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmas...@openbet.com and delete it from your system as well as any
copies. The content of e-mails as well as traffic data may be monitored by
OpenBet for employment and security purposes. To protect the environment
please do not print this e-mail unless necessary. OpenBet Ltd. Registered
Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
United Kingdom. A company registered in England and Wales. Registered no.
3134634. VAT no. GB927523612

which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?

[kant kodali]

which port should I use 9091 or 9092 or 2181 to send messages through kafka
when using a client Library?
I start kafka as follows:
sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo
./bin/kafka-server-start.sh config/server.properties

and I dont see any process running on 9091 or 9092 however lot of client library
examples have a consumer client pointing to 9092. for example here
https://github.com/apache/kafka/blob/trunk/examples/src/main/java/kafka/examples/Producer.java#L34
shouldn't both producer and consumer point to zookeeper port 2181? which I am
assuming will do the lookup?
Thanks,Kant

Re: which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?

[Ali Akhtar]

Examine server.properties and see which port you're using in there

On Thu, Sep 15, 2016 at 3:52 PM, kant kodali  wrote:

> which port should I use 9091 or 9092 or 2181 to send messages through kafka
> when using a client Library?
> I start kafka as follows:
> sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo
> ./bin/kafka-server-start.sh config/server.properties
>
> and I dont see any process running on 9091 or 9092 however lot of client
> library
> examples have a consumer client pointing to 9092. for example here
> https://github.com/apache/kafka/blob/trunk/examples/src/main
> /java/kafka/examples/Producer.java#L34
> shouldn't both producer and consumer point to zookeeper port 2181? which I
> am
> assuming will do the lookup?
> Thanks,Kant

Re: which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?

[kant kodali]

I haven't changed anything from
https://github.com/apache/kafka/blob/trunk/config/server.properties
and it looks like it is pointing to zookeeper.
Question:
Does producer client need to point 9092 and Consumer need to point 2181? is that
the standard? Why not both point to the same thing?






On Thu, Sep 15, 2016 4:24 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Examine server.properties and see which port you're using in there




On Thu, Sep 15, 2016 at 3:52 PM, kant kodali  wrote:





which port should I use 9091 or 9092 or 2181 to send messages through kafka



when using a client Library?



I start kafka as follows:



sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo



./bin/kafka-server-start.sh config/server.properties







and I dont see any process running on 9091 or 9092 however lot of client



library



examples have a consumer client pointing to 9092. for example here



https://github.com/apache/kafka/blob/trunk/examples/src/main



/java/kafka/examples/Producer.java#L34



shouldn't both producer and consumer point to zookeeper port 2181? which I



am



assuming will do the lookup?



Thanks,Kant

Re: which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?

[UMESH CHAUDHARY]

No that is not required, when you use new consumer API. You have to
specify bootstrap.servers,
which will have 9092 (for PLAINTEXT usually ).
In old consumer API you need zookeeper server which points on 2181.

On Thu, 15 Sep 2016 at 17:03 kant kodali  wrote:

> I haven't changed anything from
> https://github.com/apache/kafka/blob/trunk/config/server.properties
> and it looks like it is pointing to zookeeper.
> Question:
> Does producer client need to point 9092 and Consumer need to point 2181?
> is that
> the standard? Why not both point to the same thing?
>
>
>
>
>
>
> On Thu, Sep 15, 2016 4:24 AM, Ali Akhtar ali.rac...@gmail.com
> wrote:
> Examine server.properties and see which port you're using in there
>
>
>
>
> On Thu, Sep 15, 2016 at 3:52 PM, kant kodali  wrote:
>
>
>
>
> > which port should I use 9091 or 9092 or 2181 to send messages through
> kafka
>
> > when using a client Library?
>
> > I start kafka as follows:
>
> > sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo
>
> > ./bin/kafka-server-start.sh config/server.properties
>
> >
>
> > and I dont see any process running on 9091 or 9092 however lot of client
>
> > library
>
> > examples have a consumer client pointing to 9092. for example here
>
> > https://github.com/apache/kafka/blob/trunk/examples/src/main
>
> > /java/kafka/examples/Producer.java#L34
>
> > shouldn't both producer and consumer point to zookeeper port 2181? which
> I
>
> > am
>
> > assuming will do the lookup?
>
> > Thanks,Kant

Re: which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?

[kant kodali]

@Umesh According to these examples it looks like producer and consumer
specifies bootstrap.servers. What is PLAINTEXT? do I need to change something
here https://github.com/apache/kafka/blob/trunk/config/server.properties ?
because when I specify port 9092 for both producer consumer or just either of
them it doesn't seem to work. Only when I specify zookeeper port it seems to
work and I don't know why?

https://github.com/apache/kafka/blob/trunk/examples/src/main/java/kafka/examples/Producer.java#L34
https://github.com/apache/kafka/blob/trunk/examples/src/main/java/kafka/examples/Consumer.java

 





On Thu, Sep 15, 2016 8:15 AM, UMESH CHAUDHARY umesh9...@gmail.com
wrote:
No that is not required, when you use new consumer API. You have to

specify bootstrap.servers,

which will have 9092 (for PLAINTEXT usually ).

In old consumer API you need zookeeper server which points on 2181.




On Thu, 15 Sep 2016 at 17:03 kant kodali  wrote:




> I haven't changed anything from

> https://github.com/apache/kafka/blob/trunk/config/server.properties

> and it looks like it is pointing to zookeeper.

> Question:

> Does producer client need to point 9092 and Consumer need to point 2181?

> is that

> the standard? Why not both point to the same thing?

>

>

>

>

>

>

> On Thu, Sep 15, 2016 4:24 AM, Ali Akhtar ali.rac...@gmail.com

> wrote:

> Examine server.properties and see which port you're using in there

>

>

>

>

> On Thu, Sep 15, 2016 at 3:52 PM, kant kodali  wrote:

>

>

>

>

> > which port should I use 9091 or 9092 or 2181 to send messages through

> kafka

>

> > when using a client Library?

>

> > I start kafka as follows:

>

> > sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo

>

> > ./bin/kafka-server-start.sh config/server.properties

>

> >

>

> > and I dont see any process running on 9091 or 9092 however lot of client

>

> > library

>

> > examples have a consumer client pointing to 9092. for example here

>

> > https://github.com/apache/kafka/blob/trunk/examples/src/main

>

> > /java/kafka/examples/Producer.java#L34

>

> > shouldn't both producer and consumer point to zookeeper port 2181? which

> I

>

> > am

>

> > assuming will do the lookup?

>

> > Thanks,Kant

Re: Publish to 1 topic, consume from N

[Marko Bonaći]

1. You can create N topics
2. You control from producer where each message goes
3. You have consumer that fetches from M different topics:
https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe(java.util.Collection)

Isn't this architecture flexible enough for any type of use case? What do
you think cannot be achieved?

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Thu, Sep 15, 2016 at 11:01 AM, Ali Akhtar  wrote:

> It sounds like you can implement the 'mapping service'  component yourself
> using Kafka.
>
> Have all of your messages go to one kafka topic. Have one consumer group
> listening to this 'everything goes here' topic. This consumer group acts as
> your mapping service. It looks at each message, and based on your rules, it
> sends that message to a different topic for those specific rules.
>
> Then you have your consumers listening to the specific topics that they
> need to. Your mapping service does the job of redirecting messages from the
> 'everything' topic to the specific topics based on your rules.
>
> On Thu, Sep 15, 2016 at 1:43 PM, Luiz Cordeiro <
> luiz.corde...@mobilityhouse.com> wrote:
>
> > Hello,
> >
> > We’re considering migrating an AMQ-based platform to Kafka. However our
> > application logic needs an AMQ feature called Dynamic Binding, that is,
> on
> > AMQ one publishes messages to an Exchange, which can be dynamically
> > configured to deliver a copy of the message to several queues, based on
> > binding rules. So when a new client comes alive, it may create its
> binding
> > rules to specify a set of topics to listen to, and receive all the
> messages
> > from these topics on a private queue.
> >
> > I understand that Kafka neither provides this nor will, as it is not its
> > objective, but I was wondering if there’s another component, an overlay
> to
> > Kafka, that could provide this feature while using Kafka behind the
> scenes
> > for the persistence, something like this:
> >
> > Publisher --> Mapping Service --> Kafka <-- Consumers
> >  ^  |
> >  |   Binding rules  |
> >  \--/
> >
> > Are you aware of such a component? Otherwise, how would you solve this
> > issue of publish to 1 place and have it replicated on N topics.
> >
> > Best Regards,
> > Luiz
> >
>

compacted log key limits

[Wesley Chow]

Is there any guidance on a maximum number different keys in a compacted
log? Such total numbers, or "keys need to fit in memory, message data does
not", etc. Is it unreasonable to expect tens or hundreds of millions of
keys in a single topic to be handled gracefully?

Thanks,
Wes

Partition creation issues (0.9.0.1)

[Apurva Sharma]

Kafka Topic Creation issues (via Kafka-Manager with
auto.create.topics.enable = false)
Version: 0.9.0.1

We created a topic "web" via Kafka-Manager (our brokers are configured for
autocreate to be false) and then clicked on Generate Partitions and
according to the tool, the topic has been created cleanly with partitions
assigned correctly to brokers.
However, when we look into the individual broker logs:

[2016-09-15 18:46:10,268] ERROR [ReplicaFetcherThread-5-1006], Error for
partition [web,2] to broker
1006:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2016-09-15 18:46:10,391] ERROR [ReplicaFetcherThread-4-1005], Error for
partition [web,1] to broker
1005:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2016-09-15 18:46:10,391] WARN [Replica Manager on Broker 1001]: While
recording the replica LEO, the partition [web,5] hasn't been created.
(kafka.server.ReplicaManager)
[2016-09-15 18:46:10,407] ERROR [ReplicaFetcherThread-4-1004], Error for
partition [web,16] to broker
1004:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2016-09-15 18:46:10,472] ERROR [ReplicaFetcherThread-11-1003], Error for
partition [web,23] to broker
1003:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2016-09-15 18:46:10,520] WARN [Replica Manager on Broker 1001]: While
recording the replica LEO, the partition [web,29] hasn't been created.
(kafka.server.ReplicaManager)
[2016-09-15 18:46:10,667] ERROR [ReplicaFetcherThread-11-1004], Error for
partition [web,8] to broker
1004:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
server does not host this topic-partition.
(kafka.server.ReplicaFetcherThread)
[2016-09-15 18:46:10,895] WARN [Replica Manager on Broker 1001]: While
recording the replica LEO, the partition [web,21] hasn't been created.
(kafka.server.ReplicaManager)
[2016-09-15 18:46:10,931] WARN [Replica Manager on Broker 1001]: While
recording the replica LEO, the partition [web,13] hasn't been created.
(kafka.server.ReplicaManager)

Indeed, when we actually inspect if the partitions were actually created,
we see disparity with what's reported by the tool. (Many partitions are
actually not present on the brokers yet - leading me to believe that the
partitioning process is stuck)


| ZK Id | Broker Id  |According to Kafka Manager|
 Actual Partitions Created |
|-||||
|  1001 | broker-1001... | 1,2,5,8,9,13,16,21,23,29,30,31   |
web-1,web-16,web-2,web-23,web-8,web-9|
|  1002 | broker-1004... | 2,3,6,9,10,14,16,17,22,24,30,31  |
web-10,web-16,web-3,web-9|
|  1003 | broker-1002... | 3,4,7,10,11,15,17,18,23,24,25,31 |
web-10,web-11,web-3,web-4|
|  1004 | broker-1003... | 0,4,5,8,11,12,16,18,19,24,25,26  |
web-12,web-18,web-19,web-25,web-26   |
|  1005 | broker-1005... | 1,5,6,9,12,13,17,19,20,25,26,27  |
web-12,web-13,web-19,web-26,web-27,web-5 |
|  1006 | broker-1006... | 2,6,7,10,13,14,18,20,21,26,27,28 |
web-27   |
|  1007 | broker-1007... | 0,3,7,11,14,15,19,21,22,27,28,29 |
web-21,web-22,web-7  |
|  1008 | broker-1008... | 0,1,4,8,12,15,20,22,23,28,29,30  |
web-1,web-15,web-22,web-29,web-8 |

Finally, when we try to repartition the topic we get:
"Yikes! Partition reassignment currently in progress for. Aborting
operation" (This seems expected because the initial partitioning has not
completed for 3 days - so clearly it's stuck somewhere).

We have created topics with the Kafka-Manager tool before without issues,
but this has started happening consistently for the past 2-3 topics that we
tried creating through the tool.
Any help on this is greatly appreciated.

-- 
Regards,
Apurva

Schema for jsonConverter

[Srikrishna Alla]

I am trying to use jdbc connector to send records from Kafka 0.9 to DB. I
am using jsonConverter to convert the records. My connector is failing when
its checking the Schema I am using. Please let me know what is the issue
with my json schema.

Configuration used:
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
# Converter-specific settings can be passed in by prefixing the Converter's
setting with the converter we want to apply
# it to
key.converter.schemas.enable=true
value.converter.schemas.enable=true

Record that has been sent to the topic -
{"schema":{"type":"struct","fields":[{"name":"error_code","type":"string","optional":"false"},{"name":"error_time","type":"string","optional":"false"},{"name":"error_msg","type":"string","optional":"false"},{"name":"source","type":"string","optional":"false"},{"name":"criticality","type":"string","optional":"false"}]},"payload":{"error_code":"RAW104","error_time":"09/15/2016@18:00:32","error_msg":"Not
accepting","source":"APPLICATION","criticality":"WARN"}}


Error I am seeing:
[2016-09-15 18:01:07,513] ERROR Thread WorkerSinkTask-jdbc-sink-test-0
exiting with uncaught exception:
(org.apache.kafka.connect.util.ShutdownableThread:84)
*org.apache.kafka.connect.errors.DataException: Struct schema's field name
not specified properly*
   at
org.apache.kafka.connect.json.JsonConverter.asConnectSchema(JsonConverter.java:493)
   at
org.apache.kafka.connect.json.JsonConverter.jsonToConnect(JsonConverter.java:344)
   at
org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
   at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
   at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
   at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
   at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
   at
org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
Exception in thread "WorkerSinkTask-jdbc-sink-test-0"
*org.apache.kafka.connect.errors.DataException:
Struct schema's field name not specified properly*
   at
org.apache.kafka.connect.json.JsonConverter.asConnectSchema(JsonConverter.java:493)
   at
org.apache.kafka.connect.json.JsonConverter.jsonToConnect(JsonConverter.java:344)
   at
org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
   at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
   at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
   at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
   at
org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
   at
org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)

Thanks,
Sri

Re: Schema for jsonConverter

[Gwen Shapira]

Most people use JSON without schema, so you should probably change
your configuration to:

key.converter.schemas.enable=false
value.converter.schemas.enable=false

On Thu, Sep 15, 2016 at 4:04 PM, Srikrishna Alla
 wrote:
> I am trying to use jdbc connector to send records from Kafka 0.9 to DB. I
> am using jsonConverter to convert the records. My connector is failing when
> its checking the Schema I am using. Please let me know what is the issue
> with my json schema.
>
> Configuration used:
> key.converter=org.apache.kafka.connect.storage.StringConverter
> value.converter=org.apache.kafka.connect.json.JsonConverter
> # Converter-specific settings can be passed in by prefixing the Converter's
> setting with the converter we want to apply
> # it to
> key.converter.schemas.enable=true
> value.converter.schemas.enable=true
>
> Record that has been sent to the topic -
> {"schema":{"type":"struct","fields":[{"name":"error_code","type":"string","optional":"false"},{"name":"error_time","type":"string","optional":"false"},{"name":"error_msg","type":"string","optional":"false"},{"name":"source","type":"string","optional":"false"},{"name":"criticality","type":"string","optional":"false"}]},"payload":{"error_code":"RAW104","error_time":"09/15/2016@18:00:32","error_msg":"Not
> accepting","source":"APPLICATION","criticality":"WARN"}}
>
>
> Error I am seeing:
> [2016-09-15 18:01:07,513] ERROR Thread WorkerSinkTask-jdbc-sink-test-0
> exiting with uncaught exception:
> (org.apache.kafka.connect.util.ShutdownableThread:84)
> *org.apache.kafka.connect.errors.DataException: Struct schema's field name
> not specified properly*
>at
> org.apache.kafka.connect.json.JsonConverter.asConnectSchema(JsonConverter.java:493)
>at
> org.apache.kafka.connect.json.JsonConverter.jsonToConnect(JsonConverter.java:344)
>at
> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
>at
> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
> Exception in thread "WorkerSinkTask-jdbc-sink-test-0"
> *org.apache.kafka.connect.errors.DataException:
> Struct schema's field name not specified properly*
>at
> org.apache.kafka.connect.json.JsonConverter.asConnectSchema(JsonConverter.java:493)
>at
> org.apache.kafka.connect.json.JsonConverter.jsonToConnect(JsonConverter.java:344)
>at
> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
>at
> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
>at
> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
>
> Thanks,
> Sri



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Schema for jsonConverter

[Gwen Shapira]

ah, never mind - I just noticed you do use a schema... Maybe you are
running into this? https://issues.apache.org/jira/browse/KAFKA-3055

On Thu, Sep 15, 2016 at 4:20 PM, Gwen Shapira  wrote:
> Most people use JSON without schema, so you should probably change
> your configuration to:
>
> key.converter.schemas.enable=false
> value.converter.schemas.enable=false
>
> On Thu, Sep 15, 2016 at 4:04 PM, Srikrishna Alla
>  wrote:
>> I am trying to use jdbc connector to send records from Kafka 0.9 to DB. I
>> am using jsonConverter to convert the records. My connector is failing when
>> its checking the Schema I am using. Please let me know what is the issue
>> with my json schema.
>>
>> Configuration used:
>> key.converter=org.apache.kafka.connect.storage.StringConverter
>> value.converter=org.apache.kafka.connect.json.JsonConverter
>> # Converter-specific settings can be passed in by prefixing the Converter's
>> setting with the converter we want to apply
>> # it to
>> key.converter.schemas.enable=true
>> value.converter.schemas.enable=true
>>
>> Record that has been sent to the topic -
>> {"schema":{"type":"struct","fields":[{"name":"error_code","type":"string","optional":"false"},{"name":"error_time","type":"string","optional":"false"},{"name":"error_msg","type":"string","optional":"false"},{"name":"source","type":"string","optional":"false"},{"name":"criticality","type":"string","optional":"false"}]},"payload":{"error_code":"RAW104","error_time":"09/15/2016@18:00:32","error_msg":"Not
>> accepting","source":"APPLICATION","criticality":"WARN"}}
>>
>>
>> Error I am seeing:
>> [2016-09-15 18:01:07,513] ERROR Thread WorkerSinkTask-jdbc-sink-test-0
>> exiting with uncaught exception:
>> (org.apache.kafka.connect.util.ShutdownableThread:84)
>> *org.apache.kafka.connect.errors.DataException: Struct schema's field name
>> not specified properly*
>>at
>> org.apache.kafka.connect.json.JsonConverter.asConnectSchema(JsonConverter.java:493)
>>at
>> org.apache.kafka.connect.json.JsonConverter.jsonToConnect(JsonConverter.java:344)
>>at
>> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
>>at
>> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
>> Exception in thread "WorkerSinkTask-jdbc-sink-test-0"
>> *org.apache.kafka.connect.errors.DataException:
>> Struct schema's field name not specified properly*
>>at
>> org.apache.kafka.connect.json.JsonConverter.asConnectSchema(JsonConverter.java:493)
>>at
>> org.apache.kafka.connect.json.JsonConverter.jsonToConnect(JsonConverter.java:344)
>>at
>> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:334)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:266)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:175)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.iteration(WorkerSinkTaskThread.java:90)
>>at
>> org.apache.kafka.connect.runtime.WorkerSinkTaskThread.execute(WorkerSinkTaskThread.java:58)
>>at
>> org.apache.kafka.connect.util.ShutdownableThread.run(ShutdownableThread.java:82)
>>
>> Thanks,
>> Sri
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Old replica data does not clear following the retention.ms property

[Anish Mashankar]

I recently ran partition assignment on some topics. This made the replicas
of some partitions move around the cluster. It was seamless. However, when
it came to purging old logs following the retention.ms property of the
topic, the replica partitions were not clear. The leader partition,
however, was purged correctly.
As our consumers have caught up, a quick fix was to reduce retention period
to 1 day, let Kafka purge those replicas and reset the retention policy.

This looks like a bug. We had almost run out of disk space on few of our
brokers due to this.
-- 
Anish Samir Mashankar