Re: Kafka Node.js Integration Questions/Advice

[Reem Diab]

We recently released kafka-java-bridge - new nodejs high level consumer.

npm ,github


Its tested stable and

-- 
Reem Diab
Software Engineer
T: +972-74-700-4580
 
 We Create Meaningful Connections


-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Kafka Node.js Integration Questions/Advice

[Reem Diab]

We recently released kafka-java-bridge - new nodejs high level consumer.

npm ,github


Its tested stable and
-- 
Reem Diab
Software Engineer
T: +972-74-700-4580
 
 We Create Meaningful Connections


-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Kafka Node.js Integration Questions/Advice

[Christopher Alexander]

Update: Early (1 week) implementation of Node-Kafka has resulted in the 
following observations:

1. Consumer is unstable.
2. If use of Consumer is mandatory, create the Consumer in application-scope, 
not request-scope.
3. Attempt to close Consumer on application shutdown. Results of unplanned 
shutdowns have not been fully evaluated.
4. Producer is stable and works as expected.
5. If producing a message that also necessitates a Kafka response, implement a 
setTimeout interval to conform with application SLAs.

We will revisit swapping out Node-Kafka with Franz-Kafka once our middleware 
(Java) buildout is complete. Then we will compare the 2 Kafka Node clients 
under load and post our results here or in a blog.

Test, test, test. That's the take away here.

- Original Message -
From: "Apoorva Gaurav" 
To: users@kafka.apache.org
Sent: Saturday, December 22, 2012 1:54:00 PM
Subject: Re: Kafka Node.js Integration Questions/Advice

Thanks Radek, thanks David.

On Sun, Dec 23, 2012 at 12:08 AM, Radek Gruchalski <
radek.gruchal...@portico.io> wrote:

> That is exactly why we've decided to stick with java. Also support for all
> consumer settings out of the box.
>
> Kind regards,
> Radek Gruchalski
>
> On 22 Dec 2012, at 19:17, David Arthur  wrote:
>
> > FWIW, message production is quite simpler than consumption. It does
> > not require the same complex coordination as the consumers. Producers
> > only use ZooKeeper to locate available brokers
> >
> > Sent from my phone
> >
> > On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav 
> wrote:
> >
> >> Thanks Radek,
> >> We also are thinking of Java / Scala for Consumers, for Producers
> whether
> >> franz-kafka is a good choice?
> >>
> >> --
> >> Thanks & Regards,
> >> Apoorva
> >>
> >> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> >> radek.gruchal...@portico.io> wrote:
> >>
> >>> We started using node-kafka before we learned franz-kafka was
> available.
> >>> In node, franz-kafka would be my preferred choice now. But tbh, our
> >>> consumers are all java. node-kafka does not support consumer settings
> like
> >>> autooffset.reset and so on (or it is not obvious how to use those).
> >>>
> >>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
> >>> monitoring tools, which may be helpful if you want to add some scaling
> >>> logic when consumer is lagging.
> >>>
> >>> Our first choice is node too but we're consuming exclusively with java.
> >>>
> >>> Hope this helps a little.
> >>>
> >>> On 22 Dec 2012, at 05:21, Apoorva Gaurav 
> wrote:
> >>>
> >>>> Which is the best ZK based implementation of kafka in node.js. Our use
> >>> case
> >>>> is that a pool of node js http servers will be listening to clients
> which
> >>>> will send json over http. Using node js we'll do minimal decoration
> and
> >>>> compression (preferably snappy) and write to brokers. We might also
> need
> >>>> json to avro conversion but thats not a deal breaker. Consumers will
> be
> >>>> writing these events to S3 (to begin with we don't plan to maintain
> HDFS
> >>>> cluster). To begin with we'll have to support a peak load of 50K
> events /
> >>>> second, average being much less, around 2K events / second.
> Suggestions
> >>>> please. Is any one using franz-kafka in production. I'm only two days
> >>> into
> >>>> kafka so don't know a lot, but franz-kafka looks clean and easy to
> work
> >>>> with.
> >>>>
> >>>> If none of the existing node.js implementation is capable of this
> then we
> >>>> are willing to move to Scala or Java but node.js is the first choice.
> >>>>
> >>>> Thanks & Regards,
> >>>> Apoorva
> >>>>
> >>>> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
> >>>> radek.gruchal...@portico.io> wrote:
> >>>>
> >>>>> We are using https://github.com/radekg/node-kafka, occasionally
> pushing
> >>>>> about 2500 messages, 3.5K each / second. No issues so far. Different
> >>> story
> >>>>> with consumers. They are stable but under heavy load we experienced
> CPU
> >>>>> problems. I am the maintainer of that fork. The fork comes with ZK
> >>>>> integration.

Re: Kafka Node.js Integration Questions/Advice

[Apoorva Gaurav]

Thanks Radek, thanks David.

On Sun, Dec 23, 2012 at 12:08 AM, Radek Gruchalski <
radek.gruchal...@portico.io> wrote:

> That is exactly why we've decided to stick with java. Also support for all
> consumer settings out of the box.
>
> Kind regards,
> Radek Gruchalski
>
> On 22 Dec 2012, at 19:17, David Arthur  wrote:
>
> > FWIW, message production is quite simpler than consumption. It does
> > not require the same complex coordination as the consumers. Producers
> > only use ZooKeeper to locate available brokers
> >
> > Sent from my phone
> >
> > On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav 
> wrote:
> >
> >> Thanks Radek,
> >> We also are thinking of Java / Scala for Consumers, for Producers
> whether
> >> franz-kafka is a good choice?
> >>
> >> --
> >> Thanks & Regards,
> >> Apoorva
> >>
> >> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> >> radek.gruchal...@portico.io> wrote:
> >>
> >>> We started using node-kafka before we learned franz-kafka was
> available.
> >>> In node, franz-kafka would be my preferred choice now. But tbh, our
> >>> consumers are all java. node-kafka does not support consumer settings
> like
> >>> autooffset.reset and so on (or it is not obvious how to use those).
> >>>
> >>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
> >>> monitoring tools, which may be helpful if you want to add some scaling
> >>> logic when consumer is lagging.
> >>>
> >>> Our first choice is node too but we're consuming exclusively with java.
> >>>
> >>> Hope this helps a little.
> >>>
> >>> On 22 Dec 2012, at 05:21, Apoorva Gaurav 
> wrote:
> >>>
>  Which is the best ZK based implementation of kafka in node.js. Our use
> >>> case
>  is that a pool of node js http servers will be listening to clients
> which
>  will send json over http. Using node js we'll do minimal decoration
> and
>  compression (preferably snappy) and write to brokers. We might also
> need
>  json to avro conversion but thats not a deal breaker. Consumers will
> be
>  writing these events to S3 (to begin with we don't plan to maintain
> HDFS
>  cluster). To begin with we'll have to support a peak load of 50K
> events /
>  second, average being much less, around 2K events / second.
> Suggestions
>  please. Is any one using franz-kafka in production. I'm only two days
> >>> into
>  kafka so don't know a lot, but franz-kafka looks clean and easy to
> work
>  with.
> 
>  If none of the existing node.js implementation is capable of this
> then we
>  are willing to move to Scala or Java but node.js is the first choice.
> 
>  Thanks & Regards,
>  Apoorva
> 
>  On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
>  radek.gruchal...@portico.io> wrote:
> 
> > We are using https://github.com/radekg/node-kafka, occasionally
> pushing
> > about 2500 messages, 3.5K each / second. No issues so far. Different
> >>> story
> > with consumers. They are stable but under heavy load we experienced
> CPU
> > problems. I am the maintainer of that fork. The fork comes with ZK
> > integration. Another kafka module is this one:
> > https://github.com/dannycoates/franz-kafka.
> >
> > Kind regards,
> > Radek Gruchalski
> > radek.gruchal...@technicolor.com (mailto:
> >>> radek.gruchal...@technicolor.com)
> > | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
> > ra...@gruchalski.com (mailto:ra...@gruchalski.com)
> > 00447889948663
> >
> > Confidentiality:
> > This communication is intended for the above-named person and may be
> > confidential and/or legally privileged.
> > If it has come to you in error you must take no action based on it,
> nor
> > must you copy or show it to anyone; please delete/destroy and inform
> the
> > sender immediately.
> >
> >
> > On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
> >
> >> Chris,
> >>
> >> Not sure how stable those node.js clients are. In 0.8, we plan to
> > provide a
> >> native C version of the producer. A thin node.js layer can
> potentially
> >>> be
> >> built on top of that.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> >> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
> >>
> >>> During my due diligence to assess use of Kafka for both our
> activity
> > and
> >>> log message streams, I would like to ask the project committers and
> >>> community users about using Kafka with Node.js. Yes, I am aware
> that a
> >>> Kafka client exists for Node.js (
> >>> https://github.com/marcuswestin/node-kafka), which has spurred
> >>> further
> >>> interest by our front-end team. Here are my questions, excuse me if
> > they
> >>> seem "noobish".
> >>>
> >>> 1. How reliable is the Node.js client (
> >>> https://github.com/mar

Re: Kafka Node.js Integration Questions/Advice

[Radek Gruchalski]

That is exactly why we've decided to stick with java. Also support for all 
consumer settings out of the box.

Kind regards,
Radek Gruchalski

On 22 Dec 2012, at 19:17, David Arthur  wrote:

> FWIW, message production is quite simpler than consumption. It does
> not require the same complex coordination as the consumers. Producers
> only use ZooKeeper to locate available brokers
> 
> Sent from my phone
> 
> On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav  wrote:
> 
>> Thanks Radek,
>> We also are thinking of Java / Scala for Consumers, for Producers whether
>> franz-kafka is a good choice?
>> 
>> --
>> Thanks & Regards,
>> Apoorva
>> 
>> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
>> radek.gruchal...@portico.io> wrote:
>> 
>>> We started using node-kafka before we learned franz-kafka was available.
>>> In node, franz-kafka would be my preferred choice now. But tbh, our
>>> consumers are all java. node-kafka does not support consumer settings like
>>> autooffset.reset and so on (or it is not obvious how to use those).
>>> 
>>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
>>> monitoring tools, which may be helpful if you want to add some scaling
>>> logic when consumer is lagging.
>>> 
>>> Our first choice is node too but we're consuming exclusively with java.
>>> 
>>> Hope this helps a little.
>>> 
>>> On 22 Dec 2012, at 05:21, Apoorva Gaurav  wrote:
>>> 
 Which is the best ZK based implementation of kafka in node.js. Our use
>>> case
 is that a pool of node js http servers will be listening to clients which
 will send json over http. Using node js we'll do minimal decoration and
 compression (preferably snappy) and write to brokers. We might also need
 json to avro conversion but thats not a deal breaker. Consumers will be
 writing these events to S3 (to begin with we don't plan to maintain HDFS
 cluster). To begin with we'll have to support a peak load of 50K events /
 second, average being much less, around 2K events / second. Suggestions
 please. Is any one using franz-kafka in production. I'm only two days
>>> into
 kafka so don't know a lot, but franz-kafka looks clean and easy to work
 with.
 
 If none of the existing node.js implementation is capable of this then we
 are willing to move to Scala or Java but node.js is the first choice.
 
 Thanks & Regards,
 Apoorva
 
 On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
 radek.gruchal...@portico.io> wrote:
 
> We are using https://github.com/radekg/node-kafka, occasionally pushing
> about 2500 messages, 3.5K each / second. No issues so far. Different
>>> story
> with consumers. They are stable but under heavy load we experienced CPU
> problems. I am the maintainer of that fork. The fork comes with ZK
> integration. Another kafka module is this one:
> https://github.com/dannycoates/franz-kafka.
> 
> Kind regards,
> Radek Gruchalski
> radek.gruchal...@technicolor.com (mailto:
>>> radek.gruchal...@technicolor.com)
> | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
> ra...@gruchalski.com (mailto:ra...@gruchalski.com)
> 00447889948663
> 
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
> 
> 
> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
> 
>> Chris,
>> 
>> Not sure how stable those node.js clients are. In 0.8, we plan to
> provide a
>> native C version of the producer. A thin node.js layer can potentially
>>> be
>> built on top of that.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
>> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
>> 
>>> During my due diligence to assess use of Kafka for both our activity
> and
>>> log message streams, I would like to ask the project committers and
>>> community users about using Kafka with Node.js. Yes, I am aware that a
>>> Kafka client exists for Node.js (
>>> https://github.com/marcuswestin/node-kafka), which has spurred
>>> further
>>> interest by our front-end team. Here are my questions, excuse me if
> they
>>> seem "noobish".
>>> 
>>> 1. How reliable is the Node.js client (
>>> https://github.com/marcuswestin/node-kafka) in production
> applications?
>>> If there are issues, what are they (the GitHub repo currently lists
> none)?
>>> 2. To support real-time activity streams within Node.js, what is the
>>> recommended consumer polling interval?
>>> 3. General advise observations on integrating a front-end based
>>> Node.js
>>> applicat

Re: Kafka Node.js Integration Questions/Advice

[Radek Gruchalski]

We use that fork of node-kafka without any issues. We have a 3 server cluster 
setup. Single topic, 3 partitions. No issues. Franz-kafka is on our "to check" 
list but no rush yet.

Kind regards,
Radek Gruchalski

On 22 Dec 2012, at 18:59, Apoorva Gaurav  wrote:

> Thanks Radek,
> We also are thinking of Java / Scala for Consumers, for Producers whether
> franz-kafka is a good choice?
> 
> -- 
> Thanks & Regards,
> Apoorva
> 
> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> radek.gruchal...@portico.io> wrote:
> 
>> We started using node-kafka before we learned franz-kafka was available.
>> In node, franz-kafka would be my preferred choice now. But tbh, our
>> consumers are all java. node-kafka does not support consumer settings like
>> autooffset.reset and so on (or it is not obvious how to use those).
>> 
>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
>> monitoring tools, which may be helpful if you want to add some scaling
>> logic when consumer is lagging.
>> 
>> Our first choice is node too but we're consuming exclusively with java.
>> 
>> Hope this helps a little.
>> 
>> On 22 Dec 2012, at 05:21, Apoorva Gaurav  wrote:
>> 
>>> Which is the best ZK based implementation of kafka in node.js. Our use
>> case
>>> is that a pool of node js http servers will be listening to clients which
>>> will send json over http. Using node js we'll do minimal decoration and
>>> compression (preferably snappy) and write to brokers. We might also need
>>> json to avro conversion but thats not a deal breaker. Consumers will be
>>> writing these events to S3 (to begin with we don't plan to maintain HDFS
>>> cluster). To begin with we'll have to support a peak load of 50K events /
>>> second, average being much less, around 2K events / second. Suggestions
>>> please. Is any one using franz-kafka in production. I'm only two days
>> into
>>> kafka so don't know a lot, but franz-kafka looks clean and easy to work
>>> with.
>>> 
>>> If none of the existing node.js implementation is capable of this then we
>>> are willing to move to Scala or Java but node.js is the first choice.
>>> 
>>> Thanks & Regards,
>>> Apoorva
>>> 
>>> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
>>> radek.gruchal...@portico.io> wrote:
>>> 
 We are using https://github.com/radekg/node-kafka, occasionally pushing
 about 2500 messages, 3.5K each / second. No issues so far. Different
>> story
 with consumers. They are stable but under heavy load we experienced CPU
 problems. I am the maintainer of that fork. The fork comes with ZK
 integration. Another kafka module is this one:
 https://github.com/dannycoates/franz-kafka.
 
 Kind regards,
 Radek Gruchalski
 radek.gruchal...@technicolor.com (mailto:
>> radek.gruchal...@technicolor.com)
 | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
 ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 00447889948663
 
 Confidentiality:
 This communication is intended for the above-named person and may be
 confidential and/or legally privileged.
 If it has come to you in error you must take no action based on it, nor
 must you copy or show it to anyone; please delete/destroy and inform the
 sender immediately.
 
 
 On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
 
> Chris,
> 
> Not sure how stable those node.js clients are. In 0.8, we plan to
 provide a
> native C version of the producer. A thin node.js layer can potentially
>> be
> built on top of that.
> 
> Thanks,
> 
> Jun
> 
> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
> 
>> During my due diligence to assess use of Kafka for both our activity
 and
>> log message streams, I would like to ask the project committers and
>> community users about using Kafka with Node.js. Yes, I am aware that a
>> Kafka client exists for Node.js (
>> https://github.com/marcuswestin/node-kafka), which has spurred
>> further
>> interest by our front-end team. Here are my questions, excuse me if
 they
>> seem "noobish".
>> 
>> 1. How reliable is the Node.js client (
>> https://github.com/marcuswestin/node-kafka) in production
 applications?
>> If there are issues, what are they (the GitHub repo currently lists
 none)?
>> 2. To support real-time activity streams within Node.js, what is the
>> recommended consumer polling interval?
>> 3. General advise observations on integrating a front-end based
>> Node.js
>> application with Kafka mediated messaging.
>> 
>> Thanks you!
>> 
>> Chris
>> 

Re: Kafka Node.js Integration Questions/Advice

[David Arthur]

FWIW, message production is quite simpler than consumption. It does
not require the same complex coordination as the consumers. Producers
only use ZooKeeper to locate available brokers

Sent from my phone

On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav  wrote:

> Thanks Radek,
> We also are thinking of Java / Scala for Consumers, for Producers whether
> franz-kafka is a good choice?
>
> --
> Thanks & Regards,
> Apoorva
>
> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> radek.gruchal...@portico.io> wrote:
>
>> We started using node-kafka before we learned franz-kafka was available.
>> In node, franz-kafka would be my preferred choice now. But tbh, our
>> consumers are all java. node-kafka does not support consumer settings like
>> autooffset.reset and so on (or it is not obvious how to use those).
>>
>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
>> monitoring tools, which may be helpful if you want to add some scaling
>> logic when consumer is lagging.
>>
>> Our first choice is node too but we're consuming exclusively with java.
>>
>> Hope this helps a little.
>>
>> On 22 Dec 2012, at 05:21, Apoorva Gaurav  wrote:
>>
>>> Which is the best ZK based implementation of kafka in node.js. Our use
>> case
>>> is that a pool of node js http servers will be listening to clients which
>>> will send json over http. Using node js we'll do minimal decoration and
>>> compression (preferably snappy) and write to brokers. We might also need
>>> json to avro conversion but thats not a deal breaker. Consumers will be
>>> writing these events to S3 (to begin with we don't plan to maintain HDFS
>>> cluster). To begin with we'll have to support a peak load of 50K events /
>>> second, average being much less, around 2K events / second. Suggestions
>>> please. Is any one using franz-kafka in production. I'm only two days
>> into
>>> kafka so don't know a lot, but franz-kafka looks clean and easy to work
>>> with.
>>>
>>> If none of the existing node.js implementation is capable of this then we
>>> are willing to move to Scala or Java but node.js is the first choice.
>>>
>>> Thanks & Regards,
>>> Apoorva
>>>
>>> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
>>> radek.gruchal...@portico.io> wrote:
>>>
 We are using https://github.com/radekg/node-kafka, occasionally pushing
 about 2500 messages, 3.5K each / second. No issues so far. Different
>> story
 with consumers. They are stable but under heavy load we experienced CPU
 problems. I am the maintainer of that fork. The fork comes with ZK
 integration. Another kafka module is this one:
 https://github.com/dannycoates/franz-kafka.

 Kind regards,
 Radek Gruchalski
 radek.gruchal...@technicolor.com (mailto:
>> radek.gruchal...@technicolor.com)
 | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
 ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 00447889948663

 Confidentiality:
 This communication is intended for the above-named person and may be
 confidential and/or legally privileged.
 If it has come to you in error you must take no action based on it, nor
 must you copy or show it to anyone; please delete/destroy and inform the
 sender immediately.


 On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:

> Chris,
>
> Not sure how stable those node.js clients are. In 0.8, we plan to
 provide a
> native C version of the producer. A thin node.js layer can potentially
>> be
> built on top of that.
>
> Thanks,
>
> Jun
>
> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
>
>> During my due diligence to assess use of Kafka for both our activity
 and
>> log message streams, I would like to ask the project committers and
>> community users about using Kafka with Node.js. Yes, I am aware that a
>> Kafka client exists for Node.js (
>> https://github.com/marcuswestin/node-kafka), which has spurred
>> further
>> interest by our front-end team. Here are my questions, excuse me if
 they
>> seem "noobish".
>>
>> 1. How reliable is the Node.js client (
>> https://github.com/marcuswestin/node-kafka) in production
 applications?
>> If there are issues, what are they (the GitHub repo currently lists
 none)?
>> 2. To support real-time activity streams within Node.js, what is the
>> recommended consumer polling interval?
>> 3. General advise observations on integrating a front-end based
>> Node.js
>> application with Kafka mediated messaging.
>>
>> Thanks you!
>>
>> Chris
>>

Re: Kafka Node.js Integration Questions/Advice

[Apoorva Gaurav]

Thanks Radek,
We also are thinking of Java / Scala for Consumers, for Producers whether
franz-kafka is a good choice?

-- 
Thanks & Regards,
Apoorva

On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
radek.gruchal...@portico.io> wrote:

> We started using node-kafka before we learned franz-kafka was available.
> In node, franz-kafka would be my preferred choice now. But tbh, our
> consumers are all java. node-kafka does not support consumer settings like
> autooffset.reset and so on (or it is not obvious how to use those).
>
> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
> monitoring tools, which may be helpful if you want to add some scaling
> logic when consumer is lagging.
>
> Our first choice is node too but we're consuming exclusively with java.
>
> Hope this helps a little.
>
> On 22 Dec 2012, at 05:21, Apoorva Gaurav  wrote:
>
> > Which is the best ZK based implementation of kafka in node.js. Our use
> case
> > is that a pool of node js http servers will be listening to clients which
> > will send json over http. Using node js we'll do minimal decoration and
> > compression (preferably snappy) and write to brokers. We might also need
> > json to avro conversion but thats not a deal breaker. Consumers will be
> > writing these events to S3 (to begin with we don't plan to maintain HDFS
> > cluster). To begin with we'll have to support a peak load of 50K events /
> > second, average being much less, around 2K events / second. Suggestions
> > please. Is any one using franz-kafka in production. I'm only two days
> into
> > kafka so don't know a lot, but franz-kafka looks clean and easy to work
> > with.
> >
> > If none of the existing node.js implementation is capable of this then we
> > are willing to move to Scala or Java but node.js is the first choice.
> >
> > Thanks & Regards,
> > Apoorva
> >
> > On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
> > radek.gruchal...@portico.io> wrote:
> >
> >> We are using https://github.com/radekg/node-kafka, occasionally pushing
> >> about 2500 messages, 3.5K each / second. No issues so far. Different
> story
> >> with consumers. They are stable but under heavy load we experienced CPU
> >> problems. I am the maintainer of that fork. The fork comes with ZK
> >> integration. Another kafka module is this one:
> >> https://github.com/dannycoates/franz-kafka.
> >>
> >> Kind regards,
> >> Radek Gruchalski
> >> radek.gruchal...@technicolor.com (mailto:
> radek.gruchal...@technicolor.com)
> >> | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
> >> ra...@gruchalski.com (mailto:ra...@gruchalski.com)
> >> 00447889948663
> >>
> >> Confidentiality:
> >> This communication is intended for the above-named person and may be
> >> confidential and/or legally privileged.
> >> If it has come to you in error you must take no action based on it, nor
> >> must you copy or show it to anyone; please delete/destroy and inform the
> >> sender immediately.
> >>
> >>
> >> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
> >>
> >>> Chris,
> >>>
> >>> Not sure how stable those node.js clients are. In 0.8, we plan to
> >> provide a
> >>> native C version of the producer. A thin node.js layer can potentially
> be
> >>> built on top of that.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> >>> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
> >>>
>  During my due diligence to assess use of Kafka for both our activity
> >> and
>  log message streams, I would like to ask the project committers and
>  community users about using Kafka with Node.js. Yes, I am aware that a
>  Kafka client exists for Node.js (
>  https://github.com/marcuswestin/node-kafka), which has spurred
> further
>  interest by our front-end team. Here are my questions, excuse me if
> >> they
>  seem "noobish".
> 
>  1. How reliable is the Node.js client (
>  https://github.com/marcuswestin/node-kafka) in production
> >> applications?
>  If there are issues, what are they (the GitHub repo currently lists
> >> none)?
>  2. To support real-time activity streams within Node.js, what is the
>  recommended consumer polling interval?
>  3. General advise observations on integrating a front-end based
> Node.js
>  application with Kafka mediated messaging.
> 
>  Thanks you!
> 
>  Chris
> >>
> >>
>

Re: Kafka Node.js Integration Questions/Advice

[Radek Gruchalski]

We started using node-kafka before we learned franz-kafka was available. In 
node, franz-kafka would be my preferred choice now. But tbh, our consumers are 
all java. node-kafka does not support consumer settings like autooffset.reset 
and so on (or it is not obvious how to use those).

Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx 
monitoring tools, which may be helpful if you want to add some scaling logic 
when consumer is lagging.

Our first choice is node too but we're consuming exclusively with java.

Hope this helps a little.

On 22 Dec 2012, at 05:21, Apoorva Gaurav  wrote:

> Which is the best ZK based implementation of kafka in node.js. Our use case
> is that a pool of node js http servers will be listening to clients which
> will send json over http. Using node js we'll do minimal decoration and
> compression (preferably snappy) and write to brokers. We might also need
> json to avro conversion but thats not a deal breaker. Consumers will be
> writing these events to S3 (to begin with we don't plan to maintain HDFS
> cluster). To begin with we'll have to support a peak load of 50K events /
> second, average being much less, around 2K events / second. Suggestions
> please. Is any one using franz-kafka in production. I'm only two days into
> kafka so don't know a lot, but franz-kafka looks clean and easy to work
> with.
> 
> If none of the existing node.js implementation is capable of this then we
> are willing to move to Scala or Java but node.js is the first choice.
> 
> Thanks & Regards,
> Apoorva
> 
> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
> radek.gruchal...@portico.io> wrote:
> 
>> We are using https://github.com/radekg/node-kafka, occasionally pushing
>> about 2500 messages, 3.5K each / second. No issues so far. Different story
>> with consumers. They are stable but under heavy load we experienced CPU
>> problems. I am the maintainer of that fork. The fork comes with ZK
>> integration. Another kafka module is this one:
>> https://github.com/dannycoates/franz-kafka.
>> 
>> Kind regards,
>> Radek Gruchalski
>> radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com)
>> | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
>> ra...@gruchalski.com (mailto:ra...@gruchalski.com)
>> 00447889948663
>> 
>> Confidentiality:
>> This communication is intended for the above-named person and may be
>> confidential and/or legally privileged.
>> If it has come to you in error you must take no action based on it, nor
>> must you copy or show it to anyone; please delete/destroy and inform the
>> sender immediately.
>> 
>> 
>> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
>> 
>>> Chris,
>>> 
>>> Not sure how stable those node.js clients are. In 0.8, we plan to
>> provide a
>>> native C version of the producer. A thin node.js layer can potentially be
>>> built on top of that.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
>>> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
>>> 
 During my due diligence to assess use of Kafka for both our activity
>> and
 log message streams, I would like to ask the project committers and
 community users about using Kafka with Node.js. Yes, I am aware that a
 Kafka client exists for Node.js (
 https://github.com/marcuswestin/node-kafka), which has spurred further
 interest by our front-end team. Here are my questions, excuse me if
>> they
 seem "noobish".
 
 1. How reliable is the Node.js client (
 https://github.com/marcuswestin/node-kafka) in production
>> applications?
 If there are issues, what are they (the GitHub repo currently lists
>> none)?
 2. To support real-time activity streams within Node.js, what is the
 recommended consumer polling interval?
 3. General advise observations on integrating a front-end based Node.js
 application with Kafka mediated messaging.
 
 Thanks you!
 
 Chris
>> 
>> 

Re: Kafka Node.js Integration Questions/Advice

[Apoorva Gaurav]

Which is the best ZK based implementation of kafka in node.js. Our use case
is that a pool of node js http servers will be listening to clients which
will send json over http. Using node js we'll do minimal decoration and
compression (preferably snappy) and write to brokers. We might also need
json to avro conversion but thats not a deal breaker. Consumers will be
writing these events to S3 (to begin with we don't plan to maintain HDFS
cluster). To begin with we'll have to support a peak load of 50K events /
second, average being much less, around 2K events / second. Suggestions
please. Is any one using franz-kafka in production. I'm only two days into
kafka so don't know a lot, but franz-kafka looks clean and easy to work
with.

If none of the existing node.js implementation is capable of this then we
are willing to move to Scala or Java but node.js is the first choice.

Thanks & Regards,
Apoorva

On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
radek.gruchal...@portico.io> wrote:

> We are using https://github.com/radekg/node-kafka, occasionally pushing
> about 2500 messages, 3.5K each / second. No issues so far. Different story
> with consumers. They are stable but under heavy load we experienced CPU
> problems. I am the maintainer of that fork. The fork comes with ZK
> integration. Another kafka module is this one:
> https://github.com/dannycoates/franz-kafka.
>
> Kind regards,
> Radek Gruchalski
> radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com)
> | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
> ra...@gruchalski.com (mailto:ra...@gruchalski.com)
> 00447889948663
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
>
> > Chris,
> >
> > Not sure how stable those node.js clients are. In 0.8, we plan to
> provide a
> > native C version of the producer. A thin node.js layer can potentially be
> > built on top of that.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> > calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
> >
> > > During my due diligence to assess use of Kafka for both our activity
> and
> > > log message streams, I would like to ask the project committers and
> > > community users about using Kafka with Node.js. Yes, I am aware that a
> > > Kafka client exists for Node.js (
> > > https://github.com/marcuswestin/node-kafka), which has spurred further
> > > interest by our front-end team. Here are my questions, excuse me if
> they
> > > seem "noobish".
> > >
> > > 1. How reliable is the Node.js client (
> > > https://github.com/marcuswestin/node-kafka) in production
> applications?
> > > If there are issues, what are they (the GitHub repo currently lists
> none)?
> > > 2. To support real-time activity streams within Node.js, what is the
> > > recommended consumer polling interval?
> > > 3. General advise observations on integrating a front-end based Node.js
> > > application with Kafka mediated messaging.
> > >
> > > Thanks you!
> > >
> > > Chris
>
>

Re: Kafka Node.js Integration Questions/Advice

[Radek Gruchalski]

We are using https://github.com/radekg/node-kafka, occasionally pushing about 
2500 messages, 3.5K each / second. No issues so far. Different story with 
consumers. They are stable but under heavy load we experienced CPU problems. I 
am the maintainer of that fork. The fork comes with ZK integration. Another 
kafka module is this one: https://github.com/dannycoates/franz-kafka.

Kind regards,

Radek Gruchalski
radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com) | 
radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) | 

ra...@gruchalski.com
 (mailto:ra...@gruchalski.com)
00447889948663





Confidentiality:
This communication is intended for the above-named person and may be 
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must 
you copy or show it to anyone; please delete/destroy and inform the sender 
immediately.


On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:

> Chris,
>  
> Not sure how stable those node.js clients are. In 0.8, we plan to provide a
> native C version of the producer. A thin node.js layer can potentially be
> built on top of that.
>  
> Thanks,
>  
> Jun
>  
> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote:
>  
> > During my due diligence to assess use of Kafka for both our activity and
> > log message streams, I would like to ask the project committers and
> > community users about using Kafka with Node.js. Yes, I am aware that a
> > Kafka client exists for Node.js (
> > https://github.com/marcuswestin/node-kafka), which has spurred further
> > interest by our front-end team. Here are my questions, excuse me if they
> > seem "noobish".
> >  
> > 1. How reliable is the Node.js client (
> > https://github.com/marcuswestin/node-kafka) in production applications?
> > If there are issues, what are they (the GitHub repo currently lists none)?
> > 2. To support real-time activity streams within Node.js, what is the
> > recommended consumer polling interval?
> > 3. General advise observations on integrating a front-end based Node.js
> > application with Kafka mediated messaging.
> >  
> > Thanks you!
> >  
> > Chris  

Re: Kafka Node.js Integration Questions/Advice

[Jun Rao]

Chris,

Not sure how stable those node.js clients are. In 0.8, we plan to provide a
native C version of the producer. A thin node.js layer can potentially be
built on top of that.

Thanks,

Jun

On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
calexan...@gravycard.com> wrote:

> During my due diligence to assess use of Kafka for both our activity and
> log message streams, I would like to ask the project committers and
> community users about using Kafka with Node.js. Yes, I am aware that a
> Kafka client exists for Node.js (
> https://github.com/marcuswestin/node-kafka), which has spurred further
> interest by our front-end team. Here are my questions, excuse me if they
> seem "noobish".
>
> 1. How reliable is the Node.js client (
> https://github.com/marcuswestin/node-kafka) in production applications?
> If there are issues, what are they (the GitHub repo currently lists none)?
> 2. To support real-time activity streams within Node.js, what is the
> recommended consumer polling interval?
> 3. General advise observations on integrating a front-end based Node.js
> application with Kafka mediated messaging.
>
> Thanks you!
>
> Chris
>

Re: Kafka Node.js Integration Questions/Advice

[Christopher Alexander]

Thanks David. Yes, I am aware of the Prozess Node lib also. I forgot to include 
it in my posting. Good catch!

- Original Message -
From: "David Arthur" 
To: users@kafka.apache.org
Sent: Thursday, December 20, 2012 11:58:45 AM
Subject: Re: Kafka Node.js Integration Questions/Advice


On 12/20/12 11:46 AM, Christopher Alexander wrote:
> During my due diligence to assess use of Kafka for both our activity and log 
> message streams, I would like to ask the project committers and community 
> users about using Kafka with Node.js. Yes, I am aware that a Kafka client 
> exists for Node.js (https://github.com/marcuswestin/node-kafka), which has 
> spurred further interest by our front-end team. Here are my questions, excuse 
> me if they seem "noobish".
>
> 1. How reliable is the Node.js client 
> (https://github.com/marcuswestin/node-kafka) in production applications? If 
> there are issues, what are they (the GitHub repo currently lists none)?
Just FYI, there is another node.js library 
https://github.com/cainus/Prozess. I have no experience with either, so 
I cannot say how reliable they are.
> 2. To support real-time activity streams within Node.js, what is the 
> recommended consumer polling interval?
What kind of data velocity do you expect? You should only have to poll 
if your consumer catches up to the broker and there's no more data. 
Blocking/polling behavior of the consumer depends entirely on the client 
implementation.
> 3. General advise observations on integrating a front-end based Node.js 
> application with Kafka mediated messaging.
>
> Thanks you!
>
> Chris

Re: Kafka Node.js Integration Questions/Advice

[David Arthur]

On 12/20/12 11:46 AM, Christopher Alexander wrote:

During my due diligence to assess use of Kafka for both our activity and log message 
streams, I would like to ask the project committers and community users about using Kafka 
with Node.js. Yes, I am aware that a Kafka client exists for Node.js 
(https://github.com/marcuswestin/node-kafka), which has spurred further interest by our 
front-end team. Here are my questions, excuse me if they seem "noobish".

1. How reliable is the Node.js client 
(https://github.com/marcuswestin/node-kafka) in production applications? If 
there are issues, what are they (the GitHub repo currently lists none)?
Just FYI, there is another node.js library 
https://github.com/cainus/Prozess. I have no experience with either, so 
I cannot say how reliable they are.

2. To support real-time activity streams within Node.js, what is the 
recommended consumer polling interval?
What kind of data velocity do you expect? You should only have to poll 
if your consumer catches up to the broker and there's no more data. 
Blocking/polling behavior of the consumer depends entirely on the client 
implementation.

3. General advise observations on integrating a front-end based Node.js 
application with Kafka mediated messaging.

Thanks you!

Chris

Kafka Node.js Integration Questions/Advice

[Christopher Alexander]

During my due diligence to assess use of Kafka for both our activity and log 
message streams, I would like to ask the project committers and community users 
about using Kafka with Node.js. Yes, I am aware that a Kafka client exists for 
Node.js (https://github.com/marcuswestin/node-kafka), which has spurred further 
interest by our front-end team. Here are my questions, excuse me if they seem 
"noobish".

1. How reliable is the Node.js client 
(https://github.com/marcuswestin/node-kafka) in production applications? If 
there are issues, what are they (the GitHub repo currently lists none)?
2. To support real-time activity streams within Node.js, what is the 
recommended consumer polling interval?
3. General advise observations on integrating a front-end based Node.js 
application with Kafka mediated messaging.

Thanks you!

Chris