Re: Kafka Node.js Integration Questions/Advice

2016-05-18 Thread Reem Diab 
We recently released kafka-java-bridge - new nodejs high level consumer.

npm ,github


Its tested stable and

-- 
Reem Diab
Software Engineer
T: +972-74-700-4580
 
 We Create Meaningful Connections


-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Kafka Node.js Integration Questions/Advice

2016-05-18 Thread Reem Diab 
We recently released kafka-java-bridge - new nodejs high level consumer.

npm ,github


Its tested stable and
-- 
Reem Diab
Software Engineer
T: +972-74-700-4580
 
 We Create Meaningful Connections


-- 
This message may contain confidential and/or privileged information. 
If you are not the addressee or authorized to receive this on behalf of the 
addressee you must not use, copy, disclose or take action based on this 
message or any information herein. 
If you have received this message in error, please advise the sender 
immediately by reply email and delete this message. Thank you.

Re: Kafka Node.js Integration Questions/Advice

2012-12-22 Thread David Arthur 
FWIW, message production is quite simpler than consumption. It does
not require the same complex coordination as the consumers. Producers
only use ZooKeeper to locate available brokers

Sent from my phone

On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav apoorvagau...@gmail.com wrote:

 Thanks Radek,
 We also are thinking of Java / Scala for Consumers, for Producers whether
 franz-kafka is a good choice?

 --
 Thanks  Regards,
 Apoorva

 On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski 
 radek.gruchal...@portico.io wrote:

 We started using node-kafka before we learned franz-kafka was available.
 In node, franz-kafka would be my preferred choice now. But tbh, our
 consumers are all java. node-kafka does not support consumer settings like
 autooffset.reset and so on (or it is not obvious how to use those).

 Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
 monitoring tools, which may be helpful if you want to add some scaling
 logic when consumer is lagging.

 Our first choice is node too but we're consuming exclusively with java.

 Hope this helps a little.

 On 22 Dec 2012, at 05:21, Apoorva Gaurav apoorvagau...@gmail.com wrote:

 Which is the best ZK based implementation of kafka in node.js. Our use
 case
 is that a pool of node js http servers will be listening to clients which
 will send json over http. Using node js we'll do minimal decoration and
 compression (preferably snappy) and write to brokers. We might also need
 json to avro conversion but thats not a deal breaker. Consumers will be
 writing these events to S3 (to begin with we don't plan to maintain HDFS
 cluster). To begin with we'll have to support a peak load of 50K events /
 second, average being much less, around 2K events / second. Suggestions
 please. Is any one using franz-kafka in production. I'm only two days
 into
 kafka so don't know a lot, but franz-kafka looks clean and easy to work
 with.

 If none of the existing node.js implementation is capable of this then we
 are willing to move to Scala or Java but node.js is the first choice.

 Thanks  Regards,
 Apoorva

 On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski 
 radek.gruchal...@portico.io wrote:

 We are using https://github.com/radekg/node-kafka, occasionally pushing
 about 2500 messages, 3.5K each / second. No issues so far. Different
 story
 with consumers. They are stable but under heavy load we experienced CPU
 problems. I am the maintainer of that fork. The fork comes with ZK
 integration. Another kafka module is this one:
 https://github.com/dannycoates/franz-kafka.

 Kind regards,
 Radek Gruchalski
 radek.gruchal...@technicolor.com (mailto:
 radek.gruchal...@technicolor.com)
 | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
 ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 00447889948663

 Confidentiality:
 This communication is intended for the above-named person and may be
 confidential and/or legally privileged.
 If it has come to you in error you must take no action based on it, nor
 must you copy or show it to anyone; please delete/destroy and inform the
 sender immediately.


 On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:

 Chris,

 Not sure how stable those node.js clients are. In 0.8, we plan to
 provide a
 native C version of the producer. A thin node.js layer can potentially
 be
 built on top of that.

 Thanks,

 Jun

 On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander 
 calexan...@gravycard.com (mailto:calexan...@gravycard.com) wrote:

 During my due diligence to assess use of Kafka for both our activity
 and
 log message streams, I would like to ask the project committers and
 community users about using Kafka with Node.js. Yes, I am aware that a
 Kafka client exists for Node.js (
 https://github.com/marcuswestin/node-kafka), which has spurred
 further
 interest by our front-end team. Here are my questions, excuse me if
 they
 seem noobish.

 1. How reliable is the Node.js client (
 https://github.com/marcuswestin/node-kafka) in production
 applications?
 If there are issues, what are they (the GitHub repo currently lists
 none)?
 2. To support real-time activity streams within Node.js, what is the
 recommended consumer polling interval?
 3. General advise observations on integrating a front-end based
 Node.js
 application with Kafka mediated messaging.

 Thanks you!

 Chris

Re: Kafka Node.js Integration Questions/Advice

2012-12-22 Thread Radek Gruchalski 
That is exactly why we've decided to stick with java. Also support for all 
consumer settings out of the box.

Kind regards,
Radek Gruchalski

On 22 Dec 2012, at 19:17, David Arthur mum...@gmail.com wrote:

 FWIW, message production is quite simpler than consumption. It does
 not require the same complex coordination as the consumers. Producers
 only use ZooKeeper to locate available brokers
 
 Sent from my phone
 
 On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav apoorvagau...@gmail.com wrote:
 
 Thanks Radek,
 We also are thinking of Java / Scala for Consumers, for Producers whether
 franz-kafka is a good choice?
 
 --
 Thanks  Regards,
 Apoorva
 
 On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski 
 radek.gruchal...@portico.io wrote:
 
 We started using node-kafka before we learned franz-kafka was available.
 In node, franz-kafka would be my preferred choice now. But tbh, our
 consumers are all java. node-kafka does not support consumer settings like
 autooffset.reset and so on (or it is not obvious how to use those).
 
 Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
 monitoring tools, which may be helpful if you want to add some scaling
 logic when consumer is lagging.
 
 Our first choice is node too but we're consuming exclusively with java.
 
 Hope this helps a little.
 
 On 22 Dec 2012, at 05:21, Apoorva Gaurav apoorvagau...@gmail.com wrote:
 
 Which is the best ZK based implementation of kafka in node.js. Our use
 case
 is that a pool of node js http servers will be listening to clients which
 will send json over http. Using node js we'll do minimal decoration and
 compression (preferably snappy) and write to brokers. We might also need
 json to avro conversion but thats not a deal breaker. Consumers will be
 writing these events to S3 (to begin with we don't plan to maintain HDFS
 cluster). To begin with we'll have to support a peak load of 50K events /
 second, average being much less, around 2K events / second. Suggestions
 please. Is any one using franz-kafka in production. I'm only two days
 into
 kafka so don't know a lot, but franz-kafka looks clean and easy to work
 with.
 
 If none of the existing node.js implementation is capable of this then we
 are willing to move to Scala or Java but node.js is the first choice.
 
 Thanks  Regards,
 Apoorva
 
 On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski 
 radek.gruchal...@portico.io wrote:
 
 We are using https://github.com/radekg/node-kafka, occasionally pushing
 about 2500 messages, 3.5K each / second. No issues so far. Different
 story
 with consumers. They are stable but under heavy load we experienced CPU
 problems. I am the maintainer of that fork. The fork comes with ZK
 integration. Another kafka module is this one:
 https://github.com/dannycoates/franz-kafka.
 
 Kind regards,
 Radek Gruchalski
 radek.gruchal...@technicolor.com (mailto:
 radek.gruchal...@technicolor.com)
 | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) |
 ra...@gruchalski.com (mailto:ra...@gruchalski.com)
 00447889948663
 
 Confidentiality:
 This communication is intended for the above-named person and may be
 confidential and/or legally privileged.
 If it has come to you in error you must take no action based on it, nor
 must you copy or show it to anyone; please delete/destroy and inform the
 sender immediately.
 
 
 On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
 
 Chris,
 
 Not sure how stable those node.js clients are. In 0.8, we plan to
 provide a
 native C version of the producer. A thin node.js layer can potentially
 be
 built on top of that.
 
 Thanks,
 
 Jun
 
 On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander 
 calexan...@gravycard.com (mailto:calexan...@gravycard.com) wrote:
 
 During my due diligence to assess use of Kafka for both our activity
 and
 log message streams, I would like to ask the project committers and
 community users about using Kafka with Node.js. Yes, I am aware that a
 Kafka client exists for Node.js (
 https://github.com/marcuswestin/node-kafka), which has spurred
 further
 interest by our front-end team. Here are my questions, excuse me if
 they
 seem noobish.
 
 1. How reliable is the Node.js client (
 https://github.com/marcuswestin/node-kafka) in production
 applications?
 If there are issues, what are they (the GitHub repo currently lists
 none)?
 2. To support real-time activity streams within Node.js, what is the
 recommended consumer polling interval?
 3. General advise observations on integrating a front-end based
 Node.js
 application with Kafka mediated messaging.
 
 Thanks you!
 
 Chris

Re: Kafka Node.js Integration Questions/Advice

2012-12-20 Thread Christopher Alexander 
Thanks David. Yes, I am aware of the Prozess Node lib also. I forgot to include 
it in my posting. Good catch!

- Original Message -
From: David Arthur mum...@gmail.com
To: users@kafka.apache.org
Sent: Thursday, December 20, 2012 11:58:45 AM
Subject: Re: Kafka Node.js Integration Questions/Advice


On 12/20/12 11:46 AM, Christopher Alexander wrote:
 During my due diligence to assess use of Kafka for both our activity and log 
 message streams, I would like to ask the project committers and community 
 users about using Kafka with Node.js. Yes, I am aware that a Kafka client 
 exists for Node.js (https://github.com/marcuswestin/node-kafka), which has 
 spurred further interest by our front-end team. Here are my questions, excuse 
 me if they seem noobish.

 1. How reliable is the Node.js client 
 (https://github.com/marcuswestin/node-kafka) in production applications? If 
 there are issues, what are they (the GitHub repo currently lists none)?
Just FYI, there is another node.js library 
https://github.com/cainus/Prozess. I have no experience with either, so 
I cannot say how reliable they are.
 2. To support real-time activity streams within Node.js, what is the 
 recommended consumer polling interval?
What kind of data velocity do you expect? You should only have to poll 
if your consumer catches up to the broker and there's no more data. 
Blocking/polling behavior of the consumer depends entirely on the client 
implementation.
 3. General advise observations on integrating a front-end based Node.js 
 application with Kafka mediated messaging.

 Thanks you!

 Chris

Re: Kafka Node.js Integration Questions/Advice

2012-12-20 Thread Jun Rao 
Chris,

Not sure how stable those node.js clients are. In 0.8, we plan to provide a
native C version of the producer. A thin node.js layer can potentially be
built on top of that.

Thanks,

Jun

On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander 
calexan...@gravycard.com wrote:

 During my due diligence to assess use of Kafka for both our activity and
 log message streams, I would like to ask the project committers and
 community users about using Kafka with Node.js. Yes, I am aware that a
 Kafka client exists for Node.js (
 https://github.com/marcuswestin/node-kafka), which has spurred further
 interest by our front-end team. Here are my questions, excuse me if they
 seem noobish.

 1. How reliable is the Node.js client (
 https://github.com/marcuswestin/node-kafka) in production applications?
 If there are issues, what are they (the GitHub repo currently lists none)?
 2. To support real-time activity streams within Node.js, what is the
 recommended consumer polling interval?
 3. General advise observations on integrating a front-end based Node.js
 application with Kafka mediated messaging.

 Thanks you!

 Chris