Re: Node-Kafka Client Review and Question
We did an examination of the tagged branch but the version was 0.1.7 and its been static for over a year now. I will say that the Node-Kafka (v2.3) producer has been stable however. A previous thread concerning Node-Kafka client development revealed that a C library will be out for 0.8, supporting more stable library client development (hint hint too ;-) Unfortunately, we are moving rapidly with what works now to get our platform to an MVP state. Hopefully, there won't be too much refactoring when the next gen Node-Kafka client is complete. :( Chris - Original Message - From: "Taylor Gautier" To: users@kafka.apache.org Sent: Wednesday, April 24, 2013 4:39:24 PM Subject: Re: Node-Kafka Client Review and Question This implementation is what I worked on while at Tagged, which was forked from Marcus' version, but I don't think it ever merged back to Marcus': https://github.com/tagged/node-kafka It was in production for about a year when I left Tagged about 6 months ago. I know that there were some internal fixes that never made it back out. I think the version that is public is pretty stable, but it's hard for me to know since I am no longer at Tagged and don't have access to the internal repos to know for sure what fixes are still private and which have been published back. That said, I think you should definitely give this version a shot first before moving on. Finally, if I were to do it all over again, with about a solid additional year of node programming experience under my belt, I'd probably rewrite everything from the interfaces to the implementation. In my current job there's a strong possibility of us adopting a Node.js + Kafka implementation, but not for at least a few months, so I wouldn't expect to be back in the community to work on this for a little while. Also, I'm kind of waiting for 0.8 (hint hint ;-) On Wed, Apr 24, 2013 at 10:45 AM, Christian Carollo wrote: > Hi Everyone, > > I have been experimenting with the libraries listed below and experienced > the same problems. > I have not found any another other node clients. I am interested in > finding a node solution as well. > Happy to contribute on a common solution. > > Christian Carollo > > On Apr 24, 2013, at 10:19 AM, Christopher Alexander < > calexan...@gravycard.com> wrote: > > > Hi Everyone, > > > > I just wanted to follow-up on a previous thread concerning our > investigation of identifying a stable Node-Kafka client. To date we have > tested the following: > > > > 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka) > > 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka) > > 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka) > > 4. Prozess (v0.3.5, https://github.com/cainus/Prozess) > > > > Results: > > > > 1. Could not get Franz-Kafka and Prozess to work. Requires funky > dependencies. > > 2. Node-Kafka, v2.1 was successfully setup but performed less stable > than #3. > > 3. Node-Kafka, v2.3 was successfully setup, exhibited the best > performance profile but the consumer is highly inconsistent - specifically, > consumer object remained in-memory regardless what we did (i.e. var > consumer = undefined after receiving message). Nothing appears to mitigate > this and ALL consumed messaged get replayed on reception of a new message. > > > > With this said, is there a Node-Kafka client people are actually using > in production that doesn't exhibit the profiles we have seen? We have > back-tracked using Node-Kafka (v2.3) to only produce messages and rely on > Redis PubSub channels for asynchronous acking of these messages. We would > be willing to roll-up our sleeves with the community to develop a much more > stable Node-Kafka client. > > > > Kind regards, > > > > Chris Alexander > > Chief Technical Architect and Engineer > > Gravy, Inc. > >
Re: Node-Kafka Client Review and Question
This implementation is what I worked on while at Tagged, which was forked from Marcus' version, but I don't think it ever merged back to Marcus': https://github.com/tagged/node-kafka It was in production for about a year when I left Tagged about 6 months ago. I know that there were some internal fixes that never made it back out. I think the version that is public is pretty stable, but it's hard for me to know since I am no longer at Tagged and don't have access to the internal repos to know for sure what fixes are still private and which have been published back. That said, I think you should definitely give this version a shot first before moving on. Finally, if I were to do it all over again, with about a solid additional year of node programming experience under my belt, I'd probably rewrite everything from the interfaces to the implementation. In my current job there's a strong possibility of us adopting a Node.js + Kafka implementation, but not for at least a few months, so I wouldn't expect to be back in the community to work on this for a little while. Also, I'm kind of waiting for 0.8 (hint hint ;-) On Wed, Apr 24, 2013 at 10:45 AM, Christian Carollo wrote: > Hi Everyone, > > I have been experimenting with the libraries listed below and experienced > the same problems. > I have not found any another other node clients. I am interested in > finding a node solution as well. > Happy to contribute on a common solution. > > Christian Carollo > > On Apr 24, 2013, at 10:19 AM, Christopher Alexander < > calexan...@gravycard.com> wrote: > > > Hi Everyone, > > > > I just wanted to follow-up on a previous thread concerning our > investigation of identifying a stable Node-Kafka client. To date we have > tested the following: > > > > 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka) > > 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka) > > 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka) > > 4. Prozess (v0.3.5, https://github.com/cainus/Prozess) > > > > Results: > > > > 1. Could not get Franz-Kafka and Prozess to work. Requires funky > dependencies. > > 2. Node-Kafka, v2.1 was successfully setup but performed less stable > than #3. > > 3. Node-Kafka, v2.3 was successfully setup, exhibited the best > performance profile but the consumer is highly inconsistent - specifically, > consumer object remained in-memory regardless what we did (i.e. var > consumer = undefined after receiving message). Nothing appears to mitigate > this and ALL consumed messaged get replayed on reception of a new message. > > > > With this said, is there a Node-Kafka client people are actually using > in production that doesn't exhibit the profiles we have seen? We have > back-tracked using Node-Kafka (v2.3) to only produce messages and rely on > Redis PubSub channels for asynchronous acking of these messages. We would > be willing to roll-up our sleeves with the community to develop a much more > stable Node-Kafka client. > > > > Kind regards, > > > > Chris Alexander > > Chief Technical Architect and Engineer > > Gravy, Inc. > >
Re: Node-Kafka Client Review and Question
Hi Everyone, I have been experimenting with the libraries listed below and experienced the same problems. I have not found any another other node clients. I am interested in finding a node solution as well. Happy to contribute on a common solution. Christian Carollo On Apr 24, 2013, at 10:19 AM, Christopher Alexander wrote: > Hi Everyone, > > I just wanted to follow-up on a previous thread concerning our investigation > of identifying a stable Node-Kafka client. To date we have tested the > following: > > 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka) > 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka) > 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka) > 4. Prozess (v0.3.5, https://github.com/cainus/Prozess) > > Results: > > 1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies. > 2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3. > 3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance > profile but the consumer is highly inconsistent - specifically, consumer > object remained in-memory regardless what we did (i.e. var consumer = > undefined after receiving message). Nothing appears to mitigate this and ALL > consumed messaged get replayed on reception of a new message. > > With this said, is there a Node-Kafka client people are actually using in > production that doesn't exhibit the profiles we have seen? We have > back-tracked using Node-Kafka (v2.3) to only produce messages and rely on > Redis PubSub channels for asynchronous acking of these messages. We would be > willing to roll-up our sleeves with the community to develop a much more > stable Node-Kafka client. > > Kind regards, > > Chris Alexander > Chief Technical Architect and Engineer > Gravy, Inc.
Node-Kafka Client Review and Question
Hi Everyone, I just wanted to follow-up on a previous thread concerning our investigation of identifying a stable Node-Kafka client. To date we have tested the following: 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka) 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka) 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka) 4. Prozess (v0.3.5, https://github.com/cainus/Prozess) Results: 1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies. 2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3. 3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance profile but the consumer is highly inconsistent - specifically, consumer object remained in-memory regardless what we did (i.e. var consumer = undefined after receiving message). Nothing appears to mitigate this and ALL consumed messaged get replayed on reception of a new message. With this said, is there a Node-Kafka client people are actually using in production that doesn't exhibit the profiles we have seen? We have back-tracked using Node-Kafka (v2.3) to only produce messages and rely on Redis PubSub channels for asynchronous acking of these messages. We would be willing to roll-up our sleeves with the community to develop a much more stable Node-Kafka client. Kind regards, Chris Alexander Chief Technical Architect and Engineer Gravy, Inc.