Re: Kafka stats description

2013-06-15 Thread Jun Rao
The important metrics are byte rate and message rate in both the clients and the broker. Thanks, Jun On Sat, Jun 15, 2013 at 9:44 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Thanks for reply Jun ! > > But still not clear with significance of these parameters. > > > On Sat, Ju

Re: Using Kafka for "data" messages

2013-06-15 Thread Milind Parikh
You might want to take a look at mps (github.com/milindparikh/mps). As I was thinking about blogging about it, your use-case magically surfaced on the mailing list. Since one of the supposed benefits of having a framework is to enable quick building of stuff on top of it, I took a crack at your use

Re: Kafka stats description

2013-06-15 Thread Hanish Bansal
Thanks for reply Jun ! But still not clear with significance of these parameters. On Sat, Jun 15, 2013 at 9:20 PM, Jun Rao wrote: > AvgProduceRequestMs: avg time of a produce RPC request > > For mbeans with Num and Total, they represent the cumulated value since the > service is up. > > Thanks

Re: mps, an eventing framework for kafka

2013-06-15 Thread Milind Parikh
I meant to post it to the group; but forgot. We have the erlang client for kafka 0.8. It is also the part of mps. I blog about the features over here: http://milindparikh.blogspot.com/2013/06/the-erlang-client-for-apache-kafka-08.html?m=1 Thanks Milind On Jun 15, 2013 9:00 AM, "Jun Rao" wrote:

Re: mps, an eventing framework for kafka

2013-06-15 Thread Jun Rao
Milind, Thanks for sharing. How did you consume Kafka data in Erlang? Jun On Fri, Jun 14, 2013 at 10:23 PM, Milind Parikh wrote: > There is a lot of useful discussion on the user group about data messages > and eventing, I thought that I would introduce mps, an eventing framework > for kafka.

Re: Stall high-level 0.72 ConsumerConnector until all balanced? Avoid message dupes?

2013-06-15 Thread Jun Rao
If there are no offsets stored in ZK, I think it's possible to get some dups during startup. Once the offsets are in ZK, there shouldn't be dups during subsequent rebalances. Thanks, Jun On Fri, Jun 14, 2013 at 2:04 PM, Philip O'Toole wrote: > On Thu, Jun 13, 2013 at 9:15 PM, Jun Rao wrote:

Re: 0.8 backup strategy anyone?

2013-06-15 Thread Jun Rao
One possibility is to copy the data to another cluster using MirrorMaker. Thanks, Jun On Fri, Jun 14, 2013 at 10:47 AM, Scott Clasen wrote: > So despite 0.8 being a release that will give much higher availability do > people do anything at all to back up the data? > > For instance if any of

Re: Kafka stats description

2013-06-15 Thread Jun Rao
AvgProduceRequestMs: avg time of a produce RPC request For mbeans with Num and Total, they represent the cumulated value since the service is up. Thanks, Jun On Fri, Jun 14, 2013 at 8:22 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi > > I'm trying to fetch Kafka stats using