Re: storing last processed offset, recovery of failed message processing etc.

2013-12-11 Thread S Ahmed
When using ZK to keep track of last offsets metrics etc., how do you know when you are pushing your ZK cluster to its limit? Or can ZK handle thousands of writes/reads per second no problem since it is all in-memory? But even so, you need some idea on its upper limits and how close you are to

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread S Ahmed
Say am I doing this, a scenerio that I just came up with that demonstrates #2. Someone signs up on a website, and you have to: 1. create the user profile 2. send email confirmation email 3. resize avatar Now once a person registers on a website, I write a message to Kafka. Now I have 3

Re: storing last processed offset, recovery of failed message processing etc.

2013-12-09 Thread Benjamin Black
You might look at Curator http://curator.apache.org/ On Mon, Dec 9, 2013 at 12:36 PM, S Ahmed sahmed1...@gmail.com wrote: Say am I doing this, a scenerio that I just came up with that demonstrates #2. Someone signs up on a website, and you have to: 1. create the user profile 2. send