afaik you dont 'have' to store the consumed offsets in zk right, this is
only automatic with some of the clients?
why not store them in a data store that can write at the rate that you
require?
On Fri, May 17, 2013 at 2:15 PM, Withers, Robert wrote:
> Update from our OPS team, regarding zookeep
There is no particular need for storing the offsets in zookeeper. In fact
with Kafka 0.8, since partitions will be highly available, offsets could be
stored in Kafka topics. However, we haven't ironed out the design for this
yet.
Thanks,
Neha
On Fri, May 17, 2013 at 2:19 PM, Scott Clasen wrote:
to clarify, I meant that Robert could/should store the offsets in a faster
store not that kafka should default to that :)
Thanks Neha
On Fri, May 17, 2013 at 2:22 PM, Neha Narkhede wrote:
> There is no particular need for storing the offsets in zookeeper. In fact
> with Kafka 0.8, since partiti
Have you considered abstracting offset storage away so people could implement
their own?
Would you take a patch if I'd stabbed at it, and if yes, what's the process
(pardon the n00b)?
KCBO,
--
"If you can't conceal it well, expose it with all your might"
Alex Zuzin
On Friday, May 17, 2013 at
Neha,
apologies, I just re-read what I sent and realized my "you" wasn't specific
enough - it meant the Kafka team ;).
--
"If you can't conceal it well, expose it with all your might"
Alex Zuzin
On Friday, May 17, 2013 at 2:25 PM, Alex Zuzin wrote:
> Have you considered abstracting offset s
No problem. You can take a look at some of the thoughts we had on improving
the offset storage here -
https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management.
Suggestions are welcome.
Thanks,
Neha
On Fri, May 17, 2013 at 2:40 PM, Alex Zuzin wrote:
> Neha,
>
> apologies, I just re-r
Did so. The proposal looks perfectly sensible on first reading.
I understand that the patches in
https://issues.apache.org/jira/browse/KAFKA-657 are already in the trunk and
scheduled for 0.8.1? Are they going out with 0.8? If not, what's ETA for 0.8.1?
Either way, I'm going to try my hand at b
Yes, it looks spot on.
Thanks,
rob
> -Original Message-
> From: Alex Zuzin [mailto:carna...@gmail.com]
> Sent: Monday, May 20, 2013 11:37 AM
> To: users@kafka.apache.org
> Subject: Re: Update: RE: are commitOffsets botched to zookeeper?
>
> Did so. The proposal loo
