13:49:26 -0700
> Subject: Re: Reliable Message Commits
> From: wangg...@gmail.com
> To: users@kafka.apache.org
>
> If there is a crash between
>
> process(message);
>
> and
>
> iter.next(); //consumes the message now
>
> you will still have duplicates
If there is a crash between
process(message);
and
iter.next(); //consumes the message now
you will still have duplicates upon restart.
Guozhang
On Mon, Jun 23, 2014 at 12:37 PM, Achanta Vamsi Subhash <
achanta.va...@flipkart.com> wrote:
> What about ths pattern:
>
> message = iter.peek(); /
What about ths pattern:
message = iter.peek(); //gets the message
process(message);
iter.next(); //consumes the message now
consumer.commit();
On Fri, Jun 20, 2014 at 11:53 AM, Kyle Banker wrote:
> Thanks for the advice, Guozhang.
>
> Jagbir: I'll report back on my progress. I intend to have
Thanks for the advice, Guozhang.
Jagbir: I'll report back on my progress. I intend to have quite a few
threads across many machines. We'll see how well it performs with a whole
high-level consumer per thread.
On Thu, Jun 19, 2014 at 9:30 AM, Guozhang Wang wrote:
> Hello Kyle,
>
> For your firs
Hello Kyle,
For your first question, the first option would be preferable: it may use
little bit more memory, and have more ZK writes. In 0.9 though, the offsets
will be stored in Kafka servers instead of ZK, so you will no longer
bombard ZK.
For the third question, our designed usage pattern for
Hi Kyle,
Thanks for the update. Wondering if you found answer to your N-1 commit
question? If auto commit happens only at iterator.next () and onky for the N -1
message then client code can be much simpler and reliable as you mentioned. I'm
also looking forward to any post in this regard.
J
I think I've discovered the answer to my second question: according to the
code in ZookeeperConsumerConnector.scala, a rebalance derives its offsets
from what's already in Zookeeper. Therefore, uncommitted but consumed
messages from a given partition will be replayed when the partition is
reassigne
