Gah. I traced it down to an IOException that wasn't being handled (since
BlockingChannel doesn't declare that it throws them) and was swallowed in
all the log output from the test servers.
Thanks for the help!
On Sun, Jun 21, 2015 at 4:37 PM Jiangjie Qin
wrote:
> Hmm, it might be a little bit d
Hmm, it might be a little bit difficult to tell what happened without
looking at your test code.
Can you try with the settings I mentioned? We can try to use only two
threads here:
1. Producer thread: produce some messages -> stop/start Kafka server ->
produce some more messages.
2. Consumer thread
On Sun, Jun 21, 2015 at 1:10 AM Jiangjie Qin
wrote:
> Hey Noah,
>
> Carl is right about the offset. The offset to be commit should be the
> largest-consumed-offset + 1. But this should not break the at least once
> guarantee.
> From what I can see, your consumer should not skip messages. Just to
Hey Noah,
Carl is right about the offset. The offset to be commit should be the
largest-consumed-offset + 1. But this should not break the at least once
guarantee.
>From what I can see, your consumer should not skip messages. Just to make
sure I understand your test correctly,
1. There is a consum
Hi,
We have been solving this very problem in Hermes. You can see what we came
up by examining classes located here:
https://github.com/allegro/hermes/tree/master/hermes-consumers/src/main/java/pl/allegro/tech/hermes/consumers/consumer/offset
We are quite sure this gives us at-least-once guarant
>From my understanding of the code (admittedly very limited), the offset in
OffsetAndMetadata corresponds to the start of the message just obtained
from iterator.next(). So if you commit that, a restarted consumer should
get that message again. So it should actually continue at the previous
message
It is the value we get from calling MessageAndMetadata#offset() for the
last message processed. The MessageAndMetadata instance comes from the
ConsumerIterator.
On Fri, Jun 19, 2015 at 2:31 AM Carl Heymann wrote:
> How are you tracking the offsets that you manually commit? I.e. where do
> you ge
How are you tracking the offsets that you manually commit? I.e. where do
you get the metadata for the consumed messages?
On Thu, Jun 18, 2015 at 11:21 PM, noah wrote:
> We are in a situation where we need at least once delivery. We have a
> thread that pulls messages off the consumer, puts them
We are in a situation where we need at least once delivery. We have a
thread that pulls messages off the consumer, puts them in a queue where
they go through a few async steps, and then after the final step, we want
to commit the offset to the messages we have completed. There may be items
we have
