Re: Streams: Fetch Offset 0 is out of range for partition foo-0, resetting offset

2017-08-14 Thread Garrett Barton
Gouzhang, Will do, if it gets stuck in this loop again ill inspect the broker log dirs. I'm running 0.11 release right now. On Mon, Aug 14, 2017 at 4:22 PM, Guozhang Wang wrote: > Garrett, > > What I get confused is that you mentioned it start spamming the logs, means > that it falls into this

Re: Streams: Fetch Offset 0 is out of range for partition foo-0, resetting offset

2017-08-14 Thread Guozhang Wang
Garrett, What I get confused is that you mentioned it start spamming the logs, means that it falls into this endless loop of: 1) getting out-of-range exception 2) resetting offset by querying the broker of the offset 3) getting offset 0 from the broker, 4) send fetching request with starting 0, g

Re: Streams: Fetch Offset 0 is out of range for partition foo-0, resetting offset

2017-08-14 Thread Garrett Barton
​Gouzhang, Thanks for the reply!​ Based on what you said I am going to increase the log.retention.hours a bunch and see what happens, things typically break long before 48 hours, but your right the data could have expired by then too. I'll pay attention to that as well. As far as messing with

Re: Streams: Fetch Offset 0 is out of range for partition foo-0, resetting offset

2017-08-13 Thread Guozhang Wang
Hi Garrett, Since your error message says "offset X" is out of range, it means that the offset was reset to because there was no data any more on topic partition "foo-0". I suspect that is because all the log segments got truncated and the topic partition contains empty list. It is less likely cau

Streams: Fetch Offset 0 is out of range for partition foo-0, resetting offset

2017-08-09 Thread Garrett Barton
I have a small test setup with a local zk/kafka server and a streams app that loads sample data. The test setup is usually up for a day or two before a new build goes out and its blown away and loaded from scratch. Lately I've seen that after a few hours the stream app will stop processing and st