Bingo!  Thanks Dmitry, that is exactly what I'm running into.  Looks like I
just have to set offset.retention.minutes to be greater than my largest
log.retention.hours topics to be safe.

On Mon, Aug 7, 2017 at 9:03 PM, Dmitry Minkovsky <[email protected]>
wrote:

> Hi Garrett,
>
> This one
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-5510
>
> Best,
> Dmitry
>
> пн, 7 авг. 2017 г. в 14:22, Garrett Barton <[email protected]>:
>
> > Dmitry, which KIP are you referring to? I see this behavior too
> sometimes.
> >
> > On Fri, Aug 4, 2017 at 10:25 AM, Dmitry Minkovsky <[email protected]>
> > wrote:
> >
> > > Thank you Matthias and Bill,
> > >
> > > Just want to confirm that was my offsets *were *being committed but I
> was
> > > being affected by `offsets.retention.minutes` which I did not know
> > about. I
> > > set
> > >
> > > offsets.retention.minutes=2147483647
> > > offsets.retention.check.interval.ms=9223372036854775807
> > >
> > > Will keep an eye on that KIP.
> > >
> > > Best,
> > > Dmitry
> > >
> > > On Fri, Jul 21, 2017 at 4:31 AM, Matthias J. Sax <
> [email protected]>
> > > wrote:
> > >
> > > > >>>> My guess is that offsets are committed only when all tasks in
> the
> > > > >>> topology
> > > > >>>> have received input. Is this what's happening?
> > > >
> > > > No. Task offsets are committed independently from each other.
> > > >
> > > > You can you double check the logs in DEBUG mode. It indicates when
> > > > offsets get committed. Also check via `bin/kafka-consumer-groups.sh`
> > > > what offsets are committed (application.id == group.id)
> > > >
> > > > Hope this helps.
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 7/21/17 2:27 AM, Dmitry Minkovsky wrote:
> > > > > Hi Bill,
> > > > >
> > > > >> When you say "even if the application has not had data for a long
> > > time"
> > > > do
> > > > > you have a rough idea of how long?
> > > > >
> > > > > Minutes, hours
> > > > >
> > > > >> What is the value of  your
> > > > > "auto.offset.reset"  configuration?
> > > > >
> > > > > I don't specify it explicitly, but the ConsumerConfig logs indicate
> > > > > "auto.offset.reset = earliest" for all consumers the application
> > > creates.
> > > > >
> > > > > Thank you,
> > > > > Dmitry
> > > > >
> > > > >
> > > > > On Thu, Jul 20, 2017 at 8:07 PM, Bill Bejeck <[email protected]>
> > > wrote:
> > > > >
> > > > >> Hi Dmitry,
> > > > >>
> > > > >> When you say "even if the application has not had data for a long
> > > time"
> > > > do
> > > > >> you have a rough idea of how long?  What is the value of  your
> > > > >> "auto.offset.reset"  configuration?
> > > > >>
> > > > >> Thanks,
> > > > >> Bill
> > > > >>
> > > > >> On Thu, Jul 20, 2017 at 6:03 PM, Dmitry Minkovsky <
> > > [email protected]
> > > > >
> > > > >> wrote:
> > > > >>
> > > > >>> My Streams application is configured to commit offsets every
> 250ms:
> > > > >>>
> > > > >>>         Properties streamsConfig = new Properties();
> > > > >>>         streamsConfig.put(StreamsConfig.COMMIT_INTERVAL_
> MS_CONFIG,
> > > > 250);
> > > > >>>
> > > > >>>
> > > > >>> However, every time I restart my application, records that have
> > > already
> > > > >>> been processed are re-processed, even if the application has not
> > had
> > > > data
> > > > >>> for a long time.
> > > > >>>
> > > > >>> My guess is that offsets are committed only when all tasks in the
> > > > >> topology
> > > > >>> have received input. Is this what's happening?
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Thank you,
> > > > >>> Dmitry
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
>

Reply via email to