Maybe you could:
1.- Disable auto.commit on the consumer side.
2.- Consume messages, one by one in each poll.
3.- Check the record timestamp
a.- If the timestamp is >= desired window (time slot), process it and
commit offset
b.- If the timestamp is < desired window (time slot), do not process it
and do not commit offset, in the next poll the consumer will get the
message again.
Hope that helps,
--
Jonathan
On Wed, May 22, 2019 at 9:25 PM Peter Bukowinski <pmb...@gmail.com> wrote:
> I’d suggest using separate topics for messages that require delay and ones
> that do not. If you are limited to a single topic, then I’d use some other
> metadata to differentiate messages that require delayed processing from
> ones that do not. If you do not want to block the polling thread, you’ll
> need to route messages into a buffer of some sort to process them
> asynchronously.
>
> —
> Peter
>
> > On May 22, 2019, at 1:10 PM, Pavel Molchanov <
> pavel.molcha...@infodesk.com> wrote:
> >
> > This solution will block receiving polling thread for 15 minutes. Not
> good.
> >
> > What should we do if a topic has messages that should be processed
> > immediately and delayed messages at the same time?
> >
> > *Pavel Molchanov*
> >
> >
> >
> > On Wed, May 22, 2019 at 2:41 PM Peter Bukowinski <pmb...@gmail.com>
> wrote:
> >
> >> There is no out-of-the-box way to tell a consumer to not consume an
> offset
> >> until it is x minutes old. Your best bet is encode the creation time
> into
> >> the message themselves and add some processing logic into your consumer.
> >> Let’s assume your topic has a single partition or your partitions are
> keyed
> >> to guarantee message order. Your consumer could work like this in
> >> pseudo-code:
> >>
> >> consumer loop:
> >> consume message
> >> if (current time - message.timestamp) >= 15 minutes
> >> process message
> >> else
> >> sleep 15 minutes - (current time - message.timestamp)
> >> process message
> >>
> >> Since the messages enter the topic in the order they were published,
> >> pausing on the current offset should never cause a bottleneck on the
> later
> >> messages. If you fall behind, the greater than or equal to logic will
> >> prevent your consumer from pausing until it has caught up to your
> desired
> >> delay.
> >>
> >> This is a simplified scenario that may or may not map to your production
> >> use case, though.
> >>
> >> —
> >> Peter
> >>
> >>
> >>> On May 22, 2019, at 11:12 AM, Pavel Molchanov <
> >> pavel.molcha...@infodesk.com> wrote:
> >>>
> >>> Andrien,
> >>>
> >>> Thank you for asking this question! I have the same problem and wanted
> to
> >>> ask the same question. I hope that someone will answer soon.
> >>>
> >>> *Pavel Molchanov*
> >>>
> >>>
> >>>
> >>> On Wed, May 22, 2019 at 9:54 AM Adrien Ruffie <adrye...@gmail.com>
> >> wrote:
> >>>
> >>>> Hello all,
> >>>>
> >>>> I have a specific need and I don't know if a generic solution exist
> ...
> >>>> maybe you can enlighten me
> >>>>
> >>>> I need to delay each sended message about 15 mins.
> >>>> Example
> >>>> Message with offset 1 created at 2:41PM by the producer and received
> by
> >> the
> >>>> consumer at 2:56PM
> >>>> Message with offset 2 created at 2:46PM by the producer and received
> by
> >> the
> >>>> consumer at 3:01PM
> >>>> Message with offset 3 created at 2:46PM by the producer and received
> by
> >> the
> >>>> consumer at 3:01PM
> >>>> Message with offset 4 created at 3:01PM by the producer and received
> by
> >> the
> >>>> consumer at 3:16PM
> >>>> and so forth ...
> >>>>
> >>>> any option, mechanism, producer/consumer implementations already
> exist ?
> >>>>
> >>>> Thank a lot and best regards
> >>>>
> >>>> Adrian
> >>>>
> >>
> >>
>
>
--
Santilli Jonathan
