I’d suggest using separate topics for messages that require delay and ones that do not. If you are limited to a single topic, then I’d use some other metadata to differentiate messages that require delayed processing from ones that do not. If you do not want to block the polling thread, you’ll need to route messages into a buffer of some sort to process them asynchronously.
— Peter > On May 22, 2019, at 1:10 PM, Pavel Molchanov <pavel.molcha...@infodesk.com> > wrote: > > This solution will block receiving polling thread for 15 minutes. Not good. > > What should we do if a topic has messages that should be processed > immediately and delayed messages at the same time? > > *Pavel Molchanov* > > > > On Wed, May 22, 2019 at 2:41 PM Peter Bukowinski <pmb...@gmail.com> wrote: > >> There is no out-of-the-box way to tell a consumer to not consume an offset >> until it is x minutes old. Your best bet is encode the creation time into >> the message themselves and add some processing logic into your consumer. >> Let’s assume your topic has a single partition or your partitions are keyed >> to guarantee message order. Your consumer could work like this in >> pseudo-code: >> >> consumer loop: >> consume message >> if (current time - message.timestamp) >= 15 minutes >> process message >> else >> sleep 15 minutes - (current time - message.timestamp) >> process message >> >> Since the messages enter the topic in the order they were published, >> pausing on the current offset should never cause a bottleneck on the later >> messages. If you fall behind, the greater than or equal to logic will >> prevent your consumer from pausing until it has caught up to your desired >> delay. >> >> This is a simplified scenario that may or may not map to your production >> use case, though. >> >> — >> Peter >> >> >>> On May 22, 2019, at 11:12 AM, Pavel Molchanov < >> pavel.molcha...@infodesk.com> wrote: >>> >>> Andrien, >>> >>> Thank you for asking this question! I have the same problem and wanted to >>> ask the same question. I hope that someone will answer soon. >>> >>> *Pavel Molchanov* >>> >>> >>> >>> On Wed, May 22, 2019 at 9:54 AM Adrien Ruffie <adrye...@gmail.com> >> wrote: >>> >>>> Hello all, >>>> >>>> I have a specific need and I don't know if a generic solution exist ... >>>> maybe you can enlighten me >>>> >>>> I need to delay each sended message about 15 mins. >>>> Example >>>> Message with offset 1 created at 2:41PM by the producer and received by >> the >>>> consumer at 2:56PM >>>> Message with offset 2 created at 2:46PM by the producer and received by >> the >>>> consumer at 3:01PM >>>> Message with offset 3 created at 2:46PM by the producer and received by >> the >>>> consumer at 3:01PM >>>> Message with offset 4 created at 3:01PM by the producer and received by >> the >>>> consumer at 3:16PM >>>> and so forth ... >>>> >>>> any option, mechanism, producer/consumer implementations already exist ? >>>> >>>> Thank a lot and best regards >>>> >>>> Adrian >>>> >> >>
