Hey Ido. I haven't tried to do something like this, but we've worked out some future plans to do something similar, so I have a bit of interest in what you're saying.
I feel like there are some details left out from your post. Are your consumers able to keep up w/ the throughput without the sleep strategy in place? How are messages being published across partitions? Evenly? I assume the timestamps on the messages are always increasing? Where are the timestamps generated? By the client publishers? Are the clocks of all the publishers sync'd? Stephen On Sun, Sep 3, 2017 at 8:01 PM, Ido Barkan <i...@appsflyer.com> wrote: > Hey all. > We are trying to create a controlled lag (a lag of 30 secs). We are doing > that by inspecting a timestamp field for each msg on a specific topic and > doing an actual Thread.sleep for a computed time in the consumer thread > (until the msg is 30 secs old). We were hoping to see that eventually most > of the messages are 30 seconds old and the created lag stands for 30 secs > of processing rate. > This works for us in low throughput but on high rates we are starting to > witness strange behavior: lag on partitions is starting to grow in an > uncontrollable manner and becomes uneven per partition. > We have tried multiple variations of this: sleeping per bulk of msgs, > sleeping per msg, capping the sleep time etc but failed to resolve this. > > Has anyone tried to do this? any ideas? > > Thanks, > Ido > -- > Thanks, > Ido >
