Yeah that's my observation too. Basically small chunks of late data can get
added up quickly when data is read at a faster rate.
On a related note, I would expect if there is no late data produced in
Kafka, then immaterial of what rate the data is read, this problem should
not occur.
To take
One explanation would be that during catch up, data is consumer with higher
throughput because its just read from Kafka.
Hence, you'd see also more late data per minute while the job catches up
until it reads data at the rate at which it is produced into Kafka.
Would that explain your
Thanks for the explanation. I looked at this metric closely and noticed
there are some events arriving in out of order. The hypothesis I have is,
when the job is restarted, all of the small out of order chunks add up and
show a significant number. The graph below shows the number of out of order
Thanks Sihua. If it's reset to Long.MIN_VALUE I can't explain why
outOfOrderEvents are reported. Because the event time on the data will
always be greater than Long.MIN_VALUE.
Following are the steps to reproduce this scenario.
- A source to produce events with timestamps that is increasing for
Hi,
Is it possible the watermark in TimerService not getting reset when a job
is restarted from a previous checkpoint? I would expect the watermark in a
TimerService also to go back in time.
I have the following ProcessFunction implementation.
override def processElement(
e: TraceEvent,
