Github user juhoautio commented on the issue:
https://github.com/apache/flink/pull/5634
@tzulitai did you ever test your code? I tried it and it allowed watermarks
to proceed but apparently too aggressively, as it caused a lot of data loss.
I'm looking for a quick fix for this issue, as it seems that FLINK-5479
won't be fixed too soon. So I would very much like to hear if you have been
able to fix this in some lighter way.
My understanding of your PR is that it doesn't work reliably because it
just seems to add an internal timeout, that could be surpassed whenever the
consumer is for example busy consuming other partitions. Please comment if this
perception is wrong.
I'm thinking that it should instead get the information that a partition
was idle from the kafka client, and only in that case (empty result from
client) create a newer watermark for that partition. It shouldn't mark the
partition to some idle state â and shouldn't create newer watermarks
periodically without any connection to another empty result from the client.
New watermarks should be only generated as a callback of the kafka client
result..?
---
