There seems to be a lot of oddities with the auto offset committer and the watermark management as well as kafka offsets in general.
One issue I keep having is the auto committer will just not commit any offsets. So the topic will look like it's backing up. From what I've been able to trace on it it appears to be in relation to the executor/thread shutting down before the auto commit has a chance to run. Even though the min read times are set. It still prematurely shuts down. Turning auto commit interval down seems to help but doesn't resolve the issue. Just seems to allow it to correct itself much quicker. Another I just had happen is after restarting a pipeline the auto committed offsets reset to the earliest record and the pipeline appears to be working on those records. Which is odd in contrary to a lot of things. When I shut the pipeline down it was only a few thousand records behind. The consumer is configured to start at the latest offset not the earliest. Give that It would appear the recorded watermarks had an odd corruption or something where they believed they where in the past. -- Nick Verbeck - NerdyNick ---------------------------------------------------- NerdyNick.com Coloco.ubuntu-rocks.org
