bbejeck opened a new pull request, #22832:
URL: https://github.com/apache/kafka/pull/22832
After a StreamThread failure (thread replaced, task reassigned to another
thread on the same instance), the background cleaner
({{StateDirectory.cleanRemovedTasks}}) can delete the task's still-valid,
recently-committed state directory in the window between the failed thread
releasing its lock and the new thread re-acquiring it. The reassigned task then
finds no local offsets and restores its entire changelog from scratch; for a
compacted changelog whose head is being removed by retention, the
from-beginning restore can possibly be overtaken mid-flight →
{{OffsetOutOfRangeException}} → {{TaskCorruptedException}}.
Applies when the dirty close does not wipe local state — i.e. ALOS, or EOS
with transactional state stores (KIP-892). (Under EOS with non-transactional
stores the dirty close wipes at close, so this path isn't hit.)
The cleaner judges obsolescence by {{now - taskDir.lastModified() >
state.cleanup.delay.ms}}. Pre-KIP-1035 the per-commit {{.checkpoint}} file (a
direct child of the task dir,
written by rename) refreshed that mtime on every commit; KIP-1035 moved
offsets into RocksDB and writes {{.checkpoint}} only on downgrade, so nothing
refreshes the task-dir mtime during processing (RocksDB writes land in the
{{rocksdb/<store>/}} subdirectory, which doesn't bump the parent's mtime;
per-task locking is in-memory, so there's no {{.lock}} file either). The
directory looks arbitrarily stale while its task is actively committing, so the
moment its lock is released it is immediately eligible for deletion instead of
after the intended grace period.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
