nicktelford opened a new pull request, #22837: URL: https://github.com/apache/kafka/pull/22837
`OffsetOutOfRangeException` was observed during state restore in long-running soak tests. A task's state directory was deleted by the state directory cleaner roughly one second after the owning thread died and released the directory's lock, even though the task had been committing right up until then. When the task was reassigned it found an empty directory, defaulted to the changelog start offset, and a seek-to-beginning restore of a retention-trimmed changelog raced retention and produced an out-of-range fetch. The cleaner (`StateDirectory#cleanRemovedTasks`) deletes an unowned task directory once its filesystem last-modified time is older than `state.cleanup.delay.ms`. Now that changelog offsets are managed by the state stores rather than a per-task `.checkpoint` file, nothing rewrites a direct child of the task directory on commit, so an actively-committing task's directory mtime no longer advances — it stays frozen at creation time. The moment the task's lock is released, the directory looks obsolete to the cleaner and becomes eligible for immediate deletion. This refreshes the task directory's last-modified time when the state manager closes, marking the point at which the task is released. The cleaner's grace period is then measured from release, so a just-released directory survives long enough to be reassigned. The touch is best-effort (`File#setLastModified` can fail on some filesystems) and only bumps a directory that still exists. Closing the state manager is the natural place for this: it runs on every genuine task release (clean and dirty close, active and standby), is never invoked by the cleaner itself, and already holds a reference to the task directory. To keep the timestamp testable, a `Time` instance is injected into `ProcessorStateManager` (and threaded through the task creators) rather than reading the system clock directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
