Hi everyone, I'm trying to understand how StateStores work internally for some changes that I plan to propose, and I'd like some clarification around checkpoint files and position files.
It appears as though position files are relatively new, and were created as part of the IQv2 initiative, as a means to track the position of the local state store so that reads could be bound by particular positions? Checkpoint files look much older, and are managed by the Task itself (actually, ProcessorStateManager). It looks like this is used exclusively for determining a) whether to restore a store, and b) which offsets to restore from? If I've understood the above correctly, is there any scope to potentially replace checkpoint files with StateStore#position()? Regards, Nick
