[ https://issues.apache.org/jira/browse/FLINK-8531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephan Ewen closed FLINK-8531. ------------------------------- > Support separation of "Exclusive", "Shared" and "Task owned" state > ------------------------------------------------------------------ > > Key: FLINK-8531 > URL: https://issues.apache.org/jira/browse/FLINK-8531 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing > Reporter: Stephan Ewen > Assignee: Stephan Ewen > Priority: Major > Fix For: 1.5.0 > > > Currently, all state created at a certain checkpoint goes into the directory > {{chk-id}}. > With incremental checkpointing, some state is shared across checkpoint and is > referenced by newer checkpoints. That way, old {{chk-id}} directories stay > around, containing some shared chunks. That makes it both for users and > cleanup hooks hard to determine when a {{chk-x}} directory could be deleted. > The same holds for state that can only every be dropped by certain operators > on the TaskManager, never by the JobManager / CheckpointCoordinator. Examples > of that state are write ahead logs, which need to be retained until the move > to the target system is complete, which may in some cases be later then when > the checkpoint that created them is disposed. > I propose to introduce different scopes for tasks: > - **EXCLUSIVE** is for state that belongs to one checkpoint only > - **SHARED** is for state that is possibly part of multiple checkpoints > - **TASKOWNED** is for state that must never by dropped by the JobManager. > For file based checkpoint targets, I propose that we have the following > directory layout: > {code} > /user-defined-checkpoint-dir > | > + --shared/ > + --taskowned/ > + --chk-00001/ > + --chk-00002/ > + --chk-00003/ > ... > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)