[ https://issues.apache.org/jira/browse/FLINK-8531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephan Ewen resolved FLINK-8531. --------------------------------- Resolution: Fixed Fixed as of 4e481a72c1ed3cc5f177b511e5a72cd8726cf976 Consists of steps - 99495c91ecce7141ae8b2fbc96492681a9d130bd - 35c7d93ee85aa8689e804b713affa65b46af1acc - 9903c8c42793b922549835217c586c5928999ea5 - 5cc50934bdcf80ae1fa69abe69e2f214852653f9 - bb19e7f5278d43cd4fd265e3d2afa2fcc793ccf5 - 1887187f6b5c210d2091c69ef14fa8b8a5cae82c - fc21423e1f8f1a1661badef20f9c6f368f6daf8b - e0b0f45bd9c8b06bd2cda56f6859d0d3944aa00e - 4e481a72c1ed3cc5f177b511e5a72cd8726cf976 > Support separation of "Exclusive", "Shared" and "Task owned" state > ------------------------------------------------------------------ > > Key: FLINK-8531 > URL: https://issues.apache.org/jira/browse/FLINK-8531 > Project: Flink > Issue Type: Sub-task > Components: State Backends, Checkpointing > Reporter: Stephan Ewen > Assignee: Stephan Ewen > Priority: Major > Fix For: 1.5.0 > > > Currently, all state created at a certain checkpoint goes into the directory > {{chk-id}}. > With incremental checkpointing, some state is shared across checkpoint and is > referenced by newer checkpoints. That way, old {{chk-id}} directories stay > around, containing some shared chunks. That makes it both for users and > cleanup hooks hard to determine when a {{chk-x}} directory could be deleted. > The same holds for state that can only every be dropped by certain operators > on the TaskManager, never by the JobManager / CheckpointCoordinator. Examples > of that state are write ahead logs, which need to be retained until the move > to the target system is complete, which may in some cases be later then when > the checkpoint that created them is disposed. > I propose to introduce different scopes for tasks: > - **EXCLUSIVE** is for state that belongs to one checkpoint only > - **SHARED** is for state that is possibly part of multiple checkpoints > - **TASKOWNED** is for state that must never by dropped by the JobManager. > For file based checkpoint targets, I propose that we have the following > directory layout: > {code} > /user-defined-checkpoint-dir > | > + --shared/ > + --taskowned/ > + --chk-00001/ > + --chk-00002/ > + --chk-00003/ > ... > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)