Yang Wang created FLINK-13633:
---------------------------------
Summary: Move submittedJobGraph and completedCheckpoint to
cluster-id subdirectory of high-availability storage
Key: FLINK-13633
URL: https://issues.apache.org/jira/browse/FLINK-13633
Project: Flink
Issue Type: New Feature
Reporter: Yang Wang
Currently, if we enable the high-availability, the ha storage directory
structure is stored as below. The submittedJobGraph and completedCheckpoint are
directly stored under the ha storage path. It is reasonable when the flink
cluster finished normally. However, when the Yarn application is failed or
killed, the submittedJobGraph and completedCheckpoint will exist there forever.
Even we could not know which flink cluster(Yarn application) they belongs to.
So i suggest to move them into application subdirectory. Some external tools
could be used to clean up these residual files.
Also, we need to do best effort clean-up before the flink cluster finishes.
Current ha storage directory structure
{code:java}
└── /tmp/flink/ha
├── submittedJobGraphxxxx
├── completedCheckpointxxxx
├── application_xxxx_xxxx
│ ├── blob{code}
The new ha storage directory structure
{code:java}
└── /tmp/flink/ha
├── application_xxxx_xxxx
│ ├── blob
│ ├── submittedJobGraphxxxx
│ ├── completedCheckpointxxxx
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
