Boris Lublinsky created FLINK-12751:
---------------------------------------
Summary: Create file based HA support
Key: FLINK-12751
URL: https://issues.apache.org/jira/browse/FLINK-12751
Project: Flink
Issue Type: Improvement
Components: FileSystems
Affects Versions: 1.8.0, 1.9.0, 2.0.0
Environment: Flink on k8 and Mini cluster
Reporter: Boris Lublinsky
In the current Flink implementation, HA support can be implemented either using
Zookeeper or Custom Factory class.
Add HA implementation based on PVC. The idea behind this implementation
is as follows:
* Because implementation assumes a single instance of Job manager (Job manager
selection and restarts are done by K8 Deployment of 1)
URL management is done using StandaloneHaServices implementation (in the case
of cluster) and EmbeddedHaServices implementation (in the case of mini cluster)
* For management of the submitted Job Graphs, checkpoint counter and completed
checkpoint an implementation is leveraging the following file system layout
````
ha -----> root of the HA data
checkpointcounter -----> checkpoint counter folder
<job ID> -----> job id folder
<counter file> -----> counter file
<another job ID> -----> another job id folder
...........
completedCheckpoint -----> completed checkpoint folder
<job ID> -----> job id folder
<checkpoint file> -----> checkpoint file
<another checkpoint file> -----> checkpoint file
...........
<another job ID> -----> another job id folder
...........
submittedJobGraph -----> submitted graph folder
<job ID> -----> job id folder
<graph file> -----> graph file
<another job ID> -----> another job id folder
...........
````
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
