Hi Steve, HDFS can be used as checkpoint storage and plays a crucial role in Flink's fault tolerance mechanism. HDFS alone will not suffice to get a Flink HA setup, though. You also need Zookeeper for JobManager HA.
Flink configuration: high-availability: zookeeper high-availability.zookeeper.quorum: <host>:<port> high-availability.cluster-id: /cluster_one # important: customize per cluster high-availability.storageDir: hdfs:///flink/recovery Regarding the steps to setup checkpointing to HDFS. You don't need to set the "fs.default-scheme". As "state.backend" you use either "rocksdb" or "filesystem". They both use HFDS for checkpoints. You can also set "env.hadoop.conf.dir" in the Flink configuration as opposed to the environment variable. The environment variable will then be set by the Flink scripts. Of course, in a basic Kubernetes setup TaskManagers and JobManager pods should both be controlled by a Deployment, so that they are brought up again after a fault. Hope this helps, and please let me know if run into any issues. Best, Konstantin On Wed, Feb 6, 2019 at 7:47 PM Steven Nelson <snel...@sourceallies.com> wrote: > I am working on a POC High Availability installation of Flink on top of > Kubernetes with HDFS as a data storage location. I am not finding much > documentation on doing this, or I am finding the documentation in parts and > maybe getting it put together correctly. I think it falls between being an > HDFS thing and a Flink thing. > > I am deploying to Kubernetes using the flink:1.7.0-hadoop27-scala_2.11 > container off of docker hub. > > I think these are the things I need to do > 1) Setup an hdfs-site.xml file per > https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Deployment > 2) Set the HADOOP_CONF_DIR environment variable to the location of that > file per > https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#hdfs > 3) Create a flink-conf.yaml file that looks something like > fs.default-scheme: hdfs:// > state.backend: rocksdb > state.savepoints.dir: hdfs://flink/savepoints > state.checkpoints.dir: hdfs://flink/checkpoints > 4) Dance a little jig when it works. > > Has anyone set this up? If so, am I missing anything? > > -Steve > > > -- Konstantin Knauf | Solutions Architect +49 160 91394525 <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
