On Fri, Sep 30, 2016 at 9:12 AM, Konstantin Knauf <konstantin.kn...@tngtech.com> wrote: > we are running a Flink (1.1.2) Stand-Alone Cluster with JM HA, and HDFS > as checkpoint and recovery storage dir. What we see is that blobStores > are stored in HDFS as well as under the local Jobmanagers and > Taskmanagers /tmp directory. > > Is this the expected behaviour? Is there any documentation on which > blobs are stored locally and which are stored in HDFS in our case? In > particular, we would need to know when it is save to delete blobs stored > locally because there are not cleanup up by Flink and fill up the /tmp > partition eventually.
BLOBs are copied to another directory in case of HA in order to be available for other job managers that might take over. On regular termination (cancel, finish) all BLOBs should be cleaned up. With hard failures, it can happen that BLOBs are not cleaned up. Do you know in which cases you see BLOBs not being cleaned up? If it is the first one, that sounds like a bug to me. – Ufuk
