Till Rohrmann created FLINK-1268:
------------------------------------
Summary: FileOutputFormat with overwrite does not clear local
output directories
Key: FLINK-1268
URL: https://issues.apache.org/jira/browse/FLINK-1268
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
I noticed that the FileOutputFormat does not clear the output directories if it
writes to local disk. This has the consequence that previous partitions are
still contained in the directory if one decreases the DOP between subsequent
runs. If one reads the data from this directory, then more partitions will be
read in than were actually written. This can lead to a wrong user code
behaviour which is hard to debug. I'm aware that in case of a distributed
execution the TaskManagers or the Tasks have to be responsible for the cleanup
and if multiple Tasks are running on a TaskManager, then the cleanup has to be
coordinated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
