Hi, I think there is no support for appending to HDFS files in Flink yet. HDFS supports it, but there are some adjustments in the system required (not deleting / creating directories before writing; exposing the append() methods in the FS abstractions).
I'm planning to work on the FS abstractions in the next week, if I have enough time, I can also look into adding support for append(). Another approach could be adding support for recursively reading directories with the input formats. Vasia asked for this feature a few days ago on the mailing list. If we would have that feature, you could just write to a directory and read the parent directory (with all the dirs for the appends). Best, Robert On Thu, Dec 4, 2014 at 5:59 PM, Flavio Pompermaier <[email protected]> wrote: > Hi guys, > how can I efficiently appends data (as plain strings or also avro records) > to HDFS using Flink? > Do I need to use Flume or can I avoid it? > > Thanks in advance, > Flavio > >
