Christopher Hebert created BEAM-2751: ----------------------------------------
Summary: Write PCollection elements to individual files Key: BEAM-2751 URL: https://issues.apache.org/jira/browse/BEAM-2751 Project: Beam Issue Type: New Feature Components: sdk-java-core Reporter: Christopher Hebert Assignee: Davor Bonaci I'd like to write elements as individual files. Rather than smashing thousands of outputs into a handful of files as TextIO does (output-00000-of-00005, output-00001-of-00005,...), I want to write each element into unique files. So if I used WholeFileIO from [BEAM-2750] to read in three files (hi.txt, what.txt, and yes.txt) then I'd like to write the processed files out to individual files with user or data-defined filenames (like hi-modified.txt, what-modified.txt, and yes-modified.txt). With a WholeFileIO, this would look like: {code:java} PCollection<KV<String, Byte[]>> fileNamesAndBytes = p.apply("Read", WholeFileIO.read().from("/path/to/input/dir/*")); ... // Do stuff that change contents and file names ... modifedFileNamesAndBytes.apply("Write", WholeFileIO.write().to("/path/to/output/dir/")); {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)