Daniel Halperin created BEAM-57:
-----------------------------------
Summary: FileBasedSink should write intermediate output to a
different folder
Key: BEAM-57
URL: https://issues.apache.org/jira/browse/BEAM-57
Project: Beam
Issue Type: New Feature
Components: sdk-java-core
Reporter: Daniel Halperin
Assignee: Davor Bonaci
Priority: Minor
Currently, FileBasedSink writes intermediate files and final output files to
the same folder. Since deletion of intermediate files is prone to eventual
consistency issues with GCS,S3,other systems and users may use globs in
subsequent pipelines, intermediate files should be written to a different
folder.
This may be a subdirectory or a pipeline-defined temp directory (with probably
some additional subdirectories for this source).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
