[
https://issues.apache.org/jira/browse/BEAM-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Halperin updated BEAM-57:
--------------------------------
Assignee: (was: Davor Bonaci)
> FileBasedSink should write intermediate output to a different folder
> --------------------------------------------------------------------
>
> Key: BEAM-57
> URL: https://issues.apache.org/jira/browse/BEAM-57
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Daniel Halperin
> Priority: Minor
>
> Currently, FileBasedSink writes intermediate files and final output files to
> the same folder. Since deletion of intermediate files is prone to eventual
> consistency issues with GCS,S3,other systems and users may use globs in
> subsequent pipelines, intermediate files should be written to a different
> folder.
> This may be a subdirectory or a pipeline-defined temp directory (with
> probably some additional subdirectories for this source).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
