[
https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596979#comment-16596979
]
Ryan Williams commented on BEAM-5036:
-------------------------------------
fwiw, on the "write directly to final files" question, there is a trick using
multipart-upload APIs to upload a bunch of files (each as the sole "part" of a
"multipart" upload), and then do a final pass "committing" all those multipart
uploads; see [rblue/s3committer|https://github.com/rdblue/s3committer]. AFAIK
this could be used in GCS as well.
If anything fails during the initial uploads, you can refrain from doing the
"commit" pass altogether (and nothing will end up in the cloud store), or only
retry the failed files.
IIUC, as long as your final "commit" RPCs either all happen or all don't (which
is likely because they are effectively no-ops?), you should be fine.
> Optimize FileBasedSink's WriteOperation.moveToOutput()
> ------------------------------------------------------
>
> Key: BEAM-5036
> URL: https://issues.apache.org/jira/browse/BEAM-5036
> Project: Beam
> Issue Type: Improvement
> Components: io-java-files
> Affects Versions: 2.5.0
> Reporter: Jozef Vilcek
> Assignee: Tim Robertson
> Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> moveToOutput() methods in FileBasedSink.WriteOperation implements move by
> copy+delete. It would be better to use a rename() which can be much more
> effective for some filesystems.
> Filesystem must support cross-directory rename. BEAM-4861 is related to this
> for the case of HDFS filesystem.
> Feature was discussed here:
> http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
