[ https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590437#comment-16590437 ]
Reuven Lax commented on BEAM-5036: ---------------------------------- As mentioned, Beam currently allows the temp directory to be different than the target directory. I believe that HDFS does support this, so we only have to handle the case where the target directory doesn't exist. I'm not sure if all filesystems support this though. The API as provided allows users to put the temp files in a different filesystem than the target files (i.e. GCS v.s. HDFS). However in practice we don't support this, and should probably warn or fail if the user attempts to do this. > Optimize FileBasedSink's WriteOperation.moveToOutput() > ------------------------------------------------------ > > Key: BEAM-5036 > URL: https://issues.apache.org/jira/browse/BEAM-5036 > Project: Beam > Issue Type: Improvement > Components: io-java-files > Affects Versions: 2.5.0 > Reporter: Jozef Vilcek > Assignee: Tim Robertson > Priority: Major > > moveToOutput() methods in FileBasedSink.WriteOperation implements move by > copy+delete. It would be better to use a rename() which can be much more > effective for some filesystems. > Filesystem must support cross-directory rename. BEAM-4861 is related to this > for the case of HDFS filesystem. > Feature was discussed here: > http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)