[ https://issues.apache.org/jira/browse/BEAM-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593439#comment-16593439 ]
Tim Robertson edited comment on BEAM-5036 at 8/27/18 10:09 AM: --------------------------------------------------------------- Thanks [~reuvenlax] 1. Adding a cross FS check seems reasonable as a precaution. 2. Please see [this comment|https://issues.apache.org/jira/browse/BEAM-4861?focusedCommentId=16593406&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16593406] on BEAM-4861 where we have a decision to make on the HDFS parent directory not existing. Appreciate your and [~JozoVilcek] thoughts on that (and others). was (Author: timrobertson100): Thanks [~reuvenlax] 1. Adding a cross FS check seems reasonable as a precaution. 2. Please see this comment on BEAM-4861 where we have a decision to make on the HDFS parent directory not existing. Appreciate your and [~JozoVilcek] thoughts on that (and others). > Optimize FileBasedSink's WriteOperation.moveToOutput() > ------------------------------------------------------ > > Key: BEAM-5036 > URL: https://issues.apache.org/jira/browse/BEAM-5036 > Project: Beam > Issue Type: Improvement > Components: io-java-files > Affects Versions: 2.5.0 > Reporter: Jozef Vilcek > Assignee: Tim Robertson > Priority: Major > > moveToOutput() methods in FileBasedSink.WriteOperation implements move by > copy+delete. It would be better to use a rename() which can be much more > effective for some filesystems. > Filesystem must support cross-directory rename. BEAM-4861 is related to this > for the case of HDFS filesystem. > Feature was discussed here: > http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)