[ https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678768#comment-17678768 ]
ASF GitHub Bot commented on HADOOP-18596: ----------------------------------------- dannycjones commented on code in PR #5308: URL: https://github.com/apache/hadoop/pull/5308#discussion_r1081452359 ########## hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java: ########## @@ -354,7 +354,14 @@ private boolean canSkip(FileSystem sourceFS, CopyListingFileStatus source, boolean sameLength = target.getLen() == source.getLen(); boolean sameBlockSize = source.getBlockSize() == target.getBlockSize() || !preserve.contains(FileAttribute.BLOCKSIZE); - if (sameLength && sameBlockSize) { + // checksum check to be done if same file len(greater than 0), same block + // size and the target file has been updated more recently than the source + // file. + // Note: For Different cloud stores with different checksum algorithms, + // checksum comparisons are not performed so we would be depending on the + // file size and modification time. + if (sameLength && (source.getLen() > 0) && sameBlockSize && + source.getModificationTime() < target.getModificationTime()) { Review Comment: Why the addition of the `getLen() > 0`? We want to always copy if its an empty file? > Distcp -update between different cloud stores to use modification time while > checking for file skip. > ---------------------------------------------------------------------------------------------------- > > Key: HADOOP-18596 > URL: https://issues.apache.org/jira/browse/HADOOP-18596 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp > Reporter: Mehakmeet Singh > Assignee: Mehakmeet Singh > Priority: Major > Labels: pull-request-available > > Distcp -update currently relies on File size, block size, and Checksum > comparisons to figure out which files should be skipped or copied. > Since different cloud stores have different checksum algorithms we should > check for modification time as well to the checks. > This would ensure that while performing -update if the files are perceived to > be out of sync we should copy them. The machines between which the file > transfers occur should be in time sync to avoid any extra copies. > Improving testing and documentation for modification time checks between > different object stores to ensure no incorrect skipping of files. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org