[ https://issues.apache.org/jira/browse/HDFS-10338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269909#comment-15269909 ]
Ravi Prakash commented on HDFS-10338: ------------------------------------- {{ignoreFailures}} would be a tragically confusing and misnamed variable. I would propose something like {{ignoreCRCerrors}} > DistCp masks potential CRC check failures > ----------------------------------------- > > Key: HDFS-10338 > URL: https://issues.apache.org/jira/browse/HDFS-10338 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp > Affects Versions: 2.7.1 > Reporter: Elliot West > Assignee: Lin Yiqun > Attachments: HDFS-10338.001.patch, HDFS-10338.002.patch > > > There appear to be edge cases whereby CRC checks may be circumvented when > requests for checksums from the source or target file system fail. In this > event CRCs could differ between the source and target and yet the DistCp copy > would succeed, even when the 'skip CRC check' option is not being used. > The code in question is contained in the method > [{{org.apache.hadoop.tools.util.DistCpUtils#checksumsAreEqual(...)}}|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L457] > Specifically this code block suggests that if there is a failure when trying > to read the source or target checksum then the method will return {{true}} > (i.e. the checksums are equal), implying that the check succeeded. In actual > fact we just failed to obtain the checksum and could not perform the check. > {code} > try { > sourceChecksum = sourceChecksum != null ? sourceChecksum : > sourceFS.getFileChecksum(source); > targetChecksum = targetFS.getFileChecksum(target); > } catch (IOException e) { > LOG.error("Unable to retrieve checksum for " + source + " or " > + target, e); > } > return (sourceChecksum == null || targetChecksum == null || > sourceChecksum.equals(targetChecksum)); > {code} > I believe that at the very least the caught {{IOException}} should be > re-thrown. If this is not deemed desirable then I believe an option > ({{--strictCrc}}?) should be added to enforce a strict check where we require > that both the source and target CRCs are retrieved, are not null, and are > then compared for equality. If for any reason either of the CRCs retrievals > fail then an exception is thrown. > Clearly some {{FileSystems}} do not support CRCs and invocations to > {{FileSystem.getFileChecksum(...)}} return {{null}} in these instances. I > would suggest that these should fail a strict CRC check to prevent users > developing a false sense of security in their copy pipeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org