[ https://issues.apache.org/jira/browse/HADOOP-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Masatake Iwasaki updated HADOOP-13091: -------------------------------------- Target Version/s: 2.10.2 (was: 2.10.1) > DistCp masks potential CRC check failures > ----------------------------------------- > > Key: HADOOP-13091 > URL: https://issues.apache.org/jira/browse/HADOOP-13091 > Project: Hadoop Common > Issue Type: Bug > Affects Versions: 2.7.1 > Reporter: Elliot West > Assignee: Yiqun Lin > Priority: Major > Labels: distcp > Attachments: HADOOP-13091.003.patch, HADOOP-13091.004.patch, > HDFS-10338.001.patch, HDFS-10338.002.patch > > > There appear to be edge cases whereby CRC checks may be circumvented when > requests for checksums from the source or target file system fail. In this > event CRCs could differ between the source and target and yet the DistCp copy > would succeed, even when the 'skip CRC check' option is not being used. > The code in question is contained in the method > [{{org.apache.hadoop.tools.util.DistCpUtils#checksumsAreEqual(...)}}|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/DistCpUtils.java#L457] > Specifically this code block suggests that if there is a failure when trying > to read the source or target checksum then the method will return {{true}} > (i.e. the checksums are equal), implying that the check succeeded. In actual > fact we just failed to obtain the checksum and could not perform the check. > {code} > try { > sourceChecksum = sourceChecksum != null ? sourceChecksum : > sourceFS.getFileChecksum(source); > targetChecksum = targetFS.getFileChecksum(target); > } catch (IOException e) { > LOG.error("Unable to retrieve checksum for " + source + " or " > + target, e); > } > return (sourceChecksum == null || targetChecksum == null || > sourceChecksum.equals(targetChecksum)); > {code} > I believe that at the very least the caught {{IOException}} should be > re-thrown. If this is not deemed desirable then I believe an option > ({{--strictCrc}}?) should be added to enforce a strict check where we require > that both the source and target CRCs are retrieved, are not null, and are > then compared for equality. If for any reason either of the CRCs retrievals > fail then an exception is thrown. > Clearly some {{FileSystems}} do not support CRCs and invocations to > {{FileSystem.getFileChecksum(...)}} return {{null}} in these instances. I > would suggest that these should fail a strict CRC check to prevent users > developing a false sense of security in their copy pipeline. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org