[
https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336852#comment-16336852
]
Dennis Huo commented on HDFS-13056:
-----------------------------------
Attached a quick and dirty poc patch which just hard-codes using the new
COMPOSITE-CRC approach and hard codes CRC32C to prototype this feature against
branch-2.8 (happen to have more 2.8 clusters handy for testing purposes).
Verified the concept works by generating random data, putting in HDFS with
different block sizes and chunk sizes, verified different MD5MD5CRC checksums,
and then verified that the composite CRC returns the same file-level value for
all.
Before:
{code:java}
$ hadoop fs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest.dat
$ hadoop fs -cp gs://hadoop-cloud-dev-dhuo/random-crctest.dat
hdfs:///tmp/random-crctest2.dat
$ hadoop fs -Ddfs.bytes-per-checksum=1024 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest3.dat
$ hadoop fs -Ddfs.blocksize=67108864 -cp
gs://hadoop-cloud-dev-dhuo/random-crctest.dat hdfs:///tmp/random-crctest4.dat
$ hadoop fs -checksum hdfs:///tmp/random-crctest*.dat
hdfs:///tmp/random-crctest.dat MD5-of-262144MD5-of-512CRC32C
000002000000000000040000c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest2.dat MD5-of-262144MD5-of-512CRC32C
000002000000000000040000c0baeeacbc4b5a3c8af5152944fe2d79
hdfs:///tmp/random-crctest3.dat MD5-of-131072MD5-of-1024CRC32C
000004000000000000020000930b0d7ad333786a839b044ed8d18d2d
hdfs:///tmp/random-crctest4.dat MD5-of-131072MD5-of-512CRC32C
0000020000000000000200008baa940ef6ed21fb4bd6224ce917d127
{code}
After:
{code:java}
$ hadoop fs -checksum hdfs:///tmp/random-crctest*.dat
hdfs:///tmp/random-crctest.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest2.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest3.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
hdfs:///tmp/random-crctest4.dat COMPOSITE-CRC32C
4db86e2b00000000000000000000000000000000000000000000000000000000
{code}
> Expose file-level composite CRCs in HDFS which are comparable across
> different instances/layouts
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-13056
> URL: https://issues.apache.org/jira/browse/HDFS-13056
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, distcp, erasure-coding, federation, hdfs
> Affects Versions: 3.0.0
> Reporter: Dennis Huo
> Priority: Major
> Attachments: HDFS-13056-branch-2.8.poc1.patch,
> hdfs-file-composite-crc32-v1.pdf
>
>
> FileChecksum was first introduced in
> [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then
> has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are
> already stored as part of datanode metadata, and the MD5 approach is used to
> compute an aggregate value in a distributed manner, with individual datanodes
> computing the MD5-of-CRCs per-block in parallel, and the HDFS client
> computing the second-level MD5.
>
> A shortcoming of this approach which is often brought up is the fact that
> this FileChecksum is sensitive to the internal block-size and chunk-size
> configuration, and thus different HDFS files with different block/chunk
> settings cannot be compared. More commonly, one might have different HDFS
> clusters which use different block sizes, in which case any data migration
> won't be able to use the FileChecksum for distcp's rsync functionality or for
> verifying end-to-end data integrity (on top of low-level data integrity
> checks applied at data transfer time).
>
> This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430
> during the addition of checksum support for striped erasure-coded files;
> while there was some discussion of using CRC composability, it still
> ultimately settled on hierarchical MD5 approach, which also adds the problem
> that checksums of basic replicated files are not comparable to striped files.
>
> This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses
> CRC composition to remain completely chunk/block agnostic, and allows
> comparison between striped vs replicated files, between different HDFS
> instances, and possible even between HDFS and other external storage systems.
> This feature can also be added in-place to be compatible with existing block
> metadata, and doesn't need to change the normal path of chunk verification,
> so is minimally invasive. This also means even large preexisting HDFS
> deployments could adopt this feature to retroactively sync data. A detailed
> design document can be found here:
> https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
