What is the right way to use the "-crc" option with hadoop dfs -copyToLocal?
Is this the wrong list? --Tom On Tue, Jan 28, 2014 at 11:53 AM, Tom Brown <tombrow...@gmail.com> wrote: > I am archiving a large amount of data out of my HDFS file system to a > separate shared storage solution (There is not much HDFS space left in my > cluster, and upgrading it is not an option right now). > > I understand that HDFS internally manages checksums and won't succeed if > the data doesn't match the CRC, so I'm not worried about corruption when > reading from HDFS. > > However, I want to store the HDFS crc calculations alongside the data > files after exporting them. I thought the "hadoop dfs -copyToLocal -crc > <hdfs-source> <local-dest>" command would work, but it always gives me the > error "-crc option is not valid when source file system does not have crc > files" > > Can someone explain what exactly that option does, and when (if ever) it > should be used? > > Thanks in advance! > > --Tom >
