Yang Jie created HADOOP-20059:
---------------------------------
Summary: AbstractFileSystem#create skips the
blockSize/bytesPerChecksum check unless the BytesPerChecksum option is set
Key: HADOOP-20059
URL: https://issues.apache.org/jira/browse/HADOOP-20059
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Yang Jie
AbstractFileSystem#create rejects a block size that is not a multiple of the
checksum size with {{if (blockSize % bytesPerChecksum != 0)}}.
{{bytesPerChecksum}} is a local variable that stays at -1 unless the caller
passes {{CreateOpts.bytesPerChecksum}}, and {{x % -1}} is always 0, so the
check never fires in the usual case. HADOOP-8240 removed the line that used to
fill in the server default and left the check reading the old variable.
On HDFS, {{FileContext.create(p, EnumSet.of(CREATE, OVERWRITE),
CreateOpts.blockSize(1000))}} gets past AbstractFileSystem and the NameNode
creates the file, deleting the existing one because of OVERWRITE. Only then
does the DFSOutputStream constructor throw "bytesPerChecksum ... must divide
block size". The caller gets an exception, the old file is gone, and an empty
file stays under construction until lease recovery.
The fix checks against {{checksumOpt.getBytesPerChecksum()}}, the value the
stream is created with, so the create fails before it reaches the file system.
On file systems that ignore the block size (local, object stores),
FileContext#create with a block size that is not a multiple of the checksum
size (512 by default) now fails, as it did before HADOOP-8240.
DistributedFileSystem#create has the same create-then-check order inside
DFSClient; that needs a separate HDFS change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
