[
https://issues.apache.org/jira/browse/HDFS-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023251#comment-13023251
]
Nathan Roberts commented on HDFS-1846:
--------------------------------------
I agree with Todd. We need to verify the latency is not impacted. The tricky
part of this testing though is that the result can depend on lots of factors
(disk subsystem on namenode, other I/O going on at the same time for whatever
reason, etc.) I don't know if it makes sense or not but it would seem to me
that if we had a way of validating each transaction was sane before processing
it, that would be sufficient. Just a simple end-of-transaction magic cookie
would probably be sufficient. I'm not sure actually pre-allocating the blocks
actually gets us that much. In fact, it can hurt because we end up physically
writing the blocks twice.
> Don't fill preallocated portion of edits log with 0x00
> ------------------------------------------------------
>
> Key: HDFS-1846
> URL: https://issues.apache.org/jira/browse/HDFS-1846
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: hdfs-1846.0.txt
>
>
> HADOOP-2330 added a feature to preallocate space in the local file system for
> the NN transaction log. That change seeks past the current end of the file
> and writes out some data, which on most systems results in the intervening
> data in the file being filled with zeros. Most underlying file systems have
> special handling for sparse files, and don't actually allocate blocks on disk
> for blocks of a file which consist completely of 0x00.
> I've seen cases in the wild where the volume an edits dir is on fills up,
> resulting in a partial final transaction being written out to disk. If you
> examine the bytes of this (now corrupt) edits file, you'll see the partial
> final transaction followed by a lot of zeros, suggesting that the
> preallocation previously succeeded before the volume ran out of space. If we
> fill the preallocated space with something other than zeros, we'd likely see
> the failure at preallocation time, rather than transaction-writing time, and
> so cause the NN to crash earlier, without a partial transaction being written
> out.
> I also hypothesize that filling the preallocated space in the edits log with
> something other than 0x00 will result in a performance improvement in NN
> throughput. I haven't tested this yet, but I intend to as part of this JIRA.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
