[ https://issues.apache.org/jira/browse/HDFS-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13025013#comment-13025013 ]
Eli Collins commented on HDFS-1846: ----------------------------------- @atm - are these results from a system using an ssd? Would be interesting to see the same numbers on a spinning disk. > Don't fill preallocated portion of edits log with 0x00 > ------------------------------------------------------ > > Key: HDFS-1846 > URL: https://issues.apache.org/jira/browse/HDFS-1846 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node > Affects Versions: 0.23.0 > Reporter: Aaron T. Myers > Assignee: Aaron T. Myers > Attachments: hdfs-1846-perf-analysis.0.patch, hdfs-1846.0.txt > > > HADOOP-2330 added a feature to preallocate space in the local file system for > the NN transaction log. That change seeks past the current end of the file > and writes out some data, which on most systems results in the intervening > data in the file being filled with zeros. Most underlying file systems have > special handling for sparse files, and don't actually allocate blocks on disk > for blocks of a file which consist completely of 0x00. > I've seen cases in the wild where the volume an edits dir is on fills up, > resulting in a partial final transaction being written out to disk. If you > examine the bytes of this (now corrupt) edits file, you'll see the partial > final transaction followed by a lot of zeros, suggesting that the > preallocation previously succeeded before the volume ran out of space. If we > fill the preallocated space with something other than zeros, we'd likely see > the failure at preallocation time, rather than transaction-writing time, and > so cause the NN to crash earlier, without a partial transaction being written > out. > I also hypothesize that filling the preallocated space in the edits log with > something other than 0x00 will result in a performance improvement in NN > throughput. I haven't tested this yet, but I intend to as part of this JIRA. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira