[
https://issues.apache.org/jira/browse/HDFS-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294042#comment-13294042
]
Hadoop QA commented on HDFS-3510:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531883/HDFS-3510.006.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test
files.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2649//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2649//console
This message is automatically generated.
> Fix FSEditLog pre-allocation
> ----------------------------
>
> Key: HDFS-3510
> URL: https://issues.apache.org/jira/browse/HDFS-3510
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Fix For: 1.0.0, 2.0.1-alpha
>
> Attachments: HDFS-3510-b1.001.patch, HDFS-3510-b1.002.patch,
> HDFS-3510.001.patch, HDFS-3510.003.patch, HDFS-3510.004.patch,
> HDFS-3510.004.patch, HDFS-3510.006.patch
>
>
> In the FSEditLog, we want to avoid running out of space in the middle of
> writing an edit log operation to the disk. We do this by a process called
> "preallocation"-- reserving space on the disk for the upcoming edit log
> entries before beginning to write them.
> The idea is that if we're going to encounter an out-of-disk-space condition,
> we don't want it to happen in the middle of writing valid data. Instead, we
> want it to happen in the middle of writing padding bytes. The edit log uses
> bytes with the value 0xff (in decimal, -1) as padding. These bytes
> correspond to FSEditLogOp.OP_INVALID.
> The current preallocation strategy is flawed. Although we preallocate a very
> large chunk at a time-- 1 megabyte, in fact-- we only do this preallocation
> when we are more than 4096 bytes away from the end of the file. This means
> that the effective preallocation length is only 4096 bytes. A batch of edit
> log entries could easily be more than this. There is evidence that this has
> caused problems in the field for end-users.
> Here is a visual illustration of the old preallocation strategy:
> {code}
> first write
> |
> V <----- 1 MB ----->
> +--+---------------+
> |__|FFFFFFFFFFFFFFF|
> +--+---------------+
> second write
> |
> V
> +--+------+--------+
> |__|______|FFFFFFFF|
> +--+------+--------+
> third write
> |
> V
> +--+------+------+-+
> |__|______|______|_|
> +--+------+------+-+
> fourth write
> | (NOT preallocated)
> V
> +--+------+------+-+
> |__|______|______|________
> +--+------+------+-+
> fifth write
> |
> V<--- 1 MB -->
> +--+------+------+--------+---+--------+
> |__|______|______|________|___|FFFFFFFF|
> +--+------+------+--------+---+--------+
> {code}
> And here is the new preallocation strategy:
> {code}
> first write
> |
> V
> +--+
> |__|
> +--+
> second write
> |
> V
> +--+------+
> |__|______|
> +--+------+
> third write
> |
> V
> +--+------+------+
> |__|______|______|
> +--+------+------+
> fourth write
> |
> V
> +--+------+------+--------+
> |__|______|______|________|
> +--+------+------+--------+
> fifth write
> |
> V
> +--+------+------+--------+---+
> |__|______|______|________|___|
> +--+------+------+--------+---+
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
