[
https://issues.apache.org/jira/browse/HDFS-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777414#action_12777414
]
Ning Zhang commented on HDFS-767:
---------------------------------
Todd, you are right if we want to establish the time windows at the point of
the first request. And your calculated time is the total wait time. My formula
consider the next wait time which is 0..3 for the first attempt, 3..9 for 2nd
attempt and 6..15 for 3rd attempt. The total wait time range is 0..3, 3..12,
and 9..27 for the first 3 attempts.
One reason for my formula is its simplicity, we don't need to keep track of the
first trial time and current time. It only depends on 1 parameter (the # of
failures). The second reason is that we don't know how much time will be spent
from sending out the request to getting a failure. it could be tens to hundreds
of milliseconds due to network roundtrip and NameNode workload. We don't want
to add that into the consideration.
> Job failure due to BlockMissingException
> ----------------------------------------
>
> Key: HDFS-767
> URL: https://issues.apache.org/jira/browse/HDFS-767
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ning Zhang
>
> If a block is request by too many mappers/reducers (say, 3000) at the same
> time, a BlockMissingException is thrown because it exceeds the upper limit (I
> think 256 by default) of number of threads accessing the same block at the
> same time. The DFSClient wil catch that exception and retry 3 times after
> waiting for 3 seconds. Since the wait time is a fixed value, a lot of clients
> will retry at about the same time and a large portion of them get another
> failure. After 3 retries, there are about 256*4 = 1024 clients got the block.
> If the number of clients are more than that, the job will fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
