[
https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676344#action_12676344
]
rangadi edited comment on HADOOP-4584 at 2/24/09 9:39 AM:
---------------------------------------------------------------
Ideally there is no requirement for block reports. It is essentially to used as
'catch all' for various bugs and errors. (of course, it is now overloaded with
job of informing about deletions to NameNode and should be separated).
Yes, it specifically removes disk scan without fundamentally changing meaning
of block reports. Now DN informs NameNode about the the block that it thinks it
has. because :
* 'rm -r' by admin is just one form of many many things that can go wrong with
blocks on datanode. There is no perticular reason we should have this very
costly disk scan (with a global lock held) just for this.
** In fact 'rm -r' is probably the least probable error (haven't seen even
once in practice).
* We have periodic block verification that does handle various things that can
go wrong with a block (it can improve further).
** So 'rm -r' will be handled, just at the rate of rest of the block
problems.
* on the other hand many users have complained about datanode scans taking 10s
of minutes and making datanodes lose heartbeats.
** This makes the system pretty unusable and a major obstruction for
graceful degradation under load and for scalability.
** One can argue that those users should not have so many blocks. But I
think DN should still handle it to the best of it abilities and not die on them.
** Disks might be slow for many other reasons (other tasks on the machine,
etc).
* I think this is orthogonal to HADOOP-1079 since it addresses RPC and
NameNode overhead of block reports. This jira is only about DataNode side.
Yes, this is a bigger change in semantics than what we proposed earlier : to
scan the directories slowly, without holding the global lock... but offline
scan looks like a workaround for a problem that does not need to be solved. Not
scanning is much simpler than handling offline scan.
Eventually we need to reduce the frequency of block reports.. this can be done
as soon as we add acks for block deletions. This JIRA is major step in that
direction.
was (Author: rangadi):
Ideally there is no requirement for block reports. It is essentially to
used as 'catch all' for various bugs and errors. (of course, it is now
overloaded with job of informing about deletions to NameNode, this should
removed).
Yes, it specifically removes disk scan without fundamentally changing meaning
of block reports. Now DN informs NameNode about the the block that it thinks it
had. because :
* 'rm -r' by admin is just one form of many many things that can go wrong with
blocks on datanode. There is no perticular reason we should have this very
costly disk scan (with a global lock held) just for this.
** In fact 'rm -r' is probably the least probable error (haven't seen even
once in practice).
* We have periodic block verification that does handle various things that can
go wrong with a block (it can improve further).
** So 'rm -r' will be handled, just at the rate of rest of the block
problems.
* on the other hand many users have complained about datanode scans taking 10s
of minutes and making datanodes lose heartbeats.
** This makes the system pretty unusable and a major obstruction for
graceful degradation under load and for scalability.
** One can argue that those users should not have so many blocks. But I
think DN should still handle it to the best of it abilities and not die on them.
** Disks might be slow for many other reasons (other tasks on the machine,
etc).
* I think this is orthogonal to HADOOP-1079 since it addresses RPC and
NameNode overhead of block reports. This jira is only about DataNode side.
Yes, this is a bigger change in semantics than what we proposed earlier : to
scan the directories slowly, without holding the global lock... but offline
scan looks like a workaround for a problem that does not need to be solved. Not
scanning is much simpler than handling offline scan.
Eventually we need to reduce the frequency of block reports.. this can be done
as soon as we add acks for block deletions. This JIRA is major step in that
direction.
> Slow generation of blockReport at DataNode causes delay of sending heartbeat
> to NameNode
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-4584
> URL: https://issues.apache.org/jira/browse/HADOOP-4584
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Hairong Kuang
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4584.patch, 4584.patch, 4584.patch, 4584.patch,
> 4584.patch, 4584.patch
>
>
> sometimes due to disk or some other problems, datanode takes minutes or tens
> of minutes to generate a block report. It causes the datanode not able to
> send heartbeat to NameNode every 3 seconds. In the worst case, it makes
> NameNode to detect a lost heartbeat and wrongly decide that the datanode is
> dead.
> It would be nice to have two threads instead. One thread is for scanning data
> directories and generating block report, and executes the requests sent by
> NameNode; Another thread is for sending heartbeats, block reports, and
> picking up the requests from NameNode. By having these two threads, the
> sending of heartbeats will not get delayed by any slow block report or slow
> execution of NameNode requests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
