[
https://issues.apache.org/jira/browse/HADOOP-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558942#action_12558942
]
dhruba borthakur commented on HADOOP-2576:
------------------------------------------
There is a command that dumps namenode internal data structures to a log file.
When this problem ocurs, can you pl run it as
bin/hadoop dfsadmin -metasave "filename".
The specified filename will be created in the namenode's log directory. This
file will list blocks that are waiting to be replicated as well as blocks
waiting to be deleted. Using this tool we can determine if the namenode is not
purging the list of blocks to be invalidated.
> Namenode performance degradation over time
> ------------------------------------------
>
> Key: HADOOP-2576
> URL: https://issues.apache.org/jira/browse/HADOOP-2576
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Christian Kunz
> Priority: Blocker
> Fix For: 0.16.0
>
>
> We have a cluster running the same applications again and again with a high
> turnover of files.
> The performance of these applications seem to be correlated to the lifetime
> of the namenode:
> After starting the namenode, the applications need increasingly more time to
> complete, with about 50% more time after 1 week.
> During that time the namenode average cpu usage increases from typically 10%
> to 30%, memory usage nearly doubles (although the average amount of data on
> dfs stays the same), and the average load factor increases by a factor of 2-3
> (although not significantly high, <2).
> When looking at the namenode and datanode logs, I see a lot of asks to delete
> blocks coming from the namenode for blocks not in the blockmap of the
> datanodes, repeatedly for the same blocks.
> When I counted the number of blocks asked by the namenode to be deleted, I
> noticed a noticeable increase with the lifetime of the namenode (a factor of
> 2-3 after 1 week).
> This makes me wonder whether the namenode does not purge the list of invalid
> blocks from non-existing blocks.
> But independently, the namenode has a degradation issue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
