[
https://issues.apache.org/jira/browse/MAPREDUCE-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089839#comment-17089839
]
Jim Brennan commented on MAPREDUCE-7277:
----------------------------------------
Thanks for the patch [~jeagles]!
Question about readIndexFileToCache() in the case where we throw IOException if
we fail to construct the SpillRecord. In the old code, we just executed the
finally clause and then threw the IOException without doing the queue.add() and
total memory update. In the new code, we are still doing the queue.add() in
this case. Was this intended?
Also, I don't understand why you call checkTotalMemoryUsed() here - is this
left over debug code?
> IndexCache totalMemoryUsed differs from cache contents.
> -------------------------------------------------------
>
> Key: MAPREDUCE-7277
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7277
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jonathan Turner Eagles
> Assignee: Jonathan Turner Eagles
> Priority: Major
> Attachments: IndexCacheActualSize.png, MAPREDUCE-7277.001.patch
>
>
> It was observed recently in a nodemanager OOM that the memory was filled with
> SpillRecords. However, the IndexCache was only 15% full (1.5MB used on a 10MB
> configured cache size). In particular was noted that the booking variable
> totalMemoryUsed, was out of sync with the contents of the cache showing 96%
> full, thereby drastically reducing the effectiveness of the cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]