[
https://issues.apache.org/jira/browse/MAPREDUCE-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090762#comment-17090762
]
Eric Payne commented on MAPREDUCE-7277:
---------------------------------------
bq. In the following code, if ever mapId is in queue but not in cache, I think
totalMemoryUsed could still be out of sync
I think I've convinced myself that this risk is almost null since the
{{mapId}}'s {{IndexInfo}} will not be in the cache if {{mapId}} is not in the
queue.
I have an additional question, however.
- {{IndexCache#readIndexFileToCache}} / {{IndexCache#removeMap}}:
-- Is it possible for {{removeMap(mapId)}} to get called after
{{readIndexFileToCache}} adds {{mapId}}'s IndexInfo to the cache but before it
adds {{mapId}} to the queue? If so, the following code will leave its IndexInfo
in the cache (and in the queue after {{readIndexFileToCache}} switches back in):
{code:title=IndexCache#removeMap}
if (!queue.remove(mapId)) {
LOG.debug("Map ID {} not found in queue", mapId);
return;
}
{code}
> IndexCache totalMemoryUsed differs from cache contents.
> -------------------------------------------------------
>
> Key: MAPREDUCE-7277
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7277
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jonathan Turner Eagles
> Assignee: Jonathan Turner Eagles
> Priority: Major
> Attachments: IndexCacheActualSize.png, MAPREDUCE-7277.001.patch
>
>
> It was observed recently in a nodemanager OOM that the memory was filled with
> SpillRecords. However, the IndexCache was only 15% full (1.5MB used on a 10MB
> configured cache size). In particular was noted that the booking variable
> totalMemoryUsed, was out of sync with the contents of the cache showing 96%
> full, thereby drastically reducing the effectiveness of the cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]