[
https://issues.apache.org/jira/browse/MAPREDUCE-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090063#comment-17090063
]
Eric Payne commented on MAPREDUCE-7277:
---------------------------------------
Thanks a lot [~jeagles] for raising this issue and providing a patch. I am
still going through the test code, but here are my thoughts so far:
- IndexCache#readIndexFileToCache
-- Why is {{checkTotalMemoryUsed()}} called in the finally block? The return
value is not checked and AFAICT, it doesn't have any side effects.
- IndexCache#removeMap
-- In the following code, if {{mapId}} isn't in {{queue}}, does that
necessarily follow that it is not in {{cache}}? I think the answer is yes,
right? It only gets put in the {{queue}} once it's in the {{cache}}.
{code:java}
public void removeMap(String mapId) throws IOException {
if (!queue.remove(mapId)) {
LOG.debug("Map ID {} not found in queue", mapId);
return;
}
...
}
{code}
- IndexCache#freeIndexInformation:
-- In the following code, if ever {{mapId}} is in {{queue}} but not in
{{cache}}, I think {{totalMemoryUsed}} could still be out of sync, because by
the time freeIndexInformation is called, {{mapId}}'s indexinfo size should have
already been added to {{totalMemoryUsed}}. But that should never happen, right?
{{cache}} gets updated first and then {{queue}}, so if {{mapId}} is in
{{queue}}, it should also be in {{cache}}
{code:java}
private synchronized void freeIndexInformation() throws IOException {
while (totalMemoryUsed.get() > totalMemoryAllowed) {
String mapId = queue.remove();
IndexInformation info = cache.remove(mapId);
if (info == null) {
LOG.warn("Map ID " + mapId + " not found in cache");
continue;
}
...
}
}
{code}
> IndexCache totalMemoryUsed differs from cache contents.
> -------------------------------------------------------
>
> Key: MAPREDUCE-7277
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7277
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Jonathan Turner Eagles
> Assignee: Jonathan Turner Eagles
> Priority: Major
> Attachments: IndexCacheActualSize.png, MAPREDUCE-7277.001.patch
>
>
> It was observed recently in a nodemanager OOM that the memory was filled with
> SpillRecords. However, the IndexCache was only 15% full (1.5MB used on a 10MB
> configured cache size). In particular was noted that the booking variable
> totalMemoryUsed, was out of sync with the contents of the cache showing 96%
> full, thereby drastically reducing the effectiveness of the cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
