[
https://issues.apache.org/jira/browse/YARN-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13883816#comment-13883816
]
Zhijie Shen commented on YARN-1578:
-----------------------------------
It should be fine if the container is not finished. In this case, the finish
information will not be persisted into the store. Then, in the history file,
the history information entry should not exist.
However, the exception showed in the description indicates either historyData
or finishData is null. historyData cannot be null, because it's just
constructed in this method. Then, finishData is only one to be nullable, though
it shouldn't be, because mergeContainerHistoryData is called when the finish
information entry is found.
Would you please debug the following code in FileSystemApplicationHistorySever
again? Or would you please provide more log about the bug?
{code}
while ((!readStartData || !readFinishData) && hfReader.hasNext()) {
HistoryFileReader.Entry entry = hfReader.next();
if (entry.key.id.equals(containerId.toString())) {
if (entry.key.suffix.equals(START_DATA_SUFFIX)) {
ContainerStartData startData =
parseContainerStartData(entry.value);
mergeContainerHistoryData(historyData, startData);
readStartData = true;
} else if (entry.key.suffix.equals(FINISH_DATA_SUFFIX)) {
ContainerFinishData finishData =
parseContainerFinishData(entry.value);
mergeContainerHistoryData(historyData, finishData);
readFinishData = true;
}
}
}
{code}
> Fix how to handle ApplicationHistory about the container
> --------------------------------------------------------
>
> Key: YARN-1578
> URL: https://issues.apache.org/jira/browse/YARN-1578
> Project: Hadoop YARN
> Issue Type: Sub-task
> Affects Versions: YARN-321
> Reporter: Shinichi Yamashita
> Assignee: Shinichi Yamashita
> Attachments: YARN-1578.patch, screenshot.png
>
>
> I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
> After the job end and when I accessed Web UI of HistoryServer, it displayed
> "500". And HistoryServer daemon log was output as follows.
> {code}
> 2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI:
> /applicationhistory/appattempt/appattempt_1389146249925_0008_000001
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> (snip...)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
> at
> org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
> (snip...)
> {code}
> I confirmed that there was container which was not finished from
> ApplicationHistory file.
> In ResourceManager daemon log, ResourceManager reserved this container, but
> did not allocate it.
> Therefore, about a container which is not allocated, it is necessary to
> change how to handle in ApplicationHistory.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
