[
https://issues.apache.org/jira/browse/SAMZA-414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136555#comment-14136555
]
Yan Fang commented on SAMZA-414:
--------------------------------
It will have problem. Here is what happens in my test:
* After every application finishes ( when the OOM happens, YARN still thinks
the application finished ), YARN aggregates all the logs in the
/var/logs/hadoop-yarn/containers/applicationId-1234567/* . ([YARN log
aggregation|http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/])
. Basically what the aggregation does is to move/combine all the files in the
/var/logs/hadoop-yarn/containers/applicationId-1234567 to the HDFS. One
container has one HDFS file.
* then you will be able to read it with _yarn logs -applicationId applicationId_
* so if we put the dump in the $SAMZA_LOG_DIR , the dump file will be merged
with all other files in the directory (stdout, stderr, samza.log) into one HDFS
file. We will not be able to read the logs with "_yarn logs -applicationId
applicationId_" because of the binary code. Also we will not be able to read
the dump file because it's already removed from the local log directory.
Solution:
Though the explanation is a little long, the solution is simple, we can put
them into "/tmp". What happens now is that:
* run
{code}
yarn logs -applicationId application_1400958081002_0087 | egrep -e "Dumping
heap" -e "Container:"
{code}
* will see
{quote}
Container: container_1400958081002_0087_01_000012 on foo.com_8041
Dumping heap to /tmp/java_pid24129.hprof ...
Container: container_1400958081002_0087_01_000023 on foo.com_8041
Dumping heap to /tmp/java_pid24521.hprof ...
{quote}
I only tested in CDH5.
> Enable HeapDumpOnOutOfMemoryError by default
> --------------------------------------------
>
> Key: SAMZA-414
> URL: https://issues.apache.org/jira/browse/SAMZA-414
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.8.0
> Reporter: Chris Riccomini
> Assignee: Chris Riccomini
> Fix For: 0.8.0
>
> Attachments: SAMZA-414-0.patch, SAMZA-414-1.patch, log-hprof.png
>
>
> It would be nice if Samza's run-class.sh defaulted to use
> -XX:+HeapDumpOnOutOfMemoryError. According to
> [this|http://stackoverflow.com/questions/542979/using-heapdumponoutofmemoryerror-parameter-for-heap-dump-for-jboss]
> post, it puts the heap dump in CWD by default, which should be fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
