Hi Rahul,
>From the given log, I do not think YARN is killing containers due to memory
issue. Usage is under the limits. However full log is not shared, so you
can verify that when the AM launch is failed whether memory was under limit
or not.
Which application are you trying to run?
Also its better if we have "application master container" log. *sysout* or
*syserr* of that launch will have some more information.
Thanks
Sunil
On Fri, Jul 29, 2016 at 12:49 PM Rahul Chhiber <
rahul.chhi...@cumulus-systems.com> wrote:
> Hi all,
>
>
>
> I have launched an application on yarn cluster which has following config.
>
> Master (Resource Manager) - 16GB RAM + 8 vCPU
>
> Slave 1 (Node manager 1) - 8GB RAM + 4 vCPU
>
>
>
> Intermittently AM(2GB, 1 core) is exiting with code - 2 with the following
> trace. I am not able to find anything about exit code 2.
>
>
>
> Last log is
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Memory usage of ProcessTree 22504 for container-id
> container_1469709900068_0002_01_01: 203.8 MB of 2 GB physical memory
> used; 2.8 GB of 4.2 GB virtual memory used
>
>
>
> Does this have anything to do with my application logic or Is it possible
> that it is killed because of exceeding the memory limits?
>
>
>
> 2016-07-28 17:08:50,672 WARN
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Exception from container-launch with container ID:
> container_1469709900068_0002_01_01 and exit code: 2
>
> ExitCodeException exitCode=2:
>
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
>
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-07-28 17:08:50,674 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from
> container-launch.
>
> 2016-07-28 17:08:50,674 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id:
> container_1469709900068_0002_01_01
>
> 2016-07-28 17:08:50,674 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 2
>
> 2016-07-28 17:08:50,674 INFO
> org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace:
> ExitCodeException exitCode=2:
>
>
>
> Thanks,
>
> Rahul Chhiber
>
>
>
