RE: AWS EMR: Container is running beyond virtual memory limits

2016-03-11 Thread Jim 
I was under the understanding that with EMR 4.x, we had to create separate 
config files and place them on S3, then reference them when we create the 
cluster as described in detail in this first article:

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html

This next article goes into more detail.

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html

But, this is just based on reading the documentation, I have been unable to get 
the custom configs working yet.

Thanks,

Jim


From: Aniruddha Thombare [mailto:anirud...@datatorrent.com]
Sent: Friday, March 11, 2016 9:16 AM
To: users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>
Subject: Re: AWS EMR: Container is running beyond virtual memory limits


Hi,

It seems that the above mentioned configurations didn't take effect.
Those changes were made in:
/etc/Hadoop/conf/yarn-site.xml
And
malted-site.xml

@Sandeep even pidemo didn't run.

On Fri, 11 Mar 2016 8:34 pm Pradeep A. Dalvi, 
<p...@apache.org<mailto:p...@apache.org>> wrote:
We are facing following error message while starting any containers on AWS EMR.


Container [pid=8107,containerID=container_1457702160744_0001_01_07] is 
running beyond virtual memory limits. Current usage: 186.1 MB of 256 MB 
physical memory used; 2.0 GB of 1.3 GB virtual memory used. Killing container.

Dump of the process-tree for container_1457702160744_0001_01_07 :

 |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

 |- 8222 8107 8107 8107 (java) 589 62 2041503744 46944 
/usr/lib/jvm/java-openjdk/bin/java -Xmx234881024 
-Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
 
-Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
 -Ddt.cid=container_1457702160744_0001_01_07 -Dhadoop.root.logger=INFO,RFA 
-Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
 com.datatorrent.stram.engine.StreamingContainer

 |- 8107 8105 8107 8107 (bash) 1 5 115806208 705 /bin/bash -c 
/usr/lib/jvm/java-openjdk/bin/java  -Xmx234881024  
-Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
 
-Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
 -Ddt.cid=container_1457702160744_0001_01_07 -Dhadoop.root.logger=INFO,RFA 
-Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
 com.datatorrent.stram.engine.StreamingContainer 
1>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stdout
 
2>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stderr



Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

We had 1 m3.xlarge MASTER & 2 m3.xlarge CORE instances provisioned. We also 
have tried m4.4xlarge instances. EMR Task configurations can be found at 
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html

We tried changing following yarn configurations, however they did not seem to 
help much.

  
yarn.nodemanager.resource.memory-mb12288
  
yarn.scheduler.maximum-allocation-mb4096
   yarn.nodemanager.vmem-check-enabled 
false 
   yarn.nodemanager.vmem-pmem-ratio 
50


Thanks,
--
Pradeep A. Dalvi

RE: AWS EMR: Container is running beyond virtual memory limits

2016-03-11 Thread Jim 
I was under the understanding that with EMR 4.x, we had to create separate 
config files and place them on S3, then reference them when we create the 
cluster as described in detail in this first article:

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html

This next article goes into more detail.

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-configure-apps.html

But, this is just based on reading the documentation, I have been unable to get 
the custom configs working yet.

Thanks,

Jim


From: Aniruddha Thombare [mailto:anirud...@datatorrent.com]
Sent: Friday, March 11, 2016 9:16 AM
To: users@apex.incubator.apache.org
Subject: Re: AWS EMR: Container is running beyond virtual memory limits


Hi,

It seems that the above mentioned configurations didn't take effect.
Those changes were made in:
/etc/Hadoop/conf/yarn-site.xml
And
malted-site.xml

@Sandeep even pidemo didn't run.

On Fri, 11 Mar 2016 8:34 pm Pradeep A. Dalvi, 
<p...@apache.org<mailto:p...@apache.org>> wrote:
We are facing following error message while starting any containers on AWS EMR.


Container [pid=8107,containerID=container_1457702160744_0001_01_07] is 
running beyond virtual memory limits. Current usage: 186.1 MB of 256 MB 
physical memory used; 2.0 GB of 1.3 GB virtual memory used. Killing container.

Dump of the process-tree for container_1457702160744_0001_01_07 :

 |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

 |- 8222 8107 8107 8107 (java) 589 62 2041503744 46944 
/usr/lib/jvm/java-openjdk/bin/java -Xmx234881024 
-Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
 
-Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
 -Ddt.cid=container_1457702160744_0001_01_07 -Dhadoop.root.logger=INFO,RFA 
-Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
 com.datatorrent.stram.engine.StreamingContainer

 |- 8107 8105 8107 8107 (bash) 1 5 115806208 705 /bin/bash -c 
/usr/lib/jvm/java-openjdk/bin/java  -Xmx234881024  
-Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
 
-Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
 -Ddt.cid=container_1457702160744_0001_01_07 -Dhadoop.root.logger=INFO,RFA 
-Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
 com.datatorrent.stram.engine.StreamingContainer 
1>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stdout
 
2>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stderr



Container killed on request. Exit code is 143

Container exited with a non-zero exit code 143

We had 1 m3.xlarge MASTER & 2 m3.xlarge CORE instances provisioned. We also 
have tried m4.4xlarge instances. EMR Task configurations can be found at 
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html

We tried changing following yarn configurations, however they did not seem to 
help much.

  
yarn.nodemanager.resource.memory-mb12288
  
yarn.scheduler.maximum-allocation-mb4096
   yarn.nodemanager.vmem-check-enabled 
false 
   yarn.nodemanager.vmem-pmem-ratio 
50


Thanks,
--
Pradeep A. Dalvi

Re: AWS EMR: Container is running beyond virtual memory limits

2016-03-11 Thread Aniruddha Thombare 
Hi,

It seems that the above mentioned configurations didn't take effect.
Those changes were made in:
/etc/Hadoop/conf/yarn-site.xml
And
malted-site.xml

@Sandeep even pidemo didn't run.

On Fri, 11 Mar 2016 8:34 pm Pradeep A. Dalvi,  wrote:

> We are facing following error message while starting any containers on AWS
> EMR.
>
> Container [pid=8107,containerID=container_1457702160744_0001_01_07] is 
> running beyond virtual memory limits. Current usage: 186.1 MB of 256 MB 
> physical memory used; 2.0 GB of 1.3 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1457702160744_0001_01_07 :
>   |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>   |- 8222 8107 8107 8107 (java) 589 62 2041503744 46944 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx234881024 
> -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
>  
> -Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
>  -Ddt.cid=container_1457702160744_0001_01_07 
> -Dhadoop.root.logger=INFO,RFA 
> -Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
>  com.datatorrent.stram.engine.StreamingContainer
>   |- 8107 8105 8107 8107 (bash) 1 5 115806208 705 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java  -Xmx234881024  
> -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
>  
> -Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
>  -Ddt.cid=container_1457702160744_0001_01_07 
> -Dhadoop.root.logger=INFO,RFA 
> -Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
>  com.datatorrent.stram.engine.StreamingContainer 
> 1>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stderr
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> We had 1 m3.xlarge MASTER & 2 m3.xlarge CORE instances provisioned. We
> also have tried m4.4xlarge instances. EMR Task configurations can be found
> at
> http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html
>
> We tried changing following yarn configurations, however they did not seem
> to help much.
>
>   *yarn.nodemanager.resource.memory-mb*
> *12288*
>   *yarn.scheduler.maximum-allocation-mb*
> *4096*
>*yarn.nodemanager.vmem-check-enabled* 
> *false* 
>*yarn.nodemanager.vmem-pmem-ratio* *50*
> 
>
>
> Thanks,
> --
> Pradeep A. Dalvi
>

Re: AWS EMR: Container is running beyond virtual memory limits

2016-03-11 Thread Sandeep Deshmukh 
What is the memory allocated to the container?
On 11-Mar-2016 8:34 pm, "Pradeep A. Dalvi"  wrote:

> We are facing following error message while starting any containers on AWS
> EMR.
>
> Container [pid=8107,containerID=container_1457702160744_0001_01_07] is 
> running beyond virtual memory limits. Current usage: 186.1 MB of 256 MB 
> physical memory used; 2.0 GB of 1.3 GB virtual memory used. Killing container.
> Dump of the process-tree for container_1457702160744_0001_01_07 :
>   |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>   |- 8222 8107 8107 8107 (java) 589 62 2041503744 46944 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx234881024 
> -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
>  
> -Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
>  -Ddt.cid=container_1457702160744_0001_01_07 
> -Dhadoop.root.logger=INFO,RFA 
> -Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
>  com.datatorrent.stram.engine.StreamingContainer
>   |- 8107 8105 8107 8107 (bash) 1 5 115806208 705 /bin/bash -c 
> /usr/lib/jvm/java-openjdk/bin/java  -Xmx234881024  
> -Ddt.attr.APPLICATION_PATH=hdfs://ip-172-31-9-174.ec2.internal:8020/user/hadoop/datatorrent/apps/application_1457702160744_0001
>  
> -Djava.io.tmpdir=/mnt1/yarn/usercache/hadoop/appcache/application_1457702160744_0001/container_1457702160744_0001_01_07/tmp
>  -Ddt.cid=container_1457702160744_0001_01_07 
> -Dhadoop.root.logger=INFO,RFA 
> -Dhadoop.log.dir=/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07
>  com.datatorrent.stram.engine.StreamingContainer 
> 1>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stdout
>  
> 2>/var/log/hadoop-yarn/containers/application_1457702160744_0001/container_1457702160744_0001_01_07/stderr
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> We had 1 m3.xlarge MASTER & 2 m3.xlarge CORE instances provisioned. We
> also have tried m4.4xlarge instances. EMR Task configurations can be found
> at
> http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/TaskConfiguration_H2.html
>
> We tried changing following yarn configurations, however they did not seem
> to help much.
>
>   *yarn.nodemanager.resource.memory-mb*
> *12288*
>   *yarn.scheduler.maximum-allocation-mb*
> *4096*
>*yarn.nodemanager.vmem-check-enabled* 
> *false* 
>*yarn.nodemanager.vmem-pmem-ratio* *50*
> 
>
>
> Thanks,
> --
> Pradeep A. Dalvi
>