Thanks a lot Thomas & Sandesh.
I somehow missed the "troubleshooting" section of the docs. This is what I was looking for. Regards, Ananth On Fri, May 13, 2016 at 1:30 AM, Thomas Weise <thomas.we...@gmail.com> wrote: > Ananth, > > Please have a look at: > > http://docs.datatorrent.com/troubleshooting/#configuring-memory > > Thanks, > Thomas > > > On Thu, May 12, 2016 at 4:00 AM, Ananth Gundabattula < > agundabatt...@gmail.com> wrote: > >> Thanks Shubham. I shall bump up the memory a bit more. >> >> I was wondering how the operator memory relates to the YARN container >> memory settings ? Or it depends on the deployment models ? >> >> For example , if the deployment model is thread local, the YARN container >> needs to be ( considering above example ) configured for atleast memory of >> 2048 * number of operators + Buffer Server Size ? >> >> If the deployment model were not Thread local, it would make YARN >> container requirements for memory lower per container ? >> >> Regards, >> Ananth >> >> On Thu, May 12, 2016 at 7:19 PM, Shubham Pathak <shub...@datatorrent.com> >> wrote: >> >>> Hello Ananth, >>> >>> Looks like operator requires more memory. >>> You may add this property to have more memory allocated to the container. >>> >>> In properties.xml , for operator O in the application you may specify >>> the property : >>> >>> <property> >>> <name>dt.operator.*O*.attr.MEMORY_MB</name> >>> <value>2048</value> >>> </property> >>> >>> Thanks, >>> Shubham >>> >>> On Thu, May 12, 2016 at 1:35 PM, Ananth Gundabattula < >>> agundabatt...@gmail.com> wrote: >>> >>>> Hello All, >>>> >>>> I am seeing the following log from the web ui ocassionally when my >>>> operators are getting killed. Is there any way I can control the memory >>>> settings that are used to communicate with YARN when negotiating a >>>> container ? >>>> >>>> How does the typical yarn settings for a container heap and max memory >>>> relate to the Apex memory allocation model. >>>> >>>> The info messages I see in the web UI are as follows: >>>> >>>> Container [pid=14699,containerID=container_1462863487071_0015_01_000012] >>>> is running beyond physical memory limits. Current usage: 1.5 GB of 1.5 GB >>>> physical memory used; 6.1 GB of 3.1 GB virtual memory used. Killing >>>> container. >>>> Dump of the process-tree for container_1462863487071_0015_01_000012 : >>>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) >>>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE >>>> |- 14817 14699 14699 14699 (java) 1584 1654 6426968064 393896 >>>> /usr/java/default/bin/java -Xmx4429185024 >>>> -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 >>>> >>>> -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp >>>> -Ddt.cid=container_1462863487071_0015_01_000012 >>>> -Dhadoop.root.logger=INFO,RFA >>>> -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 >>>> -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO >>>> com.datatorrent.stram.engine.StreamingContainer >>>> |- 14699 14697 14699 14699 (bash) 1 2 108646400 303 /bin/bash -c >>>> /usr/java/default/bin/java -Xmx4429185024 >>>> -Ddt.attr.APPLICATION_PATH=hdfs://dwh109.qaperf2.sac.int.threatmetrix.com:8020/user/dtadmin/datatorrent/apps/application_1462863487071_0015 >>>> >>>> -Djava.io.tmpdir=/data3/yarn/nm/usercache/root/appcache/application_1462863487071_0015/container_1462863487071_0015_01_000012/tmp >>>> -Ddt.cid=container_1462863487071_0015_01_000012 >>>> -Dhadoop.root.logger=INFO,RFA >>>> -Dhadoop.log.dir=/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012 >>>> -Ddt.loggers.level=com.datatorrent.*:INFO,org.apache.*:INFO >>>> com.datatorrent.stram.engine.StreamingContainer >>>> 1>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stdout >>>> >>>> 2>/data3/yarn/container-logs/application_1462863487071_0015/container_1462863487071_0015_01_000012/stderr >>>> >>>> Container killed on request. Exit code is 143 >>>> Container exited with a non-zero exit code 143 >>>> >>>> >>>> Regards, >>>> >>>> Ananth >>>> >>>> >>>> >>> >> >
