Prabhu, You fully addressed my question and I'll follow your instructions. Many thanks and have a nice day. Guido
On Thu, Aug 15, 2019 at 8:19 PM Prabhu Josephraj <pjos...@cloudera.com> wrote: > 1. Easy way to reproduce container to exceed configured physical memory > limit is by configuring the Heap Size (500MB) of > container above the Container Size (100MB). > > yarn-site.xml: yarn.scheduler.minimum-allocation-mb 100 > mapred-site.xml: yarn.app.mapreduce.am.resource.mb 100 > yarn.app.mapreduce.am.command-opts -Xmx500m > > Note: This is only for testing purpose. Usually the Heap Size has to be > 80% of Container Size. > > 2. There is no job settings which increase the memory usage of a > container. It depends on the application code. > Try adding memory intensive code inside the MapReduce application. > > > https://alvinalexander.com/blog/post/java/java-program-consume-all-memory-ram-on-computer > > https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java > > Running pi job on a long number will also require huge memory. > > yarn jar > /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.1.jar pi 1 > 1000000000 > > There are chances that the JVM Crashes with OutOfMemory before Yarn kills > the container for exceeding memory usage. > > > On Thu, Aug 15, 2019 at 10:51 PM . . <writeme...@googlemail.com> wrote: > >> >> Prabhu, >> >> I reformulate my question: >> >> I successfully run following job: yarn jar >> /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.1.jar pi 3 >> 10 >> >> and noticed that highest node physical memory usage was alway <512MB >> during job duration; else job completed (see details below) >> >> quote.... >> Every 2.0s: yarn node -status hadoop-1.mydomain.local:44718 >> >> 19/08/15 19:10:54 INFO client.RMProxy: Connecting to ResourceManager at >> hadoop-1.mydomain.local/192.168.100.11:8032 >> Node Report : >> Node-Id : hadoop-1.mydomain.local:44718 >> Rack : /default-rack >> Node-State : RUNNING >> Node-Http-Address : hadoop-1.mydomain.local:8042 >> Last-Health-Update : Thu 15/Aug/19 07:10:22:75CEST >> Health-Report : >> Containers : 2 >> Memory-Used : 2048MB >> Memory-Capacity : 5120MB >> CPU-Used : 2 vcores >> CPU-Capacity : 6 vcores >> Node-Labels : >> Resource Utilization by Node : *PMem:471 MB*, VMem:1413 MB, >> VCores:0.80463576 >> Resource Utilization by Containers : PMem:110 MB, VMem:4014 MB, >> VCores:0.97300005 >> ...unquote >> >> My question is : which job setting may I use to force a node physical >> memory usage >512MB and force a job kill due (or thanks) pmem check. >> Hope above better explain my question ;) >> >> thanks/Guido >> >> >> On Thu, Aug 15, 2019 at 5:09 PM Prabhu Josephraj >> <pjos...@cloudera.com.invalid> wrote: >> >>> Jeff, Available node size for YARN is the value of >>> yarn.nodemanager.resource.memory-mb which is set ten times of 512MB. >>> >>> Guido, Did not get the below question, can you explain the same. >>> >>> Are you aware of any job syntax to tune the 'container physical >>> memory usage' to 'force' job kill/log? >>> >>> >>> On Thu, Aug 15, 2019 at 7:20 PM . . <writeme...@googlemail.com.invalid> >>> wrote: >>> >>>> Hi Prabhu, >>>> >>>> thanks for your explanation. It makes sense, but I wonder YARN allows >>>> you to define 'yarn.nodemanager.resource.memory-mb' higher then node >>>> physical memory w/out logging any entry under resourcemanager log. >>>> >>>> Are you aware of any job syntax to tune the 'container physical memory >>>> usage' to 'force' job kill/log? >>>> >>>> thanks/Guido >>>> >>>> >>>> >>>> On Thu, Aug 15, 2019 at 1:50 PM Prabhu Josephraj >>>> <pjos...@cloudera.com.invalid> wrote: >>>> >>>>> YARN allocates based on the configuration >>>>> (yarn.nodemanager.resource.memory-mb) user has configured. It has >>>>> allocated >>>>> the AM Container of size 1536MB as it can fit in 5120MB Available Node >>>>> Size. >>>>> >>>>> yarn.nodemanager.pmem-check-enabled will kill the container if the >>>>> physical memory usage of the container process is above >>>>> 1536MB. MR ApplicationMaster for a pi job is light weight and it won't >>>>> require that much memory and so not got killed. >>>>> >>>>> >>>>> >>>>> On Thu, Aug 15, 2019 at 4:02 PM . . <writeme...@googlemail.com.invalid> >>>>> wrote: >>>>> >>>>>> Correct: I set 'yarn.nodemanager.resource.memory-mb' ten times the >>>>>> node physical memory (512MB) and I was able to successfully execute a >>>>>> 'pi >>>>>> 1 10' mapreduce job. >>>>>> >>>>>> Since default 'yarn.app.mapreduce.am.resource.mb' value is 1536MB I >>>>>> expected the job to never start / be allocated and I have no valid >>>>>> explanation. >>>>>> >>>>>> >>>>>> On Wed, Aug 14, 2019 at 10:32 PM . . <writeme...@googlemail.com> >>>>>> wrote: >>>>>> >>>>>>> Correct: I set 'yarn.nodemanager.resource.memory-mb' ten times the >>>>>>> node physical memory (512MB) and I was able to successfully execute a >>>>>>> 'pi >>>>>>> 1 10' mapreduce job. >>>>>>> >>>>>>> Since default 'yarn.app.mapreduce.am.resource.mb' value is 1536MB I >>>>>>> expected the job to never start / be allocated and I have no valid >>>>>>> explanation. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 14, 2019 at 8:31 PM Jeff Hubbs <jhubbsl...@att.net> >>>>>>> wrote: >>>>>>> >>>>>>>> To make sure I understand...you've allocated *ten times* your >>>>>>>> physical RAM for containers? If so, I think that's your issue. >>>>>>>> >>>>>>>> For reference, under Hadoop 3.x I didn't have a cluster that would >>>>>>>> really do anything until its worker nodes had at least 8GiB. >>>>>>>> >>>>>>>> On 8/14/19 12:10 PM, . . wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I installed a basic 3 nodes Hadoop 2.9.1 cluster and playing with >>>>>>>> YARN settings. >>>>>>>> The 3 nodes has following configuration: >>>>>>>> 1 cpu / 1 core?? / 512MB RAM >>>>>>>> >>>>>>>> I wonder I was able to configure yarn-site.xml with following >>>>>>>> settings (higher than node physical limits) and successfully run a >>>>>>>> mapreduce 'pi 1 10' job >>>>>>>> >>>>>>>> quote... >>>>>>>> ?? <property> >>>>>>>> ?? ?? ?? >>>>>>>> <name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> ?? ?? <property> >>>>>>>> ?? ?? ?? ?? <name>yarn.nodemanager.resource.memory-mb</name> >>>>>>>> ?? ?? ?? ?? <value>5120</value> >>>>>>>> ?? ?? ?? ?? <description>Amount of physical memory, in MB, that can >>>>>>>> be allocated for containers. If set to -1 and >>>>>>>> yarn.nodemanager.resource.detect-hardware-capabilities is true, it is >>>>>>>> automatically calculated. In other cases, the default is >>>>>>>> 8192MB</description> >>>>>>>> ?? ?? </property> >>>>>>>> >>>>>>>> ?? ?? <property> >>>>>>>> ?? ?? ?? ?? <name>yarn.nodemanager.resource.cpu-vcores</name> >>>>>>>> ?? ?? ?? ?? <value>6</value> >>>>>>>> ?? ?? ?? ?? <description>Number of CPU cores that can be allocated >>>>>>>> for containers.</description> >>>>>>>> ?? ?? </property> >>>>>>>> ...unquote >>>>>>>> >>>>>>>> Can anyone provide an explanation please? >>>>>>>> >>>>>>>> Should 'yarn.nodemanager.vmem-check-enabled' and >>>>>>>> 'yarn.nodemanager.pmem-check-enabled' properties (set to 'true' as >>>>>>>> default) >>>>>>>> check that my YARN settings are higher than physical limits? >>>>>>>> >>>>>>>> Which mapreduce 'pi' job settings can I use, to 'force' containers >>>>>>>> to use more than node physical resources? >>>>>>>> >>>>>>>> Many thanks in advance! >>>>>>>> Guido >>>>>>>> >>>>>>>> >>>>>>>>
