I'd also recommend you clone master or use the 0.2.0 release candidate:
https://dist.apache.org/repos/dist/dev/incubator/myriad/myriad-0.2.0-incubating-rc4/
at it fixes a number of FGS specific bugs.
Darin
On Sun, Jun 5, 2016 at 1:09 PM, Darin Johnson <dbjohnson1...@gmail.com>
wrote:
> Stephen,
>
> I was able to recreate the problem (specific due to 2.7.2, they changed
> the defaults on the following two properties to true). Setting them to
> false allowed me to again run map reduce jobs. I'll try to update the
> documentation later today.
>
> <property>
>
> <name>yarn.nodemanager.pmem-check-enabled</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.vmem-check-enabled</name>
>
> <value>false</value>
>
> </property>
>
> Darin
>
> On Sun, Jun 5, 2016 at 10:30 AM, Stephen Gran <stephen.g...@piksel.com>
> wrote:
>
>> Hi,
>>
>> I think those are the properties I added when I started getting this
>> error. Removing them doesn't seem to make any difference, sadly.
>>
>> This is hadoop 2.7.2
>>
>> Cheers,
>>
>> On 05/06/16 14:45, Darin Johnson wrote:
>> > Hey Stephen,
>> >
>> > I think you're pretty close.
>> >
>> > Looking at the config I'd suggest removing these properties:
>> >
>> > <property>
>> > <name>yarn.nodemanager.resource.memory-mb</name>
>> > <value>4096</value>
>> > </property>
>> > <property>
>> > <name>yarn.scheduler.maximum-allocation-vcores</name>
>> > <value>12</value>
>> > </property>
>> > <property>
>> > <name>yarn.scheduler.maximum-allocation-mb</name>
>> > <value>8192</value>
>> > </property>
>> > <property>
>> > <name>yarn.nodemanager.vmem-check-enabled</name>
>> > <value>false</value>
>> > <description>Whether virtual memory limits will be enforced for
>> > containers</description>
>> > </property>
>> > <property>
>> > <name>yarn.nodemanager.vmem-pmem-ratio</name>
>> > <value>4</value>
>> > <description>Ratio between virtual memory to physical memory when
>> > setting memory limits for containers</description>
>> > </property>
>> >
>> > I'll try them out on my test cluster later today/tonight and see if I
>> can
>> > recreate the problem. What version of hadoop are you running? I'll
>> make
>> > sure I'm consistent with that as well.
>> >
>> > Thanks,
>> >
>> > Darin
>> > On Jun 5, 2016 8:15 AM, "Stephen Gran" <stephen.g...@piksel.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> Attached. Thanks very much for looking.
>> >>
>> >> Cheers,
>> >>
>> >> On 05/06/16 12:51, Darin Johnson wrote:
>> >>> Hey Steven can you please send your yarn-site.xml, I'm guessing
>> you're on
>> >>> the right track.
>> >>>
>> >>> Darin
>> >>> Hi,
>> >>>
>> >>> OK. That helps, thank you. I think I just misunderstood the docs (or
>> >>> they never said explicitly that you did need at least some static
>> >>> resource), and I scaled down the initial nm.medium that got started.
>> I
>> >>> get a bit further now, and jobs start but are killed with:
>> >>>
>> >>> Diagnostics: Container
>> >>> [pid=3865,containerID=container_1465112239753_0001_03_000001] is
>> running
>> >>> beyond virtual memory limits. Current usage: 50.7 MB of 0B physical
>> >>> memory used; 2.6 GB of 0B virtual memory used. Killing container
>> >>>
>> >>> When I've seen this in the past with yarn but without myriad, it was
>> >>> usually about ratios of vmem to mem and things like that - I've tried
>> >>> some of those knobs, but I didn't expect much result and didn't get
>> any.
>> >>>
>> >>> What strikes me about the error message is that the vmem and mem
>> >>> allocations are for 0.
>> >>>
>> >>> I'm sorry for asking what are probably naive questions here, I
>> couldn't
>> >>> find a different forum. If there is one, please point me there so I
>> >>> don't disrupt the dev flow here.
>> >>>
>> >>> I can see this in the logs:
>> >>>
>> >>>
>> >>> 2016-06-05 07:39:25,687 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
>> >>> container_1465112239753_0001_03_000001 Container Transitioned from NEW
>> >>> to ALLOCATED
>> >>> 2016-06-05 07:39:25,688 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
>> >>> OPERATION=AM Allocated Container TARGET=SchedulerApp
>> >>> RESULT=SUCCESS APPID=application_1465112239753_0001
>> >>> CONTAINERID=container_1465112239753_0001_03_000001
>> >>> 2016-06-05 07:39:25,688 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
>> >>> Assigned container container_1465112239753_0001_03_000001 of capacity
>> >>> <memory:0, vCores:0> on host slave2.testing.local:26688, which has 1
>> >>> containers, <memory:0, vCores:0> used and <memory:4096, vCores:1>
>> >>> available after allocation
>> >>> 2016-06-05 07:39:25,689 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM:
>> >>> Sending NMToken for nodeId : slave2.testing.local:26688 for container
>> :
>> >>> container_1465112239753_0001_03_000001
>> >>> 2016-06-05 07:39:25,696 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
>> >>> container_1465112239753_0001_03_000001 Container Transitioned from
>> >>> ALLOCATED to ACQUIRED
>> >>> 2016-06-05 07:39:25,696 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM:
>> >>> Clear node set for appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:25,696 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> Storing attempt: AppId: application_1465112239753_0001 AttemptId:
>> >>> appattempt_1465112239753_0001_000003 MasterContainer: Container:
>> >>> [ContainerId: container_1465112239753_0001_03_000001, NodeId:
>> >>> slave2.testing.local:26688, NodeHttpAddress:
>> slave2.testing.local:24387,
>> >>> Resource: <memory:0, vCores:0>, Priority: 0, Token: Token { kind:
>> >>> ContainerToken, service: 10.0.5.5:26688 }, ]
>> >>> 2016-06-05 07:39:25,697 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> appattempt_1465112239753_0001_000003 State change from SCHEDULED to
>> >>> ALLOCATED_SAVING
>> >>> 2016-06-05 07:39:25,698 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> appattempt_1465112239753_0001_000003 State change from
>> ALLOCATED_SAVING
>> >>> to ALLOCATED
>> >>> 2016-06-05 07:39:25,699 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
>> >>> Launching masterappattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:25,705 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
>> >>> Setting up container Container: [ContainerId:
>> >>> container_1465112239753_0001_03_000001, NodeId:
>> >>> slave2.testing.local:26688, NodeHttpAddress:
>> slave2.testing.local:24387,
>> >>> Resource: <memory:0, vCores:0>, Priority: 0, Token: Token { kind:
>> >>> ContainerToken, service: 10.0.5.5:26688 }, ] for AM
>> >>> appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:25,705 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
>> >>> Command to launch container container_1465112239753_0001_03_000001 :
>> >>> $JAVA_HOME/bin/java -Djava.io.tmpdir=$PWD/tmp
>> >>> -Dlog4j.configuration=container-log4j.properties
>> >>> -Dyarn.app.container.log.dir=<LOG_DIR>
>> >>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> >>> -Dhadoop.root.logfile=syslog -Xmx1024m
>> >>> org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout
>> >>> 2><LOG_DIR>/stderr
>> >>> 2016-06-05 07:39:25,706 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
>> >>> Create AMRMToken for ApplicationAttempt:
>> >>> appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:25,707 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
>> >>> Creating password for appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:25,727 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher:
>> >>> Done launching container Container: [ContainerId:
>> >>> container_1465112239753_0001_03_000001, NodeId:
>> >>> slave2.testing.local:26688, NodeHttpAddress:
>> slave2.testing.local:24387,
>> >>> Resource: <memory:0, vCores:0>, Priority: 0, Token: Token { kind:
>> >>> ContainerToken, service: 10.0.5.5:26688 }, ] for AM
>> >>> appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:25,728 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> appattempt_1465112239753_0001_000003 State change from ALLOCATED to
>> >> LAUNCHED
>> >>> 2016-06-05 07:39:25,736 WARN
>> >>> org.apache.myriad.scheduler.event.handlers.StatusUpdateEventHandler:
>> >>> Task: yarn_container_1465112239753_0001_03_000001 not found, status:
>> >>> TASK_RUNNING
>> >>> 2016-06-05 07:39:26,510 INFO org.apache.hadoop.yarn.util.RackResolver:
>> >>> Resolved slave1.testing.local to /default-rack
>> >>> 2016-06-05 07:39:26,517 WARN
>> >>> org.apache.myriad.scheduler.fgs.NMHeartBeatHandler: FineGrainedScaling
>> >>> feature got invoked for a NM with non-zero capacity. Host:
>> >>> slave1.testing.local, Mem: 4096, CPU: 0. Setting the NM's capacity to
>> >>> (0G,0CPU)
>> >>> 2016-06-05 07:39:26,517 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
>> >>> slave1.testing.local:29121 Node Transitioned from NEW to RUNNING
>> >>> 2016-06-05 07:39:26,518 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> >>> Added node slave1.testing.local:29121 cluster capacity: <memory:4096,
>> >>> vCores:1>
>> >>> 2016-06-05 07:39:26,519 INFO
>> >>> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager:
>> >>> afterSchedulerEventHandled: NM registration from node
>> >> slave1.testing.local
>> >>> 2016-06-05 07:39:26,528 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
>> >>> received container statuses on node manager register :[container_id {
>> >>> app_attempt_id { application_id { id: 1 cluster_timestamp:
>> 1465112239753
>> >>> } attemptId: 2 } id: 1 } container_state: C_RUNNING resource {
>> memory: 0
>> >>> virtual_cores: 0 } priority { priority: 0 } diagnostics: ""
>> >>> container_exit_status: -1000 creation_time: 1465112356478]
>> >>> 2016-06-05 07:39:26,530 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
>> >>> NodeManager from node slave1.testing.local(cmPort: 29121 httpPort:
>> >>> 20456) registered with capability: <memory:0, vCores:0>, assigned
>> nodeId
>> >>> slave1.testing.local:29121
>> >>> 2016-06-05 07:39:26,611 INFO
>> >>> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Setting
>> >>> capacity for node slave1.testing.local to <memory:4637, vCores:6>
>> >>> 2016-06-05 07:39:26,611 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>> >>> Update resource on node: slave1.testing.local from: <memory:0,
>> >>> vCores:0>, to: <memory:4637, vCores:6>
>> >>> 2016-06-05 07:39:26,615 INFO
>> >>> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Setting
>> >>> capacity for node slave1.testing.local to <memory:0, vCores:0>
>> >>> 2016-06-05 07:39:26,616 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler:
>> >>> Update resource on node: slave1.testing.local from: <memory:4637,
>> >>> vCores:6>, to: <memory:0, vCores:0>
>> >>> 2016-06-05 07:39:26,691 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
>> >>> container_1465112239753_0001_03_000001 Container Transitioned from
>> >>> ACQUIRED to RUNNING
>> >>> 2016-06-05 07:39:26,835 WARN
>> >>> org.apache.myriad.scheduler.event.handlers.StatusUpdateEventHandler:
>> >>> Task: yarn_container_1465112239753_0001_03_000001 not found, status:
>> >>> TASK_FINISHED
>> >>> 2016-06-05 07:39:27,603 INFO
>> >>> org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
>> >>> Received offers 1
>> >>> 2016-06-05 07:39:27,748 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
>> >>> container_1465112239753_0001_03_000001 Container Transitioned from
>> >>> RUNNING to COMPLETED
>> >>> 2016-06-05 07:39:27,748 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
>> >>> Completed container: container_1465112239753_0001_03_000001 in state:
>> >>> COMPLETED event:FINISHED
>> >>> 2016-06-05 07:39:27,748 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
>> >>> OPERATION=AM Released Container TARGET=SchedulerApp
>> >>> RESULT=SUCCESS APPID=application_1465112239753_0001
>> >>> CONTAINERID=container_1465112239753_0001_03_000001
>> >>> 2016-06-05 07:39:27,748 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
>> >>> Released container container_1465112239753_0001_03_000001 of capacity
>> >>> <memory:0, vCores:0> on host slave2.testing.local:26688, which
>> currently
>> >>> has 0 containers, <memory:0, vCores:0> used and <memory:4096,
>> vCores:1>
>> >>> available, release resources=true
>> >>> 2016-06-05 07:39:27,748 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> >>> Application attempt appattempt_1465112239753_0001_000003 released
>> >>> container container_1465112239753_0001_03_000001 on node: host:
>> >>> slave2.testing.local:26688 #containers=0 available=<memory:4096,
>> >>> vCores:1> used=<memory:0, vCores:0> with event: FINISHED
>> >>> 2016-06-05 07:39:27,749 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> Updating application attempt appattempt_1465112239753_0001_000003 with
>> >>> final state: FAILED, and exit status: -103
>> >>> 2016-06-05 07:39:27,750 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> appattempt_1465112239753_0001_000003 State change from LAUNCHED to
>> >>> FINAL_SAVING
>> >>> 2016-06-05 07:39:27,751 INFO
>> >>>
>> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
>> >>> Unregistering app attempt : appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:27,751 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager:
>> >>> Application finished, removing password for
>> >>> appattempt_1465112239753_0001_000003
>> >>> 2016-06-05 07:39:27,751 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>> >>> appattempt_1465112239753_0001_000003 State change from FINAL_SAVING to
>> >>> FAILED
>> >>> 2016-06-05 07:39:27,751 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The
>> >>> number of failed attempts is 2. The max attempts is 2
>> >>> 2016-06-05 07:39:27,753 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>> Updating
>> >>> application application_1465112239753_0001 with final state: FAILED
>> >>> 2016-06-05 07:39:27,756 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>> >>> application_1465112239753_0001 State change from ACCEPTED to
>> FINAL_SAVING
>> >>> 2016-06-05 07:39:27,757 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
>> >>> Application appattempt_1465112239753_0001_000003 is done.
>> >> finalState=FAILED
>> >>> 2016-06-05 07:39:27,757 INFO
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
>> >>> Application application_1465112239753_0001 requests cleared
>> >>> 2016-06-05 07:39:27,758 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore:
>> >>> Updating info for app: application_1465112239753_0001
>> >>> 2016-06-05 07:39:27,758 INFO
>> >>> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
>> >>> Application application_1465112239753_0001 failed 2 times due to AM
>> >>> Container for appattempt_1465112239753_0001_000003 exited with
>> >>> exitCode: -103
>> >>> For more detailed output, check application tracking
>> >>> page:
>> >>>
>> >>
>> http://master.testing.local:8088/cluster/app/application_1465112239753_0001Then
>> >>> ,
>> >>> click on links to logs of each attempt.
>> >>> Diagnostics: Container
>> >>> [pid=3865,containerID=container_1465112239753_0001_03_000001] is
>> running
>> >>> beyond virtual memory limits. Current usage: 50.7 MB of 0B physical
>> >>> memory used; 2.6 GB of 0B virtual memory used. Killing container.
>> >>> Dump of the process-tree for container_1465112239753_0001_03_000001 :
>> >>> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
>> >>> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES)
>> FULL_CMD_LINE
>> >>> |- 3873 3865 3865 3865 (java) 80 26 2770927616 12614
>> >>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java
>> >>>
>> >>
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1465112239753_0001/container_1465112239753_0001_03_000001/tmp
>> >>> -Dlog4j.configuration=container-log4j.properties
>> >>>
>> >>
>> -Dyarn.app.container.log.dir=/srv/apps/hadoop-2.7.2/logs/userlogs/application_1465112239753_0001/container_1465112239753_0001_03_000001
>> >>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> >>> -Dhadoop.root.logfile=syslog -Xmx1024m
>> >>> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>> >>> |- 3865 3863 3865 3865 (bash) 0 1 11427840 354 /bin/bash -c
>> >>> /usr/lib/jvm/java-8-openjdk-amd64/bin/java
>> >>>
>> >>
>> -Djava.io.tmpdir=/tmp/hadoop-root/nm-local-dir/usercache/root/appcache/application_1465112239753_0001/container_1465112239753_0001_03_000001/tmp
>> >>> -Dlog4j.configuration=container-log4j.properties
>> >>>
>> >>
>> -Dyarn.app.container.log.dir=/srv/apps/hadoop-2.7.2/logs/userlogs/application_1465112239753_0001/container_1465112239753_0001_03_000001
>> >>> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
>> >>> -Dhadoop.root.logfile=syslog -Xmx1024m
>> >>> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>> >>>
>> >>
>> 1>/srv/apps/hadoop-2.7.2/logs/userlogs/application_1465112239753_0001/container_1465112239753_0001_03_000001/stdout
>> >>>
>> >>
>> 2>/srv/apps/hadoop-2.7.2/logs/userlogs/application_1465112239753_0001/container_1465112239753_0001_03_000001/stderr
>> >>>
>> >>>
>> >>> Container killed on request. Exit code is 143
>> >>> Container exited with a non-zero exit code 143
>> >>> Failing this attempt. Failing the application.
>> >>>
>> >>>
>> >>>
>> >>> On 03/06/16 15:52, yuliya Feldman wrote:
>> >>>> I believe you need at least one NM that is not subject to fine grain
>> >>> scaling.
>> >>>> So far if total resources on the cluster is less then a single
>> container
>> >>> needs for AM you won't be able to submit any app.As exception below
>> tells
>> >>> you.
>> >>>> (Invalid resource request, requested memory < 0, or requested memory
>> >>> max
>> >>> configured, requestedMemory=1536, maxMemory=0
>> >>>> at)
>> >>>> I believe by default when starting Myriad cluster one NM with non 0
>> >>> capacity should start by default.
>> >>>> In addition see in RM log whether offers with resources are coming
>> to RM
>> >>> - this info should be in the log.
>> >>>>
>> >>>> From: Stephen Gran <stephen.g...@piksel.com>
>> >>>> To: "dev@myriad.incubator.apache.org" <
>> >> dev@myriad.incubator.apache.org>
>> >>>> Sent: Friday, June 3, 2016 1:29 AM
>> >>>> Subject: problem getting fine grained scaling workig
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> I'm trying to get fine grained scaling going on a test mesos
>> cluster. I
>> >>>> have a single master and 2 agents. I am running 2 node managers with
>> >>>> the zero profile, one per agent. I can see both of them in the RM UI
>> >>>> reporting correctly as having 0 resources.
>> >>>>
>> >>>> I'm getting stack traces when I try to launch a sample application,
>> >>>> though. I feel like I'm just missing something obvious somewhere -
>> can
>> >>>> anyone shed any light?
>> >>>>
>> >>>> This is on a build of yesterday's git head.
>> >>>>
>> >>>> Cheers,
>> >>>>
>> >>>> root@master:/srv/apps/hadoop# bin/yarn jar
>> >>>> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar teragen
>> 10000
>> >>>> /outDir
>> >>>> 16/06/03 08:23:33 INFO client.RMProxy: Connecting to ResourceManager
>> at
>> >>>> master.testing.local/10.0.5.3:8032
>> >>>> 16/06/03 08:23:34 INFO terasort.TeraSort: Generating 10000 using 2
>> >>>> 16/06/03 08:23:34 INFO mapreduce.JobSubmitter: number of splits:2
>> >>>> 16/06/03 08:23:34 INFO mapreduce.JobSubmitter: Submitting tokens for
>> >>>> job: job_1464902078156_0001
>> >>>> 16/06/03 08:23:35 INFO mapreduce.JobSubmitter: Cleaning up the
>> staging
>> >>>> area /tmp/hadoop-yarn/staging/root/.staging/job_1464902078156_0001
>> >>>> java.io.IOException:
>> >>>> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException:
>> >>>> Invalid resource request, requested memory < 0, or requested memory >
>> >>>> max configured, requestedMemory=1536, maxMemory=0
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:268)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:236)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:329)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:281)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:580)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
>> >>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>> >>>> at
>> >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
>> >>>> at
>> >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
>> >>>> at java.security.AccessController.doPrivileged(Native
>> Method)
>> >>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>> >>>> at
>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
>> >>>>
>> >>>> at
>> >>> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:306)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
>> >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
>> >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
>> >>>> at java.security.AccessController.doPrivileged(Native
>> Method)
>> >>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>> >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
>> >>>> at
>> >>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
>> >>>> at
>> >>> org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:301)
>> >>>> at
>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> >>>> at
>> >>> org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:305)
>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>> at java.lang.reflect.Method.invoke(Method.java:497)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
>> >>>> at
>> >>> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
>> >>>> at
>> >>> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>> at java.lang.reflect.Method.invoke(Method.java:497)
>> >>>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>> >>>> Caused by:
>> >>>> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException:
>> >>>> Invalid resource request, requested memory < 0, or requested memory >
>> >>>> max configured, requestedMemory=1536, maxMemory=0
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:268)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:236)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:329)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:281)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:580)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
>> >>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>> >>>> at
>> >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
>> >>>> at
>> >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
>> >>>> at java.security.AccessController.doPrivileged(Native
>> Method)
>> >>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>> >>>> at
>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
>> >>>>
>> >>>> at
>> >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> >>> Method)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> >>>> at
>> >>> java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>> >>>> at
>> >>>
>> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:239)
>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>> at java.lang.reflect.Method.invoke(Method.java:497)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> >>>> at com.sun.proxy.$Proxy13.submitApplication(Unknown
>> Source)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:253)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:290)
>> >>>> at
>> >>> org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290)
>> >>>> ... 24 more
>> >>>> Caused by:
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
>> >>>> Invalid resource request, requested memory < 0, or requested memory >
>> >>>> max configured, requestedMemory=1536, maxMemory=0
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:268)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:228)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:236)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.validateAndCreateResourceRequest(RMAppManager.java:385)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:329)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:281)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:580)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
>> >>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>> >>>> at
>> >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
>> >>>> at
>> >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
>> >>>> at java.security.AccessController.doPrivileged(Native
>> Method)
>> >>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>> >>>> at
>> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
>> >>>>
>> >>>> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>> >>>> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>> >>>> at com.sun.proxy.$Proxy12.submitApplication(Unknown
>> Source)
>> >>>> at
>> >>>>
>> >>>
>> >>
>> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:236)
>> >>>> ... 34 more
>> >>>>
>> >>>>
>> >>>> Cheers,
>> >>>> --
>> >>>> Stephen Gran
>> >>>> Senior Technical Architect
>> >>>>
>> >>>> picture the possibilities | piksel.com
>> >>>> This message is private and confidential. If you have received this
>> >>> message in error, please notify the sender or serviced...@piksel.com
>> and
>> >>> remove it from your system.
>> >>>>
>> >>>> Piksel Inc is a company registered in the United States New York
>> City,
>> >>> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> Stephen Gran
>> >>> Senior Technical Architect
>> >>>
>> >>> picture the possibilities | piksel.com
>> >>>
>> >>
>> >> --
>> >> Stephen Gran
>> >> Senior Technical Architect
>> >>
>> >> picture the possibilities | piksel.com
>> >>
>> >
>>
>> --
>> Stephen Gran
>> Senior Technical Architect
>>
>> picture the possibilities | piksel.com
>>
>
>
