I am trying to start a Twill Runnable with 2 cores in YARN. When I set the
number of virtual cores to 2 in my ResourceSpecification, the container
that is started in YARN ends up having only 1 core (according to its logs
and the NodeManager). It looks like Twill is asking for 2 cores in the
ApplicationMaster but YARN only returns a container with 1 core.
Therefore, I am not sure if this is a Twill or YARN problem. I am running
Twill 0.5 and Hadoop 2.6.0. In my yarn-site.xml, I am not changing any of
the configuration for virtual cores from the default. Any ideas of what
could be causing this?
Below are the logs from my Twill ApplicationMaster:
17:56:00.610 [ApplicationMasterService] INFO
o.a.t.i.a.ApplicationMasterService - Request 1 container with capability
<memory:1024, vCores:2> for runnable FluoWorker
17:56:02.616 [ApplicationMasterService] INFO
o.a.t.i.a.ApplicationMasterService - Starting runnable FluoWorker with
RunnableProcessLauncher{container=org.apache.twill.internal.yarn.Hadoop21YarnContainerInfo@74cff77f
}
I added the logging below to my Twill 0.5 branch which shows that the
container returned by YARN only has 1 core even though the request was for
2:
17:56:02.616 [ApplicationMasterService] INFO
o.a.t.i.a.ApplicationMasterService -
processLauncher.getContainerInfo().getVirtualCores() = 1
17:56:02.617 [ApplicationMasterService] INFO
o.a.t.i.a.ApplicationMasterService -
provisionRequest.getRuntimeSpec().getResourceSpecification().getVirtualCores()
= 2
