Hi Zach, I’m also a relative newbie, but I did run into this same issue. You are correct, in that your 5th job isn’t starting due to not enough resources available in the cluster, so you need to reduce the resources required.
First, in yarn-site.xml I switched over to the FairScheduler<http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>: <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property> I also added these two properties (yarn-site.xml) to control the amount of memory allocated to each job: <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>256</value> <description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>512</value> <description>Maximum limit of memory to allocate to each container request at the Resource Manager.</description> </property> Then, in each of my Samza properties files describing my jobs, I added the following two settings: yarn.container.memory.mb=512 yarn.am.container.memory.mb=256 Hope that helps! Best, Mark On Oct 6, 2014, at 6:27 AM, Zach Cox <[email protected]<mailto:[email protected]>> wrote: Hi - I'm just getting started with Samza. I got the hello-samza example working properly in the vagrant box. Then I wrote 2 new tasks, rebuilt everything and submitted them to yarn using run-job.sh. These 2 new jobs show up in the yarn web ui, however only one of them has State=RUNNING, the other just sits forever at State=ACCEPTED. The Cluster Metrics section shows some interesting things: - Apps Pending = 1 - Apps Running = 4 - Containers Running = 8 - Memory Used = 8 GB - Memory Total = 8 GB - Memory Reserved = 0 B Again I'm really new to samza & yarn, but does this mean that the node on this vagrant box has 8 GB memory available but all 8 GB is being used, so it can't run the 5th samza job? Are there 8 containers running because each Samza job has an ApplicationMaster and a SamzaContainer? Are each of those containers using 1 GB memory, and that's why all the available memory is used up? Do these containers really need 1 GB memory each? Can this be adjusted somehow? Just trying to better understand what's going on here, and see if there's a simple way to get both of my new tasks running in hello-samza. Thanks, Zach
