I updated patch (remove check) in https://issues.apache.org/jira/browse/GIRAPH-819. Could you please take a look. Should I create a separate ticket for correct memory check?
On Sat, Jan 4, 2014 at 2:19 AM, Mohammad Islam <misla...@yahoo.com> wrote: > I agree with Eric and Rafal's assessment. You can send a patch about this. > > Regards, > Mohammad > > > On Friday, January 3, 2014 1:13 PM, Rafal Wojdyla <ravwojd...@gmail.com> > wrote: > I had the same problem, and I have created this ticket: > https://issues.apache.org/jira/browse/GIRAPH-819 > First I thought it's just operator problem, then I got to the same > conclusion as you, and removed the if check. > Btw - what was the purpose of that check, in the same function we check > whether there is enough memory available. Memory check still doesn't > guaranty that the job will run though, imagine 10 really small node manager > (512MB) in total 5120MB, and giraph application that requires 2 containers > 1GB each. > > > On Fri, Jan 3, 2014 at 9:47 PM, Eric Kimbrel <lekimb...@gmail.com> wrote: > > I am running on hadoop 2.2.0-cdh5.0.0-beta-1 with pure yarn mode. I = > have noticed two issues in org.apache.giraph.yarn.GiraphYarnClient > > giraph code base is 1.1.0-SNAPSHOT, downloaded on Jan 2, 2014 > > looking at the checkPerNodeResourcesAvailable Method. > > The first issue is that the number of containers available is calculated = > using node.getNumContainers(). Looking at the yarn documentation this = > is the number of containers currently running on the node. So with a = > yarn cluster with no jobs running all nodes report 0 containers. > > > The second issue (in the same method) is this if block: > > if (workers < numContainers) { > throw new RuntimeException("Giraph job requires " + workers + > " containers to run; cluster only hosts " + numContainers); > > } > > So with the current set up if a cluster has 4 containers currently = > running and a graph job is submitted that requires 2 containers the job = > will fail saying =93Giraph job requires 2 containers to run; cluster = > only hosts 4=94. > > the if statement should be =93 workers > numContainers=94 and = > numContainers should reflect the total number of containers available, = > not the number of containers currently running. I don=92t know yarn = > well so i don=92t know if such a number is available at all. > > For the time being i plan on getting this working by bypassing the check = > all together. > > > > >
