Hi Sebastian, this is indeed a bug. Apparently some updates to the PartitionStore API introduced a regression. Armando has worked on a re-design of the DiskBackedPartitionStore, which we will be uploaded for a review as soon as we finish testing it.
Stay tuned. On Mon, Feb 10, 2014 at 6:42 PM, Sebastian Schelter <[email protected]> wrote: > Hi, > > I'm trying to process a huge graph on our cluster and therefore enabled > out-of-core. Unfortunately, I run into a NullPointerException during the > input phase. > > What can I do to fix this? I'm using the following settings: > > giraph.oneToAllMsgSending=true > giraph.isStaticGraph=true > giraph.numComputeThreads=15 > giraph.numInputThreads=15 > giraph.numOutputThreads=15 > giraph.maxNumberOfSupersteps=30 > giraph.useOutOfCoreGraph=true > > Here's the log for the exception: > > > java.lang.IllegalStateException: run: Caught an unrecoverable exception > waitFor: ExecutionException occurred while waiting for > org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6e172129 > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: java.lang.IllegalStateException: waitFor: ExecutionException > occurred while waiting for org.apache.giraph.utils.ProgressableUtils$ > FutureWaitable@6e172129 > at org.apache.giraph.utils.ProgressableUtils.waitFor( > ProgressableUtils.java:181) > at org.apache.giraph.utils.ProgressableUtils.waitForever( > ProgressableUtils.java:139) > at org.apache.giraph.utils.ProgressableUtils.waitForever( > ProgressableUtils.java:124) > at org.apache.giraph.utils.ProgressableUtils.getFutureResult( > ProgressableUtils.java:87) > at org.apache.giraph.utils.ProgressableUtils. > getResultsWithNCallables(ProgressableUtils.java:221) > at org.apache.giraph.worker.BspServiceWorker.loadInputSplits( > BspServiceWorker.java:281) > at org.apache.giraph.worker.BspServiceWorker.loadVertices( > BspServiceWorker.java:325) > at org.apache.giraph.worker.BspServiceWorker.setup( > BspServiceWorker.java:506) > at org.apache.giraph.graph.GraphTaskManager.execute( > GraphTaskManager.java:244) > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91) > ... 7 more > Caused by: java.util.concurrent.ExecutionException: > java.lang.IllegalStateException: > getOrCreatePartition: cannot retrieve partition 14 > at java.util.concurrent.FutureTask$Sync.innerGet( > FutureTask.java:262) > at java.util.concurrent.FutureTask.get(FutureTask.java:119) > at org.apache.giraph.utils.ProgressableUtils$ > FutureWaitable.waitFor(ProgressableUtils.java:300) > at org.apache.giraph.utils.ProgressableUtils.waitFor( > ProgressableUtils.java:173) > ... 16 more > Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot > retrieve partition 14 > at org.apache.giraph.partition.DiskBackedPartitionStore. > getOrCreatePartition(DiskBackedPartitionStore.java:243) > at org.apache.giraph.comm.requests.SendWorkerVerticesRequest. > doRequest(SendWorkerVerticesRequest.java:110) > at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces > sor.doRequest(NettyWorkerClientRequestProcessor.java:482) > at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces > sor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276) > at org.apache.giraph.worker.VertexInputSplitsCallable. > readInputSplit(VertexInputSplitsCallable.java:172) > at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit( > InputSplitsCallable.java:267) > at org.apache.giraph.worker.InputSplitsCallable.call( > InputSplitsCallable.java:211) > at org.apache.giraph.worker.InputSplitsCallable.call( > InputSplitsCallable.java:60) > at org.apache.giraph.utils.LogStacktraceCallable.call( > LogStacktraceCallable.java:51) > at java.util.concurrent.FutureTask$Sync.innerRun( > FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1110) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.util.concurrent.ExecutionException: > java.lang.NullPointerException > at java.util.concurrent.FutureTask$Sync.innerGet( > FutureTask.java:252) > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > at org.apache.giraph.partition.DiskBackedPartitionStore. > getOrCreatePartition(DiskBackedPartitionStore.java:228) > ... 13 more > Caused by: java.lang.NullPointerException > at org.apache.giraph.partition.DiskBackedPartitionStore$ > GetPartition.call(DiskBackedPartitionStore.java:692) > at org.apache.giraph.partition.DiskBackedPartitionStore$ > GetPartition.call(DiskBackedPartitionStore.java:658) > at java.util.concurrent.FutureTask$Sync.innerRun( > FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at org.apache.giraph.partition.DiskBackedPartitionStore$ > DirectExecutorService.execute(DiskBackedPartitionStore.java:972) > at java.util.concurrent.AbstractExecutorService.submit( > AbstractExecutorService.java:132) > ... 14 more > -- Claudio Martella
