Probably later today. On Tuesday, February 11, 2014, Sebastian Schelter <[email protected]> wrote:
> Is there an approximate date when this new DiskBackedPartitionStore will > become available? I'd be very interested to test it. > > --sebastian > > On 02/10/2014 07:33 PM, Claudio Martella wrote: > >> Hi Sebastian, >> >> this is indeed a bug. Apparently some updates to the PartitionStore API >> introduced a regression. Armando has worked on a re-design of the >> DiskBackedPartitionStore, which we will be uploaded for a review as soon >> as >> we finish testing it. >> >> Stay tuned. >> >> >> On Mon, Feb 10, 2014 at 6:42 PM, Sebastian Schelter <[email protected]> >> wrote: >> >> Hi, >>> >>> I'm trying to process a huge graph on our cluster and therefore enabled >>> out-of-core. Unfortunately, I run into a NullPointerException during the >>> input phase. >>> >>> What can I do to fix this? I'm using the following settings: >>> >>> giraph.oneToAllMsgSending=true >>> giraph.isStaticGraph=true >>> giraph.numComputeThreads=15 >>> giraph.numInputThreads=15 >>> giraph.numOutputThreads=15 >>> giraph.maxNumberOfSupersteps=30 >>> giraph.useOutOfCoreGraph=true >>> >>> Here's the log for the exception: >>> >>> >>> java.lang.IllegalStateException: run: Caught an unrecoverable exception >>> waitFor: ExecutionException occurred while waiting for >>> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6e172129 >>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper. >>> java:101) >>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask. >>> java:764) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at org.apache.hadoop.security.UserGroupInformation.doAs( >>> UserGroupInformation.java:1121) >>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException >>> occurred while waiting for org.apache.giraph.utils.ProgressableUtils$ >>> FutureWaitable@6e172129 >>> at org.apache.giraph.utils.ProgressableUtils.waitFor( >>> ProgressableUtils.java:181) >>> at org.apache.giraph.utils.ProgressableUtils.waitForever( >>> ProgressableUtils.java:139) >>> at org.apache.giraph.utils.ProgressableUtils.waitForever( >>> ProgressableUtils.java:124) >>> at org.apache.giraph.utils.ProgressableUtils.getFutureResult( >>> ProgressableUtils.java:87) >>> at org.apache.giraph.utils.ProgressableUtils. >>> getResultsWithNCallables(ProgressableUtils.java:221) >>> at org.apache.giraph.worker.BspServiceWorker.loadInputSplits( >>> BspServiceWorker.java:281) >>> at org.apache.giraph.worker.BspServiceWorker.loadVertices( >>> BspServiceWorker.java:325) >>> at org.apache.giraph.worker.BspServiceWorker.setup( >>> BspServiceWorker.java:506) >>> at org.apache.giraph.graph.GraphTaskManager.execute( >>> GraphTaskManager.java:244) >>> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91) >>> ... 7 more >>> Caused by: java.util.concurrent.ExecutionException: java.lang. >>> IllegalStateException: >>> getOrCreatePartition: cannot retrieve partition 14 >>> at java.util.concurrent.FutureTask$Sync.innerGet( >>> FutureTask.java:262) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:119) >>> at org.apache.giraph.utils.ProgressableUtils$ >>> FutureWaitable.waitFor(ProgressableUtils.java:300) >>> at org.apache.giraph.utils.ProgressableUtils.waitFor( >>> ProgressableUtils.java:173) >>> ... 16 more >>> Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot >>> retrieve partition 14 >>> at org.apache.giraph.partition.DiskBackedPartitionStore. >>> getOrCreatePartition(DiskBackedPartitionStore.java:243) >>> at org.apache.giraph.comm.requests.SendWorkerVerticesRequest. >>> doRequest(SendWorkerVerticesRequest.java:110) >>> at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces >>> sor.doRequest(NettyWorkerClientRequestProcessor.java:482) >>> at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces >>> sor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276) >>> at org.apache.giraph.worker.VertexInputSplitsCallable. >>> readInputSplit(VertexInputSplitsCallable.java:172) >>> at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit( >>> InputSplitsCallable.java:267) >>> at org.apache.giraph.worker.InputSplitsCallable.call( >>> InputSplitsCallable.java:211) >>> at org.apache.giraph.worker.InputSplitsCallable.call( >>> InputSplitsCallable.java:60) >>> at org.apache.giraph.utils.LogStacktraceCallable.call( >>> LogStacktraceCallable.java:51) >>> at java.util.concurrent.FutureTask$Sync.innerRun( >>> FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>> ThreadPoolExecutor.java:1110) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>> ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:722) >>> Caused by: java.util.concurrent.ExecutionException: >>> java.lang.NullPointerException >>> at java.util.concurrent.FutureTask$Sync.innerGet( >>> FutureTask.java:252) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >>> at org.apache.giraph.partition.DiskBackedPartitionStore. >>> getOrCreatePartition(DiskBackedPartitionStore.java:228) >>> ... 13 more >>> Caused by: java.lang.NullPointerException >>> at org.apache.giraph.partition.DiskBackedPartitionStore$ >>> GetPartition.call(DiskBackedPartitionStore.java:692) >>> at org.apache.giraph.partition.DiskBackedPartitionStore$ >>> GetPartition.call(DiskBackedPartitionStore.java:658) >>> at java.util.concurrent.FutureTask$Sync.innerRun( >>> FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at org.apache.giraph.partition.DiskBackedPartitionStore$ >>> DirectExecutorService.execute(DiskBackedPartitionStore.java:972) >>> at java.util.concurrent.AbstractExecutorService.submit( >>> AbstractExecutorService.java:132) >>> ... 14 more >>> >>> >> >> >> > -- Claudio Martella
