Hi Ramesh! Thanks for bringing this up, and thanks for trying out the new out-of-core mechanism. The new out-of-core mechanism has not been integrated with checkpointing yet. This is part of an ongoing project, and we should have the integration within a few weeks. In the meantime, you can try out-of-core without checkpointing enabled.
Best, Hassan On Saturday, May 14, 2016, Ramesh Krishnan <ramesh.154...@gmail.com> wrote: > PFA the correct logs for the concurrent exception > > 2016-05-14 19:10:55,733 ERROR [ooc-io-0] > org.apache.giraph.utils.LogStacktraceCallable: Execution of callable failed > java.lang.RuntimeException: java.io.EOFException > at > org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:76) > at > org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:30) > at > org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:392) > at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:47) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.readOutEdges(DiskBackedPartitionStore.java:286) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadInMemoryPartitionData(DiskBackedPartitionStore.java:329) > at > org.apache.giraph.ooc.data.OutOfCoreDataManager.loadPartitionData(OutOfCoreDataManager.java:195) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadPartitionData(DiskBackedPartitionStore.java:360) > at > org.apache.giraph.ooc.io.LoadPartitionIOCommand.execute(LoadPartitionIOCommand.java:64) > at > org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:72) > ... 6 more > 2016-05-14 19:10:55,737 INFO [ooc-io-0] > org.apache.giraph.ooc.OutOfCoreIOCallableFactory: afterExecute: an > out-of-core thread terminated unexpectedly with > java.util.concurrent.ExecutionException: java.lang.RuntimeException: > java.io.EOFException > 2016-05-14 19:10:55,739 INFO [checkpoint-vertices-7] > org.apache.giraph.ooc.FixedOutOfCoreEngine: getNextPartition: waiting until a > partition becomes available! > 2016-05-14 19:10:56,426 ERROR [checkpoint-vertices-6] > org.apache.giraph.utils.LogStacktraceCallable: Execution of callable failed > java.lang.RuntimeException: Job Failed due to a failure in an out-of-core IO > thread > at > org.apache.giraph.ooc.FixedOutOfCoreEngine.getNextPartition(FixedOutOfCoreEngine.java:81) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.getNextPartition(DiskBackedPartitionStore.java:187) > at > org.apache.giraph.worker.BspServiceWorker$3$1.call(BspServiceWorker.java:1398) > at > org.apache.giraph.worker.BspServiceWorker$3$1.call(BspServiceWorker.java:1392) > at > org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > > On Sun, May 15, 2016 at 1:02 AM, Ramesh Krishnan <ramesh.154...@gmail.com > <javascript:_e(%7B%7D,'cvml','ramesh.154...@gmail.com');>> wrote: > >> >> Hi Team, >> >> I have the latest build of giraph running on a 5 node cluster. When i try >> to use OutofCore Graph option for a huge data set like 600Milion edges i am >> running into >> the following exception. Please find below the script being executed and >> the exception logs. I have tried all possible ways and could not avoid this >> issue , i am really in need of your help. >> >> *Script:*hadoop jar >> /usr/local/giraph.back.1.2.0/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.7.0-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner -Dmapreduce.task.timeout=12000000 >> -Dmapred.job.tracker=ip-172-31-42-220.eu-west-1.compute.internal:8021 >> -Dmapreduce.map.memory.mb=23480 -Dmapreduce.map.java.opts=-Xmx22480m >> org.apache.giraph.examples.ConnectedComponentsComputation -vif >> org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /test/input_10M >> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> /test/ouput_10M -w 5 -ca >> giraph.userPartitionCount=150,giraph.SplitMasterWorker=true,giraph.isStaticGraph=true,giraph.maxPartitionsInMemory=10,mapred.map.max.attempts=2,giraph.maxMessagesInMemory=100,giraph.numOutputThreads=10,giraph.useOutOfCoreMessages=true,giraph.numOutputThreads=4,giraph.numInputThreads=4,giraph.useOutOfCoreGraph=true,giraph.cleanupCheckpointsAfterSuccess=true,giraph.checkpointFrequency=1 >> >> >> >> >> *Exception:hadoop jar >> /usr/local/giraph.back.1.2.0/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.7.0-jar-with-dependencies.jar >> org.apache.giraph.GiraphRunner -Dmapreduce.task.timeout=12000000 >> -Dmapred.job.tracker=ip-172-31-42-220.eu-west-1.compute.internal:8021 >> -Dmapreduce.map.memory.mb=23480 -Dmapreduce.map.java.opts=-Xmx22480m >> org.apache.giraph.examples.ConnectedComponentsComputation -vif >> org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /test/input_10M >> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >> /test/ouput_10M -w 5 -ca >> giraph.userPartitionCount=150,giraph.SplitMasterWorker=true,giraph.isStaticGraph=true,giraph.maxPartitionsInMemory=10,mapred.map.max.attempts=2,giraph.maxMessagesInMemory=100,giraph.numOutputThreads=10,giraph.useOutOfCoreMessages=true,giraph.numOutputThreads=4,giraph.numInputThreads=4,giraph.useOutOfCoreGraph=true,giraph.cleanupCheckpointsAfterSuccess=true,giraph.checkpointFrequency=1* >> >> *thanks* >> >> *Ramesh* >> >> >
