Hi Ramesh, First of all, it seems that you don't want the unsafe-serialization for some reason, and that makes the out-of-core to be very slow. (De)Serialization is one of the performance critical components in the new out-of-core design.
Second, you are using the out-of-core flags based on the documentation on the website. Most of those flags don't exist anymore, and the documentation on the website is stale. I've had a couple of diffs to clean up the out-of-core flags as well as the documentation on the website, but the team didn't let the diffs to go in the code base. I'm not sure if those diffs would be part of the code base anytime soon. Without those cleanups and changes, the out-of-core can be enabled with some other set of flags, which are pretty complicated to understand (and use)! At this point, I simply *discourage* using the new out-of-core design. You can simply use the 1.1 release and use the old out-of-core if that works for you. Hassan On Mon, Jul 25, 2016 at 9:35 PM, Ramesh Krishnan <ramesh.154...@gmail.com> wrote: > Hi Team, > I have been trying out of core graph option with 1.2 branch of giraph . I > have been > continously recieving the buffer underflow exception exactly when the > superstep1 starts. Any pointers would be really helpful here , > > Logs: > > 2016-07-25 16:08:20,664 ERROR [ooc-io-0] > org.apache.giraph.utils.LogStacktraceCallable: Execution of callable failed > java.lang.RuntimeException: call: execution of IO command > LoadPartitionIOCommand: (partitionId = 258, superstep = 1) failed! > at > org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:115) > at > org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:36) > at > org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:67) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow. > at com.esotericsoftware.kryo.io.Input.require(Input.java:199) > at com.esotericsoftware.kryo.io.UnsafeInput.readInt(UnsafeInput.java:82) > at > com.esotericsoftware.kryo.io.KryoDataInput.readInt(KryoDataInput.java:87) > at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:47) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.readOutEdges(DiskBackedPartitionStore.java:245) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadInMemoryPartitionData(DiskBackedPartitionStore.java:278) > at > org.apache.giraph.ooc.data.DiskBackedDataStore.loadPartitionDataProxy(DiskBackedDataStore.java:233) > at > org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadPartitionData(DiskBackedPartitionStore.java:311) > at > org.apache.giraph.ooc.command.LoadPartitionIOCommand.execute(LoadPartitionIOCommand.java:66) > at > org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:99) > ... 6 more > > Command: > hadoop jar > /usr/local/giraph-1.2/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner -Dmapreduce.task.timeout=12000000 > -Dmapred.job.tracker=ip-172-31-42-220.eu-west-1.compute.internal:8021 > -Dmapreduce.map.memory.mb=43480 -Dmapreduce.map.java.opts=-Xmx42480m > org.apache.giraph.examples.ConnectedComponentsComputation -vif > org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /VUID/input_2B > -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /VUID/ouput_2B -w 4 -ca > giraph.userPartitionCount=400,giraph.SplitMasterWorker=true,giraph.isStaticGraph=true,giraph.maxPartitionsInMemory=10,mapred.map.max.attempts=2,giraph.maxMessagesInMemory=100,giraph.numOutputThreads=1,giraph.useOutOfCoreMessages=true,giraph.numInputThreads=1,giraph.useOutOfCoreGraph=true,giraph.numComputeThreads=1,giraph.messagesBufferSize=8192000,giraph.partitionsDirectory=_bs,giraph.useUnsafeSerialization=false > > > Thanks, > > Ramesh > > >
