Hi folks.. While executing one algorithm with enabling checkpointing feature on,system is running out of memory. I gave following command
hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dmapred.job.map.memory.mb=1500 -Dgiraph.useSuperstepCounters=false -Dgiraph.useOutOfCoreMessages=true -Dgiraph.checkpointFrequency=1 org.apache.giraph.examples.MyShortestDistance -vif org.apache.giraph.examples.io.formats.MyShortestDistanceVertexInputFormat -vip /user/hduser/big_input/clustering -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/sp_output537/sd_output -w 1 -mc org.apache.giraph.examples.MyShortestDistance\$MyMasterCompute By executing this command /app/hadoop/tmp/dfs/name/data/current directory is consuming lots of space,approx 90GB(or may be more than this) This job execution is involving thousands of supersteps..If i do not enable checkpoint feature on then algorithm fails due to worker failure error after 19000 around supersteps. Pls suggest me how to deal with this issue?? Thanks Jyoti
