Re: out of core option
(DiskBackedPartitionStore.java:228)
... 13 more Caused by: java.lang.NullPointerException at
org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:692)
at
org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:658)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
java.util.concurrent.FutureTask.run(FutureTask.java:166) at
org.apache.giraph.partition.DiskBackedPartitionStore$DirectExecutorService.execute(DiskBackedPartitionStore.java:972)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
... 14 more
Thanks,
Sebastian
2013/12/5 Rob Vesse rve...@dotnetrdf.org
Sebastian
You've made a minor typo in the configuration setting which means you
haven't actually enabled out of core graph mode.
You have *giraph.useOutOfCoreGiraph *when it should be
*giraph.useOutOfCoreGraph
*– note that the last word is Graph not Giraph
Rob
From: Sebastian Stipkovic sebastian.stipko...@gmail.com
Reply-To: user@giraph.apache.org
Date: Thursday, 5 December 2013 20:39
To: user@giraph.apache.org
Subject: out of core option
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job
with out-of-memory-option works only well with a tiny graph (0.9 GB). What
is Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue*fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem
*mapred.task.cache.levels*2*giraph.vertexOutputFormatClass*
org.apache.giraph.examples.MyShortestPaths$MyOutputFormat
*hadoop.tmp.dir*/app/hadoop/tmp*hadoop.native.lib*true*map.sort.class*org.apache.hadoop.util.QuickSort
*dfs.namenode.decommission.nodes.per.interval*5
*dfs.https.need.client.auth*false *ipc.client.idlethreshold*4000
*dfs.datanode.data.dir.perm*755*mapred.system.dir*
${hadoop.tmp.dir}/mapred/system
*mapred.job.tracker.persist.jobstatus.hours*0*dfs.datanode.address*
0.0.0.0:50010*dfs.namenode.logging.level*info
*dfs.block.access.token.enable*
false*io.skip.checksum.errors*false*fs.default.name
http://fs.default.name* hdfs://localhost:54310
*mapred.cluster.reduce.memory.mb*-1*mapred.child.tmp* ./tmp
*fs.har.impl.disable.cache*true*dfs.safemode.threshold.pct*0.999f
*mapred.skip.reduce.max.skip.groups*0*dfs.namenode.handler.count*10
*dfs.blockreport.initialDelay* 0*mapred.heartbeats.in.second*100
*mapred.tasktracker.dns.nameserver*default*io.sort.factor* 10
*mapred.task.timeout*60*giraph.maxWorkers*1
*mapred.max.tracker.failures* 4
*hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory
*mapred.job.tracker.jobhistory.lru.cache.size* 5*fs.hdfs.impl*
org.apache.hadoop.hdfs.DistributedFileSystem
*mapred.queue.default.acl-administer-jobs* *
*dfs.block.access.key.update.interval*600
*mapred.skip.map.auto.incr.proc.count*true
*mapreduce.job.complete.cancel.delegation.tokens*true
*io.mapfile.bloom.size*1048576
*mapreduce.reduce.shuffle.connect.timeout* 18
*dfs.safemode.extension*3
*mapred.jobtracker.blacklist.fault-timeout-window*180
*tasktracker.http.threads*40*mapred.job.shuffle.merge.percent*0.66
*mapreduce.inputformat.class* org.apache.giraph.bsp.BspInputFormat
*fs.ftp.impl*org.apache.hadoop.fs.ftp.FTPFileSystem*user.name
http://user.name* hduser*mapred.output.compress*false
*io.bytes.per.checksum*512*giraph.isStaticGraph* true
*mapred.healthChecker.script.timeout*60
*topology.node.switch.mapping.impl*
org.apache.hadoop.net.ScriptBasedMapping
*dfs.https.server.keystore.resource*ssl-server.xml
*mapred.reduce.slowstart.completed.maps*0.05
*mapred.reduce.max.attempts*4*fs.ramfs.impl*
org.apache.hadoop.fs.InMemoryFileSystem
*dfs.block.access.token.lifetime* 600*dfs.name.edits.dir*
${dfs.name.dir}*mapred.skip.map.max.skip.records*0
*mapred.cluster.map.memory.mb*-1*hadoop.security.group.mapping*
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
*mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo
*mapred.jar*hdfs://localhost:54310
/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201312051827_0001/job.jar
*dfs.block.size*67108864*fs.s3.buffer.dir*${hadoop.tmp.dir}/s3
*job.end.retry.attempts* 0*fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem*mapred.local.dir.minspacestart*0
*mapred.output.compression.type*RECORD*dfs.datanode.ipc.address*
0.0.0.0:50020 *dfs.permissions*true*topology.script.number.args*100
*io.mapfile.bloom.error.rate* 0.005
*mapred.cluster.max.reduce.memory.mb*-1*mapred.max.tracker.blacklists*4
*mapred.task.profile.maps*0-2*dfs.datanode.https.address*0.0.0.0:50475
*mapred.userlog.retain.hours*24*dfs.secondary.http.address*
0.0.0.0:50090 *dfs.replication.max*512
Re: out of core option
(DiskBackedPartitionStore.java:692)
at
org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:658)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at
java.util.concurrent.FutureTask.run(FutureTask.java:166) at
org.apache.giraph.partition.DiskBackedPartitionStore$DirectExecutorService.execute(DiskBackedPartitionStore.java:972)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
... 14 more
Thanks,
Sebastian
2013/12/5 Rob Vesse rve...@dotnetrdf.org
Sebastian
You've made a minor typo in the configuration setting which means you
haven't actually enabled out of core graph mode.
You have *giraph.useOutOfCoreGiraph *when it should be
*giraph.useOutOfCoreGraph
*– note that the last word is Graph not Giraph
Rob
From: Sebastian Stipkovic sebastian.stipko...@gmail.com
Reply-To: user@giraph.apache.org
Date: Thursday, 5 December 2013 20:39
To: user@giraph.apache.org
Subject: out of core option
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job
with out-of-memory-option works only well with a tiny graph (0.9 GB). What
is Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue*fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem
*mapred.task.cache.levels*2*giraph.vertexOutputFormatClass*
org.apache.giraph.examples.MyShortestPaths$MyOutputFormat
*hadoop.tmp.dir*/app/hadoop/tmp*hadoop.native.lib*true*map.sort.class*org.apache.hadoop.util.QuickSort
*dfs.namenode.decommission.nodes.per.interval*5
*dfs.https.need.client.auth*false *ipc.client.idlethreshold*4000
*dfs.datanode.data.dir.perm*755*mapred.system.dir*
${hadoop.tmp.dir}/mapred/system
*mapred.job.tracker.persist.jobstatus.hours*0*dfs.datanode.address*
0.0.0.0:50010*dfs.namenode.logging.level*info
*dfs.block.access.token.enable*
false*io.skip.checksum.errors*false*fs.default.name
http://fs.default.name* hdfs://localhost:54310
*mapred.cluster.reduce.memory.mb*-1*mapred.child.tmp* ./tmp
*fs.har.impl.disable.cache*true*dfs.safemode.threshold.pct*0.999f
*mapred.skip.reduce.max.skip.groups*0*dfs.namenode.handler.count*10
*dfs.blockreport.initialDelay* 0*mapred.heartbeats.in.second*100
*mapred.tasktracker.dns.nameserver*default*io.sort.factor* 10
*mapred.task.timeout*60*giraph.maxWorkers*1
*mapred.max.tracker.failures* 4*hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory
*mapred.job.tracker.jobhistory.lru.cache.size* 5*fs.hdfs.impl*
org.apache.hadoop.hdfs.DistributedFileSystem
*mapred.queue.default.acl-administer-jobs* *
*dfs.block.access.key.update.interval*600
*mapred.skip.map.auto.incr.proc.count*true
*mapreduce.job.complete.cancel.delegation.tokens*true
*io.mapfile.bloom.size*1048576*mapreduce.reduce.shuffle.connect.timeout*
18*dfs.safemode.extension*3
*mapred.jobtracker.blacklist.fault-timeout-window*180
*tasktracker.http.threads*40*mapred.job.shuffle.merge.percent*0.66
*mapreduce.inputformat.class* org.apache.giraph.bsp.BspInputFormat
*fs.ftp.impl*org.apache.hadoop.fs.ftp.FTPFileSystem*user.name
http://user.name* hduser*mapred.output.compress*false
*io.bytes.per.checksum*512*giraph.isStaticGraph* true
*mapred.healthChecker.script.timeout*60
*topology.node.switch.mapping.impl*
org.apache.hadoop.net.ScriptBasedMapping
*dfs.https.server.keystore.resource*ssl-server.xml
*mapred.reduce.slowstart.completed.maps*0.05
*mapred.reduce.max.attempts*4*fs.ramfs.impl*
org.apache.hadoop.fs.InMemoryFileSystem*dfs.block.access.token.lifetime*
600*dfs.name.edits.dir*${dfs.name.dir}*mapred.skip.map.max.skip.records*
0 *mapred.cluster.map.memory.mb*-1*hadoop.security.group.mapping*
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
*mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo
*mapred.jar*hdfs://localhost:54310
/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201312051827_0001/job.jar
*dfs.block.size*67108864*fs.s3.buffer.dir*${hadoop.tmp.dir}/s3
*job.end.retry.attempts* 0*fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem*mapred.local.dir.minspacestart*0
*mapred.output.compression.type*RECORD*dfs.datanode.ipc.address*
0.0.0.0:50020 *dfs.permissions*true*topology.script.number.args*100
*io.mapfile.bloom.error.rate* 0.005*mapred.cluster.max.reduce.memory.mb*
-1*mapred.max.tracker.blacklists*4 *mapred.task.profile.maps*0-2
*dfs.datanode.https.address*0.0.0.0:50475 *mapred.userlog.retain.hours*
24*dfs.secondary.http.address*0.0.0.0:50090 *dfs.replication.max*512
*mapred.job.tracker.persist.jobstatus.active*false
*hadoop.security.authorization* false*local.cache.size*10737418240
*dfs.namenode.delegation.token.renew-interval*8640
*mapred.min.split.size*0
Re: out of core option
) at
java.util.concurrent.FutureTask.run(FutureTask.java:166) at
org.apache.giraph.partition.DiskBackedPartitionStore$DirectExecutorService.execute(DiskBackedPartitionStore.java:972)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
... 14 more
Thanks,
Sebastian
2013/12/5 Rob Vesse rve...@dotnetrdf.org
Sebastian
You've made a minor typo in the configuration setting which means you
haven't actually enabled out of core graph mode.
You have *giraph.useOutOfCoreGiraph *when it should be
*giraph.useOutOfCoreGraph
*– note that the last word is Graph not Giraph
Rob
From: Sebastian Stipkovic sebastian.stipko...@gmail.com
Reply-To: user@giraph.apache.org
Date: Thursday, 5 December 2013 20:39
To: user@giraph.apache.org
Subject: out of core option
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job
with out-of-memory-option works only well with a tiny graph (0.9 GB). What
is Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue*fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem
*mapred.task.cache.levels*2*giraph.vertexOutputFormatClass*
org.apache.giraph.examples.MyShortestPaths$MyOutputFormat
*hadoop.tmp.dir*/app/hadoop/tmp*hadoop.native.lib*true*map.sort.class*org.apache.hadoop.util.QuickSort
*dfs.namenode.decommission.nodes.per.interval*5
*dfs.https.need.client.auth*false *ipc.client.idlethreshold*4000
*dfs.datanode.data.dir.perm*755*mapred.system.dir*
${hadoop.tmp.dir}/mapred/system
*mapred.job.tracker.persist.jobstatus.hours*0*dfs.datanode.address*
0.0.0.0:50010*dfs.namenode.logging.level*info
*dfs.block.access.token.enable*
false*io.skip.checksum.errors*false*fs.default.name
http://fs.default.name* hdfs://localhost:54310
*mapred.cluster.reduce.memory.mb*-1*mapred.child.tmp* ./tmp
*fs.har.impl.disable.cache*true*dfs.safemode.threshold.pct*0.999f
*mapred.skip.reduce.max.skip.groups*0*dfs.namenode.handler.count*10
*dfs.blockreport.initialDelay* 0*mapred.heartbeats.in.second*100
*mapred.tasktracker.dns.nameserver*default*io.sort.factor* 10
*mapred.task.timeout*60*giraph.maxWorkers*1
*mapred.max.tracker.failures* 4*hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory
*mapred.job.tracker.jobhistory.lru.cache.size* 5*fs.hdfs.impl*
org.apache.hadoop.hdfs.DistributedFileSystem
*mapred.queue.default.acl-administer-jobs* *
*dfs.block.access.key.update.interval*600
*mapred.skip.map.auto.incr.proc.count*true
*mapreduce.job.complete.cancel.delegation.tokens*true
*io.mapfile.bloom.size*1048576*mapreduce.reduce.shuffle.connect.timeout*
18*dfs.safemode.extension*3
*mapred.jobtracker.blacklist.fault-timeout-window*180
*tasktracker.http.threads*40*mapred.job.shuffle.merge.percent*0.66
*mapreduce.inputformat.class* org.apache.giraph.bsp.BspInputFormat
*fs.ftp.impl*org.apache.hadoop.fs.ftp.FTPFileSystem*user.name
http://user.name* hduser*mapred.output.compress*false
*io.bytes.per.checksum*512*giraph.isStaticGraph* true
*mapred.healthChecker.script.timeout*60
*topology.node.switch.mapping.impl*
org.apache.hadoop.net.ScriptBasedMapping
*dfs.https.server.keystore.resource*ssl-server.xml
*mapred.reduce.slowstart.completed.maps*0.05 *mapred.reduce.max.attempts*
4*fs.ramfs.impl*org.apache.hadoop.fs.InMemoryFileSystem
*dfs.block.access.token.lifetime* 600*dfs.name.edits.dir*${dfs.name.dir}
*mapred.skip.map.max.skip.records*0 *mapred.cluster.map.memory.mb*-1
*hadoop.security.group.mapping*
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
*mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo
*mapred.jar*hdfs://localhost:54310
/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201312051827_0001/job.jar
*dfs.block.size*67108864*fs.s3.buffer.dir*${hadoop.tmp.dir}/s3
*job.end.retry.attempts* 0*fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem*mapred.local.dir.minspacestart*0
*mapred.output.compression.type*RECORD*dfs.datanode.ipc.address*
0.0.0.0:50020 *dfs.permissions*true*topology.script.number.args*100
*io.mapfile.bloom.error.rate* 0.005*mapred.cluster.max.reduce.memory.mb*
-1*mapred.max.tracker.blacklists*4 *mapred.task.profile.maps*0-2
*dfs.datanode.https.address*0.0.0.0:50475 *mapred.userlog.retain.hours*24
*dfs.secondary.http.address*0.0.0.0:50090 *dfs.replication.max*512
*mapred.job.tracker.persist.jobstatus.active*false
*hadoop.security.authorization* false*local.cache.size*10737418240
*dfs.namenode.delegation.token.renew-interval*8640
*mapred.min.split.size*0*mapred.map.tasks*2*mapred.child.java.opts*-Xmx4000m
*mapreduce.job.counters.limit*120*dfs.https.client.keystore.resource*
ssl-client.xml *mapred.job.queue.name http://mapred.job.queue.name*
default*dfs.https.address
Re: out of core option
Each worker is allocated *mapred.child.java.opts *memory, which in your
case is 4000M. Check if your server doesn't have enough memory for 2
Mappers. Also the out of memory option is available in two forms.
1. Out of core graph
2. Out of core messages.
Currently you are setting only the out of core graph and not the out of
core messages. Enable both of them. More information about options can be
found here: http://giraph.apache.org/options.html
set -D giraph.useOutOfCoreGraph=true -D giraph.useOutOfCoreMessages=true
while passing options to GiraphRunner.
Thanks,
Ameya
On Thu, Dec 5, 2013 at 12:39 PM, Sebastian Stipkovic
sebastian.stipko...@gmail.com wrote:
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job with
out-of-memory-option works only well with a tiny graph (0.9 GB). What is
Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue *fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem
*mapred.task.cache.levels*2 *giraph.vertexOutputFormatClass*
org.apache.giraph.examples.MyShortestPaths$MyOutputFormat
*hadoop.tmp.dir*/app/hadoop/tmp *hadoop.native.lib*true *map.sort.class*
org.apache.hadoop.util.QuickSort
*dfs.namenode.decommission.nodes.per.interval*5
*dfs.https.need.client.auth*false *ipc.client.idlethreshold*4000
*dfs.datanode.data.dir.perm*755 *mapred.system.dir*
${hadoop.tmp.dir}/mapred/system
*mapred.job.tracker.persist.jobstatus.hours*0 *dfs.datanode.address*
0.0.0.0:50010 *dfs.namenode.logging.level*info
*dfs.block.access.token.enable*false *io.skip.checksum.errors*false
*fs.default.name
http://fs.default.name*hdfs://localhost:54310
*mapred.cluster.reduce.memory.mb*-1 *mapred.child.tmp*./tmp
*fs.har.impl.disable.cache*true *dfs.safemode.threshold.pct*0.999f
*mapred.skip.reduce.max.skip.groups*0 *dfs.namenode.handler.count*10
*dfs.blockreport.initialDelay*0 *mapred.heartbeats.in.second*100
*mapred.tasktracker.dns.nameserver*default *io.sort.factor*10
*mapred.task.timeout*60 *giraph.maxWorkers*1
*mapred.max.tracker.failures*4 *hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory
*mapred.job.tracker.jobhistory.lru.cache.size*5 *fs.hdfs.impl*
org.apache.hadoop.hdfs.DistributedFileSystem
*mapred.queue.default.acl-administer-jobs**
*dfs.block.access.key.update.interval*600
*mapred.skip.map.auto.incr.proc.count*true
*mapreduce.job.complete.cancel.delegation.tokens*true
*io.mapfile.bloom.size*1048576 *mapreduce.reduce.shuffle.connect.timeout*
18 *dfs.safemode.extension*3
*mapred.jobtracker.blacklist.fault-timeout-window*180
*tasktracker.http.threads*40 *mapred.job.shuffle.merge.percent*0.66
*mapreduce.inputformat.class*org.apache.giraph.bsp.BspInputFormat
*fs.ftp.impl*org.apache.hadoop.fs.ftp.FTPFileSystem *user.name
http://user.name*hduser *mapred.output.compress*false
*io.bytes.per.checksum*512 *giraph.isStaticGraph*true
*mapred.healthChecker.script.timeout*60
*topology.node.switch.mapping.impl*
org.apache.hadoop.net.ScriptBasedMapping
*dfs.https.server.keystore.resource*ssl-server.xml
*mapred.reduce.slowstart.completed.maps*0.05 *mapred.reduce.max.attempts*
4 *fs.ramfs.impl*org.apache.hadoop.fs.InMemoryFileSystem
*dfs.block.access.token.lifetime*600 *dfs.name.edits.dir*${dfs.name.dir}
*mapred.skip.map.max.skip.records*0 *mapred.cluster.map.memory.mb*-1
*hadoop.security.group.mapping*
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
*mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo
*mapred.jar*
hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201312051827_0001/job.jar
*dfs.block.size*67108864 *fs.s3.buffer.dir*${hadoop.tmp.dir}/s3
*job.end.retry.attempts*0 *fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem *mapred.local.dir.minspacestart*0
*mapred.output.compression.type*RECORD *dfs.datanode.ipc.address*
0.0.0.0:50020 *dfs.permissions*true *topology.script.number.args*100
*io.mapfile.bloom.error.rate*0.005 *mapred.cluster.max.reduce.memory.mb*
-1 *mapred.max.tracker.blacklists*4 *mapred.task.profile.maps*0-2
*dfs.datanode.https.address*0.0.0.0:50475 *mapred.userlog.retain.hours*24
*dfs.secondary.http.address*0.0.0.0:50090 *dfs.replication.max*512
*mapred.job.tracker.persist.jobstatus.active*false
*hadoop.security.authorization*false *local.cache.size*10737418240
*dfs.namenode.delegation.token.renew-interval*8640
*mapred.min.split.size*0 *mapred.map.tasks*2 *mapred.child.java.opts*
-Xmx4000m *mapreduce.job.counters.limit*120
*dfs.https.client.keystore.resource*ssl-client.xml *mapred.job.queue.name
http://mapred.job.queue.name*default *dfs.https.address*0.0.0.0:50470
Re: out of core option
Sebastian
You've made a minor typo in the configuration setting which means you
haven't actually enabled out of core graph mode.
You have giraph.useOutOfCoreGiraph when it should be
giraph.useOutOfCoreGraph note that the last word is Graph not Giraph
Rob
From: Sebastian Stipkovic sebastian.stipko...@gmail.com
Reply-To: user@giraph.apache.org
Date: Thursday, 5 December 2013 20:39
To: user@giraph.apache.org
Subject: out of core option
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job with
out-of-memory-option works only well with a tiny graph (0.9 GB). What is
Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue
fs.s3n.implorg.apache.hadoop.fs.s3native.NativeS3FileSystem
mapred.task.cache.levels2
giraph.vertexOutputFormatClassorg.apache.giraph.examples.MyShortestPaths$MyOut
putFormat
hadoop.tmp.dir/app/hadoop/tmp
hadoop.native.libtrue
map.sort.classorg.apache.hadoop.util.QuickSort
dfs.namenode.decommission.nodes.per.interval5
dfs.https.need.client.authfalse
ipc.client.idlethreshold4000
dfs.datanode.data.dir.perm755
mapred.system.dir${hadoop.tmp.dir}/mapred/system
mapred.job.tracker.persist.jobstatus.hours0
dfs.datanode.address0.0.0.0:50010 http://0.0.0.0:50010
dfs.namenode.logging.levelinfo
dfs.block.access.token.enablefalse
io.skip.checksum.errorsfalse
fs.default.name http://fs.default.name hdfs://localhost:54310
mapred.cluster.reduce.memory.mb-1
mapred.child.tmp./tmp
fs.har.impl.disable.cachetrue
dfs.safemode.threshold.pct0.999f
mapred.skip.reduce.max.skip.groups0
dfs.namenode.handler.count10
dfs.blockreport.initialDelay0
mapred.heartbeats.in.second100
mapred.tasktracker.dns.nameserverdefault
io.sort.factor10
mapred.task.timeout60
giraph.maxWorkers1
mapred.max.tracker.failures4
hadoop.rpc.socket.factory.class.defaultorg.apache.hadoop.net.StandardSocketFac
tory
mapred.job.tracker.jobhistory.lru.cache.size5
fs.hdfs.implorg.apache.hadoop.hdfs.DistributedFileSystem
mapred.queue.default.acl-administer-jobs*
dfs.block.access.key.update.interval600
mapred.skip.map.auto.incr.proc.counttrue
mapreduce.job.complete.cancel.delegation.tokenstrue
io.mapfile.bloom.size1048576
mapreduce.reduce.shuffle.connect.timeout18
dfs.safemode.extension3
mapred.jobtracker.blacklist.fault-timeout-window180
tasktracker.http.threads40
mapred.job.shuffle.merge.percent0.66
mapreduce.inputformat.classorg.apache.giraph.bsp.BspInputFormat
fs.ftp.implorg.apache.hadoop.fs.ftp.FTPFileSystem
user.name http://user.name hduser
mapred.output.compressfalse
io.bytes.per.checksum512
giraph.isStaticGraphtrue
mapred.healthChecker.script.timeout60
topology.node.switch.mapping.implorg.apache.hadoop.net.ScriptBasedMapping
dfs.https.server.keystore.resourcessl-server.xml
mapred.reduce.slowstart.completed.maps0.05
mapred.reduce.max.attempts4
fs.ramfs.implorg.apache.hadoop.fs.InMemoryFileSystem
dfs.block.access.token.lifetime600
dfs.name.edits.dir${dfs.name.dir}
mapred.skip.map.max.skip.records0
mapred.cluster.map.memory.mb-1
hadoop.security.group.mappingorg.apache.hadoop.security.ShellBasedUnixGroupsMa
pping
mapred.job.tracker.persist.jobstatus.dir/jobtracker/jobsInfo
mapred.jarhdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging
/job_201312051827_0001/job.jar
dfs.block.size67108864
fs.s3.buffer.dir${hadoop.tmp.dir}/s3
job.end.retry.attempts0
fs.file.implorg.apache.hadoop.fs.LocalFileSystem
mapred.local.dir.minspacestart0
mapred.output.compression.typeRECORD
dfs.datanode.ipc.address0.0.0.0:50020 http://0.0.0.0:50020
dfs.permissionstrue
topology.script.number.args100
io.mapfile.bloom.error.rate0.005
mapred.cluster.max.reduce.memory.mb-1
mapred.max.tracker.blacklists4
mapred.task.profile.maps0-2
dfs.datanode.https.address0.0.0.0:50475 http://0.0.0.0:50475
mapred.userlog.retain.hours24
dfs.secondary.http.address0.0.0.0:50090 http://0.0.0.0:50090
dfs.replication.max512
mapred.job.tracker.persist.jobstatus.activefalse
hadoop.security.authorizationfalse
local.cache.size10737418240
dfs.namenode.delegation.token.renew-interval8640
mapred.min.split.size0
mapred.map.tasks2
mapred.child.java.opts-Xmx4000m
mapreduce.job.counters.limit120
dfs.https.client.keystore.resourcessl-client.xml
mapred.job.queue.name http://mapred.job.queue.name default
dfs.https.address0.0.0.0:50470 http://0.0.0.0:50470
mapred.job.tracker.retiredjobs.cache.size1000
dfs.balance.bandwidthPerSec1048576
ipc.server.listen.queue.size128
mapred.inmem.merge.threshold1000
job.end.retry.interval3
mapred.skip.attempts.to.start.skipping2
fs.checkpoint.dir${hadoop.tmp.dir}/dfs/namesecondary
mapred.reduce.tasks0
mapred.merge.recordsBeforeProgress1
mapred.userlog.limit.kb0
Re: out of core option
Hi Ameya, thanks for the answer. My allocated memory was too high. My server has altogether 4000M. I have turned the memory down to 2000M for each Mapper. Now I have set both out of core options and get the following exception: 2013-12-05 23:10:18,568 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201312052304_0001_m_01_0' to tip task_201312052304_0001_m_01, for tracker 'tracker_hduser:localhost/ 127.0.0.1:39793' 2013-12-05 23:10:27,645 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201312052304_0001_m_01_0: java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@62bf5822 at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@62bf5822 at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:181) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:139) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:124) at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:87) at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:221) at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:281) at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:325) at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:506) at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:244) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91) ... 7 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: getOrCreatePartition: cannot retrieve partition 0 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:300) at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:173) ... 16 more Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot retrieve partition 0 at org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:243) at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:110) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:482) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276) at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:172) at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:267) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:211) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:228) ... 13 more Caused by: java.lang.NullPointerException at org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:692) at org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:658) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at org.apache.giraph.partition.DiskBackedPartitionStore$DirectExecutorService.execute(DiskBackedPartitionStore.java:972) at
Re: out of core option
in the configuration setting which means you
haven't actually enabled out of core graph mode.
You have *giraph.useOutOfCoreGiraph *when it should be
*giraph.useOutOfCoreGraph
*– note that the last word is Graph not Giraph
Rob
From: Sebastian Stipkovic sebastian.stipko...@gmail.com
Reply-To: user@giraph.apache.org
Date: Thursday, 5 December 2013 20:39
To: user@giraph.apache.org
Subject: out of core option
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job with
out-of-memory-option works only well with a tiny graph (0.9 GB). What is
Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue*fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem
*mapred.task.cache.levels*2*giraph.vertexOutputFormatClass*
org.apache.giraph.examples.MyShortestPaths$MyOutputFormat*hadoop.tmp.dir*
/app/hadoop/tmp*hadoop.native.lib*true*map.sort.class*
org.apache.hadoop.util.QuickSort
*dfs.namenode.decommission.nodes.per.interval*5
*dfs.https.need.client.auth*false*ipc.client.idlethreshold*4000
*dfs.datanode.data.dir.perm*755*mapred.system.dir*
${hadoop.tmp.dir}/mapred/system
*mapred.job.tracker.persist.jobstatus.hours*0*dfs.datanode.address*
0.0.0.0:50010*dfs.namenode.logging.level*info
*dfs.block.access.token.enable*false*io.skip.checksum.errors*false*fs.default.name
http://fs.default.name*hdfs://localhost:54310
*mapred.cluster.reduce.memory.mb*-1*mapred.child.tmp*./tmp
*fs.har.impl.disable.cache*true*dfs.safemode.threshold.pct*0.999f
*mapred.skip.reduce.max.skip.groups*0*dfs.namenode.handler.count*10
*dfs.blockreport.initialDelay*0*mapred.heartbeats.in.second*100
*mapred.tasktracker.dns.nameserver*default*io.sort.factor*10
*mapred.task.timeout*60*giraph.maxWorkers*1
*mapred.max.tracker.failures*4*hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory
*mapred.job.tracker.jobhistory.lru.cache.size*5*fs.hdfs.impl*
org.apache.hadoop.hdfs.DistributedFileSystem
*mapred.queue.default.acl-administer-jobs**
*dfs.block.access.key.update.interval*600
*mapred.skip.map.auto.incr.proc.count*true
*mapreduce.job.complete.cancel.delegation.tokens*true
*io.mapfile.bloom.size*1048576*mapreduce.reduce.shuffle.connect.timeout*
18*dfs.safemode.extension*3
*mapred.jobtracker.blacklist.fault-timeout-window*180
*tasktracker.http.threads*40*mapred.job.shuffle.merge.percent*0.66
*mapreduce.inputformat.class*org.apache.giraph.bsp.BspInputFormat
*fs.ftp.impl*org.apache.hadoop.fs.ftp.FTPFileSystem*user.name
http://user.name*hduser*mapred.output.compress*false
*io.bytes.per.checksum*512*giraph.isStaticGraph*true
*mapred.healthChecker.script.timeout*60
*topology.node.switch.mapping.impl*
org.apache.hadoop.net.ScriptBasedMapping
*dfs.https.server.keystore.resource*ssl-server.xml
*mapred.reduce.slowstart.completed.maps*0.05*mapred.reduce.max.attempts*4
*fs.ramfs.impl*org.apache.hadoop.fs.InMemoryFileSystem
*dfs.block.access.token.lifetime*600*dfs.name.edits.dir*${dfs.name.dir}
*mapred.skip.map.max.skip.records*0*mapred.cluster.map.memory.mb*-1
*hadoop.security.group.mapping*
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
*mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo*mapred.jar*
hdfs://localhost:54310
/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201312051827_0001/job.jar
*dfs.block.size*67108864*fs.s3.buffer.dir*${hadoop.tmp.dir}/s3
*job.end.retry.attempts*0*fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem*mapred.local.dir.minspacestart*0
*mapred.output.compression.type*RECORD*dfs.datanode.ipc.address*
0.0.0.0:50020*dfs.permissions*true*topology.script.number.args*100
*io.mapfile.bloom.error.rate*0.005*mapred.cluster.max.reduce.memory.mb*-1
*mapred.max.tracker.blacklists*4*mapred.task.profile.maps*0-2
*dfs.datanode.https.address*0.0.0.0:50475*mapred.userlog.retain.hours*24
*dfs.secondary.http.address*0.0.0.0:50090*dfs.replication.max*512
*mapred.job.tracker.persist.jobstatus.active*false
*hadoop.security.authorization*false*local.cache.size*10737418240
*dfs.namenode.delegation.token.renew-interval*8640
*mapred.min.split.size*0*mapred.map.tasks*2*mapred.child.java.opts*
-Xmx4000m*mapreduce.job.counters.limit*120
*dfs.https.client.keystore.resource*ssl-client.xml*mapred.job.queue.name
http://mapred.job.queue.name*default*dfs.https.address*0.0.0.0:50470
*mapred.job.tracker.retiredjobs.cache.size*1000
*dfs.balance.bandwidthPerSec*1048576*ipc.server.listen.queue.size*128
*mapred.inmem.merge.threshold*1000*job.end.retry.interval*3
*mapred.skip.attempts.to.start.skipping*2*fs.checkpoint.dir*
${hadoop.tmp.dir}/dfs/namesecondary*mapred.reduce.tasks*0
*mapred.merge.recordsBeforeProgress*1*mapred.userlog.limit.kb*0
