Re: out of core option
Each worker is allocated *mapred.child.java.opts *memory, which in your
case is 4000M. Check if your server doesn't have enough memory for 2
Mappers. Also the out of memory option is available in two forms.
1. Out of core graph
2. Out of core messages.
Currently you are setting only the out of core graph and not the out of
core messages. Enable both of them. More information about options can be
found here: http://giraph.apache.org/options.html
set -D giraph.useOutOfCoreGraph=true -D giraph.useOutOfCoreMessages=true
while passing options to GiraphRunner.
Thanks,
Ameya
On Thu, Dec 5, 2013 at 12:39 PM, Sebastian Stipkovic
sebastian.stipko...@gmail.com wrote:
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job with
out-of-memory-option works only well with a tiny graph (0.9 GB). What is
Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue *fs.s3n.impl*org.apache.hadoop.fs.s3native.NativeS3FileSystem
*mapred.task.cache.levels*2 *giraph.vertexOutputFormatClass*
org.apache.giraph.examples.MyShortestPaths$MyOutputFormat
*hadoop.tmp.dir*/app/hadoop/tmp *hadoop.native.lib*true *map.sort.class*
org.apache.hadoop.util.QuickSort
*dfs.namenode.decommission.nodes.per.interval*5
*dfs.https.need.client.auth*false *ipc.client.idlethreshold*4000
*dfs.datanode.data.dir.perm*755 *mapred.system.dir*
${hadoop.tmp.dir}/mapred/system
*mapred.job.tracker.persist.jobstatus.hours*0 *dfs.datanode.address*
0.0.0.0:50010 *dfs.namenode.logging.level*info
*dfs.block.access.token.enable*false *io.skip.checksum.errors*false
*fs.default.name
http://fs.default.name*hdfs://localhost:54310
*mapred.cluster.reduce.memory.mb*-1 *mapred.child.tmp*./tmp
*fs.har.impl.disable.cache*true *dfs.safemode.threshold.pct*0.999f
*mapred.skip.reduce.max.skip.groups*0 *dfs.namenode.handler.count*10
*dfs.blockreport.initialDelay*0 *mapred.heartbeats.in.second*100
*mapred.tasktracker.dns.nameserver*default *io.sort.factor*10
*mapred.task.timeout*60 *giraph.maxWorkers*1
*mapred.max.tracker.failures*4 *hadoop.rpc.socket.factory.class.default*
org.apache.hadoop.net.StandardSocketFactory
*mapred.job.tracker.jobhistory.lru.cache.size*5 *fs.hdfs.impl*
org.apache.hadoop.hdfs.DistributedFileSystem
*mapred.queue.default.acl-administer-jobs**
*dfs.block.access.key.update.interval*600
*mapred.skip.map.auto.incr.proc.count*true
*mapreduce.job.complete.cancel.delegation.tokens*true
*io.mapfile.bloom.size*1048576 *mapreduce.reduce.shuffle.connect.timeout*
18 *dfs.safemode.extension*3
*mapred.jobtracker.blacklist.fault-timeout-window*180
*tasktracker.http.threads*40 *mapred.job.shuffle.merge.percent*0.66
*mapreduce.inputformat.class*org.apache.giraph.bsp.BspInputFormat
*fs.ftp.impl*org.apache.hadoop.fs.ftp.FTPFileSystem *user.name
http://user.name*hduser *mapred.output.compress*false
*io.bytes.per.checksum*512 *giraph.isStaticGraph*true
*mapred.healthChecker.script.timeout*60
*topology.node.switch.mapping.impl*
org.apache.hadoop.net.ScriptBasedMapping
*dfs.https.server.keystore.resource*ssl-server.xml
*mapred.reduce.slowstart.completed.maps*0.05 *mapred.reduce.max.attempts*
4 *fs.ramfs.impl*org.apache.hadoop.fs.InMemoryFileSystem
*dfs.block.access.token.lifetime*600 *dfs.name.edits.dir*${dfs.name.dir}
*mapred.skip.map.max.skip.records*0 *mapred.cluster.map.memory.mb*-1
*hadoop.security.group.mapping*
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
*mapred.job.tracker.persist.jobstatus.dir*/jobtracker/jobsInfo
*mapred.jar*
hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201312051827_0001/job.jar
*dfs.block.size*67108864 *fs.s3.buffer.dir*${hadoop.tmp.dir}/s3
*job.end.retry.attempts*0 *fs.file.impl*
org.apache.hadoop.fs.LocalFileSystem *mapred.local.dir.minspacestart*0
*mapred.output.compression.type*RECORD *dfs.datanode.ipc.address*
0.0.0.0:50020 *dfs.permissions*true *topology.script.number.args*100
*io.mapfile.bloom.error.rate*0.005 *mapred.cluster.max.reduce.memory.mb*
-1 *mapred.max.tracker.blacklists*4 *mapred.task.profile.maps*0-2
*dfs.datanode.https.address*0.0.0.0:50475 *mapred.userlog.retain.hours*24
*dfs.secondary.http.address*0.0.0.0:50090 *dfs.replication.max*512
*mapred.job.tracker.persist.jobstatus.active*false
*hadoop.security.authorization*false *local.cache.size*10737418240
*dfs.namenode.delegation.token.renew-interval*8640
*mapred.min.split.size*0 *mapred.map.tasks*2 *mapred.child.java.opts*
-Xmx4000m *mapreduce.job.counters.limit*120
*dfs.https.client.keystore.resource*ssl-client.xml *mapred.job.queue.name
http://mapred.job.queue.name*default *dfs.https.address*0.0.0.0:50470
Re: out of core option
Sebastian
You've made a minor typo in the configuration setting which means you
haven't actually enabled out of core graph mode.
You have giraph.useOutOfCoreGiraph when it should be
giraph.useOutOfCoreGraph note that the last word is Graph not Giraph
Rob
From: Sebastian Stipkovic sebastian.stipko...@gmail.com
Reply-To: user@giraph.apache.org
Date: Thursday, 5 December 2013 20:39
To: user@giraph.apache.org
Subject: out of core option
Hello,
I had setup giraph 1.1.0 with hadoop-0.20.203.0rc1 on a single
node cluster. It computes a tiny graph successful. But if the
input graph is huge (5 GB), I get an OutOfMemory(Garbage Collector)
exception, although I had turned on the out-of-memory-option. The job with
out-of-memory-option works only well with a tiny graph (0.9 GB). What is
Wrong? Does I have to do furthermore configurations?
My Configurations are as follows:
namevalue
fs.s3n.implorg.apache.hadoop.fs.s3native.NativeS3FileSystem
mapred.task.cache.levels2
giraph.vertexOutputFormatClassorg.apache.giraph.examples.MyShortestPaths$MyOut
putFormat
hadoop.tmp.dir/app/hadoop/tmp
hadoop.native.libtrue
map.sort.classorg.apache.hadoop.util.QuickSort
dfs.namenode.decommission.nodes.per.interval5
dfs.https.need.client.authfalse
ipc.client.idlethreshold4000
dfs.datanode.data.dir.perm755
mapred.system.dir${hadoop.tmp.dir}/mapred/system
mapred.job.tracker.persist.jobstatus.hours0
dfs.datanode.address0.0.0.0:50010 http://0.0.0.0:50010
dfs.namenode.logging.levelinfo
dfs.block.access.token.enablefalse
io.skip.checksum.errorsfalse
fs.default.name http://fs.default.name hdfs://localhost:54310
mapred.cluster.reduce.memory.mb-1
mapred.child.tmp./tmp
fs.har.impl.disable.cachetrue
dfs.safemode.threshold.pct0.999f
mapred.skip.reduce.max.skip.groups0
dfs.namenode.handler.count10
dfs.blockreport.initialDelay0
mapred.heartbeats.in.second100
mapred.tasktracker.dns.nameserverdefault
io.sort.factor10
mapred.task.timeout60
giraph.maxWorkers1
mapred.max.tracker.failures4
hadoop.rpc.socket.factory.class.defaultorg.apache.hadoop.net.StandardSocketFac
tory
mapred.job.tracker.jobhistory.lru.cache.size5
fs.hdfs.implorg.apache.hadoop.hdfs.DistributedFileSystem
mapred.queue.default.acl-administer-jobs*
dfs.block.access.key.update.interval600
mapred.skip.map.auto.incr.proc.counttrue
mapreduce.job.complete.cancel.delegation.tokenstrue
io.mapfile.bloom.size1048576
mapreduce.reduce.shuffle.connect.timeout18
dfs.safemode.extension3
mapred.jobtracker.blacklist.fault-timeout-window180
tasktracker.http.threads40
mapred.job.shuffle.merge.percent0.66
mapreduce.inputformat.classorg.apache.giraph.bsp.BspInputFormat
fs.ftp.implorg.apache.hadoop.fs.ftp.FTPFileSystem
user.name http://user.name hduser
mapred.output.compressfalse
io.bytes.per.checksum512
giraph.isStaticGraphtrue
mapred.healthChecker.script.timeout60
topology.node.switch.mapping.implorg.apache.hadoop.net.ScriptBasedMapping
dfs.https.server.keystore.resourcessl-server.xml
mapred.reduce.slowstart.completed.maps0.05
mapred.reduce.max.attempts4
fs.ramfs.implorg.apache.hadoop.fs.InMemoryFileSystem
dfs.block.access.token.lifetime600
dfs.name.edits.dir${dfs.name.dir}
mapred.skip.map.max.skip.records0
mapred.cluster.map.memory.mb-1
hadoop.security.group.mappingorg.apache.hadoop.security.ShellBasedUnixGroupsMa
pping
mapred.job.tracker.persist.jobstatus.dir/jobtracker/jobsInfo
mapred.jarhdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging
/job_201312051827_0001/job.jar
dfs.block.size67108864
fs.s3.buffer.dir${hadoop.tmp.dir}/s3
job.end.retry.attempts0
fs.file.implorg.apache.hadoop.fs.LocalFileSystem
mapred.local.dir.minspacestart0
mapred.output.compression.typeRECORD
dfs.datanode.ipc.address0.0.0.0:50020 http://0.0.0.0:50020
dfs.permissionstrue
topology.script.number.args100
io.mapfile.bloom.error.rate0.005
mapred.cluster.max.reduce.memory.mb-1
mapred.max.tracker.blacklists4
mapred.task.profile.maps0-2
dfs.datanode.https.address0.0.0.0:50475 http://0.0.0.0:50475
mapred.userlog.retain.hours24
dfs.secondary.http.address0.0.0.0:50090 http://0.0.0.0:50090
dfs.replication.max512
mapred.job.tracker.persist.jobstatus.activefalse
hadoop.security.authorizationfalse
local.cache.size10737418240
dfs.namenode.delegation.token.renew-interval8640
mapred.min.split.size0
mapred.map.tasks2
mapred.child.java.opts-Xmx4000m
mapreduce.job.counters.limit120
dfs.https.client.keystore.resourcessl-client.xml
mapred.job.queue.name http://mapred.job.queue.name default
dfs.https.address0.0.0.0:50470 http://0.0.0.0:50470
mapred.job.tracker.retiredjobs.cache.size1000
dfs.balance.bandwidthPerSec1048576
ipc.server.listen.queue.size128
mapred.inmem.merge.threshold1000
job.end.retry.interval3
mapred.skip.attempts.to.start.skipping2
fs.checkpoint.dir${hadoop.tmp.dir}/dfs/namesecondary
mapred.reduce.tasks0
mapred.merge.recordsBeforeProgress1
mapred.userlog.limit.kb0
Re: out of core option
Hi Ameya, thanks for the answer. My allocated memory was too high. My server has altogether 4000M. I have turned the memory down to 2000M for each Mapper. Now I have set both out of core options and get the following exception: 2013-12-05 23:10:18,568 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201312052304_0001_m_01_0' to tip task_201312052304_0001_m_01, for tracker 'tracker_hduser:localhost/ 127.0.0.1:39793' 2013-12-05 23:10:27,645 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201312052304_0001_m_01_0: java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@62bf5822 at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@62bf5822 at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:181) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:139) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:124) at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:87) at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:221) at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:281) at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:325) at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:506) at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:244) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91) ... 7 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: getOrCreatePartition: cannot retrieve partition 0 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:300) at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:173) ... 16 more Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot retrieve partition 0 at org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:243) at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:110) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:482) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276) at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:172) at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:267) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:211) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:228) ... 13 more Caused by: java.lang.NullPointerException at org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:692) at org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:658) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at org.apache.giraph.partition.DiskBackedPartitionStore$DirectExecutorService.execute(DiskBackedPartitionStore.java:972) at
Re: out of core option
Hi Rob, embarrassing. You are right. But now I get with the correct option the following exception: 2013-12-05 23:10:18,568 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201312052304_0001_m_01_0' to tip task_201312052304_0001_m_01, for tracker 'tracker_hduser:localhost/ 127.0.0.1:39793' 2013-12-05 23:10:27,645 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201312052304_0001_m_01_0: java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@62bf5822 at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@62bf5822 at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:181) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:139) at org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:124) at org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:87) at org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:221) at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:281) at org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:325) at org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:506) at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:244) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91) ... 7 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: getOrCreatePartition: cannot retrieve partition 0 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:300) at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:173) ... 16 more Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot retrieve partition 0 at org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:243) at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:110) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.doRequest(NettyWorkerClientRequestProcessor.java:482) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276) at org.apache.giraph.worker.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:172) at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:267) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:211) at org.apache.giraph.worker.InputSplitsCallable.call(InputSplitsCallable.java:60) at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:228) ... 13 more Caused by: java.lang.NullPointerException at org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:692) at org.apache.giraph.partition.DiskBackedPartitionStore$GetPartition.call(DiskBackedPartitionStore.java:658) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at org.apache.giraph.partition.DiskBackedPartitionStore$DirectExecutorService.execute(DiskBackedPartitionStore.java:972) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ... 14 more Thanks, Sebastian 2013/12/5 Rob Vesse rve...@dotnetrdf.org Sebastian You've made a minor typo
Re: Issue while executing SimpleShortestPathsComputation
Hi, I also got this error when tried to build Giraph on top of hadoop-2.2.0 using similar commands. Can anyone help? Thanks, Yong 2013/12/6 Rohit Bhat rohitbhat.n...@gmail.com I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2 commands and got similar errors in both the cases: 1. mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests 2. mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install package [INFO] Checking dependency versions [ERROR] Found a problem with the direct dependency commons-httpclient:commons-httpclient of the current project Expected version is 3.0.1 Resolved version is 3.0.1 Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-lang:commons-lang Resolved version is 2.4 Version 2.4 was expected by artifact: commons-configuration:commons-configuration Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-logging:commons-logging Resolved version is 1.0.4 Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency io.netty:netty Resolved version is 3.5.3.Final Version 3.6.2.Final was expected by artifacts: org.apache. hadoop:hadoop-mapreduce-client-common, org.apache. hadoop:hadoop-mapreduce-client-core [ERROR] Found a problem with the dependency org.apache.zookeeper:zookeeper Resolved version is 3.3.3 Version 3.4.5 was expected by artifacts: org.apache.hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common Am i missing something? On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar ameya.vilan...@gmail.comwrote: Yes. Use this: mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang manuellag...@gmail.comwrote: I get the same error when I compile Giraph against the default hadoop version (0.20.203.0), but my project that uses Giraph uses a more recent hadoop version. Did you set the hadoop version via a maven profile when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)? Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn profile in the pom.xml. I personally haven't had success building it though. On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat rohitbhat.n...@gmail.comwrote: Hi, I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have installed Giraph on top of it. I have been trying to execute SimpleShortestPathsComputation example but i am getting the error: java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time! I fixed this by adding an entry in mapred-site.xml for mapreduce.jobtracker.address but now i am getting a different error: *Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected* at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) *Is Giraph compatible with yarn? If yes, am i missing anything?* Thanks, Rohit
Re: Issue while executing SimpleShortestPathsComputation
Hi, I think it might be the Giraph version problem. When I tried to build giraph *1.0.0* on hadoop2.0.3-alpha, I can compile successfully by *mvn -Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*. The command failed when I used giraph*1.1.0*. Thanks, Yong 2013/12/6 Yong Guo guoyongn...@gmail.com Hi, I also got this error when tried to build Giraph on top of hadoop-2.2.0 using similar commands. Can anyone help? Thanks, Yong 2013/12/6 Rohit Bhat rohitbhat.n...@gmail.com I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2 commands and got similar errors in both the cases: 1. mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package -DskipTests 2. mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install package [INFO] Checking dependency versions [ERROR] Found a problem with the direct dependency commons-httpclient:commons-httpclient of the current project Expected version is 3.0.1 Resolved version is 3.0.1 Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-lang:commons-lang Resolved version is 2.4 Version 2.4 was expected by artifact: commons-configuration:commons-configuration Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-logging:commons-logging Resolved version is 1.0.4 Version 1.1.1 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency io.netty:netty Resolved version is 3.5.3.Final Version 3.6.2.Final was expected by artifacts: org.apache. hadoop:hadoop-mapreduce-client-common, org.apache. hadoop:hadoop-mapreduce-client-core [ERROR] Found a problem with the dependency org.apache. zookeeper:zookeeper Resolved version is 3.3.3 Version 3.4.5 was expected by artifacts: org.apache. hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common Am i missing something? On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar ameya.vilan...@gmail.comwrote: Yes. Use this: mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang manuellag...@gmail.comwrote: I get the same error when I compile Giraph against the default hadoop version (0.20.203.0), but my project that uses Giraph uses a more recent hadoop version. Did you set the hadoop version via a maven profile when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)? Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn profile in the pom.xml. I personally haven't had success building it though. On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat rohitbhat.n...@gmail.comwrote: Hi, I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have installed Giraph on top of it. I have been trying to execute SimpleShortestPathsComputation example but i am getting the error: java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time! I fixed this by adding an entry in mapred-site.xml for mapreduce.jobtracker.address but now i am getting a different error: *Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected* at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) *Is Giraph compatible with yarn? If yes, am i missing anything?* Thanks, Rohit
Re: Issue while executing SimpleShortestPathsComputation
I am using giraph 1.0.0 but with yarn (hadoop 2.2.0) On Thu, Dec 5, 2013 at 5:49 PM, Yong Guo guoyongn...@gmail.com wrote: Hi, I think it might be the Giraph version problem. When I tried to build giraph*1.0.0* on hadoop2.0.3-alpha, I can compile successfully by *mvn -Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*. The command failed when I used giraph*1.1.0*. Thanks, Yong 2013/12/6 Yong Guo guoyongn...@gmail.com Hi, I also got this error when tried to build Giraph on top of hadoop-2.2.0 using similar commands. Can anyone help? Thanks, Yong 2013/12/6 Rohit Bhat rohitbhat.n...@gmail.com I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2 commands and got similar errors in both the cases: 1. mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package - DskipTests 2. mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install package [INFO] Checking dependency versions [ERROR] Found a problem with the direct dependency commons-httpclient:commons-httpclient of the current project Expected version is 3.0.1 Resolved version is 3.0.1 Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-lang:commons-lang Resolved version is 2.4 Version 2.4 was expected by artifact: commons-configuration:commons-configuration Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-logging:commons-logging Resolved version is 1.0.4 Version 1.1.1 was expected by artifact: org.apache. hadoop:hadoop-common [ERROR] Found a problem with the dependency io.netty:netty Resolved version is 3.5.3.Final Version 3.6.2.Final was expected by artifacts: org.apache. hadoop:hadoop-mapreduce-client-common, org.apache. hadoop:hadoop-mapreduce-client-core [ERROR] Found a problem with the dependency org.apache. zookeeper:zookeeper Resolved version is 3.3.3 Version 3.4.5 was expected by artifacts: org.apache. hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common Am i missing something? On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar ameya.vilan...@gmail.com wrote: Yes. Use this: mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang manuellag...@gmail.comwrote: I get the same error when I compile Giraph against the default hadoop version (0.20.203.0), but my project that uses Giraph uses a more recent hadoop version. Did you set the hadoop version via a maven profile when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)? Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn profile in the pom.xml. I personally haven't had success building it though. On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat rohitbhat.n...@gmail.comwrote: Hi, I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have installed Giraph on top of it. I have been trying to execute SimpleShortestPathsComputation example but i am getting the error: java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time! I fixed this by adding an entry in mapred-site.xml for mapreduce.jobtracker.address but now i am getting a different error: *Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected* at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) *Is Giraph compatible with yarn? If yes, am i missing anything?* Thanks, Rohit
Re: Issue while executing SimpleShortestPathsComputation
Can someone please confirm if giraph 1.0.0 works with hadoop-2.2.0? On Thu, Dec 5, 2013 at 5:58 PM, Rohit Bhat rohitbhat.n...@gmail.com wrote: I am using giraph 1.0.0 but with yarn (hadoop 2.2.0) On Thu, Dec 5, 2013 at 5:49 PM, Yong Guo guoyongn...@gmail.com wrote: Hi, I think it might be the Giraph version problem. When I tried to build giraph*1.0.0* on hadoop2.0.3-alpha, I can compile successfully by *mvn -Phadoop_yarn -Dhadoop.version=2.0.3-alpha clean package -DskipTests*. The command failed when I used giraph*1.1.0*. Thanks, Yong 2013/12/6 Yong Guo guoyongn...@gmail.com Hi, I also got this error when tried to build Giraph on top of hadoop-2.2.0 using similar commands. Can anyone help? Thanks, Yong 2013/12/6 Rohit Bhat rohitbhat.n...@gmail.com I already have yarn (hadoop_2.2.0) on my machine. I tried the following 2 commands and got similar errors in both the cases: 1. mvn -Phadoop_yarn -Dhadoop.version=2.2.0 -X clean package - DskipTests 2. mvn -P2.2.0 -DskipTests -Dmaven.javadoc.skip=true clean install package [INFO] Checking dependency versions [ERROR] Found a problem with the direct dependency commons-httpclient:commons-httpclient of the current project Expected version is 3.0.1 Resolved version is 3.0.1 Version 3.1 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-lang:commons-lang Resolved version is 2.4 Version 2.4 was expected by artifact: commons-configuration:commons-configuration Version 2.5 was expected by artifact: org.apache.hadoop:hadoop-common [ERROR] Found a problem with the dependency commons-logging:commons-logging Resolved version is 1.0.4 Version 1.1.1 was expected by artifact: org.apache. hadoop:hadoop-common [ERROR] Found a problem with the dependency io.netty:netty Resolved version is 3.5.3.Final Version 3.6.2.Final was expected by artifacts: org.apache. hadoop:hadoop-mapreduce-client-common, org.apache. hadoop:hadoop-mapreduce-client-core [ERROR] Found a problem with the dependency org.apache. zookeeper:zookeeper Resolved version is 3.3.3 Version 3.4.5 was expected by artifacts: org.apache. hadoop:hadoop-common, org.apache.hadoop:hadoop-mapreduce-client-common Am i missing something? On Mon, Dec 2, 2013 at 7:21 PM, Ameya Vilankar ameya.vilan...@gmail.com wrote: Yes. Use this: mvn -Phadoop_cdh4.4.0 -DskipTests -Dmaven.javadoc.skip=true package On Mon, Dec 2, 2013 at 5:18 PM, Manuel Lagang manuellag...@gmail.comwrote: I get the same error when I compile Giraph against the default hadoop version (0.20.203.0), but my project that uses Giraph uses a more recent hadoop version. Did you set the hadoop version via a maven profile when compiling Giraph (e.g. mvn -Phadoop_1.0 compile for hadoop 1.0)? Presumably, there's a way to compile Giraph for Yarn. There's even a Yarn profile in the pom.xml. I personally haven't had success building it though. On Mon, Dec 2, 2013 at 4:30 PM, Rohit Bhat rohitbhat.n...@gmail.comwrote: Hi, I am new to Giraph. I have installed Yarn (hadoop 2.2.0) and have installed Giraph on top of it. I have been trying to execute SimpleShortestPathsComputation example but i am getting the error: java.lang.IllegalArgumentException: checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot run in split master / worker mode since there is only 1 task at a time! I fixed this by adding an entry in mapred-site.xml for mapreduce.jobtracker.address but now i am getting a different error: *Exception in thread main java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected* at org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
