Hi Matthew I use giraph 1.1.0 and cdh4 hadoop 0.23 the patch of Marco:
http://www.mail-archive.com/**user@giraph.apache.org/**msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff> and build mvn clean install -P hadoop_cdh4.1.2 Il giorno 07/ott/2013 20:20, "Matthew Laird" <lai...@sfu.ca> ha scritto: > Hello, so I'm watching this thread with interest as well, since that's the > exact example I'll be trying to run as well, and I see Silvio appears to be > asking about running Giraph 1.0 against Hadoop 2.0.0. > > I'm curious, is that a known working combination, 1.0 vs 2.0.x? And if > so, are there patches to apply first and what maven command line should I > build with? (ie. will a -Dhadoop.version=2.0.5 give the needed MR based API > or will it use YARN?) I figure it's better to ask than aimlessly try all > the combinations. :) > > Thanks! > > On 13-10-07 11:01 AM, Claudio Martella wrote: > >> try to increase the mappers' heap space through mapred.child.java.opts. >> >> >> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio >> <silvio.digrego...@gmail.com >> <mailto:silvio.digregorio@**gmail.com<silvio.digrego...@gmail.com>>> >> wrote: >> >> >> I think now my issue is another, thanks to all. >> >> >> >> >> java.lang.**IllegalStateException: run: Caught an unrecoverable >> exception waitFor: ExecutionException occurred while waiting for >> org.apache.giraph.utils.**ProgressableUtils$**FutureWaitable@73a01e8a >> at org.apache.giraph.graph.**GraphMapper.run(GraphMapper.** >> java:102) >> >> >> >> ...... >> Caused by: java.lang.**IllegalStateException: waitFor: >> ExecutionException occurred while waiting for org.apache.giraph.utils.** >> ProgressableUtils$**FutureWaitable@73a01e8a >> at org.apache.giraph.utils.**ProgressableUtils.waitFor(** >> ProgressableUtils.java:151) >> >> >> >> ... 7 more >> Caused by: java.util.concurrent.**ExecutionException: >> java.lang.OutOfMemoryError:***Java heap space* >> at org.apache.giraph.utils.**ProgressableUtils.waitFor(** >> ProgressableUtils.java:143) >> >> >> >> ... 13 more >> >> >> >> 2013/10/7 Claudio Martella <claudio.marte...@gmail.com >> <mailto:claudio.martella@**gmail.com <claudio.marte...@gmail.com>>> >> >> OK, thanks. >> >> I really have to push that patch in. >> >> >> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio >> <silvio.digrego...@gmail.com >> <mailto:silvio.digregorio@**gmail.com<silvio.digrego...@gmail.com>>> >> wrote: >> >> yes i do, >> i have seen this in your post in: >> http://www.mail-archive.com/**user@giraph.apache.org/** >> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html> >> >> excuse me if i had checked in the mail-achive first I would >> have avoided the last post. >> >> Now zk issue are resolved. >> >> >> 2013/10/7 Claudio Martella <claudio.marte...@gmail.com >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> >> >> >> Can you try applying this one first? >> >> <<< < http://www.mail-archive.com/**user@giraph.apache.org/ >> **msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff> >> <<<< >> >> >> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio >> <silvio.digrego...@gmail.com >> >> <mailto:silvio.digregorio@**gmail.com<silvio.digrego...@gmail.com>>> >> wrote: >> >> >> >> >> >> >> >> *As i said i have builded* >> >> >> >> >> /giraph-examples-1.0.0-for-** >> hadoop-2.0.0-cdh4.1.2-jar-**with-dependencies.jar/ >> >> >> >> >> >> >> >> >> >> >> >> >> >> *for cdh4, successfully. The job start to monitoring >> the success rate:* >> >> >> >> >> >> >> >> >> >> >> >> >> >> /13/10/07 08:28:45 INFO mapred.JobClient: map 0% >> reduce 0%/ >> >> >> >> >> *but then* >> >> >> >> >> /Error running child >> java.lang.**IllegalStateException: run: Caught an >> unrecoverable exception java.io.FileNotFoundException: File >> _bsp/_defaultZkManagerDir/job_**201309181636_0678/_zkServer does not >> exist. >> / >> >> >> >> >> >> >> >> >> >> >> >> >> >> /.../ >> >> >> >> >> >> >> >> /Caused by: java.lang.RuntimeException: >> java.io.FileNotFoundException: File >> _bsp/_defaultZkManagerDir/job_**201309181636_0678/_zkServer >> does not exist./ >> >> >> >> >> >> >> >> >> >> >> 2013/10/5 Silvio Di gregorio >> <silvio.digrego...@gmail.com >> >> <mailto:silvio.digregorio@**gmail.com<silvio.digrego...@gmail.com> >> >> >> >> I ha ve build w/ hadoop_cdh4.1.2 parameter. >> Something is changed, monday i report the >> result. Now the farm is closed. >> >> Il giorno 05/ott/2013 14:06, "Claudio Martella" >> <claudio.marte...@gmail.com >> >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com>>> >> ha scritto: >> >> Oh, right, -vof is in trunk. Anyway it looks >> like you built giraph for the wrong profile. >> You mentioned you're running on 2.0, but >> your giraph is built for 0.20.203 >> <tel:0.20.203>. try building with a profile >> for your hadoop version. >> >> >> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di >> gregorio <silvio.digrego...@gmail.com >> >> <mailto:silvio.digregorio@**gmail.com<silvio.digrego...@gmail.com>>> >> wrote: >> >> >> org.apache.commons.cli.** >> UnrecognizedOptionException: >> Unrecognized option: -vof >> >> in 1.0 version is >> -of,--outputFormat <arg> >> Vertex output format >> -op,--outputPath <arg> >> Vertex output path >> >> >> >> 2013/10/4 Claudio Martella >> <claudio.marte...@gmail.com >> >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> >> >> >> did you try the argument (-vof) i >> suggested? >> >> >> On Fri, Oct 4, 2013 at 2:13 PM, >> Silvio Di gregorio >> <silvio.digrego...@gmail.com >> >> <mailto:silvio.digregorio@**gmail.com<silvio.digrego...@gmail.com>>> >> wrote: >> >> >> i've specified -of >> org.apache.giraph.io.formats.** >> IdWithValueTextOutputFormat >> >> but the same error was produced >> >> Exception in thread "main" >> java.lang.** >> IncompatibleClassChangeError: >> Found interface >> org.apache.hadoop.mapreduce.** >> JobContext, >> but class was expected >> at >> org.apache.giraph.bsp.** >> BspOutputFormat.**checkOutputSpecs(**BspOutputFormat.java:43) >> at >> org.apache.hadoop.mapred.** >> JobClient$2.run(JobClient.**java:984) >> at >> org.apache.hadoop.mapred.** >> JobClient$2.run(JobClient.**java:945) >> at >> java.security.** >> AccessController.doPrivileged(**Native >> Method) >> at >> javax.security.auth.Subject.** >> doAs(Subject.java:396) >> at >> org.apache.hadoop.security.** >> UserGroupInformation.doAs(**UserGroupInformation.java:**1408) >> at >> org.apache.hadoop.mapred.** >> JobClient.submitJobInternal(**JobClient.java:945) >> at >> org.apache.hadoop.mapreduce.** >> Job.submit(Job.java:566) >> at >> org.apache.hadoop.mapreduce.** >> Job.waitForCompletion(Job.**java:596) >> at >> org.apache.giraph.job.** >> GiraphJob.run(GiraphJob.java:**237) >> at >> org.apache.giraph.** >> GiraphRunner.run(GiraphRunner.**java:94) >> at >> org.apache.hadoop.util.** >> ToolRunner.run(ToolRunner.**java:70) >> at >> org.apache.hadoop.util.** >> ToolRunner.run(ToolRunner.**java:84) >> at >> org.apache.giraph.** >> GiraphRunner.main(**GiraphRunner.java:124) >> at >> sun.reflect.** >> NativeMethodAccessorImpl.**invoke0(Native >> Method) >> at >> sun.reflect.** >> NativeMethodAccessorImpl.**invoke(**NativeMethodAccessorImpl.java:**39) >> at >> sun.reflect.** >> DelegatingMethodAccessorImpl.**invoke(**DelegatingMethodAccessorImpl.** >> java:25) >> at >> java.lang.reflect.Method.** >> invoke(Method.java:597) >> at >> org.apache.hadoop.util.RunJar.** >> main(RunJar.java:208) >> >> >> >> 2013/10/4 Claudio Martella >> <claudio.marte...@gmail.com >> <mailto:claudio.martella@** >> gmail.com <claudio.marte...@gmail.com>>> >> >> Hi, >> >> you need to specify the >> vertex outputformat class >> (-vof option), e.g. >> org.apache.giraph.io.formats. >> **IdWithValueTextOutputFormat. >> >> >> On Fri, Oct 4, 2013 at 1:06 >> PM, Silvio Di gregorio >> <silvio.digrego...@gmail.com >> <mailto:silvio.digregorio@** >> gmail.com <silvio.digrego...@gmail.com>>> >> wrote: >> >> >> Hi, >> >> I hope I have sent to >> the right address. >> >> i have a graph (directed >> and unweighted) stored >> in hdfs like a adjacency >> list (140Milions of >> edges 6Milions of vertex) >> >> node<tab>neighbors >> >> 23 2 1 3 4 3 >> >> 1 999 99923 >> <tel:1%20999%2099923> >> 909 ....... >> >> ...... >> >> hadoop version Hadoop >> 2.0.0-cdh4.3.0 - java 1.6 >> >> >> I have executed the >> giraph-1.0 connected >> components example, in >> this fashion >> >> > hadoop jar >> >> /usr/local/giraph/giraph-**examples/target/giraph-** >> examples-1.0.0-for-hadoop-0.20.203.0 >> <tel:0.20.203.0 >> >-jar-with-dependencies.**jar >> >> org.apache.giraph.GiraphRunner >> >> org.apache.giraph.examples.**ConnectedComponentsVertex >> -vif >> >> org.apache.giraph.io.formats.**IntIntNullTextInputFormat >> -vip >> >> /user/hdfs/lista_adj_txt >> -op connectedgiraph >> --workers 4 >> >> and then fail with: >> >> 13/10/04 09:28:29 INFO >> utils.ConfigurationUtils: >> No >> edge input format >> specified. Ensure your >> InputFormat does not >> require one. >> >> 13/10/04 09:28:29 INFO >> utils.ConfigurationUtils: >> No >> output format specified. >> Ensure your OutputFormat >> does not require one. >> >> 13/10/04 09:28:30 INFO >> job.GiraphJob: run: >> Since checkpointing is >> disabled (default), do >> not allow any task >> retries (setting >> mapred.map.max.attempts >> = 0, old value = 4) >> >> 13/10/04 09:28:31 WARN >> mapred.JobClient: Use >> GenericOptionsParser for >> parsing the arguments. >> Applications should >> implement Tool for the >> same. >> >> 13/10/04 09:28:31 INFO >> mapred.JobClient: >> Cleaning up the staging >> area >> hdfs:// >> srv-bigdata-dev-01.int.**sose.it:8020/user/hdfs/.** >> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535> >> < >> http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.** >> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535> >> > >> >> Exception in thread >> "main" >> java.lang.** >> IncompatibleClassChangeError: >> Found interface >> >> org.apache.hadoop.mapreduce.**JobContext, >> but class was expected >> >> at >> org.apache.giraph.bsp.** >> BspOutputFormat.**checkOutputSpecs(**BspOutputFormat.java:43) >> >> ...... >> >> Thanks in advance >> >> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> <mailto:claudio.martella@** >> gmail.com <claudio.marte...@gmail.com>> >> >> >> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> > >> >> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> > >> >> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> > >> >> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> > >> >> >> >> >> >> -- >> Claudio Martella >> claudio.marte...@gmail.com >> <mailto:claudio.martella@**gmail.com<claudio.marte...@gmail.com> >> > >> > > -- > Matthew Laird > Lead Software Developer, Bioinformatics > Brinkman Laboratory > Simon Fraser University, Burnaby, BC, Canada >
