Hi again I have found a solution to my "problem". I simply added myJobs.jar to the HADOOP_CLASSPATH. Also "-libjars" parameter is not needed any more. But it still puzzles me why the previous "solution" "suddenly" stopped working (I haven't update the giraph revision etc).
HADOOP_CLASSPATH contains girpah.jar and myJobs.jar regards marcin 2012/11/8 Marcin Biczak <marcinbic...@gmail.com> > Hi > > I have very strange problem regarding ClassNotFoundException for vertex > output format (my own implementation). When I run my program I get an > exception: > > java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: > org.test.giraph.utils.writers.generic.SamplingOutputFormat > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898) > at > org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:134) > at > org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56) > at org.apache.hadoop.mapred.Task.initialize(Task.java:490) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:352) > at org.apache.hadoop.mapred.Child$4.run(Child.java:259) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:253) > > I have came across this exception earlier and I've solved it with the use > of "-libjars" parameter in which I pass the path to > giraph-0.2-SNAPSHOT-jar-with-dependencies.jar and myJob.jar (which I > execute). This solution has worked very nice for me, I used it in a number > of programs (with my own InputFormat and OutputFormat), till today. I've > created a small program (TestVertex) to test one solution and I used a > NewOutputFormat created for the test program. When I received > ClassNotFoundException I thought I have made a mistake somewhere, so I've > checked. I couldn't find any bugs, so I replaced the NewOutputFormat with a > one used across other programs (OldOutputFormat). I got the same exception. > for the old one as well. I have cleaned and build the whole myJob.jar and I > was still getting the exception (both for New and Old). So I tested the > original program (same myJob.jar) in which I'm using the OldOutputFormat > and the program works. Since then I tried couple of OutputFormat and none > of them worked (ClassNotFoundException). Then I commented the whole "body" > of TestVertex and I've paste some old working code (with OldOutputFormat). > The only difference between TestVertex and old working code is the class > name. Final I received the ClassNotFoundException again (again the original > code works, I've tested it). It looks like only the TestVertex class has > this problem, no matter what OutputFormat I will use I am always getting > the exception (all other programs which use my OutputFormats work > correctly). > > Another interesting aspect is the fact that if I use Giraph provided > OutputFormat (for example JsonBase64VertexOutputFormat) I get a > ClassNotFoundException but for the TestVertex. I've checked the myJob.jar > and all mentioned classes are present in the jar in correct places. > > I run out of ideas any suggestions will be greatly appreciated. > > Some conf info > I am using an older Giraph revision (1340869) and Hadoop 0.20.203 in > pseudo-distributed mode. > I have created the HADOOP_CLASSPATH which holds path to > giraph-0.2-SNAPSHOT-jar-with-dependencies.jar. > I execute job in the following way: hadoop jar myJob.jar > org.test.giraph.Test -libjars > /path/to/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar,/path/to/myJob.jar > /path/to/input /path/to/output <job params> > > regards > marcin > -- martin