Zhengguo 'Mike' SUN wrote:
Hi, Peeyush,
I guess I didn't make myself clear. I am trying to run a Hadoop pipes job with
a combination of Java classes and C++ classes. So the command I am using is
like:
hadoop pipes -conf myconf.xml -inputformat MyInputFormat.class -input in
-output out
And it threw ClassNotFoundException for my InputSplit class.
As I understand "hadoop jar" is used to run a jar file, which is not my case. And there
is a -jar option in "hadoop pipes". But, unfortunately, it is not working for me. So the
question I want to ask is how to include customized Java classes, such as MyInputSplit, in a pipes
job?
You are right. -jar option also doesn't add the jar file to classpath on
the client-side. You can use -libjars option with 0.19. Then, the
command looks like
hadoop pipes -conf myconf.xml -libjars <jarfile> -inputformat
MyInputFormat.class -input in -output out
I don't see a way to do this in 0.17.*, one way could be you add it
explicitly to the classpath on client-side, and add it through the
option -jar for the job.
Thanks,
Amareshwari
Thanks,
Mike
________________________________
From: Peeyush Bishnoi <[EMAIL PROTECTED]>
To: core-user@hadoop.apache.org; core-user@hadoop.apache.org
Sent: Wednesday, October 29, 2008 12:52:18 PM
Subject: RE: How do I include customized InputFormat, InputSplit and
RecordReader in a C++ pipes job?
Hello Zhengguo ,
Yes , -libjars is the new feature in Hadoop. This feature has been available from Hadoop-0.17.x , but it is more stable from hadoop 0.18.x
example to use -libjars...
hadoop jar -libjars <comma seperated list of jars> ...
Thanks ,
---
Peeyush
-----Original Message-----
From: Zhengguo 'Mike' SUN [mailto:[EMAIL PROTECTED]
Sent: Wed 10/29/2008 9:22 AM
To: core-user@hadoop.apache.org
Subject: Re: How do I include customized InputFormat, InputSplit and
RecordReader in a C++ pipes job?
Hi, Amareshwari,
Is -libjars a new option in Hadoop 0.19? I am using 0.17.2. The only option I
see is -jar, which didn't work for me. And besides passing them as jar file, is
there any other ways to do that?
Thanks
Mike
________________________________
From: Amareshwari Sriramadasu <[EMAIL PROTECTED]>
To: core-user@hadoop.apache.org
Sent: Tuesday, October 28, 2008 11:58:33 PM
Subject: Re: How do I include customized InputFormat, InputSplit and
RecordReader in a C++ pipes job?
Hi,
How are you passing your classes to the pipes job? If you are passing
them as a jar file, you can use -libjars option. From branch 0.19, the
libjar files are added to the client classpath also.
Thanks
Amareshwari
Zhengguo 'Mike' SUN wrote:
Hi,
I implemented customized classes for InputFormat, InputSplit and RecordReader
in Java and was trying to use them in a C++ pipes job. The customized
InputFormat class could be included using the -inputformat option, but it threw
ClassNotFoundException for my customized InputSplit class. It seemed the
classpath has not been correctly set. Is there any way that let me include my
customized classes in a pipes job?
