Thanks Amareshwari, I find it & I'm sorry it results in another error:
bash-3.2$ bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=true -libjars
/home/hadoop/project/hadoop-0.20.2/hadoop-0.20.2-test.jar -inputformat
org.apache.hadoop.mapred.pipes.WordCountInputFormat -input gutenberg
-output gutenberg-out101 -program bin/wordcount-nopipe
11/03/31 16:36:26 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
hdfs://ws-test:54310/user/hadoop/gutenberg, expected: file:
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:47)
at
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:273)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:721)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:746)
at
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:465)
at
org.apache.hadoop.mapred.pipes.WordCountInputFormat.getSplits(WordCountInputFormat.java:57)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
at
org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248)
at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
Best regards, Adarsh
Amareshwari Sri Ramadasu wrote:
Adarsh,
The inputformat is present in test jar. So, pass -libjars <full path to
testjar> to your command. libjars option should be passed before program specific
options. So, it should be just after your -D parameters.
-Amareshwari
On 3/31/11 3:45 PM, "Adarsh Sharma" <adarsh.sha...@orkash.com> wrote:
Amareshwari Sri Ramadasu wrote:
Re: Hadoop Pipes Error You can not run it with TextInputFormat. You should run
it with org.apache.hadoop.mapred.pipes .WordCountInputFormat. You can pass the
input format by passing it in -inputformat option.
I did not try it myself, but it should work.
Here is the command that I am trying and it results in exception:
bash-3.2$ bin/hadoop pipes -D hadoop.pipes.java.recordreader=true -D
hadoop.pipes.java.recordwriter=true -inputformat
org.apache.hadoop.mapred.pipes.WordCountInputFormat -input gutenberg -output
gutenberg-out101 -program bin/wordcount-nopipe
Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.hadoop.mapred.pipes.WordCountInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762)
at org.apache.hadoop.mapred.pipes.Submitter.getClass(Submitter.java:372)
at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:421)
at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
Thanks , Adarsh
