Could you please state exactly the steps you do in setting up the pipes run?
The two critical things to watch are: a/ where did you load the executable on hdfs e.g., $ ls pipes-bin genreads pair-reads seqal $ hadoop dfs -put pipes-bin pipes-bin $ hadoop dfs -ls hdfs://host:53897/user/zag/pipes-bin Found 3 items -rw-r--r-- 3 zag supergroup 480 2010-03-17 12:28 /user/zag/pipes-bin/genreads -rw-r--r-- 3 zag supergroup 692 2010-03-17 15:02 /user/zag/pipes-bin/pair_reads -rw-r--r-- 3 zag supergroup 477 2010-03-17 12:33 /user/zag/pipes-bin/seqal b/ how you started the program $ hadoop pipes -D hadoop.pipes.executable=pipes-bin/genreads -D hadoop.pipes.java.recordreader=true -D hadoop.pipes.java.recordwriter=true -input input -output output hope this is clear enough. --gianluigi On Wed, 2010-03-31 at 06:57 -0700, Keith Wiley wrote: > On 2010, Mar 31, at 4:25 AM, Gianluigi Zanetti wrote: > > > What happens if you try this: > > > > $ hadoop fs -rmr HDFSPATH/output ; hadoop pipes -D > > hadoop.pipes.executable=EXECUTABLE -D > > hadoop.pipes.java.recordreader=true -D > > hadoop.pipes.java.recordwriter=true -input HDFSPATH/input -output > > HDFSPATH/output > > > Not good news. This is what I got: > > $ hadoop pipes -D hadoop.pipes.executable=/Users/keithwiley/Astro_LSST/ > hadoop-0.20.1+152/Mosaic/clue/Mosaic/src/cpp/Mosaic -D > hadoop.pipes.java.recordreader=true -D > hadoop.pipes.java.recordwriter=true -input /uwphysics/kwiley/mosaic/ > input -output /uwphysics/kwiley/mosaic/output > Exception in thread "main" java.io.FileNotFoundException: File does > not exist: /Users/keithwiley/Astro_LSST/hadoop-0.20.1+152/Mosaic/clue/ > Mosaic/src/cpp/Mosaic > at > org > .apache > .hadoop > .hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java: > 457) > at > org > .apache > .hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java: > 509) > at > org > .apache > .hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:681) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:802) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:771) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1290) > at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:248) > at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479) > at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494) > > Incidentally, just in case you're wondering: > $ ls -l /Users/keithwiley/Astro_LSST/hadoop-0.20.1+152/Mosaic/clue/ > Mosaic/src/cpp/ > total 800 > 368 -rwxr-xr-x 1 keithwiley keithwiley 185184 Mar 29 19:08 Mosaic* > ...other files... > > The path is obviously correct on my local machine. The only > explanation is that Hadoop is looking for it on HDFS under that path. > > I'm desperate. I don't understand why I'm the only person who can get > this working. Could you please describe to me the set of commands you > use to run a pipes program on a fully distributed cluster? > > ________________________________________________________________________________ > Keith Wiley kwi...@keithwiley.com keithwiley.com > music.keithwiley.com > > "The easy confidence with which I know another man's religion is folly > teaches > me to suspect that my own is also." > -- Mark Twain > ________________________________________________________________________________ >
