I tested that example as well and I'm getting the same exception with this in stderr:
Hadoop Pipes Exception: failed to open hdfs://compute-0-7-2:54310/in/file at /home/scaino/hadoop-2.2.0-maven/hadoop-tools/hadoop-pipes/src/main/native/examples/impl/wordcount-nopipe.cc:82 in WordCountReader I found out that wordcount-nopipe is picky for its input format, here is the full explanation: http://lucene.472066.n3.nabble.com/pipe-application-error-td650185.html On 17 December 2013 17:36, Mauro Del Rio <mdrio1...@gmail.com> wrote: > Ok, I had some problems with configuration and host resolution and I fixed > them. I was able to run successfully the simple wordcount example, but I > failed running wordcount-nopipe.cc. > This is the stack trace: > > Error: java.io.IOException: pipe child exception > at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:226) > at > org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:102) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) > Caused by: java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:250) > at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) > at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) > at > org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:125) > > > Any suggestions? > > > 2013/12/17 Silvina Caíno Lores <silvi.ca...@gmail.com> > >> I'm having similar problems with pipes, mostly because of issues with the >> native shared libraries that leave the job stuck either at 0%-0% or before >> launch (because the resource manager gets stuck as well and crashes). >> >> I found that out by looking at the stderr logs by the way. >> >> Let us know if you progress, good luck! >> >> Best, >> Silvina >> >> >> On 16 December 2013 19:32, Vinod Kumar Vavilapalli < >> vino...@hortonworks.com> wrote: >> >>> You should navigate to the ResourceManager UI following the link and see >>> what is happening on the ResourceManager as well as the application-master. >>> Check if any nodes are active first. Then look at ResourceManager and >>> NodeManager logs. >>> >>> +Vinod >>> >>> On Dec 16, 2013, at 10:29 AM, Mauro Del Rio <mdrio1...@gmail.com> wrote: >>> >>> I installed hadoop 2.2.0 on a small cluster, just two nodes. I run a >>> simple wordcount in c++ with pipes, this time there was no exception, but >>> the job didn't finish. This is the output on the shell where I launched >>> pipes: >>> >>> mauro@mauro-VirtualBox:~/hadoop-2.2.0$ bin/mapred pipes -program wc >>> -input test.sh -output out >>> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / >>> 0.0.0.0:9052 >>> 13/12/16 18:51:41 INFO client.RMProxy: Connecting to ResourceManager at / >>> 0.0.0.0:9052 >>> 13/12/16 18:51:41 WARN mapreduce.JobSubmitter: No job jar file set. >>> User classes may not be found. See Job or Job#setJar(String). >>> 13/12/16 18:51:41 INFO mapred.FileInputFormat: Total input paths to >>> process : 1 >>> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: number of splits:2 >>> 13/12/16 18:51:41 INFO Configuration.deprecation: user.name is >>> deprecated. Instead, use mapreduce.job.user.name >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.cache.files.filesizes is deprecated. Instead, use >>> mapreduce.job.cache.files.filesizes >>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.cache.files is >>> deprecated. Instead, use mapreduce.job.cache.files >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.pipes.user.inputformat is deprecated. Instead, use >>> mapreduce.pipes.inputformat >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.output.value.class is deprecated. Instead, use >>> mapreduce.job.output.value.class >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.mapoutput.value.class is deprecated. Instead, use >>> mapreduce.map.output.value.class >>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.input.dir is >>> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir >>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.output.dir is >>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir >>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.map.tasks is >>> deprecated. Instead, use mapreduce.job.maps >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> hadoop.pipes.partitioner is deprecated. Instead, use >>> mapreduce.pipes.partitioner >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> hadoop.pipes.executable is deprecated. Instead, use >>> mapreduce.pipes.executable >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.cache.files.timestamps is deprecated. Instead, use >>> mapreduce.job.cache.files.timestamps >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.output.key.class is deprecated. Instead, use >>> mapreduce.job.output.key.class >>> 13/12/16 18:51:41 INFO Configuration.deprecation: >>> mapred.mapoutput.key.class is deprecated. Instead, use >>> mapreduce.map.output.key.class >>> 13/12/16 18:51:41 INFO Configuration.deprecation: mapred.working.dir is >>> deprecated. Instead, use mapreduce.job.working.dir >>> 13/12/16 18:51:41 INFO mapreduce.JobSubmitter: Submitting tokens for >>> job: job_1387213974967_0003 >>> 13/12/16 18:51:42 INFO mapred.YARNRunner: Job jar is not present. Not >>> adding any jar to the list of resources. >>> 13/12/16 18:51:42 INFO impl.YarnClientImpl: Submitted application >>> application_1387213974967_0003 to ResourceManager at /0.0.0.0:9052 >>> 13/12/16 18:51:42 INFO mapreduce.Job: The url to track the job: >>> http://mauro-VirtualBox:8088/proxy/application_1387213974967_0003/<http://mauro-virtualbox:8088/proxy/application_1387213974967_0003/> >>> 13/12/16 18:51:42 INFO mapreduce.Job: Running job: job_1387213974967_0003 >>> >>> >>> The job status from bin/mapred job -list is PREP. I didn't find any >>> interesting information in logs file. >>> >>> >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >>> >> >> > > > -- > Mauro >
