We have set up a single node cluster to test pipes examples. The
wordcount-simple example works. However, the wordcount-nopipe fails. The
wordcount-nopipe example was invoked with the following commandline:
"bin/hadoop pipes -conf src/examples/pipes/conf/word-nopipe.xml -input
input/ -output out"
The word-nopipe.xml conf has the "hadoop.pipes.executable" property set.
All other properties set to default.
Are we using the correct commandline to execute the
wordcount-nopipe example?
I am using hadoop-0.17.0 release. We have found a previous post which
seems to be related but we're not sure how to resolve it.
Owen O'Malley said:
>The nopipe example needs more documentation.
>It assumes that it is run with the InputFormat from
>src/test/org/apache/hadoop/mapred/pipes/
>WordCountInputFormat.java, which has a very specific input split
>format. By running with a TextInputFormat, it will send binary bytes
>as the input split and won't work right. The nopipe example should
>probably be recoded to use libhdfs too, but that is more complicated
>to get running as a unit test. Also note that since the C++ example
>is using local file reads, it will only work on a cluster if you have
>nfs or something working across the cluster.
Here is the output:
WARN mapred.JobClient: No job jar file set. User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
INFO mapred.FileInputFormat: Total input paths to process
: 3
INFO mapred.JobClient: Running job: job_200803032218_0004
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.JobClient: Task Id :
task_200808281013_0009_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at
org.apache.hadoop.mapred.pipes.Application.abort(Application.java
:139)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(
PipesMapRunner.java:83)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
:2124)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:243)
at
org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java
:324)
at
org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java
:345)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(
BinaryProtocol.java:112)
.....
: Hadoop Pipes Exception: failed to open
at /usr/local/hadoop/src/examples/pipes/impl/wordcount-nopipe.cc:66 in
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
Thanks.
Marc
