Re: Pipes example wordcount-nopipe.cc failed when reading from input splits
Hi,
Here is some discussion on how to run wordcount-nopipe :
http://www.nabble.com/pipe-application-error-td13840804.html
Probably makes sense for your question.
Thanks
Amareshwari
11 Nov. wrote:
I traced into the c++ recordreader code:
WordCountReader(HadoopPipes::MapContext& context) {
std::string filename;
HadoopUtils::StringInStream stream(context.getInputSplit());
HadoopUtils::deserializeString(filename, stream);
struct stat statResult;
stat(filename.c_str(), &statResult);
bytesTotal = statResult.st_size;
bytesRead = 0;
cout << filename<:
hi colleagues,
I have set up the single node cluster to test pipes examples.
wordcount-simple and wordcount-part work just fine. but
wordcount-nopipe can't run. Here is my commnad line:
bin/hadoop pipes -conf src/examples/pipes/conf/word-nopipe.xml -input
input/ -output out-dir-nopipe1
and here is the error message printed on my console:
08/03/03 23:23:06 WARN mapred.JobClient: No job jar file set. User
classes may not be found. See JobConf(Class) or JobConf#setJar(String).
08/03/03 23:23:06 INFO mapred.FileInputFormat: Total input paths to
process : 1
08/03/03 23:23:07 INFO mapred.JobClient: Running job:
job_200803032218_0004
08/03/03 23:23:08 INFO mapred.JobClient: map 0% reduce 0%
08/03/03 23:23:11 INFO mapred.JobClient: Task Id :
task_200803032218_0004_m_00_0, Status : FAILED
java.io.IOException: pipe child exception
at org.apache.hadoop.mapred.pipes.Application.abort(
Application.java:138)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(
PipesMapRunner.java:83)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
at org.apache.hadoop.mapred.TaskTracker$Child.main(
TaskTracker.java:1787)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:250)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java
:313)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java
:335)
at
org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(
BinaryProtocol.java:112)
task_200803032218_0004_m_00_0:
task_200803032218_0004_m_00_0:
task_200803032218_0004_m_00_0:
task_200803032218_0004_m_00_0: Hadoop Pipes Exception: failed to open
at /home/hadoop/hadoop-0.15.2-single-cluster
/src/examples/pipes/impl/wordcount-nopipe.cc:67 in
WordCountReader::WordCountReader(HadoopPipes::MapContext&)
Could anybody tell me how to fix this? That will be appreciated.
Thanks a lot!
Re: Bugs in 0.16.0?
Holden Robbins wrote: Hello, I'm just starting to dig into Hadoop and testing it's feasibility for large scale development work. I was wondering if anyone else being affected by these issues using hadoop 0.16.0? I searched Jira, and I'm not sure if I saw anything that specifically fit some of these: 1) The symlinks for the distributed cache in the task directory are being created as 'null' directory links (stated another way, the name of the symbolic link in the directory is the string literal "null"). Am I doing something wrong to cause this, or do not many people use this functionality? If you want create symlinks for distributed cache the url has to have symlink field like hdfs://host:port/#. And mapred.create.symlink must be set to "yes". If mapred.create.symlink is yes and link field is not provided, distributed cache will create a symlink with literal "null" as you said. 2) I'm running into an issue where the job is giving errors in the form: 08/03/01 09:44:25 INFO mapred.JobClient: Task Id : task_200803010908_0001_r_02_0, Status : FAILED Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. 08/03/01 09:44:25 WARN mapred.JobClient: Error reading task outputGo-Box4 08/03/01 09:44:25 WARN mapred.JobClient: Error reading task outputGo-Box4 The jobs appear to never finish the reducing once this happens. The tasks themselves are long running map tasks (up to 10 minutes per input), as far as I understand from the Jira posts this is related to the MAX_FAILED_UNIQUE_FETCHES being hard coded to 4? Is there a known work around or fix in the pipeline? Possible related jira post: https://issues.apache.org/jira/browse/HADOOP-2220 Improving the way the shuffling mechanism works may also help? https://issues.apache.org/jira/browse/HADOOP-1339 I've tried setting: mapred.reduce.copy.backoff 1440 The maximum amount of time (in seconds) a reducer spends on fetching one map output before declaring it as failed. which should be 24 minutes, with no effect. 3) Lastly, it would seem beneficial for jobs that have significant startup overhead and memory requirements to not be run in separate JVMs for each task. Along these lines, it looks like someone submitted a patch for JVM-reuse a while back, but it wasn't commited? https://issues.apache.org/jira/browse/HADOOP-249 Probably a question for the dev mailing list, but if I wanted to modify hadoop to allow threading tasks, rather than running independent JVMs, is there any reason someone hasn't done this yet? Or am I overlooking something? Thanks, -Holden
Re: the best way to kill a bad job?
You can kill job using the command bin/hadoop job -kill . You need not shutdown the cluster. Thanks, Amareshwari. Jim the Standing Bear wrote: What is the best way to kill a bad job (e.g. an infinite loop)? The job I was running went into an infinite loop and I had to stop it with ctrl-c on the master node. Then I used bin/stop-all.sh hoping to stop the entire hadoop system... But that didn't work. I had to manually log into every node and use kill -9 to terminate all these java processes. There's gotta be a better way of doing this - I cannot imagine doing this for a 1000 node cluster... Can someone enlighten me please. Thanks. -- Jim
Re: How to write a matrix multiplication program in C ++ using Hadoop?
You can use Hadoop pipes for c++ map/reduce. Some info available at http://hadoop.apache.org/core/docs/r0.15.3/api/org/apache/hadoop/mapred/pipes/package-summary.html Hadoop wrote: I do not see any code at the link : https://issues.apache.org/jira/browse/HADOOP-2515 and when I tried to register to login, I got following the error message: Errors: Your session has timed out I have found this link at Haoop web site: http://wiki.apache.org/hadoop/Matrix?highlight=%28matri%29 but it does not solve my problem for three reasons: 1-The code is for shared-memory architecture but I am looking for a code for multicomputer (commodity cluster) architecture 2- The code generate random numbers for the matrix but I am looking for a code to read the inputs form text files and write the result to a text file 3- How can I write the code in c++ and run it on Haoop? edward yoon-3 wrote: Please see https://issues.apache.org/jira/browse/HADOOP-2515 On 1/26/08, Hadoop <[EMAIL PROTECTED]> wrote: How can I write a C++ program that reads two text files (MatriX1 and Matrix2), multiplies the two matrices, and writes the result in a text file (Matrix3) using Hadoop? -- View this message in context: http://www.nabble.com/How-to-write-a-matrix-multiplication-program-in-C-%2B%2B-using-Hadoop--tp15013753p15013753.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com. -- B. Regards, Edward yoon @ NHN, corp.
Re: distributed cache
jerrro wrote: Hello, Is there a way to use Distributed Cache with a pipes (C++ code) job? I want to be able to access a file on the local disk all over the data nodes, so hadoop would copy it to all data nodes before a map reduce job. Thanks. Hi, First of all you need to copy the files to the dfs. And then add the file to the distributed cache. You can give comma sepearted values of the files or archives to be added to distributed cache, for "mapred.cache.files" and "mapred.cache.archives" in the conf file. Ex: mapred.cache.files /files/file1,/files/file2.txt The files in distributed cache mapred.cache.archives /archives/arc1.zip,/archives/arc2.jar The archives in distributed cache You can also give URIs of the file names. You can give the URI as hdfs://#, here mapred will create a symlink with name "link" in the working directory. Hope this clarifies Thanks Amareshwari
