Hadoop pipes (wordcount) example failing (with hadoop 0.21.0)

2011-07-27 Thread Decimus Phostle
Hello Folks, I am trying to run a simplistic Hadoop pipes program (the typical wordcount example). Unfortunately I am seeing a bunch of errors while running it as follows: $ bin/hadoop pipes -Dhadoop.pipes.java.recordreader=true \ -Dhadoop.pipes.java.recordwriter=true \

Re: Deep Magic on the distributed classpath

2011-07-27 Thread John Armstrong
On Wed, 27 Jul 2011 10:58:17 -0400, David Rosenstrauch wrote: > There is another, easier approach: if your app inherits from the Tool > class / runs via ToolRunner, then your app can inherit the -libjars > command line functionality itself. This is true; the problem with this approach is that

Re: Deep Magic on the distributed classpath

2011-07-27 Thread David Rosenstrauch
On 07/27/2011 10:39 AM, John Armstrong wrote: So I think I've figured out how to fix my problem with putting files on the distributed classpath by digging through the code Hadoop uses to process -libjars. There is another, easier approach: if your app inherits from the Tool class / runs via T

Re: Merge Reducers Outputs

2011-07-27 Thread David Rosenstrauch
On 07/26/2011 06:52 PM, Mohamed Riadh Trad wrote: Dear All, Is it possible to set up a task with multiple reducers and merge reducers outputs into one single file? Bests, Trad Mohamed Riadh, M.Sc, Ing. Not within the map-reduce job, but you can merge it after the job is done. At my previo

Deep Magic on the distributed classpath

2011-07-27 Thread John Armstrong
So I think I've figured out how to fix my problem with putting files on the distributed classpath by digging through the code Hadoop uses to process -libjars. If I say DistributedCache.addFileToClassPath(hdfsFile,conf); then hdfsFile is added to the distributed cache, but doesn't show upon the c

Re: Adding files to map/reduce classpath

2011-07-27 Thread John Armstrong
On Tue, 26 Jul 2011 12:35:48 -0700, Shrijeet Paliwal wrote: > ** > See if this (very old) reply from Mikhail helps. > http://search-hadoop.com/m/QFVD1kEmQT > Here is the patch he is referring to. > http://m1.archiveorange.com/m/att/RNVYm/ArchiveOrange_8dEcdJI4bXFkKHBnsll8YzTc8u8a.patch > > **repl

Re:Re: what happen in my hadoop cluster?

2011-07-27 Thread 周俊清
Yes,I can see all the data node from web page:http://dn224.pengyun.org:50070/dfsnodelist.jsp? -- 周俊清 2ho...@163.com 在 2011-07-27 15:30:37，"Harsh J" 写道： >Are all your DataNodes up? > >2011/7/27 周俊清 <2ho...@163.com>: >> hello everyone, >> I got an exception from

Multiple avro outputs from a reducer

2011-07-27 Thread Vyacheslav Zholudev
Hi, I'm using the avro format both for input and output, for a mapper and a reducer. I would like to output multiple avro items with different schemata. For sequence files I would use the MultipleOutputs class from the mapreduce package. I looked into the same class but from the old package "mapr

what happen in my hadoop cluster?

2011-07-27 Thread 周俊清
hello everyone, I got an exception from my jobtracker's log file as follow: 2011-07-27 01:58:04,197 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory 2011-07-27 01:58:04,230 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory:hdfs://dn224.peng

Re: what happen in my hadoop cluster?

2011-07-27 Thread Arun C Murthy
Moving to hdfs-user@, bcc mapreduce-user@. You NameNode isn't coming out of safemode since all the datanodes haven't rejoined the cluster... > The ratio of reported blocks 0.2915 has not reached the threshold 0.9990. > Safe mode will be turned off automatically. Can you check why your datanod

Re: what happen in my hadoop cluster?

2011-07-27 Thread Harsh J
Are all your DataNodes up? 2011/7/27 周俊清 <2ho...@163.com>: > hello everyone, >     I got an exception from my jobtracker's log file as follow: > 2011-07-27 01:58:04,197 INFO org.apache.hadoop.mapred.JobTracker: Cleaning > up the system directory > 2011-07-27 01:58:04,230 INFO org.apache.hadoop.map