Re: Availability of Job traces or logs

Amar, I am attempting to write a new scheduler for Hadoop and test it using Mumak. 1> I want to test its behaviour under different size of jobs traces(meaning number of jobs say 5,10,25,50,100) under different number of nodes. Till now i was using only the test/data given by mumak which has 19 jo

rename after copying on HDFS does not succeed

Hi , I create a copy of a file in HDFS using org.apache.hadoop.fs.FileUtil.copy(dfs, file1, dfs, file2, false, false, dfs.getConf()); boolean succ = dfs.rename(new Path(file2), new Path(file3)); The rename always return false, any suggestions? Is that because FileUtil.copy() is not closing

Re: Hadoop Streaming

Oh, I see the line wrapped. My bad. Either way, I think the NLineInputFormat is what you need. I'm assuming you want one line of input to execute on one mapper. Thanks, Tom On Sat, Dec 3, 2011 at 7:57 PM, Daniel Yehdego wrote: > > TOM, > What the HADOOP script do is ...read each line from th

RE: Hadoop Streaming

TOM, What the HADOOP script do is ...read each line from the STDIN and execute the program pknotsRG. tmp.txt is a temporary file. the script is like this: #!/bin/sh rm -f temp.txt;while read line doecho $line >> temp.txt;doneexec /data/yehdego/hadoop-0.20.2/PKNOTSRG/

Re: Hadoop Cluster Quick Setup Script

Thanks Ayon. Unfortunately my Cloud software not supported yet. https://issues.apache.org/jira/browse/WHIRR-388 )-: Thanks for the suggestion, though. We may ultimately contribute code ... > Some nice guys at HortonWorks told me yesterday about Apache Whirr. Do you > think this will > help you

Re: Hadoop Streaming

Hi Daniel, I see from your other thread that your HADOOP script has a line like: #!/bin/shrm -f temp.txt I'm not sure what that is, exactly. I suspect the -f is reading from some file and the while loop you had listed read from stdin it seems. What does your input look like? I think what's ha

RE: Hadoop Streaming

Thanks Tom for your reply, I think my code is reading from stdin. Because I tried it locally using the following command and its running: $ bin/hadoop fs -cat /user/yehdego/Hadoop-Data-New/RF00171_A.bpseqL3G1_seg_Optimized_Method.txt | head -2 | ./HADOOP But when I tried streaming , it faile

Re: Hadoop Streaming

So that code 126 should be kicked out by your program - do you know what that means? Your code can read from stdin? Thanks, Tom On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego wrote: > > I have the following error in running hadoop streaming, > PipeMapRed\.waitOutputThreads(): subprocess failed

Hadoop Streaming

I have the following error in running hadoop streaming, PipeMapRed\.waitOutputThreads(): subprocess failed with code 126at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java

Re: Hadoop Cluster Quick Setup Script

Edmon, I made some effort but got bored eventually 'cause of no interest. I think i made some progress and perhaps you can take it forward from there in MAPREDUCE-3131 I am ready to help incase anything I can with. Also it works perfect for a

Re: Hadoop Cluster Quick Setup Script

Some nice guys at HortonWorks told me yesterday about Apache Whirr. Do you think this will help you? http://whirr.apache.org/docs/0.6.0/quick-start-guide.html   -Ayon See My Photos on Flickr Also check out my Blog for answers to commonly asked questions. From:

Hadoop Cluster Quick Setup Script

Does anyone have or know of a simple (Apache) Hadoop cluster script that sets up Hadoop accross the cluster using some reasonable default values and across set of IPs. I am wanting to install a min five virtual node cluster and perhaps grow it larger. I would like to use some script that pulls com

Re: Creating a custom trace file using Rumen

Arun, > Is there any way to create a new trace file from job history logs with custom > set of split locations ? Rumen doesn't support this. You have to manually edit the trace file. > Is there any way to generate trace with extra fields ? Rumen is a tool which converts Hadoop MapReduce logs int