Oh, I see the line wrapped. My bad. Either way, I think the NLineInputFormat is what you need. I'm assuming you want one line of input to execute on one mapper.
Thanks, Tom On Sat, Dec 3, 2011 at 7:57 PM, Daniel Yehdego <dtyehd...@miners.utep.edu> wrote: > > TOM, > What the HADOOP script do is ...read each line from the STDIN and execute the > program pknotsRG. tmp.txt is a temporary file. > the script is like this: > #!/bin/sh > rm -f temp.txt; while read line > do echo $line >> temp.txt; done exec > /data/yehdego/hadoop-0.20.2/PKNOTSRG/src/pknotsRG -k 0 -F temp.txt; > >> Date: Sat, 3 Dec 2011 19:49:46 -0800 >> Subject: Re: Hadoop Streaming >> From: t...@supertom.com >> To: common-user@hadoop.apache.org >> >> Hi Daniel, >> >> I see from your other thread that your HADOOP script has a line like: >> >> #!/bin/shrm -f temp.txt >> >> I'm not sure what that is, exactly. I suspect the -f is reading from >> some file and the while loop you had listed read from stdin it seems. >> >> What does your input look like? I think what's happening is that you >> might be expecting lines of input and you're getting splits. What >> does your input look like? >> >> You might want to try this: >> -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat >> >> Thanks, >> >> Tom >> >> >> >> >> On Sat, Dec 3, 2011 at 7:22 PM, Daniel Yehdego >> <dtyehd...@miners.utep.edu> wrote: >> > >> > Thanks Tom for your reply, >> > I think my code is reading from stdin. Because I tried it locally using >> > the following command and its running: >> > $ bin/hadoop fs -cat >> > /user/yehdego/Hadoop-Data-New/RF00171_A.bpseqL3G1_seg_Optimized_Method.txt >> > | head -2 | ./HADOOP >> > >> > But when I tried streaming , it failed and gave me the error code 126. >> > >> >> Date: Sat, 3 Dec 2011 19:14:20 -0800 >> >> Subject: Re: Hadoop Streaming >> >> From: t...@supertom.com >> >> To: common-user@hadoop.apache.org >> >> >> >> So that code 126 should be kicked out by your program - do you know >> >> what that means? >> >> >> >> Your code can read from stdin? >> >> >> >> Thanks, >> >> >> >> Tom >> >> >> >> On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego >> >> <dtyehd...@miners.utep.edu> wrote: >> >> > >> >> > I have the following error in running hadoop streaming, >> >> > PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 >> >> > at >> >> > org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) >> >> > at >> >> > org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) >> >> > at >> >> > org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) >> >> > at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) >> >> > at >> >> > org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) >> >> > at >> >> > org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) >> >> > at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at >> >> > org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170) >> >> > I couldn't find out any other error information. >> >> > Any help ? >> >> > >> > >
