RE: HDFS error
Thanks for you help Samuel. I was having problem in both writing and reading. Did run the fsck and removed some damaged files and restarted the dfs. Seems to be OK now. Not exactly sure what happened though. Thanks, Htin -Original Message- From: Samuel Guo [mailto:[EMAIL PROTECTED] Sent: Thursday, October 09, 2008 6:15 PM To: core-user@hadoop.apache.org Subject: Re: HDFS error Does this happen when you want to write some files to HDFS? if it is so, plz check that you have enough space in the disks of your datanode. if this happened when you want to read some files in HDFS, maybe you can run fsck to check if the file is healthy. hope it will be helpful. On Fri, Oct 10, 2008 at 8:21 AM, Htin Hlaing <[EMAIL PROTECTED]> wrote: > Hello - I am experiencing the following HDFS problem across the clusters > sharing the DFS. It's not specific to this particular data node ip > address but the exception is across all other data nodes as well. Any > help is appreciated. > > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.50.80.108:50010 > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Abandoning > block blk_2383100013215057496 > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Waiting to > find target node: 10.50.80.112:50010 > 2008-10-09 14:14:16,604 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:14:54,370 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:17:19,619 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:17:57,385 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:20:25,634 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:21:09,401 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:23:28,649 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > > Is there a knowledge base that I can search in the old posts to the > mailing list? > > Thanks, > Htin >
save mp3 file Using Hadoop
i am a beginner,i want to save a lot of Mp3 files into Hadoop and using Tomcat to service this files. 1.i want to using org.apache.hadoop.io.MapFile, key is mp3 name's MD5,value is Mp3 stream data. mp3 's filesize is nearly 3M. i split 20G per one MapFile.There are 30~50 MapFile. 2. i put these MapFile into x linux server. 3. my user send Mp3' name request to Tomcat ,Tomcat using Mp3 name 's MD5 to find Mp3 file in one linux server . question: 1> is it right way to use hadoop? 2> if 20 user access one linux server in one time,maybe 20 Mp3 files load in memory,20* 3M memory is being eat? is it right? how to solve this problem ?
Re: How to make LZO work?
Thanks, Arun. Does that mean I have to rebuild the native library? Also, the LZO installation puts liblzo2.a and liblzo2.la under /usr/local/lib. There is no liblzo2.so there. Do I need to rename them to liblzo2.so somehow? --- On Thu, 10/9/08, Arun C Murthy <[EMAIL PROTECTED]> wrote: > From: Arun C Murthy <[EMAIL PROTECTED]> > Subject: Re: How to make LZO work? > To: core-user@hadoop.apache.org > Date: Thursday, October 9, 2008, 6:35 PM > On Oct 9, 2008, at 5:58 PM, Songting Chen wrote: > > > Hi, > > I have installed lzo-2.03 to my Linux box. > > But still my code for writing a SequenceFile using > LZOcodec returns > > the following error: > > > > util.NativeCodeLoader: Loaded the native-hadoop > library > > java.lang.UnsatisfiedLinkError: Cannot load > liblzo2.so! > > > > What needs to be done to make this work? > > > > http://hadoop.apache.org/core/docs/current/native_libraries.html > > Arun
Re: How to make LZO work?
On Oct 9, 2008, at 5:58 PM, Songting Chen wrote: Hi, I have installed lzo-2.03 to my Linux box. But still my code for writing a SequenceFile using LZOcodec returns the following error: util.NativeCodeLoader: Loaded the native-hadoop library java.lang.UnsatisfiedLinkError: Cannot load liblzo2.so! What needs to be done to make this work? http://hadoop.apache.org/core/docs/current/native_libraries.html Arun
Re: HDFS error
Does this happen when you want to write some files to HDFS? if it is so, plz check that you have enough space in the disks of your datanode. if this happened when you want to read some files in HDFS, maybe you can run fsck to check if the file is healthy. hope it will be helpful. On Fri, Oct 10, 2008 at 8:21 AM, Htin Hlaing <[EMAIL PROTECTED]> wrote: > Hello - I am experiencing the following HDFS problem across the clusters > sharing the DFS. It's not specific to this particular data node ip > address but the exception is across all other data nodes as well. Any > help is appreciated. > > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.50.80.108:50010 > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Abandoning > block blk_2383100013215057496 > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Waiting to > find target node: 10.50.80.112:50010 > 2008-10-09 14:14:16,604 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:14:54,370 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:17:19,619 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:17:57,385 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:20:25,634 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:21:09,401 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:23:28,649 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > > Is there a knowledge base that I can search in the old posts to the > mailing list? > > Thanks, > Htin >
How to make LZO work?
Hi, I have installed lzo-2.03 to my Linux box. But still my code for writing a SequenceFile using LZOcodec returns the following error: util.NativeCodeLoader: Loaded the native-hadoop library java.lang.UnsatisfiedLinkError: Cannot load liblzo2.so! What needs to be done to make this work? Thanks a lot, -Songting
Re: HDFS error
No live nodes contain current block It seem that you want to access a Block which isn't exist in the cluster. 2008/10/10 Htin Hlaing <[EMAIL PROTECTED]> > Hello - I am experiencing the following HDFS problem across the clusters > sharing the DFS. It's not specific to this particular data node ip > address but the exception is across all other data nodes as well. Any > help is appreciated. > > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Exception in > createBlockOutputStream java.io.IOException: Bad connect ack with > firstBadLink 10.50.80.108:50010 > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Abandoning > block blk_2383100013215057496 > 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Waiting to > find target node: 10.50.80.112:50010 > 2008-10-09 14:14:16,604 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:14:54,370 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:17:19,619 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:17:57,385 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:20:25,634 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:21:09,401 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_-4901580690304720524 from any node: java.io.IOException: > No live nodes contain current block > 2008-10-09 14:23:28,649 INFO org.apache.hadoop.fs.DFSClient: Could not > obtain block blk_3359685166656187008 from any node: java.io.IOException: > No live nodes contain current block > > Is there a knowledge base that I can search in the old posts to the > mailing list? > > Thanks, > Htin > -- Sorry for my English!! 明 Please help me correct my English expression and error in syntax
HDFS error
Hello - I am experiencing the following HDFS problem across the clusters sharing the DFS. It's not specific to this particular data node ip address but the exception is across all other data nodes as well. Any help is appreciated. 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.50.80.108:50010 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Abandoning block blk_2383100013215057496 2008-10-09 14:13:59,732 INFO org.apache.hadoop.fs.DFSClient: Waiting to find target node: 10.50.80.112:50010 2008-10-09 14:14:16,604 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_3359685166656187008 from any node: java.io.IOException: No live nodes contain current block 2008-10-09 14:14:54,370 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_-4901580690304720524 from any node: java.io.IOException: No live nodes contain current block 2008-10-09 14:17:19,619 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_3359685166656187008 from any node: java.io.IOException: No live nodes contain current block 2008-10-09 14:17:57,385 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_-4901580690304720524 from any node: java.io.IOException: No live nodes contain current block 2008-10-09 14:20:25,634 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_3359685166656187008 from any node: java.io.IOException: No live nodes contain current block 2008-10-09 14:21:09,401 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_-4901580690304720524 from any node: java.io.IOException: No live nodes contain current block 2008-10-09 14:23:28,649 INFO org.apache.hadoop.fs.DFSClient: Could not obtain block blk_3359685166656187008 from any node: java.io.IOException: No live nodes contain current block Is there a knowledge base that I can search in the old posts to the mailing list? Thanks, Htin
Hadoop User Group (Bay Area) Oct 15th
The next Bay Area User Group meeting is scheduled for October 15th at Yahoo! 2821 Mission College Blvd, Santa Clara, Building 1, Training Rooms 3 & 4 from 6:00-7:30 pm. Agenda: - Exploiting database join techniques for analytics with Hadoop: Jun Rao, IBM - Jaql Update: Kevin Beyer, IBM - Experiences moving a Petabyte Data Center: Sriram Rao, Quantcast Look forward to seeing you there! Ajay
Keep jobcache files around
Hello, I wish to keep my jobcache files after the run. I'm using a program which can't read from STDIN (i'm hadoop streaming) so i've written a python wrapper to create a file and pass the file to the program. However, though the python file runs (and maybe the program) i'm not getting the desired results. Nothing fails, and even though I've kept keep.failed.tasks=true (-jobconf mapred.reduce.tasks=0 -jobconf keep.failed.tasks.files=1 in the streaming command line) nothing is preserved i.e the jobcache folders(no attempt_200810091420_0004_m_03_3*** folders) are deletecd from the task nodes. How can I keep them, even when nothing fails? Regards Saptarshi Saptarshi Guha | [EMAIL PROTECTED] | http://www.stat.purdue.edu/~sguha
Re: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory
Thanks Alexander!! On Thu, Oct 9, 2008 at 4:49 PM, Alexander Aristov <[EMAIL PROTECTED]> wrote: > I received such errors when I overloaded data nodes. You may increase swap > space or run less tasks. > > Alexander > > 2008/10/9 Edward J. Yoon <[EMAIL PROTECTED]> > >> Hi, >> >> I received below message. Can anyone explain this? >> >> 08/10/09 11:53:33 INFO mapred.JobClient: Task Id : >> task_200810081842_0004_m_00_0, Status : FAILED >> java.io.IOException: Cannot run program "bash": java.io.IOException: >> error=12, Cannot allocate memory >>at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) >>at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) >>at org.apache.hadoop.util.Shell.run(Shell.java:134) >>at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) >>at >> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:296) >>at >> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) >>at >> org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) >>at >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:734) >>at >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:694) >>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220) >>at >> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) >> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot >> allocate memory >>at java.lang.UNIXProcess.(UNIXProcess.java:148) >>at java.lang.ProcessImpl.start(ProcessImpl.java:65) >>at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) >>... 10 more >> >> -- >> Best regards, Edward J. Yoon >> [EMAIL PROTECTED] >> http://blog.udanax.org >> > > > > -- > Best Regards > Alexander Aristov > -- Best regards, Edward J. Yoon [EMAIL PROTECTED] http://blog.udanax.org
Re: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory
I received such errors when I overloaded data nodes. You may increase swap space or run less tasks. Alexander 2008/10/9 Edward J. Yoon <[EMAIL PROTECTED]> > Hi, > > I received below message. Can anyone explain this? > > 08/10/09 11:53:33 INFO mapred.JobClient: Task Id : > task_200810081842_0004_m_00_0, Status : FAILED > java.io.IOException: Cannot run program "bash": java.io.IOException: > error=12, Cannot allocate memory >at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) >at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) >at org.apache.hadoop.util.Shell.run(Shell.java:134) >at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) >at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:296) >at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) >at > org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) >at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:734) >at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:694) >at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220) >at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) > Caused by: java.io.IOException: java.io.IOException: error=12, Cannot > allocate memory >at java.lang.UNIXProcess.(UNIXProcess.java:148) >at java.lang.ProcessImpl.start(ProcessImpl.java:65) >at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) >... 10 more > > -- > Best regards, Edward J. Yoon > [EMAIL PROTECTED] > http://blog.udanax.org > -- Best Regards Alexander Aristov
