RE: problem about put a lot of files
Hi The max open files have limit in LINUX box. Please using ulimit to view and modify the limit 1.view limit # ulimit -a 2.modify limit For example # ulimit -n 10240 Best wish > -Original Message- > From: stchu [mailto:stchu.cl...@gmail.com] > Sent: Monday, June 22, 2009 12:57 PM > To: core-user@hadoop.apache.org > Subject: problem about put a lot of files > > Hi, > Is there any restriction on the amount of putting files? I tried to > put/copyFromLocal about 50,573 files to HDFS, but I faced a problem: > == > == > 09/06/22 11:34:34 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010 > 09/06/22 11:34:34 INFO dfs.DFSClient: Abandoning block > blk_8245450203753506945_65955 > 09/06/22 11:34:40 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010 > 09/06/22 11:34:40 INFO dfs.DFSClient: Abandoning block > blk_-8257846965500649510_65956 > 09/06/22 11:34:46 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010 > 09/06/22 11:34:46 INFO dfs.DFSClient: Abandoning block > blk_4751737303082929912_65956 > 09/06/22 11:34:56 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010 > 09/06/22 11:34:56 INFO dfs.DFSClient: Abandoning block > blk_5912850890372596972_66040 > 09/06/22 11:35:02 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.193:51010 > 09/06/22 11:35:02 INFO dfs.DFSClient: Abandoning block > blk_6609198685444611538_66040 > 09/06/22 11:35:08 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.193:51010 > 09/06/22 11:35:08 INFO dfs.DFSClient: Abandoning block > blk_6696101244177965180_66040 > 09/06/22 11:35:17 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010 > 09/06/22 11:35:17 INFO dfs.DFSClient: Abandoning block > blk_-5430033105510098342_66105 > 09/06/22 11:35:26 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink 140.96.89.57:51010 > 09/06/22 11:35:26 INFO dfs.DFSClient: Abandoning block > blk_5325140471333041601_66165 > 09/06/22 11:35:32 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.205:51010 > 09/06/22 11:35:32 INFO dfs.DFSClient: Abandoning block > blk_1121864992752821949_66165 > 09/06/22 11:35:39 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.205:51010 > 09/06/22 11:35:39 INFO dfs.DFSClient: Abandoning block > blk_-2096783021040778965_66184 > 09/06/22 11:35:45 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.205:51010 > 09/06/22 11:35:45 INFO dfs.DFSClient: Abandoning block > blk_6949821898790162970_66184 > 09/06/22 11:35:51 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.205:51010 > 09/06/22 11:35:51 INFO dfs.DFSClient: Abandoning block > blk_4708848202696905125_66184 > 09/06/22 11:35:57 INFO dfs.DFSClient: Exception in > createBlockOutputStream > java.io.IOException: Bad connect ack with firstBadLink > 140.96.89.205:51010 > 09/06/22 11:35:57 INFO dfs.DFSClient: Abandoning block > blk_8031882012801762201_66184 > 09/06/22 11:36:03 WARN dfs.DFSClient: DataStreamer Exception: > java.io.IOException: Unable to create new block. > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream( > DFSClient.java:2359) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient. > java:1745) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl > ient.java:1922) > > 09/06/22 11:36:03 WARN dfs.DFSClient: Error Recovery for block > blk_8031882012801762201_66184 bad datanode[2] > put: Could not get block locations. Aborting... > Exception closing file /osmFiles/a/109103.gpx.txt > java.io.IOException: Could not get block locations. Aborting... > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(D > FSClient.java:2153) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient. > java:1745) > at > org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCl > ient.java:1899) > > == > = >
can't edit the file that mounted by fuse_dfs by editor
Hey all I was trying to edit the file that mounted by fuse_dfs by vi editor, but the contents could not save. The command is like the following: [had...@vm-centos-5-shu-4 src]$ vi /mnt/dfs/test.txt The error message from system log (/var/log/messages) is the following: Feb 12 09:53:48 VM-CentOS-5-SHU-4 fuse_dfs: ERROR: could not connect open file fuse_dfs.c:1340 I using the hadoop0.19.0 and fuse-dfs version 26 with centos5.2. Does anyone have an idea as to what could be wrong! Thanks! zhuweimin
RE: Reduce won't start until Map stage reaches 100%?
Hi I think the number of your job's reduce task is 1 because if the number of reduce task is 1 then reduce stage does not start until Map stage 100% completion. zhuweimin -Original Message- From: Taeho Kang [mailto:tka...@gmail.com] Sent: Monday, February 09, 2009 4:26 PM To: hadoop-u...@lucene.apache.org Subject: Reduce won't start until Map stage reaches 100%? Dear All, With Hadoop 0.19.0, Reduce stage does not start until Map stage gets to the 100% completion. Has anyone faced the similar situation? ... ... - map 90% reduce 0% - map 91% reduce 0% - map 92% reduce 0% - map 93% reduce 0% - map 94% reduce 0% - map 95% reduce 0% - map 96% reduce 0% - map 97% reduce 0% - map 98% reduce 0% - map 99% reduce 0% - map 100% reduce 0% - map 100% reduce 1% - map 100% reduce 2% - map 100% reduce 3% - map 100% reduce 4% - map 100% reduce 5% - map 100% reduce 6% - map 100% reduce 7% - map 100% reduce 8% - map 100% reduce 9% Thank you all in advance, /Taeho
RE: The error occurred when a lot of files created use fuse-dfs
Brian Thank you very much. The version of Hadoop is 0.19.0,I think 4616 and 4635 patches is necessary. I will try it. -Original Message- From: Brian Bockelman [mailto:bbock...@cse.unl.edu] Sent: Monday, December 15, 2008 10:00 PM To: core-user@hadoop.apache.org Subject: Re: The error occurred when a lot of files created use fuse-dfs Hey, What version of Hadoop are you running? Have you taken a look at HADOOP-4775? https://issues.apache.org/jira/browse/HADOOP-4775 Basically, fuse-dfs is not usable on Hadoop 0.19.0 without a patch. Brian On Dec 15, 2008, at 12:24 AM, zhuweimin wrote: > Dear fuse-dfs users > > I copy 1000 files into hadoop from local disk use fuse-dfs, > Display the following error when the 600th files are copied: > > cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/ > 10m_33.dat': > Input/output error > cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/ > 10m_34.dat': > Input/output error > cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/ > 10m_35.dat': > Input/output error > cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/ > 10m_36.dat': > Input/output error > cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/ > 10m_37.dat': > Input/output error > cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/ > 10m_38.dat': > Input/output error > ... > > It is necessary to remount the fuse-dfs. > > Do you think about of the error. > > thanks > >
The error occurred when a lot of files created use fuse-dfs
Dear fuse-dfs users I copy 1000 files into hadoop from local disk use fuse-dfs, Display the following error when the 600th files are copied: cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/10m_33.dat': Input/output error cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/10m_34.dat': Input/output error cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/10m_35.dat': Input/output error cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/10m_36.dat': Input/output error cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/10m_37.dat': Input/output error cp: cannot create regular file `/mnt/dfs/user/hadoop/fuse3/10m/10m_38.dat': Input/output error ... It is necessary to remount the fuse-dfs. Do you think about of the error. thanks
RE: Did Hadoop support gz/zip format file?
Hello I have a requirement of use file with the extension .Z(the file is created by UNIX compress command) in Hadoop. It will too be automatically recognized / handled ? Any suggestion on how to handle the .Z file in the map task? Thanks Best Regards Alamo -Original Message- From: Abdul Qadeer [mailto:[EMAIL PROTECTED] Sent: Monday, November 03, 2008 4:19 PM To: core-user@hadoop.apache.org Subject: Re: Did Hadoop support gz/zip format file? Hadoop supports gzip format by means of its gzip codecs. If you meant to use default input format / record reader of Hadoop, it will automatically recognize / handle your gzip input files. And if you are using some other input formats / record readers, you can use the gzip codec provided by Hadoop to de-compress the data and then interpret it in your record reader. On Sun, Nov 2, 2008 at 7:49 PM, howardh <[EMAIL PROTECTED]> wrote: > Hello, > > I have a requirement of use gz/zip format in Hadoop, After these days > research and learning, It seems Hadoop didn't support gz/zip file yet, > is that true? > Like I'm going to create file with gz format, and read it later by > FileSystem interface. Does it feasible? Experts! Could you show me > some advices? > > Best Regards > 2008-11-03 > > > > howardh >
