Hi Prem, My guess is that your Linux filesystem on this partition is corrupt. Check dmesg for output indicating fs-level errors.
-Todd On Mon, Jun 6, 2011 at 1:23 PM, Jain, Prem <premanshu.j...@netapp.com>wrote: > Mapuser or hdfs user didn't seem to help, so I switched to root: > > [root@hadoop20 mapred]# ls -la /part/data > total 0 > drwx------ 3 hdfs hadoop 16 Jun 6 10:22 . > drwxrwxrwx 4 hdfs hadoop 47 May 26 18:36 .. > drwxr-xr-x 4 mapred mapred 35 May 26 21:02 tmp > [root@hadoop20 mapred]# > > [root@hadoop20 mapred]# pwd > > /part/data/tmp/distcache/642114211252449475_2038269146_799583695/hmaster/user/mapred > [root@hadoop20 mapred]# ls -la > total 0 > drwxr-xr-x 3 mapred mapred 22 Jun 6 12:46 . > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 .. > ?--------- ? ? ? ? ? input-dir > > > > -----Original Message----- > From: Marcos Ortiz [mailto:mlor...@uci.cu] > Sent: Monday, June 06, 2011 1:17 PM > To: hdfs-user@hadoop.apache.org > Cc: Jain, Prem > Subject: Re: cant remove files from tmp > > * Why are using he root user for these operations? > * Which are your permisions on your data directory? (ls -la /part/data)? > > Regards > > El 6/6/2011 3:41 PM, Jain, Prem escribió: > > I have a wrecked datanode which is giving me hard time restarting. It > > keeps complaining of Datanode dead, pid file exists. I already tried > > deleting the files but seems like the files are corrupted and don't > > allow me delete. > > > > ____________________________________________________________________ > > > > Here is the log: > > ____________________________________________________________________ > > > > /************************************************************ > > STARTUP_MSG: Starting DataNode > > STARTUP_MSG: host = hadoop20/192.168.1.190 > > STARTUP_MSG: args = [] > > STARTUP_MSG: version = 0.20.2-cdh3u0 > > STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; > > compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011 > > ************************************************************/ > > 2011-06-06 09:11:01,232 INFO > > org.apache.hadoop.security.UserGroupInformation: JAAS Configuration > > already set up for Hadoop, not re-installing. > > 2011-06-06 09:11:01,369 ERROR > > org.apache.hadoop.hdfs.server.datanode.DataNode: > > org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access > > `/part/data/tmp/distcache/642114211252449475_2038269146_79 > > 9583695/hmaster/user/mapred/input-dir': No such file or directory > > du: cannot read directory > > `/part/data/tmp/mapred/jobcache/job_201105261845_0005': Permission > > denied > > > > > > _________________________________ > > Here is the file I can't delete > > _________________________________ > > [root@hadoop20 distcache]# pwd > > /part/data/tmp/distcache > > [root@hadoop20 distcache]# ls -la > > total 0 > > drwxr-xr-x 3 mapred mapred 52 May 26 21:36 . > > drwxr-xr-x 4 mapred mapred 35 May 26 21:02 .. > > drwxr-xr-x 3 mapred mapred 20 May 26 21:17 > > 642114211252449475_2038269146_799583695 > > [root@hadoop20 distcache]# cd * > > [root@hadoop20 642114211252449475_2038269146_799583695]# ls -la > > total 0 > > drwxr-xr-x 3 mapred mapred 20 May 26 21:17 . > > drwxr-xr-x 3 mapred mapred 52 May 26 21:36 .. > > drwxr-xr-x 3 mapred mapred 17 May 26 21:17 hmaster > > [root@hadoop20 642114211252449475_2038269146_799583695]# cd h* > > [root@hadoop20 hmaster]# ls > > user > > [root@hadoop20 hmaster]# cd * > > [root@hadoop20 user]# ls -la > > total 0 > > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 . > > drwxr-xr-x 3 mapred mapred 17 May 26 21:17 .. > > drwxr-xr-x 3 mapred mapred 22 May 26 21:17 mapred > > [root@hadoop20 user]# cd m* > > [root@hadoop20 mapred]# ls -la > > total 0 > > drwxr-xr-x 3 mapred mapred 22 May 26 21:17 . > > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 .. > > ?--------- ? ? ? ? ? input-dir > > [root@hadoop20 mapred]# rm input-dir > > rm: cannot lstat `input-dir': No such file or directory > > [root@hadoop20 mapred]# touch * > > [root@hadoop20 mapred]# ls > > input-dir input-dir > > [root@hadoop20 mapred]# rm * > > rm: remove regular empty file `input-dir'? y > > rm: cannot lstat `input-dir': No such file or directory > > [root@hadoop20 mapred]# ls -la > > total 0 > > drwxr-xr-x 3 mapred mapred 22 Jun 6 12:45 . > > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 .. > > ?--------- ? ? ? ? ? input-dir > > [root@hadoop20 mapred]# > > > > -- > Marcos Luís Ortíz Valmaseda > Software Engineer (UCI) > http://marcosluis2186.posterous.com > http://twitter.com/marcosluis2186 > > > -- Todd Lipcon Software Engineer, Cloudera
