Re: Hodoop namenode startup problem
I've fixed the problem. The reason namenode won't start was that I accidentally started the cluster with root account. This somehow changed the ownership of some hadoop-related files(ex: log files, and hadoop.tmp.dir/dfs/name/current/edits) from hadoop:hadoop to root:root. After I fixed the ownership issue, everything went fine. Thanks for the concern. 2010/8/5 Harsh J qwertyman...@gmail.com Could you check the NameNode/SecondaryNameNode logs and try to find the exact issue? Post the errors (if) it contains here, so we can try to help you better. On Thu, Aug 5, 2010 at 9:39 AM, edward choi mp2...@gmail.com wrote: I am currently stuck with hadoop namenode that won't start. When I type start-all.sh, everything prints out fine. But when I type jps, only JobTracker is activated. When this happens, I usually format the namenode. But the problem is that there are 500gigs of date in HDFS. So I really want to save the data. Can anyone give help please? -- Harsh J www.harshj.com
hdfs space problem.
I run a 512 node hadoop cluster. Yesterday I moved 30Gb of compressed data from a NFS mounted partition by running on the namenode hadoop fs -copyFromLocal /mnt/data/data1 /mnt/data/data2 mnt/data/data3 hdfs:/data When the job completed the local disk on the namenode was 40% full ( Most of it used by the dfs dierctories) while the others had 1% disk utilization. Just to see if there was an issue, I deleted the hdfs:/data directory and restarted the move from a datanode. Once again the disk space on that data node was substantially over utilized. I would have assumed that the disk space would be more or less uniformly consumed on all the data nodes. Is there a reason why one disk would be over utilized? Do I have to run balancer everytime I copy data? Am I missing something? Raj
centralized record reader in new API
Hi all, to create a RecordReader in new API, we needs a TaskAttemptContext object, which seems to me the RecordReader should only be created on each split that has been assigned a task ID. However, I want to do a centralized sampling and create record reader on some splits before the job is submitted. What I am doing is create a dummy TaskAttemptContext and use it to create record reader, but not sure whether there is some side-effects. Is there any better way to do this? Why we are not supposed to create record reader centrally as indicated by the new API? Thanks, -Gang
RE: hdfs space problem.
when you copy files and have a local datanode - first copy will end up there. Just stop datanode at the node from which you copy files, and they will end up on random nodes. Also don't run datanode at the same machine as namenode. -Original Message- From: Raj V [mailto:rajv...@yahoo.com] Sent: Thursday, August 05, 2010 8:33 AM To: common-user@hadoop.apache.org Subject: hdfs space problem. I run a 512 node hadoop cluster. Yesterday I moved 30Gb of compressed data from a NFS mounted partition by running on the namenode hadoop fs -copyFromLocal /mnt/data/data1 /mnt/data/data2 mnt/data/data3 hdfs:/data When the job completed the local disk on the namenode was 40% full ( Most of it used by the dfs dierctories) while the others had 1% disk utilization. Just to see if there was an issue, I deleted the hdfs:/data directory and restarted the move from a datanode. Once again the disk space on that data node was substantially over utilized. I would have assumed that the disk space would be more or less uniformly consumed on all the data nodes. Is there a reason why one disk would be over utilized? Do I have to run balancer everytime I copy data? Am I missing something? Raj
Problems running hadoop on Amazon Elastic MapReduce
I am a complete newbie to hadoop. I'm running a job on 19 Amazon Elastic MapReduce servers and am trying to understand two separate issues. 1) The job is ending with an error ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution, encountered a Hadoop error. I do not have the hadoop log files and will have to rerun the job with different settings to obtain them. Once I have them, I'll add them to the posting. 2) The job seems to complete %19 within 1 minute and then takes 15 minutes to complete another %20. It then takes over 40 minutes to complete the last %60. Why the sudden slow down? Am I misunderstanding the messages? The following is the ouput from the job: 10/08/05 16:39:30 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 2010-08-05 16:39:30,688 [main] INFO org.apache.pig.Main - Logging error messages to: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log 2010-08-05 16:39:31,663 [main] INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening '/scripts/calculateTrackingErrors.pig' for reading 2010-08-05 16:39:32,071 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://domU-12-31-39-09-F1-D2.compute-1.internal:9000 2010-08-05 16:39:32,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: domU-12-31-39-09-F1-D2.compute-1.internal:9001 2010-08-05 16:39:33,360 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for covMatrix 2010-08-05 16:39:33,360 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for covMatrix 2010-08-05 16:39:33,362 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for fundExposures2 2010-08-05 16:39:33,362 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for fundExposures2 2010-08-05 16:39:33,363 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for fundExposures1 2010-08-05 16:39:33,363 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for fundExposures1 2010-08-05 16:39:33,469 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 1 time(s). 2010-08-05 16:39:33,484 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage-POForEach to POJoinPackage 2010-08-05 16:39:33,485 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage-POForEach to POJoinPackage 2010-08-05 16:39:33,486 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage-POForEach to POJoinPackage 2010-08-05 16:39:33,495 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5 2010-08-05 16:39:33,496 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - Merged 0 out of total 3 MR operators. 2010-08-05 16:39:33,496 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5 2010-08-05 16:39:33,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode. 2010-08-05 16:39:33,502 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode. 2010-08-05 16:39:34,465 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-05 16:39:34,559 [Thread-12] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-05 16:39:35,585 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Cannot get jobid for this job 2010-08-05 16:39:42,847 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2010-08-05 16:39:55,415 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 5% complete 2010-08-05 16:39:56,464 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 10% complete 2010-08-05 16:40:08,548 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 11% complete 2010-08-05 16:40:11,561
RE: Problems running hadoop on Amazon Elastic MapReduce
Hi, EMR has a live debug option in the panel, you will find the logs there as well. Ankit -Original Message- From: grabbler [mailto:twiza...@gmail.com] Sent: Thursday, August 05, 2010 2:37 PM To: core-u...@hadoop.apache.org Subject: Problems running hadoop on Amazon Elastic MapReduce I am a complete newbie to hadoop. I'm running a job on 19 Amazon Elastic MapReduce servers and am trying to understand two separate issues. 1) The job is ending with an error ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 6015: During execution, encountered a Hadoop error. I do not have the hadoop log files and will have to rerun the job with different settings to obtain them. Once I have them, I'll add them to the posting. 2) The job seems to complete %19 within 1 minute and then takes 15 minutes to complete another %20. It then takes over 40 minutes to complete the last %60. Why the sudden slow down? Am I misunderstanding the messages? The following is the ouput from the job: 10/08/05 16:39:30 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively 2010-08-05 16:39:30,688 [main] INFO org.apache.pig.Main - Logging error messages to: /mnt/var/lib/hadoop/steps/2/pig_1281026370670.log 2010-08-05 16:39:31,663 [main] INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening '/scripts/calculateTrackingErrors.pig' for reading 2010-08-05 16:39:32,071 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://domU-12-31-39-09-F1-D2.compute-1.internal:9000 2010-08-05 16:39:32,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: domU-12-31-39-09-F1-D2.compute-1.internal:9001 2010-08-05 16:39:33,360 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for covMatrix 2010-08-05 16:39:33,360 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for covMatrix 2010-08-05 16:39:33,362 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for fundExposures2 2010-08-05 16:39:33,362 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for fundExposures2 2010-08-05 16:39:33,363 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for fundExposures1 2010-08-05 16:39:33,363 [main] INFO org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for fundExposures1 2010-08-05 16:39:33,469 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 1 time(s). 2010-08-05 16:39:33,484 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage-POForEach to POJoinPackage 2010-08-05 16:39:33,485 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage-POForEach to POJoinPackage 2010-08-05 16:39:33,486 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage-POForEach to POJoinPackage 2010-08-05 16:39:33,495 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5 2010-08-05 16:39:33,496 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - Merged 0 out of total 3 MR operators. 2010-08-05 16:39:33,496 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5 2010-08-05 16:39:33,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode. 2010-08-05 16:39:33,502 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.AccumulatorOptimizer - Reducer is to run in accumulative mode. 2010-08-05 16:39:34,465 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job 2010-08-05 16:39:34,559 [Thread-12] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 2010-08-05 16:39:35,585 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Cannot get jobid for this job 2010-08-05 16:39:42,847 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2010-08-05 16:39:55,415 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 5%
Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1
We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your thoughts are regarding which one would be best for our Hadoop distribution and OS (Ubuntu 8.04 64-bit). In particular, hadoop-gpl-compression (http://code.google.com/p/hadoop-gpl-compression) vs. hadoop-lzo (http://github.com/kevinweil/hadoop-lzo). Some of what appear to be the better instructions/guides out there: * Josh Patterson's reply on June 25th to the Newbie to HDFS compression thread -- http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3caanlktileo-q8useip8y3na9pdyhlyufippr0in0lk...@mail.gmail.com%3e * hadoop-gpl-compression FAQ -- http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ * Hadoop at Twitter (part 1): Splittable LZO Compression blog post -- http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Thanks in advance, -Bobby
Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1
Please take questions on Cloudera Distro to their internal lists. On Aug 5, 2010, at 3:52 PM, Bobby Dennett wrote: We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your thoughts are regarding which one would be best for our Hadoop distribution and OS (Ubuntu 8.04 64-bit). In particular, hadoop-gpl-compression (http://code.google.com/p/hadoop-gpl-compression) vs. hadoop-lzo (http://github.com/kevinweil/hadoop-lzo). Some of what appear to be the better instructions/guides out there: * Josh Patterson's reply on June 25th to the Newbie to HDFS compression thread -- http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3caanlktileo-q8useip8y3na9pdyhlyufippr0in0lk...@mail.gmail.com%3e * hadoop-gpl-compression FAQ -- http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ * Hadoop at Twitter (part 1): Splittable LZO Compression blog post -- http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Thanks in advance, -Bobby
Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1
Bobby, We're working hard to make compression easier, the biggest hurdle currently is the licensing issues around the LZO codec libs (GPL, which is not compatible with ASF bsd-style license). Outside of making the changes to the mapred-site.xml file, with your setup would do you view as the biggest pain point? Josh Patterson Cloudera On Thu, Aug 5, 2010 at 6:52 PM, Bobby Dennett bdennett+softw...@gmail.com wrote: We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your thoughts are regarding which one would be best for our Hadoop distribution and OS (Ubuntu 8.04 64-bit). In particular, hadoop-gpl-compression (http://code.google.com/p/hadoop-gpl-compression) vs. hadoop-lzo (http://github.com/kevinweil/hadoop-lzo). Some of what appear to be the better instructions/guides out there: * Josh Patterson's reply on June 25th to the Newbie to HDFS compression thread -- http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3caanlktileo-q8useip8y3na9pdyhlyufippr0in0lk...@mail.gmail.com%3e * hadoop-gpl-compression FAQ -- http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ * Hadoop at Twitter (part 1): Splittable LZO Compression blog post -- http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Thanks in advance, -Bobby
Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1
Hi Josh, No real pain points... just trying to investigate/research the best way to create the necessary libraries and jar files to support LZO compression in Hadoop. In particular, there are the 2 repositories to build from and I am trying to find out if one should be used over the other. For instance, in your previous posting, you refer to hadoop-gpl-compression while the Twitter blog post from last year mentions the Hadoop-LZO project. Briefly looking, it seems Hadoop-LZO is preferable but we're curious if there are any caveats/gotchas we should be aware of. Thanks, -Bobby On Thu, Aug 5, 2010 at 15:59, Josh Patterson j...@cloudera.com wrote: Bobby, We're working hard to make compression easier, the biggest hurdle currently is the licensing issues around the LZO codec libs (GPL, which is not compatible with ASF bsd-style license). Outside of making the changes to the mapred-site.xml file, with your setup would do you view as the biggest pain point? Josh Patterson Cloudera On Thu, Aug 5, 2010 at 6:52 PM, Bobby Dennett bdennett+softw...@gmail.com wrote: We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your thoughts are regarding which one would be best for our Hadoop distribution and OS (Ubuntu 8.04 64-bit). In particular, hadoop-gpl-compression (http://code.google.com/p/hadoop-gpl-compression) vs. hadoop-lzo (http://github.com/kevinweil/hadoop-lzo). Some of what appear to be the better instructions/guides out there: * Josh Patterson's reply on June 25th to the Newbie to HDFS compression thread -- http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3caanlktileo-q8useip8y3na9pdyhlyufippr0in0lk...@mail.gmail.com%3e * hadoop-gpl-compression FAQ -- http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ * Hadoop at Twitter (part 1): Splittable LZO Compression blog post -- http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Thanks in advance, -Bobby
Re: Enabling LZO compression of map outputs in Cloudera Hadoop 0.20.1
On Thu, Aug 5, 2010 at 4:52 PM, Bobby Dennett bdenn...@gmail.com wrote: Hi Josh, No real pain points... just trying to investigate/research the best way to create the necessary libraries and jar files to support LZO compression in Hadoop. In particular, there are the 2 repositories to build from and I am trying to find out if one should be used over the other. For instance, in your previous posting, you refer to hadoop-gpl-compression while the Twitter blog post from last year mentions the Hadoop-LZO project. Briefly looking, it seems Hadoop-LZO is preferable but we're curious if there are any caveats/gotchas we should be aware of. Yes, definitely use the hadoop-lzo project from github -- either from my repo or from kevinweil's (the two are kept in sync) The repo on Google Code has a number of known bugs, which is why we forked it over to github last year. -Todd On Thu, Aug 5, 2010 at 15:59, Josh Patterson j...@cloudera.com wrote: Bobby, We're working hard to make compression easier, the biggest hurdle currently is the licensing issues around the LZO codec libs (GPL, which is not compatible with ASF bsd-style license). Outside of making the changes to the mapred-site.xml file, with your setup would do you view as the biggest pain point? Josh Patterson Cloudera On Thu, Aug 5, 2010 at 6:52 PM, Bobby Dennett bdennett+softw...@gmail.com bdennett%2bsoftw...@gmail.com wrote: We are looking to enable LZO compression of the map outputs on our Cloudera 0.20.1 cluster. It seems there are various sets of instructions available and I am curious what your thoughts are regarding which one would be best for our Hadoop distribution and OS (Ubuntu 8.04 64-bit). In particular, hadoop-gpl-compression (http://code.google.com/p/hadoop-gpl-compression) vs. hadoop-lzo (http://github.com/kevinweil/hadoop-lzo). Some of what appear to be the better instructions/guides out there: * Josh Patterson's reply on June 25th to the Newbie to HDFS compression thread -- http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201006.mbox/%3caanlktileo-q8useip8y3na9pdyhlyufippr0in0lk...@mail.gmail.com%3e * hadoop-gpl-compression FAQ -- http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ * Hadoop at Twitter (part 1): Splittable LZO Compression blog post -- http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Thanks in advance, -Bobby -- Todd Lipcon Software Engineer, Cloudera
Best way to reduce a 8-node cluster in half and get hdfs to come out of safe mode
As part of our experimentation, the plan is to pull 4 slave nodes out of a 8-slave/1-master cluster. With replication factor set to 3, I thought losing half of the cluster may be too much for hdfs to recover. Thus I copied out all relevant data from hdfs to local disk and reconfigure the cluster. The 4 slave nodes started okay but hdfs never left safe mode. The nn.log has the following line. What is the best way to deal with this? Shall I restart the cluster with 8-node and then delete /data/hadoop-hadoop/mapred/system? Or shall I reformat hdfs? Thanks. 2010-08-05 22:28:12,921 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.128.135.100 cmd=listStatus src=/data/hadoop-hadoop/mapred/system dst=nullperm=null 2010-08-05 22:28:12,923 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9000, call delete(/data/hadoop-hadoop/mapred/system, true) from 10.128.135.100:52368: error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /data/hadoop-hadoop/mapred/system. Name node is in safe mode. The reported blocks 64 needs additional 3 blocks to reach the threshold 0.9990 of total blocks 68. Safe mode will be turned off automatically. org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /data/hadoop-hadoop/mapred/system. Name node is in safe mode. The reported blocks 64 needs additional 3 blocks to reach the threshold 0.9990 of total blocks 68. Safe mode will be turned off automatically. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1741) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1721) at org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:565) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:968) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:962)
