Hi folks , I have received this error in the hadoop cluster. Has anybody anybody seen this . Any solution. --------------------- namenode errors and warning ------------------
2010-06-17 18:35:19,970 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place enough replicas, still in need of 1 2010-06-17 18:35:19,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.13.53.127:50010 to delete blk_-5851901968674962584_28816 blk_-5704762921662804278_28806 blk_-3249260253840047858_28816 blk_-6785644112288085978_28816 blk_-5991019206688403229_28806 blk_3615269337590779570_28806 blk_-1181514666968321558_28806 blk_7264052281086809701_28816 blk_663323733333557946_28781 blk_5673553217372004676_28816 blk_3626732201125971756_28806 blk_8707314634323417208_28816 blk_-6887179921277000590_28812 blk_-3816018501532347017_28816 blk_-7238540159191278524_28816 blk_-7658691789280578071_28806 blk_3236948345360966700_28816 blk_-6254109926970173434_28816 blk_5195430663886472445_28816 blk_-412573738832546671_28816 blk_8109473696289375765_28806 blk_-5810373118380394822_28816 blk_2743979608816557043_28806 blk_5715402730928093013_28816 blk_8464503407273685120_28806 blk_-5062152410152898606_28816 blk_-3177617939821456482_28810 blk_-5555427680789719632_28816 blk_-5789474774332045366_28806 blk_-5623935836874134523_28806 blk_-4337702879554700741_28806 blk_-8461056881907762006_28806 blk_9026822631507898528_28816 blk_7156167055494887627_28816 blk_-8277210689227324100_28806 blk_-3751852098002911697_28806 blk_-5585636802791023762_28806 blk_7895962823848655899_28806 blk_-990310910545048635_28816 blk_-9154860186123815888_28806 blk_-4227488676014658152_28806 blk_8936924022792123139_28816 blk_-769244617229872944_28816 blk_2688806916705051324_28816 blk_-4338943142397506199_28816 blk_-509403461442878177_28806 blk_-4842888174619311915_28815 blk_-3952777283518699846_28806 blk_-5861717742640359549_28806 blk_-3952576171995960772_28816 blk_-163789650674468878_28816 blk_-4592795213928607535_28806 blk_2333271861949132522_28816 blk_4198208275476238889_28816 blk_360552057991500806_28816 blk_5491773302900671409_28806 blk_4047676415653175342_28806 blk_-1004260037826453175_28806 blk_2767440385264091225_28816 blk_5476421277228240141_28816 blk_4407005806101654351_28816 blk_4195254311 970318771_28816 blk_-737647080244712447_28806 blk_7438398943! 33029124 8_28806 blk_9065794226469565702_28806 blk_-909573278974779662_28816 blk_-720057463076992315_28806 blk_-3706189772409350949_28816 blk_-7669121304118161359_28816 blk_5626381233374602629_28816 blk_3226781108133048989_28806 blk_4534712638713702794_28816 blk_-2501367440326715009_28816 blk_1182598706752583066_28806 blk_2899109514732336036_28813 blk_-3583975497823582887_28806 blk_1637573118385058229_28816 blk_6123281255845765914_28816 blk_-7325625322659708014_28816 blk_5881568644646580533_28816 blk_-3117891195870623616_28816 blk_4293852484387807756_28806 blk_1053300130581981980_28781 blk_2196698001825902338_28809 2010-06-17 18:35:19,974 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 10.13.53.128:50010 to delete blk_-5851901968674962584_28816 blk_-5991019206688403229_28806 blk_-3249260253840047858_28816 blk_-1074403248426101769_28816 blk_1102552384917878009_28816 blk_663323733333557946_28781 blk_7264052281086809701_28816 blk_-9186151702356219036_28806 blk_-6887179921277000590_28812 blk_-7403187390102989856_28814 blk_-3816018501532347017_28816 blk_-3256647808151302716_28806 blk_5195430663886472445_28816 blk_-6254109926970173434_28816 blk_-412573738832546671_28816 blk_8109473696289375765_28806 blk_2743979608816557043_28806 blk_8464503407273685120_28806 blk_-1927566815393964066_28813 blk_-910165942916728217_28816 blk_6991381516571090085_28816 blk_-5062152410152898606_28816 blk_-5555427680789719632_28816 blk_-347418184816360801_28806 blk_9026822631507898528_28816 blk_8645717205835711727_28816 blk_-8277210689227324100_28806 blk_-6820670802505184305_28809 blk_7156167055494887627_28816 blk_-3751852098002911697_28806 blk_2230966748060172045_28816 blk_-9154860186123815888_28806 blk_-4227488676014658152_28806 blk_8936924022792123139_28816 blk_-3952777283518699846_28806 blk_-163789650674468878_28816 blk_2333271861949132522_28816 blk_5491773302900671409_28806 blk_4047676415653175342_28806 blk_7147022655053153746_28806 blk_5476421277228240141_28816 blk_3601702425471675469_28806 blk_-1812525521195434130_28809 blk_6208014296194515686_28816 blk_-3588835264704726772_28816 blk_5487381930889488938_28806 blk_1786827488554715485_28806 blk_6381806163859245084_28806 blk_-909573278974779662_28816 blk_-720057463076992315_28806 blk_-3706189772409350949_28816 blk_5626381233374602629_28816 blk_8812361351759351875_28806 blk_3226781108133048989_28806 blk_2626414889098125951_28816 blk_5642048906322764709_28806 blk_-3583975497823582887_28806 blk_1637573118385058229_28816 blk_-7325625322659708014_28816 blk_4293852484387807756_28806 blk_7869940696281550339_28806 blk_2196698001825902 338_28809 blk_-6887326635813252129_28816 blk_-57047629216628! 04278_28 806 blk_3615269337590779570_28806 blk_-6785644112288085978_28816 blk_3880871563914097435_28806 blk_-1181514666968321558_28806 blk_3626732201125971756_28806 blk_-5111860011657356431_28812 blk_5673553217372004676_28816 blk_1144230701070834232_28816 blk_8707314634323417208_28816 blk_-2478324223605041774_28808 blk_-7238540159191278524_28816 blk_-7658691789280578071_28806 blk_3236948345360966700_28816 blk_744702532009597216_28816 blk_-5810373118380394822_28816 blk_2247163771290221731_28808 blk_5715402730928093013_28816 blk_-3177617939821456482_28810 blk_-5789474774332045366_28806 blk_-5623935836874134523_28806 blk_-4682289668534212518_28816 blk_-4337702879554700741_28806 blk_-8461056881907762006_28806 blk_3225744222537772912_28816 blk_-7377461388357398241_28816 blk_-5585636802791023762_28806 blk_7787185831361186700_28816 blk_7895962823848655899_28806 blk_-990310910545048635_28816 blk_-1876758814669931378_28816 blk_-769244617229872944_28816 blk_-497469852942635196_28806 blk_-433894 3142397506199_28816 blk_2688806916705051324_28816 blk_-6654089746586683607_28816 blk_-509403461442878177_28806 2010-06-17 18:35:19,975 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.13.53.127:50010 is added to blk_151014847393206974_28824 size 41837 2010-06-17 18:35:19,978 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_0068/jo b.xml is closed by DFSClient_-1596407549 2010-06-17 18:35:19,986 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.13.53.127 cmd=open src=/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_006 8/job.xml dst=null perm=null 2010-06-17 18:35:20,078 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.13.53.127 cmd=open src=/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_006 8/job.jar dst=null perm=null 2010-06-17 18:35:21,581 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.13.53.127 cmd=create src=/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-mast er_1276071808358_job_201006090123_0068_hadoop_I2ALogArchiver dst=null perm=hadoop:supergroup:rw-r--r-- 2010-06-17 18:35:21,710 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.13.53.127 cmd=create src=/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-mast er_1276071808358_job_201006090123_0068_conf.xml dst=null perm=hadoop:supergroup:rw-r--r-- 2010-06-17 18:35:21,731 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-master_1 276071808358_job_201006090123_0068_conf.xml. blk_1315602836759907786_28826 2010-06-17 18:35:21,748 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.13.53.128:50010 is added to blk_1315602836759907786_28826 size 41823 -- 2010-06-17 18:45:11,591 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: PendingReplicationMonitor timed out block blk_-2915452315391280184_28822 2010-06-17 18:50:03,643 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 676 Total time for transactions(ms): 11Number of transactions batched in Syncs: 4 Number of syncs: 69 SyncTimes(ms): 3632 2010-06-17 18:50:03,645 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.13.53.129 cmd=create src=/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0 dst=null perm=hadoop:supergroup:rw-r--r-- 2010-06-17 18:50:03,659 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0. blk_-5308288487531094520_28832 2010-06-17 18:50:04,363 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.13.53.156:50010 is added to blk_-5308288487531094520_28832 size 31804984 2010-06-17 18:50:04,363 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.13.53.128:50010 is added to blk_-5308288487531094520_28832 size 31804984 2010-06-17 18:50:04,364 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.13.53.129:50010 is added to blk_-5308288487531094520_28832 size 31804984 2010-06-17 18:50:04,366 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0 is closed by DFSClient_-1916945035 2010-06-17 18:50:06,172 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hadoop,hadoop ip=/10.13.53.129 cmd=create src=/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182531-access.log.0 dst=null perm=hadoop:supergroup:rw-r--r-- 2010-06-17 18:50:06,187 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182531-access.log.0. blk_1130168086083501329_28833 2010-06-17 18:50:07,024 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 10.13.53.127:50010 is added to blk_1130168086083501329_28833 size 32278706 --------------------- jobtracker errors and warning ------------------ 2010-06-17 17:00:06,382 WARN org.apache.hadoop.conf.Configuration: /var/lib/hadoop-0.20/cache/hadoop/mapred/local/jobTracker/job_2010060901 23_0067.xml:a attempt to override final parameter: dfs.data.dir; Ignoring. 2010-06-17 17:00:06,448 DEBUG org.apache.hadoop.mapred.JobTracker: Per-Task memory configuration is not set on JT. Not checking the job for invalid memory requirements. 2010-06-17 17:00:06,449 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201006090123_0067 2010-06-17 17:00:06,451 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201006090123_0067 2010-06-17 17:00:06,698 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201006090123_0067 = 8493462348. Number of splits = 288 2010-06-17 17:00:06,698 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0067_m_000000 has split on node:/default-rack/10.13.53.127 2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0067_m_000000 has split on node:/default-rack/10.13.53.156 2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0067_m_000000 has split on node:/default-rack/10.13.53.128 2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0067_m_000001 has split on node:/default-rack/10.13.53.127 2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0067_m_000001 has split on node:/default-rack/10.13.53.129 2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0067_m_000001 has split on node:/default-rack/10.13.53.128 -- 2010-06-17 18:35:20,073 WARN org.apache.hadoop.conf.Configuration: /var/lib/hadoop-0.20/cache/hadoop/mapred/local/jobTracker/job_2010060901 23_0068.xml:a attempt to override final parameter: dfs.data.dir; Ignoring. 2010-06-17 18:35:20,132 DEBUG org.apache.hadoop.mapred.JobTracker: Per-Task memory configuration is not set on JT. Not checking the job for invalid memory requirements. 2010-06-17 18:35:20,133 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201006090123_0068 2010-06-17 18:35:20,134 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201006090123_0068 2010-06-17 18:35:21,123 DEBUG org.apache.hadoop.mapred.JobTracker: Got heartbeat from: tracker_10.13.53.128:localhost.localdomain/127.0.0.1:35313 (restarted: false initialContact: false acceptNewTasks: true) with responseId: -32475 2010-06-17 18:35:21,186 DEBUG org.apache.hadoop.mapred.JobTracker: Got heartbeat from: tracker_10.13.53.127:localhost.localdomain/127.0.0.1:45581 (restarted: false initialContact: false acceptNewTasks: true) with responseId: -32074 2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201006090123_0068 = 33327229. Number of splits = 288 2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0068_m_000000 has split on node:/default-rack/10.13.53.127 2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0068_m_000000 has split on node:/default-rack/10.13.53.156 2010-06-17 18:35:21,777 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0068_m_000000 has split on node:/default-rack/10.13.53.129 2010-06-17 18:35:21,777 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201006090123_0068_m_000001 has split on node:/default-rack/10.13.53.127 --------------------- datanode errors and warning ------------------ Thanks & Regards Bikash Singhal "Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment." www.cybage.com
