[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk failed during different HDFS operation

2011-09-26 Thread Suresh Srinivas (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2359:
--

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> NPE found in Datanode log while Disk failed during different HDFS operation
> ---
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: data-node
>Affects Versions: 0.20.205.0
>Reporter: Rajit
>Assignee: Jonathan Eagles
> Fix For: 0.20.205.0
>
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3" 
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one 
> Datanode, by making Data Directory permission 000, The distcp job is 
> successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp  /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at 
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:, 
> storageID=xx--xx.xxx.xxx.xxx--xxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk failed during different HDFS operation

2011-09-26 Thread Suresh Srinivas (Updated) (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas updated HDFS-2359:
--

Fix Version/s: 0.20.205.0
 Hadoop Flags: Reviewed

> NPE found in Datanode log while Disk failed during different HDFS operation
> ---
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: data-node
>Affects Versions: 0.20.205.0
>Reporter: Rajit
>Assignee: Jonathan Eagles
> Fix For: 0.20.205.0
>
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3" 
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one 
> Datanode, by making Data Directory permission 000, The distcp job is 
> successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp  /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at 
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:, 
> storageID=xx--xx.xxx.xxx.xxx--xxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk failed during different HDFS operation

2011-09-26 Thread Jonathan Eagles (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated HDFS-2359:
--

Status: Patch Available  (was: Open)

> NPE found in Datanode log while Disk failed during different HDFS operation
> ---
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: data-node
>Affects Versions: 0.20.205.0
>Reporter: Rajit
>Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3" 
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one 
> Datanode, by making Data Directory permission 000, The distcp job is 
> successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp  /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at 
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:, 
> storageID=xx--xx.xxx.xxx.xxx--xxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk failed during different HDFS operation

2011-09-26 Thread Jonathan Eagles (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated HDFS-2359:
--

Attachment: HDFS-2359-branch-0.20-security.patch

> NPE found in Datanode log while Disk failed during different HDFS operation
> ---
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: data-node
>Affects Versions: 0.20.205.0
>Reporter: Rajit
>Assignee: Jonathan Eagles
> Attachments: HDFS-2359-branch-0.20-security.patch
>
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3" 
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one 
> Datanode, by making Data Directory permission 000, The distcp job is 
> successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp  /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at 
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:, 
> storageID=xx--xx.xxx.xxx.xxx--xxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk failed during different HDFS operation

2011-09-23 Thread Rajit (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajit updated HDFS-2359:


Affects Version/s: 0.20.205.0

Suresh, thanks for correcting. I am seeing this in .20.205 unreleased version

> NPE found in Datanode log while Disk failed during different HDFS operation
> ---
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: data-node
>Affects Versions: 0.20.205.0
>Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3" 
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one 
> Datanode, by making Data Directory permission 000, The distcp job is 
> successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp  /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at 
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:, 
> storageID=xx--xx.xxx.xxx.xxx--xxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Updated] (HDFS-2359) NPE found in Datanode log while Disk failed during different HDFS operation

2011-09-22 Thread Rajit (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajit updated HDFS-2359:


Summary: NPE found in Datanode log while Disk failed during different HDFS 
operation  (was: NPE found in Namenode log while Disk failed during distcp )

Thanks Aaron, its Datanode , my bad

> NPE found in Datanode log while Disk failed during different HDFS operation
> ---
>
> Key: HDFS-2359
> URL: https://issues.apache.org/jira/browse/HDFS-2359
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: data-node
>Reporter: Rajit
>
> Scenario:
> I have a cluster of 4 DN ,each of them have 12disks.
> In hdfs-site.xml I have "dfs.datanode.failed.volumes.tolerated=3" 
> During the execution of distcp (hdfs->hdfs), I am failing 3 disks in one 
> Datanode, by making Data Directory permission 000, The distcp job is 
> successful but , I am getting some NullPointerException in Datanode log
> In one thread
> $hadoop distcp  /user/$HADOOPQA_USER/data1 /user/$HADOOPQA_USER/data3
> In another thread in a datanode
> $ chmod 000 /xyz/{0,1,2}/hadoop/var/hdfs/data
> where [ dfs.data.dir is set as /xyz/{0..11}/hadoop/var/hdfs/data ]
> Log Snippet from the Datanode
> =
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7065198814142552283_62557. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7066946313092770579_39189. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7070305189404753930_49359. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:40,327 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at 
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:1820)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1074)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.processCommand(DataNode.java:1036)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:891)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1419)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:41,304 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(xx.xxx.xxx.xxx:, 
> storageID=xx--xx.xxx.xxx.xxx--xxx, infoPort=1006,
> ipcPort=8020):DataXceiver
> java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner$LogFileHandler.appendLine(DataBlockScanner.java:788)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.updateScanStatusInternal(DataBlockScanner.java:365)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifiedByClient(DataBlockScanner.java:308)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:205)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
> at java.lang.Thread.run(Thread.java:619)
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7071818644980664768_40827. BlockInfo not found in volumeMap.
> 2011-09-19 12:43:43,313 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Unexpected error trying to delete block
> blk_7073840977856837621_62108. BlockInfo not found in volumeMap.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira