Re: mismatch corrupt blocks from fsck and dfadmin report.

[Mungeol Heo]

In my case, the "Blocks with corrupt replicas", which from "hdfs dfsadmin
-report" command, became 0 after a while.

On Tue, Dec 27, 2016 at 4:15 PM, Mungeol Heo  wrote:

> +1
>
> On Tue, Jan 12, 2016 at 7:20 PM, 권병창  wrote:
>
>> Hi. hadooper!
>>
>>
>>
>> I use hadoop-2.7.1 and  my cluster has 130 nodes.
>>
>>
>>
>> recently I am facing a problem.
>>
>>
>>
>> I have found corrupt block by nagios.
>>
>>
>>
>> nagios request http://namenode01:50070/jmx?qry=Hadoop:service=NameN
>> ode,name=FSNamesystem.
>>
>>
>>
>> below is result.  notice CorruptBlocks is 1.
>>
>>  {
>>
>>   "beans" : [ {
>> "name" : "Hadoop:service=NameNode,name=FSNamesystem",
>> "modelerType" : "FSNamesystem",
>> "tag.Context" : "dfs",
>> "tag.HAState" : "active",
>> "tag.Hostname" : "css0700.nhnsystem.com",
>> "MissingBlocks" : 0,
>> "MissingReplOneBlocks" : 0,
>> "ExpiredHeartbeats" : 10,
>> "TransactionsSinceLastCheckpoint" : 820630,
>> "TransactionsSinceLastLogRoll" : 1916,
>> "LastWrittenTransactionId" : 376685578,
>> "LastCheckpointTime" : 1452583950883,
>> "CapacityTotal" : 1650893130075660,
>> "CapacityTotalGB" : 1537514.0,
>> "CapacityUsed" : 1237079990848257,
>> "CapacityUsedGB" : 1152121.0,
>> "CapacityRemaining" : 410189981364473,
>> "CapacityRemainingGB" : 382019.0,
>> "CapacityUsedNonDFS" : 3623157862930,
>> "TotalLoad" : 6717,
>> "SnapshottableDirectories" : 0,
>> "Snapshots" : 0,
>> "BlocksTotal" : 4034155,
>> "FilesTotal" : 2866690,
>> "PendingReplicationBlocks" : 0,
>> "UnderReplicatedBlocks" : 0,
>> "CorruptBlocks" : 1,
>> "ScheduledReplicationBlocks" : 0,
>> "PendingDeletionBlocks" : 0,
>> "ExcessBlocks" : 87,
>> "PostponedMisreplicatedBlocks" : 0,
>> "PendingDataNodeMessageCount" : 0,
>> "MillisSinceLastLoadedEdits" : 0,
>> "BlockCapacity" : 67108864,
>> "StaleDataNodes" : 0,
>> "TotalFiles" : 2866690
>>   } ]
>> }
>>
>>
>>
>>
>>
>> however  'hdfs fsck / -list-corruptfileblocks' does not found.
>>
>>
>>
>> below is fsck result.
>>
>>  The filesystem under path '/' has 0 CORRUPT files
>>
>>
>>
>> 'hdfs dfsadmin -report'  is similar first result.
>>
>>  Configured Capacity: 1650892318461452 (1.47 PB)
>>
>> Present Capacity: 1647353181258422 (1.46 PB)
>>
>> DFS Remaining: 408711410865856 (371.72 TB)
>>
>> DFS Used: 1238641770392566 (1.10 PB)
>>
>> DFS Used%: 75.19%
>>
>> Under replicated blocks: 0
>>
>> Blocks with corrupt replicas: 1
>>
>> Missing blocks: 0
>>
>> Missing blocks (with replication factor 1): 0
>>
>>
>>
>> My question is
>>
>>
>>
>> 1. why different result?
>>
>> 2. How do I find corrupt filename? I wonder which file is corrupt.
>>
>>
>>
>> Thank you.
>>
>>
>>
>
>

Re: mismatch corrupt blocks from fsck and dfadmin report.

[Mungeol Heo]

+1

On Tue, Jan 12, 2016 at 7:20 PM, 권병창  wrote:

> Hi. hadooper!
>
>
>
> I use hadoop-2.7.1 and  my cluster has 130 nodes.
>
>
>
> recently I am facing a problem.
>
>
>
> I have found corrupt block by nagios.
>
>
>
> nagios request http://namenode01:50070/jmx?qry=Hadoop:service=
> NameNode,name=FSNamesystem.
>
>
>
> below is result.  notice CorruptBlocks is 1.
>
>  {
>
>   "beans" : [ {
> "name" : "Hadoop:service=NameNode,name=FSNamesystem",
> "modelerType" : "FSNamesystem",
> "tag.Context" : "dfs",
> "tag.HAState" : "active",
> "tag.Hostname" : "css0700.nhnsystem.com",
> "MissingBlocks" : 0,
> "MissingReplOneBlocks" : 0,
> "ExpiredHeartbeats" : 10,
> "TransactionsSinceLastCheckpoint" : 820630,
> "TransactionsSinceLastLogRoll" : 1916,
> "LastWrittenTransactionId" : 376685578,
> "LastCheckpointTime" : 1452583950883,
> "CapacityTotal" : 1650893130075660,
> "CapacityTotalGB" : 1537514.0,
> "CapacityUsed" : 1237079990848257,
> "CapacityUsedGB" : 1152121.0,
> "CapacityRemaining" : 410189981364473,
> "CapacityRemainingGB" : 382019.0,
> "CapacityUsedNonDFS" : 3623157862930,
> "TotalLoad" : 6717,
> "SnapshottableDirectories" : 0,
> "Snapshots" : 0,
> "BlocksTotal" : 4034155,
> "FilesTotal" : 2866690,
> "PendingReplicationBlocks" : 0,
> "UnderReplicatedBlocks" : 0,
> "CorruptBlocks" : 1,
> "ScheduledReplicationBlocks" : 0,
> "PendingDeletionBlocks" : 0,
> "ExcessBlocks" : 87,
> "PostponedMisreplicatedBlocks" : 0,
> "PendingDataNodeMessageCount" : 0,
> "MillisSinceLastLoadedEdits" : 0,
> "BlockCapacity" : 67108864,
> "StaleDataNodes" : 0,
> "TotalFiles" : 2866690
>   } ]
> }
>
>
>
>
>
> however  'hdfs fsck / -list-corruptfileblocks' does not found.
>
>
>
> below is fsck result.
>
>  The filesystem under path '/' has 0 CORRUPT files
>
>
>
> 'hdfs dfsadmin -report'  is similar first result.
>
>  Configured Capacity: 1650892318461452 (1.47 PB)
>
> Present Capacity: 1647353181258422 (1.46 PB)
>
> DFS Remaining: 408711410865856 (371.72 TB)
>
> DFS Used: 1238641770392566 (1.10 PB)
>
> DFS Used%: 75.19%
>
> Under replicated blocks: 0
>
> Blocks with corrupt replicas: 1
>
> Missing blocks: 0
>
> Missing blocks (with replication factor 1): 0
>
>
>
> My question is
>
>
>
> 1. why different result?
>
> 2. How do I find corrupt filename? I wonder which file is corrupt.
>
>
>
> Thank you.
>
>
>

mismatch corrupt blocks from fsck and dfadmin report.

[권병창]

Hi. hadooper!
 
I use hadoop-2.7.1 and  my cluster has 130 nodes.
 
recently I am facing a problem.
 
I have found corrupt block by nagios.
 
nagios request 
http://namenode01:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem. 
 
below is result.  notice CorruptBlocks is 1. 

 {
  "beans" : [ {
"name" : "Hadoop:service=NameNode,name=FSNamesystem",
"modelerType" : "FSNamesystem",
"tag.Context" : "dfs",
"tag.HAState" : "active",
"tag.Hostname" : "css0700.nhnsystem.com",
"MissingBlocks" : 0,
"MissingReplOneBlocks" : 0,
"ExpiredHeartbeats" : 10,
"TransactionsSinceLastCheckpoint" : 820630,
"TransactionsSinceLastLogRoll" : 1916,
"LastWrittenTransactionId" : 376685578,
"LastCheckpointTime" : 1452583950883,
"CapacityTotal" : 1650893130075660,
"CapacityTotalGB" : 1537514.0,
"CapacityUsed" : 1237079990848257,
"CapacityUsedGB" : 1152121.0,
"CapacityRemaining" : 410189981364473,
"CapacityRemainingGB" : 382019.0,
"CapacityUsedNonDFS" : 3623157862930,
"TotalLoad" : 6717,
"SnapshottableDirectories" : 0,
"Snapshots" : 0,
"BlocksTotal" : 4034155,
"FilesTotal" : 2866690,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"CorruptBlocks" : 1,
"ScheduledReplicationBlocks" : 0,
"PendingDeletionBlocks" : 0,
"ExcessBlocks" : 87,
"PostponedMisreplicatedBlocks" : 0,
"PendingDataNodeMessageCount" : 0,
"MillisSinceLastLoadedEdits" : 0,
"BlockCapacity" : 67108864,
"StaleDataNodes" : 0,
"TotalFiles" : 2866690
  } ]
}


 
 
however  'hdfs fsck / -list-corruptfileblocks' does not found.
 
below is fsck result.

 The filesystem under path '/' has 0 CORRUPT files



 
'hdfs dfsadmin -report'  is similar first result.

 Configured Capacity: 1650892318461452 (1.47 PB)
Present Capacity: 1647353181258422 (1.46 PB)
DFS Remaining: 408711410865856 (371.72 TB)
DFS Used: 1238641770392566 (1.10 PB)
DFS Used%: 75.19%
Under replicated blocks: 0
Blocks with corrupt replicas: 1
Missing blocks: 0
Missing blocks (with replication factor 1): 0



 
My question is
 
1. why different result?
2. How do I find corrupt filename? I wonder which file is corrupt.
 
Thank you.