[jira] [Commented] (HDFS-4229) HDFS can report negative DFS Used on clusters with very small amounts of data

2012-12-15 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13533098#comment-13533098
 ] 

Hadoop QA commented on HDFS-4229:
-

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12561135/HDFS-4229-0.patch.txt
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/3670//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3670//console

This message is automatically generated.

 HDFS can report negative DFS Used on clusters with very small amounts of data
 -

 Key: HDFS-4229
 URL: https://issues.apache.org/jira/browse/HDFS-4229
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.0.2-alpha
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
 Attachments: HDFS-4229-0.patch.txt


 I had a near empty HDFS instance where I was creating a file and deleting it 
 very quickly. I noticed that HDFS sometimes reported a negative DFS used.
 {noformat}
 root@brock0-1 ~]# sudo -u hdfs -i hdfs dfsadmin -report
 Configured Capacity: 97233235968 (90.56 GB)
 Present Capacity: 84289609707 (78.5 GB)
 DFS Remaining: 84426645504 (78.63 GB)
 DFS Used: -137035797 (-133824.02 KB)
 DFS Used%: -0.16%
 Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
 -
 Datanodes available: 1 (1 total, 0 dead)
 Live datanodes:
 Name: 127.0.0.1:50010 (localhost)
 Hostname: brock0-1.ent.cloudera.com
 Decommission Status : Normal
 Configured Capacity: 97233235968 (90.56 GB)
 DFS Used: -137035797 (-133824.02 KB)
 Non DFS Used: 12943626261 (12.05 GB)
 DFS Remaining: 84426645504 (78.63 GB)
 DFS Used%: -0.14%
 DFS Remaining%: 86.83%
 Last contact: Thu Nov 22 18:25:37 PST 2012
 [root@brock0-1 ~]# sudo -u hdfs -i hdfs dfsadmin -report
 Configured Capacity: 97233235968 (90.56 GB)
 Present Capacity: 84426973184 (78.63 GB)
 DFS Remaining: 84426629120 (78.63 GB)
 DFS Used: 344064 (336 KB)
 DFS Used%: 0%
 Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
 -
 Datanodes available: 1 (1 total, 0 dead)
 Live datanodes:
 Name: 127.0.0.1:50010 (localhost)
 Hostname: brock0-1.ent.cloudera.com
 Decommission Status : Normal
 Configured Capacity: 97233235968 (90.56 GB)
 DFS Used: 344064 (336 KB)
 Non DFS Used: 12806262784 (11.93 GB)
 DFS Remaining: 84426629120 (78.63 GB)
 DFS Used%: 0%
 DFS Remaining%: 86.83%
 Last contact: Thu Nov 22 18:28:47 PST 2012
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4229) HDFS can report negative DFS Used on clusters with very small amounts of data

2012-11-26 Thread Brock Noland (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504201#comment-13504201
 ] 

Brock Noland commented on HDFS-4229:


Looks like there might be a race in FsDatasetAsyncDiskService - 
ReplicaFileDeleteTask.run

Scenario:

1) blockFile.delete()
2) DU thread runs setting disk usage number which (deleted block not included)
3) volume.decDfsUsed(...) decrements the block from the used amount

 HDFS can report negative DFS Used on clusters with very small amounts of data
 -

 Key: HDFS-4229
 URL: https://issues.apache.org/jira/browse/HDFS-4229
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 2.0.2-alpha
Reporter: Brock Noland
Priority: Minor

 I had a near empty HDFS instance where I was creating a file and deleting it 
 very quickly. I noticed that HDFS sometimes reported a negative DFS used.
 {noformat}
 root@brock0-1 ~]# sudo -u hdfs -i hdfs dfsadmin -report
 Configured Capacity: 97233235968 (90.56 GB)
 Present Capacity: 84289609707 (78.5 GB)
 DFS Remaining: 84426645504 (78.63 GB)
 DFS Used: -137035797 (-133824.02 KB)
 DFS Used%: -0.16%
 Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
 -
 Datanodes available: 1 (1 total, 0 dead)
 Live datanodes:
 Name: 127.0.0.1:50010 (localhost)
 Hostname: brock0-1.ent.cloudera.com
 Decommission Status : Normal
 Configured Capacity: 97233235968 (90.56 GB)
 DFS Used: -137035797 (-133824.02 KB)
 Non DFS Used: 12943626261 (12.05 GB)
 DFS Remaining: 84426645504 (78.63 GB)
 DFS Used%: -0.14%
 DFS Remaining%: 86.83%
 Last contact: Thu Nov 22 18:25:37 PST 2012
 [root@brock0-1 ~]# sudo -u hdfs -i hdfs dfsadmin -report
 Configured Capacity: 97233235968 (90.56 GB)
 Present Capacity: 84426973184 (78.63 GB)
 DFS Remaining: 84426629120 (78.63 GB)
 DFS Used: 344064 (336 KB)
 DFS Used%: 0%
 Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
 -
 Datanodes available: 1 (1 total, 0 dead)
 Live datanodes:
 Name: 127.0.0.1:50010 (localhost)
 Hostname: brock0-1.ent.cloudera.com
 Decommission Status : Normal
 Configured Capacity: 97233235968 (90.56 GB)
 DFS Used: 344064 (336 KB)
 Non DFS Used: 12806262784 (11.93 GB)
 DFS Remaining: 84426629120 (78.63 GB)
 DFS Used%: 0%
 DFS Remaining%: 86.83%
 Last contact: Thu Nov 22 18:28:47 PST 2012
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4229) HDFS can report negative DFS Used on clusters with very small amounts of data

2012-11-26 Thread Brock Noland (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504205#comment-13504205
 ] 

Brock Noland commented on HDFS-4229:


From a user perspective, I am not expecting a negative number and as such, a 
simple solution such as Math.max(used, 0) would be acceptable.

 HDFS can report negative DFS Used on clusters with very small amounts of data
 -

 Key: HDFS-4229
 URL: https://issues.apache.org/jira/browse/HDFS-4229
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 2.0.2-alpha
Reporter: Brock Noland
Priority: Minor

 I had a near empty HDFS instance where I was creating a file and deleting it 
 very quickly. I noticed that HDFS sometimes reported a negative DFS used.
 {noformat}
 root@brock0-1 ~]# sudo -u hdfs -i hdfs dfsadmin -report
 Configured Capacity: 97233235968 (90.56 GB)
 Present Capacity: 84289609707 (78.5 GB)
 DFS Remaining: 84426645504 (78.63 GB)
 DFS Used: -137035797 (-133824.02 KB)
 DFS Used%: -0.16%
 Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
 -
 Datanodes available: 1 (1 total, 0 dead)
 Live datanodes:
 Name: 127.0.0.1:50010 (localhost)
 Hostname: brock0-1.ent.cloudera.com
 Decommission Status : Normal
 Configured Capacity: 97233235968 (90.56 GB)
 DFS Used: -137035797 (-133824.02 KB)
 Non DFS Used: 12943626261 (12.05 GB)
 DFS Remaining: 84426645504 (78.63 GB)
 DFS Used%: -0.14%
 DFS Remaining%: 86.83%
 Last contact: Thu Nov 22 18:25:37 PST 2012
 [root@brock0-1 ~]# sudo -u hdfs -i hdfs dfsadmin -report
 Configured Capacity: 97233235968 (90.56 GB)
 Present Capacity: 84426973184 (78.63 GB)
 DFS Remaining: 84426629120 (78.63 GB)
 DFS Used: 344064 (336 KB)
 DFS Used%: 0%
 Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
 -
 Datanodes available: 1 (1 total, 0 dead)
 Live datanodes:
 Name: 127.0.0.1:50010 (localhost)
 Hostname: brock0-1.ent.cloudera.com
 Decommission Status : Normal
 Configured Capacity: 97233235968 (90.56 GB)
 DFS Used: 344064 (336 KB)
 Non DFS Used: 12806262784 (11.93 GB)
 DFS Remaining: 84426629120 (78.63 GB)
 DFS Used%: 0%
 DFS Remaining%: 86.83%
 Last contact: Thu Nov 22 18:28:47 PST 2012
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira