[ https://issues.apache.org/jira/browse/HDFS-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Deepak Chander updated HDFS-11191: ---------------------------------- Description: In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system. hdfs@kimtest1:~> hdfs dfsadmin -report Configured Capacity: 239942369274 (223.46 GB) Present Capacity: 207894724602 (193.62 GB) DFS Remaining: 207894552570 (193.62 GB) DFS Used: 172032 (168 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 172.26.79.87:50010 (kimtest3) Hostname: kimtest3 Decommission Status : Normal Configured Capacity: 79980789758 (74.49 GB) DFS Used: 57344 (56 KB) Non DFS Used: 9528000512 (8.87 GB) DFS Remaining: 70452731902 (65.61 GB) DFS Used%: 0.00% DFS Remaining%: 88.09% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 06:59:02 PST 2016 Name: 172.26.80.38:50010 (kimtest4) Hostname: kimtest4 Decommission Status : Normal Configured Capacity: 79980789758 (74.49 GB) DFS Used: 57344 (56 KB) Non DFS Used: 13010952192 (12.12 GB) DFS Remaining: 66969780222 (62.37 GB) DFS Used%: 0.00% DFS Remaining%: 83.73% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 06:59:02 PST 2016 Name: 172.26.79.86:50010 (kimtest2) Hostname: kimtest2 Decommission Status : Normal Configured Capacity: 79980789758 (74.49 GB) DFS Used: 57344 (56 KB) Non DFS Used: 9508691968 (8.86 GB) DFS Remaining: 70472040446 (65.63 GB) DFS Used%: 0.00% DFS Remaining%: 88.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 06:59:02 PST 2016 If you see my datanode root file system size its only 38GB kimtest3:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 38G 2.6G 33G 8% / kimtest4:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 38G 4.2G 32G 12% / kimtest2:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 38G 2.6G 33G 8% / The below is from hdfs-site.xml file <property> <name>dfs.datanode.data.dir</name> <value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value> </property> I have removed the other directory grid1 and restarted datanode process. <property> <name>dfs.datanode.data.dir</name> <value>file:///grid/hadoop/hdfs/dn</value> </property> Now the size is reflecting correctly hdfs@kimtest1:/grid> hdfs dfsadmin -report Configured Capacity: 119971184637 (111.73 GB) Present Capacity: 103947243517 (96.81 GB) DFS Remaining: 103947157501 (96.81 GB) DFS Used: 86016 (84 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 172.26.79.87:50010 (kimtest3) Hostname: kimtest3 Decommission Status : Normal Configured Capacity: 39990394879 (37.24 GB) DFS Used: 28672 (28 KB) Non DFS Used: 4764057600 (4.44 GB) DFS Remaining: 35226308607 (32.81 GB) DFS Used%: 0.00% DFS Remaining%: 88.09% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 07:34:02 PST 2016 Name: 172.26.80.38:50010 (kimtest4) Hostname: kimtest4 Decommission Status : Normal Configured Capacity: 39990394879 (37.24 GB) DFS Used: 28672 (28 KB) Non DFS Used: 6505525248 (6.06 GB) DFS Remaining: 33484840959 (31.19 GB) DFS Used%: 0.00% DFS Remaining%: 83.73% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 07:34:01 PST 2016 Name: 172.26.79.86:50010 (kimtest2) Hostname: kimtest2 Decommission Status : Normal Configured Capacity: 39990394879 (37.24 GB) DFS Used: 28672 (28 KB) Non DFS Used: 4754358272 (4.43 GB) DFS Remaining: 35236007935 (32.82 GB) DFS Used%: 0.00% DFS Remaining%: 88.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 07:34:02 PST 2016 was: In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if the dfs.datanode.data.dir is configured with two directories from the same file system. hdfs@kimtest1:~> hdfs dfsadmin -report Configured Capacity: 239942369274 (223.46 GB) Present Capacity: 207894724602 (193.62 GB) DFS Remaining: 207894552570 (193.62 GB) DFS Used: 172032 (168 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 172.26.79.87:50010 (kimtest3) Hostname: kimtest3 Decommission Status : Normal Configured Capacity: 79980789758 (74.49 GB) DFS Used: 57344 (56 KB) Non DFS Used: 9528000512 (8.87 GB) DFS Remaining: 70452731902 (65.61 GB) DFS Used%: 0.00% DFS Remaining%: 88.09% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 06:59:02 PST 2016 Name: 172.26.80.38:50010 (kimtest4) Hostname: kimtest4 Decommission Status : Normal Configured Capacity: 79980789758 (74.49 GB) DFS Used: 57344 (56 KB) Non DFS Used: 13010952192 (12.12 GB) DFS Remaining: 66969780222 (62.37 GB) DFS Used%: 0.00% DFS Remaining%: 83.73% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 06:59:02 PST 2016 Name: 172.26.79.86:50010 (kimtest2) Hostname: kimtest2 Decommission Status : Normal Configured Capacity: 79980789758 (74.49 GB) DFS Used: 57344 (56 KB) Non DFS Used: 9508691968 (8.86 GB) DFS Remaining: 70472040446 (65.63 GB) DFS Used%: 0.00% DFS Remaining%: 88.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 06:59:02 PST 2016 If you see my datanode root file system size its only 38GB kimtest3:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 38G 2.6G 33G 8% / kimtest4:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 38G 4.2G 32G 12% / kimtest2:~ # df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-root 38G 2.6G 33G 8% / The below is from hdfs-site.xml file <property> <name>dfs.datanode.data.dir</name> <value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value> </property> I have removed the other directory grid1 and restarted datanode process. <property> <name>dfs.datanode.data.dir</name> <value>file:///grid/hadoop/hdfs/dn</value> </property> Now the size is reflecting correctly hdfs@kimtest1:/grid> hdfs dfsadmin -report Configured Capacity: 119971184637 (111.73 GB) Present Capacity: 103947243517 (96.81 GB) DFS Remaining: 103947157501 (96.81 GB) DFS Used: 86016 (84 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 172.26.79.87:50010 (kimtest3) Hostname: kimtest3 Decommission Status : Normal Configured Capacity: 39990394879 (37.24 GB) DFS Used: 28672 (28 KB) Non DFS Used: 4764057600 (4.44 GB) DFS Remaining: 35226308607 (32.81 GB) DFS Used%: 0.00% DFS Remaining%: 88.09% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 07:34:02 PST 2016 Name: 172.26.80.38:50010 (kimtest4) Hostname: kimtest4 Decommission Status : Normal Configured Capacity: 39990394879 (37.24 GB) DFS Used: 28672 (28 KB) Non DFS Used: 6505525248 (6.06 GB) DFS Remaining: 33484840959 (31.19 GB) DFS Used%: 0.00% DFS Remaining%: 83.73% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 07:34:01 PST 2016 Name: 172.26.79.86:50010 (kimtest2) Hostname: kimtest2 Decommission Status : Normal Configured Capacity: 39990394879 (37.24 GB) DFS Used: 28672 (28 KB) Non DFS Used: 4754358272 (4.43 GB) DFS Remaining: 35236007935 (32.82 GB) DFS Used%: 0.00% DFS Remaining%: 88.11% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Nov 29 07:34:02 PST 2016 If the file system is same for the directories listed under dfs.datanode.data.dir, we expect to see the original size of the file system. The code should differentiate between the file systems. > In the command “hdfs dfsadmin -report” The Configured Capacity is misleading > if the dfs.datanode.data.dir is configured with two directories from the same > file system. > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-11191 > URL: https://issues.apache.org/jira/browse/HDFS-11191 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs > Affects Versions: 2.5.0 > Environment: SLES 11SP3 > HDP 2.5.0 > Reporter: Deepak Chander > > In the command “hdfs dfsadmin -report” The Configured Capacity is misleading > if the dfs.datanode.data.dir is configured with two directories from the same > file system. > hdfs@kimtest1:~> hdfs dfsadmin -report > Configured Capacity: 239942369274 (223.46 GB) > Present Capacity: 207894724602 (193.62 GB) > DFS Remaining: 207894552570 (193.62 GB) > DFS Used: 172032 (168 KB) > DFS Used%: 0.00% > Under replicated blocks: 0 > Blocks with corrupt replicas: 0 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > ------------------------------------------------- > Live datanodes (3): > Name: 172.26.79.87:50010 (kimtest3) > Hostname: kimtest3 > Decommission Status : Normal > Configured Capacity: 79980789758 (74.49 GB) > DFS Used: 57344 (56 KB) > Non DFS Used: 9528000512 (8.87 GB) > DFS Remaining: 70452731902 (65.61 GB) > DFS Used%: 0.00% > DFS Remaining%: 88.09% > Configured Cache Capacity: 0 (0 B) > Cache Used: 0 (0 B) > Cache Remaining: 0 (0 B) > Cache Used%: 100.00% > Cache Remaining%: 0.00% > Xceivers: 2 > Last contact: Tue Nov 29 06:59:02 PST 2016 > Name: 172.26.80.38:50010 (kimtest4) > Hostname: kimtest4 > Decommission Status : Normal > Configured Capacity: 79980789758 (74.49 GB) > DFS Used: 57344 (56 KB) > Non DFS Used: 13010952192 (12.12 GB) > DFS Remaining: 66969780222 (62.37 GB) > DFS Used%: 0.00% > DFS Remaining%: 83.73% > Configured Cache Capacity: 0 (0 B) > Cache Used: 0 (0 B) > Cache Remaining: 0 (0 B) > Cache Used%: 100.00% > Cache Remaining%: 0.00% > Xceivers: 2 > Last contact: Tue Nov 29 06:59:02 PST 2016 > Name: 172.26.79.86:50010 (kimtest2) > Hostname: kimtest2 > Decommission Status : Normal > Configured Capacity: 79980789758 (74.49 GB) > DFS Used: 57344 (56 KB) > Non DFS Used: 9508691968 (8.86 GB) > DFS Remaining: 70472040446 (65.63 GB) > DFS Used%: 0.00% > DFS Remaining%: 88.11% > Configured Cache Capacity: 0 (0 B) > Cache Used: 0 (0 B) > Cache Remaining: 0 (0 B) > Cache Used%: 100.00% > Cache Remaining%: 0.00% > Xceivers: 2 > Last contact: Tue Nov 29 06:59:02 PST 2016 > If you see my datanode root file system size its only 38GB > kimtest3:~ # df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/system-root 38G 2.6G 33G 8% / > kimtest4:~ # df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/system-root 38G 4.2G 32G 12% / > kimtest2:~ # df -h / > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/system-root 38G 2.6G 33G 8% / > The below is from hdfs-site.xml file > <property> > <name>dfs.datanode.data.dir</name> > <value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value> > </property> > I have removed the other directory grid1 and restarted datanode process. > <property> > <name>dfs.datanode.data.dir</name> > <value>file:///grid/hadoop/hdfs/dn</value> > </property> > Now the size is reflecting correctly > hdfs@kimtest1:/grid> hdfs dfsadmin -report > Configured Capacity: 119971184637 (111.73 GB) > Present Capacity: 103947243517 (96.81 GB) > DFS Remaining: 103947157501 (96.81 GB) > DFS Used: 86016 (84 KB) > DFS Used%: 0.00% > Under replicated blocks: 0 > Blocks with corrupt replicas: 0 > Missing blocks: 0 > Missing blocks (with replication factor 1): 0 > ------------------------------------------------- > Live datanodes (3): > Name: 172.26.79.87:50010 (kimtest3) > Hostname: kimtest3 > Decommission Status : Normal > Configured Capacity: 39990394879 (37.24 GB) > DFS Used: 28672 (28 KB) > Non DFS Used: 4764057600 (4.44 GB) > DFS Remaining: 35226308607 (32.81 GB) > DFS Used%: 0.00% > DFS Remaining%: 88.09% > Configured Cache Capacity: 0 (0 B) > Cache Used: 0 (0 B) > Cache Remaining: 0 (0 B) > Cache Used%: 100.00% > Cache Remaining%: 0.00% > Xceivers: 2 > Last contact: Tue Nov 29 07:34:02 PST 2016 > Name: 172.26.80.38:50010 (kimtest4) > Hostname: kimtest4 > Decommission Status : Normal > Configured Capacity: 39990394879 (37.24 GB) > DFS Used: 28672 (28 KB) > Non DFS Used: 6505525248 (6.06 GB) > DFS Remaining: 33484840959 (31.19 GB) > DFS Used%: 0.00% > DFS Remaining%: 83.73% > Configured Cache Capacity: 0 (0 B) > Cache Used: 0 (0 B) > Cache Remaining: 0 (0 B) > Cache Used%: 100.00% > Cache Remaining%: 0.00% > Xceivers: 2 > Last contact: Tue Nov 29 07:34:01 PST 2016 > Name: 172.26.79.86:50010 (kimtest2) > Hostname: kimtest2 > Decommission Status : Normal > Configured Capacity: 39990394879 (37.24 GB) > DFS Used: 28672 (28 KB) > Non DFS Used: 4754358272 (4.43 GB) > DFS Remaining: 35236007935 (32.82 GB) > DFS Used%: 0.00% > DFS Remaining%: 88.11% > Configured Cache Capacity: 0 (0 B) > Cache Used: 0 (0 B) > Cache Remaining: 0 (0 B) > Cache Used%: 100.00% > Cache Remaining%: 0.00% > Xceivers: 2 > Last contact: Tue Nov 29 07:34:02 PST 2016 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org