[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381483#comment-16381483
]
genericqa commented on HDFS-13207:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
35s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}174m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}242m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13207 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912505/HDFS-13207.01.patch |
| Optional Tests |
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381281#comment-16381281 ] Bharat Viswanadham commented on HDFS-13207: --- [~xyao] Attached the patch to address 1 and 3 comments. > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch, HDFS-13207.01.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381198#comment-16381198
]
Bharat Viswanadham commented on HDFS-13207:
---
Hi [~xyao]
Thank you for the comments.
{quote} 1. In the patch, we added the block pool used per datanode. Should we
add "block pool used" to the summary portion before the "Live datanodes" as
well?
{quote}
I think, we can add blockPoolUsed, which will show cumulative blockPoolUsed for
this namespace.
{quote}2. If there are multiple namespaces and block pools, is it possible to
separate the usage of different block pool?
{quote}
As DFSAdmin report command is made from one namespace, it will show
blockPoolused for that namespace. As DatanodeDescriptor maintains blockpoolused
for the namespace it belongs.
{quote}3. Should we default to turn this off and show the "block pool used"
only when federation is configured or a CLI switch like "-blockPoolUsed" is
provided?
{quote}
I think it is a good idea, will do that. I will try to use a switch
blockPoolUsed for dfsadmin -report command to do this.
> Add blockpool used for DFSAdmin Report command
> ---
>
> Key: HDFS-13207
> URL: https://issues.apache.org/jira/browse/HDFS-13207
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13207.00.patch
>
>
> Add block pool used for dfsadmin -report command.
> This will be helpful on a federated cluster to know DFS used by a particular
> namespace.
>
> On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381048#comment-16381048 ] Xiaoyu Yao commented on HDFS-13207: --- Thanks [~bharatviswa] for reporting the issue and posting the patch. I just have few questions: 1. In the patch, we added the block pool used per datanode. Should we add "block pool used" to the summary portion before the "Live datanodes" as well? 2. If there are multiple namespaces and block pools, is it possible to separate the usage of different block pool? 3. Should we default to turn this off and show the "block pool used" only when federation is configured or a CLI switch like "-blockPoolUsed" is provided? > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381046#comment-16381046
]
genericqa commented on HDFS-13207:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
31s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13207 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912479/HDFS-13207.00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5b6a9b0a4da3 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3100903 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23246/testReport/ |
| Max. process+thread count | 319 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23246/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add blockpool used for DFSAdmin
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380976#comment-16380976
]
Bharat Viswanadham commented on HDFS-13207:
---
Existing TestDFSAdmin test cases, covers this.
Tested it on a Federated cluster.
The output is as below:
{code:java}
hdfs@373dc353753f:/$ /hadoop/bin/hdfs dfsadmin -report
2018-02-28 20:15:14,153 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Configured Capacity: 1995884724224 (1.82 TB)
Present Capacity: 1117203682022 (1.02 TB)
DFS Remaining: 1117203578880 (1.02 TB)
DFS Used: 103142 (100.72 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Pending deletion blocks: 0
-
Live datanodes (4):
Name: xxx
Hostname: 373dc353753f
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 78566 (76.72 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219416452378 (204.35 GB)
DFS Remaining: 279292506112 (260.11 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:13 UTC 2018
Last Block Report: Wed Feb 28 20:14:43 UTC 2018
Name: xxx
Hostname: 57a3a51fff80
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 8192 (8 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219416522752 (204.35 GB)
DFS Remaining: 279292506112 (260.11 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:12 UTC 2018
Last Block Report: Wed Feb 28 20:14:42 UTC 2018
Name: xxx
Hostname: 0865db3359a9
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 8192 (8 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219382968320 (204.32 GB)
DFS Remaining: 279326060544 (260.14 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.98%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:12 UTC 2018
Last Block Report: Wed Feb 28 20:14:42 UTC 2018
Name: xxx
Hostname: e7d4bc7c9353
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 8192 (8 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219416522752 (204.35 GB)
DFS Remaining: 279292506112 (260.11 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:12 UTC 2018
Last Block Report: Wed Feb 28 20:14:42 UTC 2018
{code}
> Add blockpool used for DFSAdmin Report command
> ---
>
> Key: HDFS-13207
> URL: https://issues.apache.org/jira/browse/HDFS-13207
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13207.00.patch
>
>
> Add block pool used for dfsadmin -report command.
> This will be helpful on a federated cluster to know DFS used by a particular
> namespace.
>
> On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
