[jira] [Commented] (HDFS-11548) Ozone: SCM: Add node pool management API
[ https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455017#comment-16455017 ] Hudson commented on HDFS-11548: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-11548. Ozone: SCM: Add node pool management API. Contributed by (omalley: rev 3598d4b8cfbef5ac01e14665784feaa410e7da1a) * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/NodeStat.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeStat.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestNodeManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestSCMNodePoolManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodePoolManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/exceptions/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/exceptions/SCMException.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/NodePoolManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestContainerPlacement.java > Ozone: SCM: Add node pool management API > > > Key: HDFS-11548 > URL: https://issues.apache.org/jira/browse/HDFS-11548 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-11548-HDFS-7240.001.patch, > HDFS-11548-HDFS-7240.002.patch, HDFS-11548-HDFS-7240.003.patch, > HDFS-11548-HDFS-7240.004.patch > > > The idea is to group registered nodes into pools of fixed size (say 24 nodes > per pool) so that the container allocation and report can all be handled > independently on a pool basis by SCM. > The initial patch will implement the following Node Pool API that > 1) add node to a node pool > 2) remove a node from a pool > 3) get the pool name that a node belongs to > 4) get all the pool names > 5) get all nodes of a pool > The integration with SCM container allocation can be all nodes in a single > default pool upon registration. We will provide a CLI to manage multiple > pools and support for pool definition file later. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11548) Ozone: SCM: Add node pool management API
[ https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450607#comment-16450607 ] Hudson commented on HDFS-11548: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-11548. Ozone: SCM: Add node pool management API. Contributed by (aengineer: rev 8f4d8c43f84defd5550c3635051b056afc9ec57f) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodeStat.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestContainerPlacement.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestNodeManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/exceptions/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/SCMNodePoolManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/scm/node/TestSCMNodePoolManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/NodePoolManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/utils/LevelDBStore.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/exceptions/SCMException.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/scm/node/NodeStat.java > Ozone: SCM: Add node pool management API > > > Key: HDFS-11548 > URL: https://issues.apache.org/jira/browse/HDFS-11548 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-11548-HDFS-7240.001.patch, > HDFS-11548-HDFS-7240.002.patch, HDFS-11548-HDFS-7240.003.patch, > HDFS-11548-HDFS-7240.004.patch > > > The idea is to group registered nodes into pools of fixed size (say 24 nodes > per pool) so that the container allocation and report can all be handled > independently on a pool basis by SCM. > The initial patch will implement the following Node Pool API that > 1) add node to a node pool > 2) remove a node from a pool > 3) get the pool name that a node belongs to > 4) get all the pool names > 5) get all nodes of a pool > The integration with SCM container allocation can be all nodes in a single > default pool upon registration. We will provide a CLI to manage multiple > pools and support for pool definition file later. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11548) Ozone: SCM: Add node pool management API
[
https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944278#comment-15944278
]
Hadoop QA commented on HDFS-11548:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
57s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}152m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.cblock.TestCBlockCLI |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.cblock.TestCBlockServer |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| Timed out junit tests | org.apache.hadoop.ozone.scm.node.TestNodeManager |
| | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11548 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860720/HDFS-11548-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 039399e78f94 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / ed14373 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18853/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18853/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Bui
[jira] [Commented] (HDFS-11548) OZone: SCM: Add node pool management API
[ https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944114#comment-15944114 ] Anu Engineer commented on HDFS-11548: - [~xyao] Thanks for the update. +1, Pending Jenkins. > OZone: SCM: Add node pool management API > > > Key: HDFS-11548 > URL: https://issues.apache.org/jira/browse/HDFS-11548 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-11548-HDFS-7240.001.patch, > HDFS-11548-HDFS-7240.002.patch, HDFS-11548-HDFS-7240.003.patch, > HDFS-11548-HDFS-7240.004.patch > > > The idea is to group registered nodes into pools of fixed size (say 24 nodes > per pool) so that the container allocation and report can all be handled > independently on a pool basis by SCM. > The initial patch will implement the following Node Pool API that > 1) add node to a node pool > 2) remove a node from a pool > 3) get the pool name that a node belongs to > 4) get all the pool names > 5) get all nodes of a pool > The integration with SCM container allocation can be all nodes in a single > default pool upon registration. We will provide a CLI to manage multiple > pools and support for pool definition file later. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11548) OZone: SCM: Add node pool management API
[
https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943933#comment-15943933
]
Anu Engineer commented on HDFS-11548:
-
Thank you for updating the patch. Some very minor comments below.
* Could we please rewrite this pattern ?
{code}
try {
lock.writeLock().lock();
{code}
* Replace this pattern with
{code}
lock.writeLock().lock();
try {
...
} finally {
lock.writeLock().unlock();
}
That seems to be the Java idiom.
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/Lock.html
{code}
* nit: Preconditions.CheckNotNull(pool);
{code}
public void addNode(final String pool, final DatanodeID node)
{code}
Same feedback in removeNode, checkNotNull on node.
* Hard coded nodeCount in {{testDefaultNodePoolReload}}
> OZone: SCM: Add node pool management API
>
>
> Key: HDFS-11548
> URL: https://issues.apache.org/jira/browse/HDFS-11548
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11548-HDFS-7240.001.patch,
> HDFS-11548-HDFS-7240.002.patch, HDFS-11548-HDFS-7240.003.patch
>
>
> The idea is to group registered nodes into pools of fixed size (say 24 nodes
> per pool) so that the container allocation and report can all be handled
> independently on a pool basis by SCM.
> The initial patch will implement the following Node Pool API that
> 1) add node to a node pool
> 2) remove a node from a pool
> 3) get the pool name that a node belongs to
> 4) get all the pool names
> 5) get all nodes of a pool
> The integration with SCM container allocation can be all nodes in a single
> default pool upon registration. We will provide a CLI to manage multiple
> pools and support for pool definition file later.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11548) OZone: SCM: Add node pool management API
[
https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943843#comment-15943843
]
Hadoop QA commented on HDFS-11548:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.cblock.TestCBlockServerPersistence |
| | hadoop.cblock.TestCBlockServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11548 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860681/HDFS-11548-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3d4bb98a3b77 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / ed14373 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/18847/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/18847/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18847/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> OZone: SCM: Add node pool management API
>
>
> Key: HDFS-11548
> URL: https://issues.apache.org/jira/browse/HDFS-11548
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: X
[jira] [Commented] (HDFS-11548) OZone: SCM: Add node pool management API
[
https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942655#comment-15942655
]
Hadoop QA commented on HDFS-11548:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
16s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Format-string method String.format(String, Object[]) called with format
string "Unable to find node {} from pool {} in DB." wants 0 arguments but is
given 2 in
org.apache.hadoop.ozone.scm.node.SCMNodePoolManager.removeNode(String,
DatanodeID) At SCMNodePoolManager.java:with format string "Unable to find node
{} from pool {} in DB." wants 0 arguments but is given 2 in
org.apache.hadoop.ozone.scm.node.SCMNodePoolManager.removeNode(String,
DatanodeID) At SCMNodePoolManager.java:[line 180] |
| | Format-string method String.format(String, Object[]) called with format
string "Unable to find node {} from pool {} in MAP." wants 0 arguments but is
given 2 in
org.apache.hadoop.ozone.scm.node.SCMNodePoolManager.removeNode(String,
DatanodeID) At SCMNodePoolManager.java:with format string "Unable to find node
{} from pool {} in MAP." wants 0 arguments but is given 2 in
org.apache.hadoop.ozone.scm.node.SCMNodePoolManager.removeNode(String,
DatanodeID) At SCMNodePoolManager.java:[line 191] |
| Failed junit tests | hadoop.ozone.scm.node.TestSCMNodePoolManager |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.cblock.TestCBlockCLI |
| | hadoop.cblock.TestCBlockServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11548 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12860586/HDFS
[jira] [Commented] (HDFS-11548) OZone: SCM: Add node pool management API
[
https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935131#comment-15935131
]
Anu Engineer commented on HDFS-11548:
-
[~xyao] Thank you for adding this feature. Some of my next patches depend on
this feature. Some comments below.
* public SCMNodePoolManager(final OzoneConfiguration conf)
-- remove double brackets ?
{code}
if ((scmMetaDataDir == null)) {
{code}
* TestSCMNodePoolManager::testDefaultNodePool
-- we declare nodeCount, but hard code 4 in comparisons. We have this in few
places in this test. Can we please use nodeCount ?
{code}
assertEquals(4, nodesRetrieved.size());
{code}
-- Looks like {{Iterators.elementsEqual}} returns a boolean. So we should
assertTrue on the return value from those statements. I am getting some
failures when I add that assertion.
-- Same comments for the other test too.
* SCMNodeManager.java : 2 Check Style warnings.
-- Unused import - org.apache.hadoop.conf.Configuration. (25:8)
-- Empty statement. (177:57)
* SCMNodePoolManager.java: Line 127
-- Shouldn't we fail SCM load at this point of time ?
{{LOG.error("Loading node pool error " + e);}}
* removeNode() : Line 171
{code}
if (kData == null) {
throw new StorageContainerException("Unable to find the key.",
ContainerProtos.Result.NO_SUCH_KEY);
}
{code}
-- I think we should have a new SCM server only exception to propagate this
kind of error. The container protos are part of wire protocol that we use to
communicate with datanode containers. While the semantics is appropriate here,
I am worried that someone looking at logs might get confused that the error
came from a datanode. Same for all uses of StorageContainerException in this
file.
> OZone: SCM: Add node pool management API
>
>
> Key: HDFS-11548
> URL: https://issues.apache.org/jira/browse/HDFS-11548
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11548-HDFS-7240.001.patch
>
>
> The idea is to group registered nodes into pools of fixed size (say 24 nodes
> per pool) so that the container allocation and report can all be handled
> independently on a pool basis by SCM.
> The initial patch will implement the following Node Pool API that
> 1) add node to a node pool
> 2) remove a node from a pool
> 3) get the pool name that a node belongs to
> 4) get all the pool names
> 5) get all nodes of a pool
> The integration with SCM container allocation can be all nodes in a single
> default pool upon registration. We will provide a CLI to manage multiple
> pools and support for pool definition file later.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11548) OZone: SCM: Add node pool management API
[
https://issues.apache.org/jira/browse/HDFS-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15931864#comment-15931864
]
Hadoop QA commented on HDFS-11548:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestDatanodeReport |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.ozone.scm.TestAllocateContainer |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStream |
| | hadoop.hdfs.TestFileStatusWithECPolicy |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.TestSetTimes |
| | hadoop.hdfs.TestListFilesInDFS |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.TestMiniDFSCluster |
