[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454986#comment-16454986 ] Hudson commented on HDFS-11138: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-11138. Block Storage: add block storage server. Contributed by Chen (omalley: rev 9997d36eb6059cb2fe29babf91a5f57bd324c0fb) * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockConfigKeys.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockConfiguration.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockClientServerProtocol.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/VolumeDescriptor.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/IStorageClient.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockServiceProtocol.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/TestCBlockServer.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/StorageManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/util/ContainerLookUpService.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolServerSideTranslatorPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/VolumeInfo.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockServiceProtocolServerSideTranslatorPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/exception/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/ContainerDescriptor.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockServiceProtocolPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/exception/CBlockException.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/util/MockStorageClient.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/MountVolumeResponse.java > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450544#comment-16450544 ] Hudson commented on HDFS-11138: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-11138. Block Storage: add block storage server. Contributed by Chen (aengineer: rev 334113adab57d81db82868fdd82636a2f9562be8) * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockServiceProtocol.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/util/ContainerLookUpService.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockServiceProtocolServerSideTranslatorPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockConfiguration.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/exception/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/util/MockStorageClient.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/IStorageClient.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/ContainerDescriptor.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/TestCBlockServer.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/exception/CBlockException.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/VolumeDescriptor.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/VolumeInfo.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockConfigKeys.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockClientServerProtocol.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/MountVolumeResponse.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/StorageManager.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolServerSideTranslatorPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockServiceProtocolPB.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/package-info.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/package-info.java > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677911#comment-15677911
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839610/HDFS-11138-HDFS-7240.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b553fc17dd47 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 5ceca1b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automa
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677871#comment-15677871 ] Anu Engineer commented on HDFS-11138: - +1, Thanks for updating this patch. I will commit this patch to ozone branch shortly. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677614#comment-15677614
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839597/HDFS-11138-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 78f71dbd9b2b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 5ceca1b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreC
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675342#comment-15675342 ] Anu Engineer commented on HDFS-11138: - Let us keep the map for time being. We can revisit this later. It probably has no material impact. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675319#comment-15675319 ] Chen Liang commented on HDFS-11138: --- Thanks [~anu] for the comments! Again here are some thoughts I had (mainly related to 3 & 4 & 6). And will address the rest in the next path. I know that keeping a map AND a list in StorageManager is not quite clean. The idea was that I wanted to allow two ways of accessing containers: 1. access them from the first to the last, based on their order index - which requires a list 2. access a specific a container with a given containerID - which requires a map This was my very first design when writing the class. So far, we access all containers at once on mount time by returning all the containers along with their order, in which case a list is sufficient. I wanted to leave the possibility for accessing a container by key on server side, but I can't really think of a use case for the time being... Do you think this access-by-key is not worth keeping? If so, I'll just go remove it. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675206#comment-15675206
]
Anu Engineer commented on HDFS-11138:
-
[~vagarychen] Thanks for updating the patch. Overall it is looking quite good.
Some very minor comments and questions.
1. Can you please clarify these comments. It is hard to know the difference
between local client and cblock client.
{noformat}
CBlockServiceProtocolPB - The protocol that local client tool uses to talk to
CBlock server.
CBlockClientServerProtocolPB - Protocol that cblock client uses to talk to
cblock server.
{noformat}
2. Don't understand this logging message here. if user does not specify a block
size we will use this value. But why log that in startup ?
{noformat}
LOG.info("CBlock manager initialized, with default block size:
{}",DFS_CBLOCK_SERVICE_BLOCK_SIZE_DEFAULT);
{noformat}
3. In {{VolumeDescriptor.java}} shouldn't this {{private List
containerIdOrdered;}} be {{private List
containerIdOrdered;}}
4. Also a related question, does it make sense to make {{addContainer}}
automatically call {{setContainerIDs}}. That way the user of this API does not
need to know about keeping these two lists. In {{StorageManager.java}} that
would allow you to eliminate maintaining an extra list.
5. public static VolumeDescriptor parse(String jsonString) -- do we need
this ?
6. in {{CBlockClientServerProtocolServerSideTranslatorPB#mountVolume}}
{noformat}
List containers = result.getContainerList();
for (int i=0; i Block Storage : add block storage server
>
>
> Key: HDFS-11138
> URL: https://issues.apache.org/jira/browse/HDFS-11138
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11138-HDFS-7240.001.patch,
> HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch,
> HDFS-11138-HDFS-7240.004.patch
>
>
> This JIRA adds the skeleton for server side code, which is one of the core
> components in block storage. For performance concerns, the server does not
> handle any actual read/write operation but serving primarily as a meta data
> server. It provides four APIs:
> # create volume : which will call into underlying container layer to allocate
> containers
> # delete volume : delete a specific volume (as well as its containers)
> # info volume : return information of a specific volume
> # list volume : list all volumes
> Note that this is still subject to potentially major changes. Features such
> as persistence are missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675154#comment-15675154 ] Chen Liang commented on HDFS-11138: --- The findbug warnings are for the protobuf-generated classes. The failed tests are unrelated. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674751#comment-15674751
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
4s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839422/HDFS-11138-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cd5c5e973b3e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / d10f39e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17601/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17601/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17601/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17601/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Block Storage : add block storage server
>
>
>
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15672424#comment-15672424
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 104 unchanged -
1 fixed = 105 total (was 105) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839278/HDFS-11138-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ae59688ed69f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / d10f39e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17595/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/17595/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17595/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17595/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console outp
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15672125#comment-15672125
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
55s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s{color}
| {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 104 unchanged -
1 fixed = 105 total (was 105) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839249/HDFS-11138-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cd39328ae89b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / d10f39e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17593/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/17593/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17593/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17593/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17593/conso
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671496#comment-15671496
]
Anu Engineer commented on HDFS-11138:
-
Thought this might help you, checkstyle warnings:
{noformat}
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockConfigKeys.java:23:public
class CBlockConfigKeys {:1: Utility classes should not have a public or
default constructor.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockConfiguration.java:22:public
class CBlockConfiguration extends Configuration {: Missing a Javadoc comment.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockManager.java:127:
(conf, CBlockClientServerProtocolPB.class,:9: '(' should be on the
previous line.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/exception/CBlockException.java:0::
Missing package-info.java file.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/exception/CBlockException.java:22:public
class CBlockException extends IOException {: Missing a Javadoc comment.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/ContainerDescriptor.java:0::
Missing package-info.java file.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/meta/VolumeDescriptor.java:177:
private String getBlockKey(String volumeName, long blockId) {:37:
'volumeName' hides a field.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockClientServerProtocol.java:0::
Missing package-info.java file.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockClientServerProtocol.java:22:public
interface CBlockClientServerProtocol {: Missing a Javadoc comment.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/CBlockServiceProtocol.java:26:/**:
First sentence should end with a period.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/MountVolumeResponse.java:5:public
class MountVolumeResponse {: Missing a Javadoc comment.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/proto/MountVolumeResponse.java:6:
public static final long versionID = 1L;:28: Name 'versionID' must match
pattern '^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$'.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolPB.java:0::
Missing package-info.java file.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolPB.java:24:@ProtocolInfo(protocolName
=: Missing a Javadoc comment.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolServerSideTranslatorPB.java:30:@InterfaceAudience.Private:
Missing a Javadoc comment.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/protocolPB/CBlockClientServerProtocolServerSideTranslatorPB.java:62:
for (int i=0;i Block Storage : add block storage server
>
>
> Key: HDFS-11138
> URL: https://issues.apache.org/jira/browse/HDFS-11138
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11138-HDFS-7240.001.patch
>
>
> This JIRA adds the skeleton for server side code, which is one of the core
> components in block storage. For performance concerns, the server does not
> handle any actual read/write operation but serving primarily as a meta data
> server. It provides four APIs:
> # create volume : which will call into underlying container layer to allocate
> containers
> # delete volume : delete a specific volume (as well as its containers)
> # info volume : return information of a specific volume
> # list volume : list all volumes
> Note that this is still subject to potentially major changes. Features such
> as persistence are missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671319#comment-15671319 ] Anu Engineer commented on HDFS-11138: - bq. I actually do not have any strong opinion on this, if it is making more sense to require setting force flag all the time, then I can just make this change. I think if user specifies it we should use it, that is like most unix command if the user specifies something like -f on command line, then we set force to true, otherwise it always defaults to false. That way we don't really need a config setting. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671324#comment-15671324 ] Chen Liang commented on HDFS-11138: --- bq. I think if user specifies it we should use it, that is like most unix command if the user specifies something like -f on command line, then we set force to true, otherwise it always defaults to false. That way we don't really need a config setting. I agree this is probably the best way to go, thanks! > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671311#comment-15671311 ] Chen Liang commented on HDFS-11138: --- I actually do not have any strong opinion on this, if it is making more sense to require setting force flag all the time, then I can just make this change. Thanks for the suggestion. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671293#comment-15671293 ] Anu Engineer commented on HDFS-11138: - bq. The idea is that the force flag is only an optional flag, i.e. the call may not have this flag. In which case a default value will be used. This key is to allow users specify different default behavior when force flag is missing. This being said, user parameter, if given, takes precedence. So do expect the user to say force=false in case the cluster wide setting is true ? How does the user know what is the cluster wide setting ? Isn't easier to just return an error and let user use force if needed ? I am not sure I see the benefit of having a cluster wide default. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671281#comment-15671281 ] Chen Liang commented on HDFS-11138: --- Thanks [~anu] for the comments! Here were my thoughts on some of the comments, and will address the rest and submit another patch: (#5) bq. Slightly confused about how we use this, We have RPCs which take forceDelete as a parameter, we also have forceDelete as a config key. I am trying to decide which one takes precedence and why we need both. The idea is that the force flag is only an optional flag, i.e. the call may not have this flag. In which case a default value will be used. This key is to allow users specify different default behavior when force flag is missing. This being said, user parameter, if given, takes precedence. (#11) bq. VolumeDescriptor.java I am not able to understand why we have to maintain this map. Can you please explain the use case ? private HashMap containerMap; This is where server side maintains the containers of a volume. Say, when users request infoContainers, this is where server gets all containers and iterate on them. The ideal here was that, in addition to just an id of a container, server also maintains some information about each container, including a handler to allow communication to the container on info request, and other information to keep in persistent storage and failover, all this wrapped in ContainerDescriptor. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669116#comment-15669116
]
Anu Engineer commented on HDFS-11138:
-
Thank you for the patch, overall it looks very good. I have a bunch of very
minor comments. Since the patch is big, I might add more comments little later.
Please feel free to post a second patch if you like. Also for some reason we
did not get a Jenkins run, which would have given you all the CheckStyle and
other issues.
1. Could you please run checkstyle. I see a bunch of checkstyle warnings.
2. {{CBlockConfigKeys.java:21}} Comment about ozone, I think you meant cBlocks.
3. {{CBlockConfigKeys.java:39}}
//The port on CBlockManager node for jSCSI to ask
{noformat}
public static final int DFS_CBLOCK_JSCSI_PORT_DEFAULT = 50701;
public static final int DFS_CBLOCK_RPCSERVICE_PORT_DEFAULT = 50700;
{noformat}
I realize that you are trying to be consistent, however trunk has changed its
port map.
Namenode ports
50470 --> 9871
50070 --> 9870
8020 --> 9820
Secondary NN ports
---
50091 --> 9869
50090 --> 9868
Datanode ports
---
50020 --> 9867
50010 --> 9866
50475 --> 9865
50075 --> 9864
I know ozone still uses the old port map, we might have to go an fix it. You
might want to use a port that is free but closer to 98xx series.
4. {{public static final String DFS_CBLOCK_RPCSERVICE_IP_DEFAULT =
"127.0.0.1";}}
Did you want this to be 127.0.0.1 or 0.0.0.0 ? I would think you might want to
listen on 0.0.0.0
5.
{noformat}
58public static final String DFS_CBLOCK_SERVICE_DELETE_FORCE_KEY =
59"dfs.cblock.service.delete-force";
60public static final boolean DFS_CBLOCK_SERVICE_DELETE_FORCE_DEFAULT =
61false;
{noformat}
Slightly confused about how we use this, We have RPCs which take forceDelete as
a parameter, we also have forceDelete as a config key. I am trying to decide
which one takes precedence and why we need both.
6. More of a comment I not asking for a change here. We probably have 2 RPCs
from jSCSI server to cBlock server. getContainers and getLease. While I like
the fact that you have separated the interface that CLI depends upon and what
jscsi depends upon, I was wondering it is extra work. But now that it is done,
we should probably put that in.
7. {{ public void join()}} After we catch an exception, would you please add an
interrupt call ?
{noformat}
catch (InterruptedException e) {
Thread.currentThread().interrupt();
{noformat}
This is one of the dark corners of Java threading.
8. {{start, stop and join}} We have null check only in stop, I don't know if we
need it.
9. Nit: Log.info , you can use arguments instead of + operator.
10. {{ContainerDescriptor}} Can we please add an Index variable that tells us
what index this container is in the list of containers.
11. {{VolumeDescriptor.java}}
I am not able to understand why we have to maintain this map. Can you please
explain the use case ?
{{private HashMap containerMap;}}
12. May I suggest that instead of converting this class to JSON and then
persisting it to local file -- Which we happen to do in the ozone test
implementation, I would suggest that we take the protobuf class as is -- and
the do toByteArray -- which will give you a byte stream which you can easily
persist to LevelDB. This works well for both keys and values.
13. We have a debugging line left out in the code: Line 204:
{{System.err.println("VolumeDesc...parse():" + jsonString);}}
14. {{CBlockServiceProtocol.java}}
{noformat}
33void createVolume(String userName, String volumeName,
34long volumeSize, int blockSize) throws IOException;
35
36void createVolume(String userName, String volumeName,
37long volumeSize) throws IOException;
{noformat}
Do we need both version of create ? if we set up the blockSize to have a
default size, then either the client can set the right value or you can rely on
default values in the protoc. Same comment about delete.
You can easily remove the second call by moving this to client side.
{noformat}
try {
51if (request.hasBlockSize()) {
52 impl.createVolume(request.getUserName(),
request.getVolumeName(),
53 request.getVolumeSize(), request.getBlockSize());
54} else{
55 impl.createVolume(request.getUserName(),
request.getVolumeName(),
56 request.getVolumeSize());
57}
{noformat}
15. {{MountVolumeResponse}} Return a structure with index in
{{getContainerList}}. Please add some comments to this class, checkstyle is
going to complain about this. Also missing Licence header in this class.
16. {{CBlockClientServerProtocolServerSideTranslatorPB}}
{noformat}
62 for (int i=0;i Block Storage : add block storage server
> ---
