[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455385#comment-16455385
]
Hudson commented on HDFS-13022:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14070/])
HDFS-13022. Block Storage: Kubernetes dynamic persistent volume (omalley: rev
377b31ffa1234889d55c1d15832c87bfcef818ba)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/dynamicprovisioner/expected1-pv.json
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/kubernetes/package-info.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/StorageManager.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/ozone-default.xml
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/dynamicprovisioner/input1-pvc.json
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/kubernetes/DynamicProvisioner.java
* (edit) hadoop-minicluster/pom.xml
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/kubernetes/TestDynamicProvisioner.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/cli/CBlockCli.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/cblock/CBlockConfigKeys.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockManager.java
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch, HDFS-13022-HDFS-7240.005.patch,
> HDFS-13022-HDFS-7240.006.patch, HDFS-13022-HDFS-7240.007.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451169#comment-16451169
]
Hudson commented on HDFS-13022:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14057/])
HDFS-13022. Block Storage: Kubernetes dynamic persistent volume (msingh: rev
eb5e66a1c46f3b542b22ee1e2046d1c728abc479)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/kubernetes/package-info.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/dynamicprovisioner/expected1-pv.json
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/ozone-default.xml
* (edit) hadoop-minicluster/pom.xml
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/cli/CBlockCli.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cblock/kubernetes/TestDynamicProvisioner.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/kubernetes/DynamicProvisioner.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/dynamicprovisioner/input1-pvc.json
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/cblock/CBlockConfigKeys.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/OzonePropertyTag.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/storage/StorageManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/cblock/CBlockManager.java
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch, HDFS-13022-HDFS-7240.005.patch,
> HDFS-13022-HDFS-7240.006.patch, HDFS-13022-HDFS-7240.007.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359883#comment-16359883
]
Mukul Kumar Singh commented on HDFS-13022:
--
Thanks for updating the patch [~elek].
+1, v7 patch looks good to me.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch, HDFS-13022-HDFS-7240.005.patch,
> HDFS-13022-HDFS-7240.006.patch, HDFS-13022-HDFS-7240.007.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359632#comment-16359632
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-minicluster {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
44s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s{color} | {color:green} root: The patch generated 0 new + 1 unchanged - 1
fixed = 1 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-minicluster {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
59s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-minicluster in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason ||
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359471#comment-16359471
]
Elek, Marton commented on HDFS-13022:
-
I uploaded a new version with fixed maven dependency: shadedclient check also
should pass now (I excluded the new dependency from minicluster). Previous
tests were unrelated the next run hopefully also prove this (no code has been
changed)
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch, HDFS-13022-HDFS-7240.005.patch,
> HDFS-13022-HDFS-7240.006.patch, HDFS-13022-HDFS-7240.007.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358924#comment-16358924
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
17s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
7s{color} | {color:green} root: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
49s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
2s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}244m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.ozone.container.replication.TestContainerReplicationManager |
| | hadoop.ozone.scm.container.TestContainerStateManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13022 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358422#comment-16358422
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 7s{color} | {color:orange} root: The patch generated 2 new + 0 unchanged -
1 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
51s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-13022 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909924/HDFS-13022-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 333206f0fc2c
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358209#comment-16358209
]
Elek, Marton commented on HDFS-13022:
-
Thank you very much the review [~msingh].
1. they should be fixed now
([commit1|https://github.com/elek/hadoop/commit/3a9031d60c21c687e79ee04e0f2bdd64c42c0e3b],
[commit2|https://github.com/elek/hadoop/commit/2c68fd70a7e69299cbd38858592defd435bc1862]
)
2,3: Fixed:
[commit1|https://github.com/elek/hadoop/commit/9a68d59080d419c0e5df99c96f61126c8609a5b3]
4. This is just a test file which won't be included in any release, so IMHO we
shouldn't. I checked other json files, and there was no asf header + I got no
RAT warning.
5. Good question, but fortunatelly we don't need to add anything. See my
analysis bellow. There was one problematic dependenciy (system-rules). I
[excluded|https://github.com/elek/hadoop/commit/bd8a27853e248e3dae976d2400748eda6c121b45]
it from the pom.xml and created a
[PR|https://github.com/kubernetes-client/java/pull/182] to use it with test
scope in the future.
6. yep, I removed it.
About the licences:
These are the new jar files which are introduced:
{code:java}
[INFO] +- io.kubernetes:client-java:jar:1.0.0-beta1:compile
[INFO] | +- io.kubernetes:client-java-api:jar:1.0.0-beta1:compile
[INFO] | | +- com.squareup.okhttp:logging-interceptor:jar:2.7.5:compile
[INFO] | | \- joda-time:joda-time:jar:2.9.9:compile
[INFO] | +- io.kubernetes:client-java-proto:jar:1.0.0-beta1:compile
[INFO] | +- org.yaml:snakeyaml:jar:1.16:compile
[INFO] | \- com.squareup.okhttp:okhttp-ws:jar:2.7.5:compile
{code}
OkHttp, Joda time and snakeyaml are not new dependencies (included
hadoop-azure-datalake, and hadoop-project)
And anyway: all of them are licenced under Apache licence without copyright
notice:
* Kubernetes Java client: Apache licence without notice:
[https://github.com/kubernetes-client/java/blob/master/LICENSE]
* Okhttp: Apache licence without copyright notice:
[https://github.com/square/okhttp/blob/master/LICENSE.txt]
* SnakeYaml: Apache licence without copyright notice:
[https://bitbucket.org/asomov/snakeyaml/src/5d89bc78c3e9c6313f7111d596fe5b454d66d7fc/LICENSE.txt?at=default=file-view-default]
* Joda-Time: [http://joda-time.sourceforge.net/license.html]
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355486#comment-16355486
]
Mukul Kumar Singh commented on HDFS-13022:
--
Thanks for the updated patch [~elek], The new patch looks really good. Some
minor comments.
1) Please fix the findbugs and checkstyle issues
2) Nitpick, CBlockManager.java Please move the static imports to the section
with static imports, this needs to be done for DynamicProvisioner.java
3) DynamicProvisioner.java#stop, should join on the watcher thread as well.
4) We should add ASF license to json files as well ?
5) Should LICENSE.txt should also be updated as part of this change
6) I feel that the file ozone-site.xml is not needed, can we set the required
fields in the test(TestDynamicProvisioner) ?
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354861#comment-16354861
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
11s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
13s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 12s{color} | {color:orange} root: The patch generated 10 new + 0 unchanged -
1 fixed = 10 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 13m
18s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
39s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
49s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}148m 4s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
41s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}269m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Write to static field
org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.running from instance
method org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.stop() At
DynamicProvisioner.java:from instance method
org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.stop() At
DynamicProvisioner.java:[line 230] |
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
|
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354562#comment-16354562
]
Elek, Marton commented on HDFS-13022:
-
Moving back to PATCH_AVAILABLE. This is the last version, with some minor
changes:
* I created separated thread for the kubernetes watcher
* I improved the filter of the event handling (only the PVCs with the right
annotations are handled)
* I improved the exception handling (the timeout exception is normal as the
wather uses long polling)
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch,
> HDFS-13022-HDFS-7240.004.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340110#comment-16340110
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 10s{color} | {color:orange} root: The patch generated 10 new + 0 unchanged -
1 fixed = 10 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m
52s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}146m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
47s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}280m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Write to static field
org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.running from instance
method org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.stop() At
DynamicProvisioner.java:from instance method
org.apache.hadoop.cblock.kubernetes.DynamicProvisioner.stop() At
DynamicProvisioner.java:[line 193] |
| Failed junit tests |
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339479#comment-16339479
]
Elek, Marton commented on HDFS-13022:
-
Thank you very much the review, [~msingh]. You are all right. I addressed all
the comments (license, javadoc, init, testname,...)
Also fixed a few typo like Cblock -> CBlock and tested and fixed the default
configuration values (TestOzoneConfigurationFields).
New version is uploaded.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch, HDFS-13022-HDFS-7240.003.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338856#comment-16338856
]
Mukul Kumar Singh commented on HDFS-13022:
--
Thanks for working on this [~elek]. the patch looks really good to me. Some
comments though. All the new files are missing the ASF license header.
1) DynamicProvisioner is missing the ASF license header.
2) DynamicProvisioner.java:50, there is a "" header should we have an ending
"<\p>" somewhere as well
3) DynamicProvisioner.java:108. I feel that we should have a separate init
function which should be called from CblockManager, the constructor does not
need an init function.
4) DynamicProvisioner.java:193, nitpick: Can we have all the arguments in one
line ?
5) DynamicProvisionerTest is missing the ASF license header.
6) Can we rename this file to "TestDynamicProvisioner" as all the other hadoop
tests are named?
7) ozone-default.xml - dfs.cblock.kubernetes.dynamic-provisioner.enabled is
repeated twice
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332479#comment-16332479
]
genericqa commented on HDFS-13022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-13022 does not apply to HDFS-7240. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13022 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906843/HDFS-13022-HDFS-7240.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22719/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332463#comment-16332463
]
Elek, Marton commented on HDFS-13022:
-
I cleaned up the first version:
* Unit test for the transformation logic
* Proper configuration key handling (everything is in the ozone-defaults.xml)
* Correct handling for the pvc size (Gi, Ti)
Now it's stable, but can't be used without HDFS-13018 and HDFS-13017.
To test:
1. First of all you need a kubernetes cluster and a deployed cblock service. I
used my helm charts from [https://github.com/flokkr/charts:]
{code:java}
helm upgrade --install demo ./cblock --set image.tag=iscsi --set
serviceAccount=cblock{code}
2. You need the cluster ip of the iscsi server. Edit the configmap and set the
dfs.cblock.iscsi.advertised.ip key to the value of the iscsi service (kubectl
get svc)
3. You need a storage definition (kubectl apply -f storage.yaml)
{code:java}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cblock
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: hadoop.apache.org/cblock{code}
4. And you can try to create a new persistent container:
{code:java}
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: persistent
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: persistent
labels:
app: persistent
spec:
replicas: 1
selector:
matchLabels:
app: persistent
template:
metadata:
labels:
app: persistent
spec:
containers:
- name: persistent
image: alpine
command: ["cat"]
tty: true
stding: true
volumeMounts:
- mountPath: "/data"
name: iscsi
volumes:
- name: iscsi
persistentVolumeClaim:
claimName: persistent{code}
As a result you can see a good PersistentVolume/PersistentVolumeClaim and after
a few minutes (first formatting could be slow) the running pod with the
attached volume.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch,
> HDFS-13022-HDFS-7240.002.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13022) Block Storage: Kubernetes dynamic persistent volume provisioner
[
https://issues.apache.org/jira/browse/HDFS-13022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326401#comment-16326401
]
Elek, Marton commented on HDFS-13022:
-
Uploaded a *work in progress* version. I tested it with onprem kubernetes
cluster and worked well.
Few TODOs:
* Handle the volume size parsing (Gi, GB, M, Mi,...)
* Turn it off by default
The cblock deletion could be implemented in a followup Jira to keep it simple.
> Block Storage: Kubernetes dynamic persistent volume provisioner
> ---
>
> Key: HDFS-13022
> URL: https://issues.apache.org/jira/browse/HDFS-13022
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: HDFS-7240
>Affects Versions: HDFS-7240
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDFS-13022-HDFS-7240.001.patch
>
>
> {color:#FF}{color}
> With HDFS-13017 and HDFS-13018 the cblock/jscsi server could be used in a
> kubernetes cluster as the backend for iscsi persistent volumes.
> Unfortunatelly we need to create all the required cblocks manually with 'hdfs
> cblok -c user volume...' for all the Persistent Volumes.
>
> But it could be handled with a simple optional component. An additional
> service could listen on the kubernetes event stream. In case of new
> PersistentVolumeClaim (where the storageClassName is cblock) the cblock
> server could create cblock in advance AND create the persistent volume could
> be created.
>
> The code is very simple, and this additional component could be optional in
> the cblock server.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
