[jira] [Updated] (HDFS-11445) FSCK shows overall health stauts as corrupt even one replica is corrupt
[
https://issues.apache.org/jira/browse/HDFS-11445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11445:
Attachment: HDFS-11445-004.patch
[~jingzhao] thanks for review..
Updated the patch to Address the {{check-style}}.
> FSCK shows overall health stauts as corrupt even one replica is corrupt
> ---
>
> Key: HDFS-11445
> URL: https://issues.apache.org/jira/browse/HDFS-11445
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-11445-002.patch, HDFS-11445-003.patch,
> HDFS-11445-004.patch, HDFS-11445.patch
>
>
> In the following scenario,FSCK shows overall health status as corrupt even
> it's has one good replica.
> 1. Create file with 2 RF.
> 2. Shutdown one DN
> 3. Append to file again.
> 4. Restart the DN
> 5. After block report, check Fsck
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11794) Add ec sub command -listCodec to show currently supported ec codecs
[ https://issues.apache.org/jira/browse/HDFS-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018663#comment-16018663 ] SammiChen commented on HDFS-11794: -- Double checked failed Unit test. They are irrelevant. > Add ec sub command -listCodec to show currently supported ec codecs > --- > > Key: HDFS-11794 > URL: https://issues.apache.org/jira/browse/HDFS-11794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: SammiChen >Assignee: SammiChen > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11794.001.patch, HDFS-11794.002.patch, > HDFS-11794.003.patch > > > Add ec sub command -listCodec to show currently supported ec codecs -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not remove chosen node from healthy list.
[
https://issues.apache.org/jira/browse/HDFS-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018634#comment-16018634
]
Xiaoyu Yao commented on HDFS-11860:
---
This patch fixed the random failure of
TestContainerPlacement#testContainerPlacementCapacity.
> Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not
> remove chosen node from healthy list.
> -
>
> Key: HDFS-11860
> URL: https://issues.apache.org/jira/browse/HDFS-11860
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11860-HDFS-7240.001.patch
>
>
> This was caught in Jenkins run randomly. After debugging, found the cause is
> the
> logic when two random index happens to be the same below where the node id
> was returned without being removed from the healthy list for next round of
> selection. As a result, there could be duplicated datanodes chosen for the
> pipeline and the machine list size smaller than expected. I will post a fix
> soon.
> {code}
> SCMContainerPlacementCapacity#chooseNode
> // There is a possibility that both numbers will be same.
> // if that is so, we just return the node.
> if (firstNodeNdx == secondNodeNdx) {
> return healthyNodes.get(firstNodeNdx);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not remove chosen node from healthy list.
[
https://issues.apache.org/jira/browse/HDFS-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018593#comment-16018593
]
Hadoop QA commented on HDFS-11860:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
22s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11860 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869130/HDFS-11860-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 547f2728c930 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b4e5c55 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19526/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
[jira] [Updated] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not remove chosen node from healthy list.
[
https://issues.apache.org/jira/browse/HDFS-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11860:
--
Description:
This was caught in Jenkins run randomly. After debugging, found the cause is
the
logic when two random index happens to be the same below where the node id was
returned without being removed from the healthy list for next round of
selection. As a result, there could be duplicated datanodes chosen for the
pipeline and the machine list size smaller than expected. I will post a fix
soon.
{code}
SCMContainerPlacementCapacity#chooseNode
// There is a possibility that both numbers will be same.
// if that is so, we just return the node.
if (firstNodeNdx == secondNodeNdx) {
return healthyNodes.get(firstNodeNdx);
}
{code}
was:
This was caught in Jenkins run. After debugging, found the cause is the
logic below where the node was returned without being removed from the healthy
list for next round. As a result, there could be duplicated datanodes chosen
with pipeline size smaller than expected. I will post a fix soon.
{code}
SCMContainerPlacementCapacity#chooseNode
// There is a possibility that both numbers will be same.
// if that is so, we just return the node.
if (firstNodeNdx == secondNodeNdx) {
return healthyNodes.get(firstNodeNdx);
}
{code}
> Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not
> remove chosen node from healthy list.
> -
>
> Key: HDFS-11860
> URL: https://issues.apache.org/jira/browse/HDFS-11860
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11860-HDFS-7240.001.patch
>
>
> This was caught in Jenkins run randomly. After debugging, found the cause is
> the
> logic when two random index happens to be the same below where the node id
> was returned without being removed from the healthy list for next round of
> selection. As a result, there could be duplicated datanodes chosen for the
> pipeline and the machine list size smaller than expected. I will post a fix
> soon.
> {code}
> SCMContainerPlacementCapacity#chooseNode
> // There is a possibility that both numbers will be same.
> // if that is so, we just return the node.
> if (firstNodeNdx == secondNodeNdx) {
> return healthyNodes.get(firstNodeNdx);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not remove chosen node from healthy list.
[
https://issues.apache.org/jira/browse/HDFS-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11860:
--
Summary: Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes
does not remove chosen node from healthy list. (was: Ozone: SCM:
SCMContainerPlacementCapacity#chooseNode chosen node is not removed from
healthy list.)
> Ozone: SCM: SCMContainerPlacementCapacity#chooseNode sometimes does not
> remove chosen node from healthy list.
> -
>
> Key: HDFS-11860
> URL: https://issues.apache.org/jira/browse/HDFS-11860
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11860-HDFS-7240.001.patch
>
>
> This was caught in Jenkins run. After debugging, found the cause is the
> logic below where the node was returned without being removed from the
> healthy list for next round. As a result, there could be duplicated datanodes
> chosen with pipeline size smaller than expected. I will post a fix soon.
> {code}
> SCMContainerPlacementCapacity#chooseNode
> // There is a possibility that both numbers will be same.
> // if that is so, we just return the node.
> if (firstNodeNdx == secondNodeNdx) {
> return healthyNodes.get(firstNodeNdx);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode chosen node is not removed from healthy list.
[
https://issues.apache.org/jira/browse/HDFS-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11860:
--
Attachment: HDFS-11860-HDFS-7240.001.patch
> Ozone: SCM: SCMContainerPlacementCapacity#chooseNode chosen node is not
> removed from healthy list.
> --
>
> Key: HDFS-11860
> URL: https://issues.apache.org/jira/browse/HDFS-11860
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11860-HDFS-7240.001.patch
>
>
> This was caught in Jenkins run. After debugging, found the cause is the
> logic below where the node was returned without being removed from the
> healthy list for next round. As a result, there could be duplicated datanodes
> chosen with pipeline size smaller than expected. I will post a fix soon.
> {code}
> SCMContainerPlacementCapacity#chooseNode
> // There is a possibility that both numbers will be same.
> // if that is so, we just return the node.
> if (firstNodeNdx == secondNodeNdx) {
> return healthyNodes.get(firstNodeNdx);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode chosen node is not removed from healthy list.
[
https://issues.apache.org/jira/browse/HDFS-11860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11860:
--
Status: Patch Available (was: Open)
> Ozone: SCM: SCMContainerPlacementCapacity#chooseNode chosen node is not
> removed from healthy list.
> --
>
> Key: HDFS-11860
> URL: https://issues.apache.org/jira/browse/HDFS-11860
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11860-HDFS-7240.001.patch
>
>
> This was caught in Jenkins run. After debugging, found the cause is the
> logic below where the node was returned without being removed from the
> healthy list for next round. As a result, there could be duplicated datanodes
> chosen with pipeline size smaller than expected. I will post a fix soon.
> {code}
> SCMContainerPlacementCapacity#chooseNode
> // There is a possibility that both numbers will be same.
> // if that is so, we just return the node.
> if (firstNodeNdx == secondNodeNdx) {
> return healthyNodes.get(firstNodeNdx);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11860) Ozone: SCM: SCMContainerPlacementCapacity#chooseNode chosen node is not removed from healthy list.
Xiaoyu Yao created HDFS-11860:
-
Summary: Ozone: SCM: SCMContainerPlacementCapacity#chooseNode
chosen node is not removed from healthy list.
Key: HDFS-11860
URL: https://issues.apache.org/jira/browse/HDFS-11860
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
This was caught in Jenkins run. After debugging, found the cause is the
logic below where the node was returned without being removed from the healthy
list for next round. As a result, there could be duplicated datanodes chosen
with pipeline size smaller than expected. I will post a fix soon.
{code}
SCMContainerPlacementCapacity#chooseNode
// There is a possibility that both numbers will be same.
// if that is so, we just return the node.
if (firstNodeNdx == secondNodeNdx) {
return healthyNodes.get(firstNodeNdx);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11859) Ozone : separate blockLocationProtocol out of containerLocationProtocol
[
https://issues.apache.org/jira/browse/HDFS-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018511#comment-16018511
]
Hadoop QA commented on HDFS-11859:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
39s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
57s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.ozone.web.TestOzoneWebAccess |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.web.client.TestVolume |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.TestContainerOperations |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
| | hadoop.ozone.web.TestDistributedOzoneVolumes |
| | hadoop.ozone.scm.TestSCMMXBean |
| |
[jira] [Updated] (HDFS-11859) Ozone : separate blockLocationProtocol out of containerLocationProtocol
[
https://issues.apache.org/jira/browse/HDFS-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11859:
--
Attachment: HDFS-11859-HDFS-7240.004.patch
> Ozone : separate blockLocationProtocol out of containerLocationProtocol
> ---
>
> Key: HDFS-11859
> URL: https://issues.apache.org/jira/browse/HDFS-11859
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11859-HDFS-7240.001.patch,
> HDFS-11859-HDFS-7240.002.patch, HDFS-11859-HDFS-7240.003.patch,
> HDFS-11859-HDFS-7240.004.patch
>
>
> Currently StorageLocationProtcol contains two types of operations: container
> related operations and block related operations. Although there is
> {{ScmBlockLocationProtocol}} for block operations, only
> {{StorageContainerLocationProtocolServerSideTranslatorPB}} is making the
> distinguish.
> This JIRA tries to make the separation complete and thorough for all places.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11844) Ozone: Recover SCM state when SCM is restarted
[
https://issues.apache.org/jira/browse/HDFS-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018445#comment-16018445
]
Xiaoyu Yao commented on HDFS-11844:
---
Thanks [~cheersyang] for reporting the issue. ContainerManagerImpl manages
containers on DN side. It will be reloaded upon DN restart.
SCM maintains its own container DB (see ContainerMapping.java) and will reload
upon restart as well. However, the get container command works only against a
closed container. (according to the spec). I think we have an open JIRA for
implement get container CLI.
Can you elaborate on 4. "Get existing container state via SCM CLI, this step
will fail with container doesn't exist error."
> Ozone: Recover SCM state when SCM is restarted
> --
>
> Key: HDFS-11844
> URL: https://issues.apache.org/jira/browse/HDFS-11844
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, scm
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> SCM losses its state once being restarted. This issue can be found by a
> simple test with following steps
> # Start NN, DN, SCM
> # Create several containers via SCM CLI
> # Restart SCM
> # Get existing container state via SCM CLI, this step will fail with
> container doesn't exist error.
> {{ContainerManagerImpl}} maintains a cache of container mapping
> {{containerMap}}, if SCM is restarted, this information is lost. We need a
> way to restore the state from DB in a background thread.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11859) Ozone : separate blockLocationProtocol out of containerLocationProtocol
[
https://issues.apache.org/jira/browse/HDFS-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018444#comment-16018444
]
Hadoop QA commented on HDFS-11859:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
9s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.ozone.web.TestOzoneWebAccess |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.web.client.TestVolume |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.TestContainerOperations |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.ozone.web.TestDistributedOzoneVolumes |
| | hadoop.ozone.scm.TestSCMMXBean |
| |
[jira] [Commented] (HDFS-11770) Ozone: KSM: Add setVolumeProperty
[ https://issues.apache.org/jira/browse/HDFS-11770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018435#comment-16018435 ] Xiaoyu Yao commented on HDFS-11770: --- Thanks [~msingh] for the update with additional tests. The patch looks pretty good to me. Here a few minor issues: KeySpaceManagerProtocol.java NIT: line 70 extra "the" can be removed. KeySpaceManagerProtocolClientSideTranslatorPB.java NIT: line 210 same as above KeySpaceManager.java NIT: line 296 same as above VolumeManagerImpl.java Line 107-111: in delVolumeFromOwnerList() when all the volume of a user is deleted, should we remove the empty user entry instead of updating the empty user entry? DistributedStorageHandler.java Line 164-167: We should set the Quota in the VolumeInfo returned here to validate the SetQuota that has been implemented here. TestKeySpaceManager.java Can we add test for SetQuota with the getVolumeInfo support? Checkstyle from latest Jenkins. > Ozone: KSM: Add setVolumeProperty > - > > Key: HDFS-11770 > URL: https://issues.apache.org/jira/browse/HDFS-11770 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-11770-HDFS-7240.001.patch, > HDFS-11770-HDFS-7240.002.patch, HDFS-11770-HDFS-7240.003.patch, > HDFS-11770-HDFS-7240.005.patch > > > SetVolumeProperty allows Ozone administrators to change the ownership of a > volume and quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11859) Ozone : separate blockLocationProtocol out of containerLocationProtocol
[
https://issues.apache.org/jira/browse/HDFS-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-11859:
--
Attachment: HDFS-11859-HDFS-7240.003.patch
> Ozone : separate blockLocationProtocol out of containerLocationProtocol
> ---
>
> Key: HDFS-11859
> URL: https://issues.apache.org/jira/browse/HDFS-11859
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Chen Liang
>Assignee: Xiaoyu Yao
> Attachments: HDFS-11859-HDFS-7240.001.patch,
> HDFS-11859-HDFS-7240.002.patch, HDFS-11859-HDFS-7240.003.patch
>
>
> Currently StorageLocationProtcol contains two types of operations: container
> related operations and block related operations. Although there is
> {{ScmBlockLocationProtocol}} for block operations, only
> {{StorageContainerLocationProtocolServerSideTranslatorPB}} is making the
> distinguish.
> This JIRA tries to make the separation complete and thorough for all places.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11829) Backport HDFS-8312 to branch 2.8.1
[ https://issues.apache.org/jira/browse/HDFS-11829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11829: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.1 Release Note: HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang via Eric Yang) Status: Resolved (was: Patch Available) > Backport HDFS-8312 to branch 2.8.1 > -- > > Key: HDFS-11829 > URL: https://issues.apache.org/jira/browse/HDFS-11829 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Critical > Labels: trash > Fix For: 2.8.1 > > Attachments: HDFS-8312-branch-2.8.1.001.patch > > > Back port HDFS-8312 to branch-2.8 as it is a critical permission flaw. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11859) Ozone : separate blockLocationProtocol out of containerLocationProtocol
[
https://issues.apache.org/jira/browse/HDFS-11859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16018345#comment-16018345
]
Hadoop QA commented on HDFS-11859:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in HDFS-7240
has 2 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.scm.TestSCMMXBean |
| | hadoop.ozone.web.TestDistributedOzoneVolumes |
| | hadoop.cblock.TestCBlockCLI |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.ozone.web.client.TestVolume |
| |
