[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554619#comment-16554619 ] Tsz Wo Nicholas Sze commented on HDDS-181: -- I have filed HDDS-288. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch, HDDS-181.05.patch, HDDS-181.06.patch, HDDS-181.07.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554616#comment-16554616
]
Tsz Wo Nicholas Sze commented on HDDS-181:
--
- OpenContainerBlockMap should not be synchronized for a better performance.
- There is a memory leak in removeContainer(..) -- it sets the entry to null
instead of removing it.
- addChunkToMap may add the same chunk twice. See the comments below.
{code}
keyDataSet.putIfAbsent(blockID.getLocalID(), getKeyData(info, blockID));
// (1) when id is absent, it puts
keyDataSet.computeIfPresent(blockID.getLocalID(), (key, value) -> { //
(2) now, the id is present, it adds again.
value.addChunk(info);
return value;
});
{code}
> CloseContainer should commit all pending open Keys on a datanode
>
>
> Key: HDDS-181
> URL: https://issues.apache.org/jira/browse/HDDS-181
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch,
> HDDS-181.04.patch, HDDS-181.05.patch, HDDS-181.06.patch, HDDS-181.07.patch
>
>
> A close container command arrives in the Datanode by the SCM heartBeat
> response.It will then be queued up over the ratis pipeline. Once the command
> execution starts inside the Datanode, it will mark the container in CLOSING
> State. All the pending open keys for the container now will be committed
> followed by the transition of the container state from CLOSING to CLOSED. For
> achieving this, all the open keys for a container need to be tracked.
> This Jira aims to address this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552304#comment-16552304 ] Hudson commented on HDDS-181: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14614 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14614/]) HDDS-181. CloseContainer should commit all pending open Keys on a (msingh: rev bbe2f6225ea500651de04c064f7b847be18e5b66) * (edit) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/container/common/helpers/KeyData.java * (add) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestCloseContainerHandler.java * (add) hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/OpenContainerBlockMap.java > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch, HDDS-181.05.patch, HDDS-181.06.patch, HDDS-181.07.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551941#comment-16551941 ] Mukul Kumar Singh commented on HDDS-181: Thanks for the updated patch [~shashikant]. +1, The v7 patch looks good to me. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch, HDDS-181.05.patch, HDDS-181.06.patch, HDDS-181.07.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551654#comment-16551654
]
genericqa commented on HDDS-181:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
7s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} container-service in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 36s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}132m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestBlockDeletingService |
| |
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-181 |
| JIRA Patch URL |
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551649#comment-16551649
]
genericqa commented on HDDS-181:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 28m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} container-service in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 43s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestBlockDeletingService |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-181 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932548/HDDS-181.06.patch |
|
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551612#comment-16551612 ] Shashikant Banerjee commented on HDDS-181: -- patch v7 fixes a bug in "handleWriteChunk" in KeyValueHandler where we need to update the openContainerBlockMap only while WRITE_DATA stage not during COMMIT_DATA phase as it will lead to duplicating the chunkInfo in the openContainerBlockMap. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch, HDDS-181.05.patch, HDDS-181.06.patch, HDDS-181.07.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551585#comment-16551585
]
genericqa commented on HDDS-181:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 55s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} container-service in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 50s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.ozone.container.common.TestBlockDeletingService |
| |
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
|
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| |
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551552#comment-16551552 ] Shashikant Banerjee commented on HDDS-181: -- Thanks [~msingh], for the review. patch v5 addresses your review comments. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch, HDDS-181.05.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551144#comment-16551144 ] Mukul Kumar Singh commented on HDDS-181: Thanks for the updated patch [~shashikant]. Please find my updated comments as following. 1) I feel that OpenContainerBlockMap can be initialized inside the KeyValueContainer. As this is a per container type property. 2) KeyValueHandler.java:378, the comment can be re-phrased to "remove the container from open block map once, all the blocks have been committed and the container is closed" 3) KeyValueHandler.java:411, lets move the commit key before we update the counter at 409. 4) KeyValueHandler.java:426, NIT: pendinKeys -> pendingKeys. 5) KeyValueHandler.java:491, I feel this is not required. If a block has not been comitted, then the client is never call deleteblock for the block. At this time, the entries in the container if any are present are garbage. Lets raise a jira to deal with this garbage later. 6) OpenContainerBlockMap.java, removeContainer and removeContainerFromKeyMap are both removing the container. Please remove of these functions. Also in removeFromKeyMap, the call to remove the container, can call one of these functions to remove the final container. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548799#comment-16548799 ] Shashikant Banerjee commented on HDDS-181: -- Resubmitting the patch for getting jenkins run. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch, > HDDS-181.04.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548132#comment-16548132
]
genericqa commented on HDDS-181:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
7s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDDS-181 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932062/HDDS-181.04.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/538/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> CloseContainer should commit all pending open Keys on a datanode
>
>
> Key: HDDS-181
> URL: https://issues.apache.org/jira/browse/HDDS-181
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch,
> HDDS-181.04.patch
>
>
> A close container command arrives in the Datanode by the SCM heartBeat
> response.It will then be queued up over the ratis pipeline. Once the command
> execution starts inside the Datanode, it will mark the container in CLOSING
> State. All the pending open keys for the container now will be committed
> followed by the transition of the container state from CLOSING to CLOSED. For
> achieving this, all the open keys for a container need to be tracked.
> This Jira aims to address this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547651#comment-16547651
]
Shashikant Banerjee commented on HDDS-181:
--
Thanks [~msingh] for the review.
{quote}At this time, let us also raise jira to track keys which are never
committed by client because the client went away. Such keys should be garbage
collected.
{quote}
Right now, on a Datanode all the open keys in a container will be tracked by
openContainerBlockMap. Even if the client goes away, the pending keys will
still be there in the openContainerBlockMap and will be committed once the
container gets closed. So essenially, this map will have those keys for which
the client never issue a commit operation till the container close arrives.
Let's discuss on this more in order to gain more clarity on this.
{quote}The current patch maintains an in-memory map of the open keys. This
information should be persisted as well to allow the keys to be committed after
a restart.
{quote}
I think we can reconstruct this map by basically reading the individual chunk
file and updating the sizes in the map for all the blocks for which the block
entry does not exists in the container Db on datanode. This info will be lost
only when the Datanode dies, which will trigger closing of open containers on
other datanodes and the replicating the container by Replica Manger. In such
cases , if the datanode comes back up, we need this info for garbage collection
and we might not need this info for committing the open blocks once again. I
think we can do this in a separate jira(for now, It has been mentioned as a
TODO item in the code).
Rest all the review comments are addressed in patch v4.
> CloseContainer should commit all pending open Keys on a datanode
>
>
> Key: HDDS-181
> URL: https://issues.apache.org/jira/browse/HDDS-181
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch
>
>
> A close container command arrives in the Datanode by the SCM heartBeat
> response.It will then be queued up over the ratis pipeline. Once the command
> execution starts inside the Datanode, it will mark the container in CLOSING
> State. All the pending open keys for the container now will be committed
> followed by the transition of the container state from CLOSING to CLOSED. For
> achieving this, all the open keys for a container need to be tracked.
> This Jira aims to address this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546152#comment-16546152 ] Mukul Kumar Singh commented on HDDS-181: Thanks for working on this issue [~shashikant]. The patch generally looks good to me. Please find my comment as following General comments: 1) At this time, let us also raise jira to track keys which are never committed by client because the client went away. Such keys should be garbage collected. 2) The current patch maintains an in-memory map of the open keys. This information should be persisted as well to allow the keys to be committed after a restart. KeyValueHandler.java:344, The deletion should happen before the lock is released. KeyValueHandler.java:412, commitKey internally calls putkey twice, this should be done only once. KeyValueHandler.java:493, removeFromKeyMap should be called before the keyManagerDelete as the keyManager delete will fail for any kind of pending keys. KeyValueHandler.java:649, lets add a comment here explaining that as the chunks are committed as part of putSmallFile, there is no need to maintain these keys in the openKeyMap. OpenContainerBlockMap.java: Lets add a constructor to this class. OpenContainerBlockMap.java: Lets keep the signature of addChunkToMap and deleteChunkFromMap with blockID first and info as second parameter. Nits: Handler.java: 64, more than 80 characters. KeyData.java:144, typo in removes KeyData.java: 151-154, misaligned comments > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[
https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542092#comment-16542092
]
genericqa commented on HDDS-181:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 34m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 35m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 35m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} tools in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 39s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.TestStorageContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDDS-181 |
| JIRA Patch URL |
[jira] [Commented] (HDDS-181) CloseContainer should commit all pending open Keys on a datanode
[ https://issues.apache.org/jira/browse/HDDS-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539885#comment-16539885 ] Shashikant Banerjee commented on HDDS-181: -- Patch v3 fixes the findBug and one test failure related to the patch. The other test failure is not related. > CloseContainer should commit all pending open Keys on a datanode > > > Key: HDDS-181 > URL: https://issues.apache.org/jira/browse/HDDS-181 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-181.01.patch, HDDS-181.02.patch, HDDS-181.03.patch > > > A close container command arrives in the Datanode by the SCM heartBeat > response.It will then be queued up over the ratis pipeline. Once the command > execution starts inside the Datanode, it will mark the container in CLOSING > State. All the pending open keys for the container now will be committed > followed by the transition of the container state from CLOSING to CLOSED. For > achieving this, all the open keys for a container need to be tracked. > This Jira aims to address this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
