[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660638#comment-16660638 ] Nanda kumar commented on HDDS-676: -- Thanks [~shashikant] for providing the patch for ozone-0.3 branch. +1, LGTM. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-676-ozone-0.3.000.patch, HDDS-676.001.patch, > HDDS-676.002.patch, HDDS-676.003.patch, HDDS-676.004.patch, > HDDS-676.005.patch, HDDS-676.006.patch, HDDS-676.007.patch, HDDS-676.008.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660622#comment-16660622
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} ozone-0.3 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
41s{color} | {color:green} ozone-0.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
18s{color} | {color:green} ozone-0.3 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
3s{color} | {color:green} ozone-0.3 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} ozone-0.3 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} ozone-0.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
12s{color} | {color:green} ozone-0.3 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
0s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 4s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.om.TestOmMetrics |
| | hadoop.hdds.scm.container.TestContainerStateManagerIntegration |
| |
hadoop.ozone.container.common.statemachine.commandhandler.Te
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659837#comment-16659837
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 7m
0s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 0s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 7s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ozShell.TestOzoneShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDDS-676 |
| JIRA Patch URL |
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659818#comment-16659818 ] Hudson commented on HDDS-676: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15289 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15289/]) HDDS-676. Enable Read from open Containers via Standalone Protocol. (aengineer: rev e3cca1204874d37b48095c8ff9a44c1f16dc15ed) * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientRatis.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ChunkGroupInputStream.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientManager.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ChunkGroupOutputStream.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/Pipeline.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneCluster.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClient.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, > HDDS-676.006.patch, HDDS-676.007.patch, HDDS-676.008.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659767#comment-16659767 ] Anu Engineer commented on HDDS-676: --- +1, I will commit this shortly. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, > HDDS-676.006.patch, HDDS-676.007.patch, HDDS-676.008.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659673#comment-16659673 ] Shashikant Banerjee commented on HDDS-676: -- Patch v8 addresses the review comments in *testPutKeyAndGetKey.* > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, > HDDS-676.006.patch, HDDS-676.007.patch, HDDS-676.008.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659639#comment-16659639
]
Shashikant Banerjee commented on HDDS-676:
--
Thanks[~anu], for the review comments.
{code:java}
I agree with this premise; that is we only talk to next data node if we get a
failure on the first data node.
If that is the case, do we need all this Async framework changes, hash tables
etc?
{code}
if we get a failure/connection issues with one of a datanode, we failover to
the next datanode. If we don't maintain the state of the active channels for
communication in the hash map, so that when we close the client we close all
the conections. If we don't maintain the state, we need to close the
connections in active read path as a part of handling the exception. Connection
Errors can be transient. Also, multiple ozone clients can use the same
XceiverClient instance as we maintain a client cache, so immediately closing
the connection in case one client op fails.
HashMap will also be helpful if we get the leader info cached, so that we will
use that specific channel to execute first.
Regarding the async framework change, there is functionally no change in the
code.It has been just split to 2 functions so while executing the command we
execute on a specific channel.
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch,
> HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch,
> HDDS-676.006.patch, HDDS-676.007.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659617#comment-16659617
]
Anu Engineer commented on HDDS-676:
---
[~shashikant] The changes look much better, thanks for separating this into 2
different patches.
Some very minor comments below:
*testPutKeyAndGetKey*
1. {{long currentTime = Time.now();}} // Unused variable.
2. {{OzoneKeyDetails keyDetails = (OzoneKeyDetails)
bucket.getKey(keyName);}} //Casting is not needed
3. It might be a good idea to verify the exception string.
{{catch (Exception e)}}
*XcieverClientGrpc.java*
{{responseProto = sendCommandAsync(request, dn).get();}}
I agree with this premise; that is we only talk to next data node if we get a
failure on the first data node.
If that is the case, do we need all this Async framework changes, hash tables
etc?
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch,
> HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch,
> HDDS-676.006.patch, HDDS-676.007.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658997#comment-16658997
]
Hadoop QA commented on HDDS-676:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
57s{color} | {color:green} integration-test in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDDS-676 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944994/HDDS-676.007.patch |
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658831#comment-16658831 ] Shashikant Banerjee commented on HDDS-676: -- Thanks [~anu], for the review comments. I have created a new Jira HDDS-708 which will have the changes required for reading and validating BCSID while reading the block from container db on datanodes. Patch v7 attached has the changes required to enable read from open containers by using standalone grpc client. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, > HDDS-676.006.patch, HDDS-676.007.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658592#comment-16658592 ] Anu Engineer commented on HDDS-676: --- I also feel that this patch mixes tow independent notions. The actual usage of BCSID and the changes in grpc client. Can you please break this up into 2 separate JIRAs, please? First one, that focuses only on the BCSID issues and another one that actually allows us to read those changes using grpc client? > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, HDDS-676.006.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658586#comment-16658586 ] Anu Engineer commented on HDDS-676: --- bq. I feel blockID is used to uniquely identify a block in the system but the BCSID of a block may be different in different non closed container replica. BCSID should not be necessarily a part of BlockId. I am trying to understand this more. Block Commit Sequence ID is associated to a block? That let us identity the generation number of a block. Why is it not the part of the identity of a block? > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, HDDS-676.006.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657922#comment-16657922
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
18s{color} | {color:green} integration-test in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{c
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657885#comment-16657885 ] Shashikant Banerjee commented on HDDS-676: -- patch v6 addresses the checkstyle issues. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, HDDS-676.006.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657831#comment-16657831
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 21m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 21m 16s{color}
| {color:red} root generated 3 new + 1312 unchanged - 0 fixed = 1315 total (was
1312) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 45s{color} | {color:orange} root: The patch generated 3 new + 3 unchanged -
0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
52s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 47s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657779#comment-16657779
]
Shashikant Banerjee commented on HDDS-676:
--
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different open container replica. BCSID should
not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i don't have this block for an open container
replica, It is difficult to figure out at client whether block got deleted or
the block commit transaction has not yet happened on this node(in case delete
is allowed on open containers). But for closed container replica we may not
need to retry. I think we need to think through this more as the retry
behaviour might be different for open and closed container replica here.
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch,
> HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657454#comment-16657454
]
Anu Engineer commented on HDDS-676:
---
1. message GetBlockRequestProto {
required DatanodeBlockID blockID = 1;
*required* uint64 blockCommitSequenceId = 2;
}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block -- say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.
2. More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
3. BlockManagerImpl.java#putBlock
{code}
long blockCommitSequenceId = data.getBlockCommitSequenceId();
long blockCommitSequenceIdValue = getBlockCommitSequenceId(db);
{code}
My apologies, but this code gives me a headache. I understand the context and
the why, But a function with variables *blockCommitSequenceId*,
*blockCommitSequenceIdValue* and *blockCommitSequenceIdKey* causes a stack
overflow for me :(
4. BlockManagerImpl.java#getBlock:Line 155
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
5. ContainerProtcolCalls.java#readSmallFile:Line 394
{code}
// by default, set the bcsId to be 0
.setBlockCommitSequenceId(0);
{code}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
// default blockCommitSequenceId for any block is 0. It the putBlock
// request is not coming via Ratis(for test scenarios), it will be 0.
// In such cases, we should overwrite the block as well
if (blockCommitSequenceIdValue != null && blockCommitSequenceId != 0) {
if (blockCommitSequenceId <= Longs
.fromByteArray(blockCommitSequenceIdValue)) {
if (blockCommitSequenceId != 0) {
if (blockCommitSequenceId <= blockCommitSequenceIdValue) {
// Since the blockCommitSequenceId stored in the db is greater than
// equal to blockCommitSequenceId to be updated, it means the putBlock
// transaction is reapplied in the ContainerStateMachine on restart.
// It also implies that the given block must already exist in the db.
// just log and return
LOG.warn("blockCommitSequenceId " + Longs
.fromByteArray(blockCommitSequenceIdValue)
LOG.warn("blockCommitSequenceId " + blockCommitSequenceIdValue
+ " in the Container Db is greater than" + " the supplied value "
+ blockCommitSequenceId + " .Ignoring it");
return data.getSize();
}
}
{code}
6. XceieverClientGrpc.java#sendCommand Can we please separate out the trying
for different datanodes vs. connecting and reading data. That make this into 2
functions. One that connects and reads, another which tries all nodes till we
get a successful read.
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
7. nit: The fact that we have to maintain dnIndex and Size might be a good
indication that we need old school for-loop at line 154 instead of a for-each
loop.
8. Nit: XceiverClientRatis#watchForCommit - if this is a test only function
perhaps have in a test helper file?
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch,
> HDDS-676.003.patch, HDDS-676.004.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656786#comment-16656786 ] Shashikant Banerjee commented on HDDS-676: -- Test failures and fundbug are not related to the patch. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656783#comment-16656783
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 29m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 37s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
52s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 32s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} |
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656483#comment-16656483 ] Shashikant Banerjee commented on HDDS-676: -- Patch v4 addresses the javaodc, checkstyle and failed test cases. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656415#comment-16656415
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 21m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 16s{color} | {color:orange} root: The patch generated 20 new + 3 unchanged -
0 fixed = 23 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
39s{color} | {color:red} hadoop-hdds_container-service generated 1 new + 4
unchanged - 0 fixed = 5 total (was 4) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 52s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
28s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 20s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656361#comment-16656361
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 18m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 33s{color} | {color:orange} root: The patch generated 20 new + 3 unchanged -
0 fixed = 23 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
40s{color} | {color:red} hadoop-hdds_container-service generated 1 new + 4
unchanged - 0 fixed = 5 total (was 4) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 11s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {col
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656286#comment-16656286 ] Shashikant Banerjee commented on HDDS-676: -- cleaned up ReadSmallFile command handling related changes in Patch v3. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656270#comment-16656270
]
Shashikant Banerjee commented on HDDS-676:
--
Thanks [~jnp], for the review comments.
{code:java}
What is the reason for changing clientCache key from PipelineId to string?
{code}
By default, SCM always gives a Ratis pipeline for open containers and when a
XceiverClient instance gets created, it always placed in the clinet cache based
on the pipeline ID. Since, while doing a Read Op we always want to use a
Standalone pipeline with the same pipelineId which SCM provides, the idea is
get a XceiverClientGrpc instance with the same pipleineId, thus having the same
set of datanodes which will be used by Ratis pipeline as well having the same
pipleineID, Changing key in ClinetCahe from pipelineId to a String which is
combination of pipelineId and type gives us the flexiblity to have create two
different types of pipeline with the same pipelineID.
{code:java}
The changes in XceiverClientRatis are only for testing?
{code}
Yes, for now.
{code:java}
It is minor but I feel we should not make type mutable in the Pipeline class.
We could clone the Pipeline object to change the type.
{code}
This will be addressed with HDDS-694.
{code:java}
In ContainerStateMachine changes don't look related to this Jira as they are
about put-small-files. If yes, we should put them in a separate jira.
{code}
Opened HDDS-697 for the same.
Rest of the review comments are addressed in the patch along with the
checkstyle fixes. Javadoc issues seem to be unrelated.
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655990#comment-16655990
]
Jitendra Nath Pandey commented on HDDS-676:
---
# What is the reason for changing clientCache key from PipelineId to string?
# XceiverClientGrpc#isConnected : The three channels may each be in different
state.
# dnIndex is used in only one method, no need to be a class attribute.
# In XceiverClientGrpc, we should create the channel and stub, just before
making the cal in {{sendCommand}}, otherwise in majority of the cases, we would
be opening unnecessary connection.
# The changes in XceiverClientRatis are only for testing?
# It is minor but I feel we should not make {{type}} mutable in the Pipeline
class. We could clone the {{Pipeline}} object to change the type.
# In {{ContainerStateMachine}} changes don't look related to this Jira as they
are about put-small-files. If yes, we should put them in a separate jira.
# Instead of INVALID_BCSID , we should call it UNKNOWN_BCSID. Client has
obtained it from OM, so it is likely a valid one, but container doesn't have
it. We should also put the containerBcs in the error message.
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654995#comment-16654995
]
Tsz Wo Nicholas Sze commented on HDDS-676:
--
{code}
[WARNING]
Dependency convergence error for
org.apache.hadoop:hadoop-annotations:3.3.0-SNAPSHOT paths to dependency are:
+-org.apache.hadoop:hadoop-hdds-common:0.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-common:3.3.0-20181017.235917-140
+-org.apache.hadoop:hadoop-annotations:3.3.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-hdds-common:0.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-common:3.3.0-20181017.235917-140
+-org.apache.hadoop:hadoop-annotations:3.3.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-hdds-common:0.4.0-SNAPSHOT
+-org.apache.hadoop:hadoop-annotations:3.3.0-20181017.235840-140
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
failed with message:
{code}
It seems that the pom files have some bugs; filed HDDS-691.
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654532#comment-16654532
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 49m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 47m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
11s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
15s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
44s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
45s{color} | {color:red} client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
41s{color} | {color:red} client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
45s{color} | {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 34m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 34m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 34m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
5m 33s{color} | {color:orange} root: The patch generated 23 new + 3 unchanged -
0 fixed = 26 total (was 3) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
13s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
5s{color} | {color:red} client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m
19s{color} | {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} container-service in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
30s{color} | {color:red} client in the patch failed. {color} |
| {c
