[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455179#comment-16455179
]
Hudson commented on HDFS-11580:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14070/])
HDFS-11580. Ozone: Support asynchronus client API for SCM and (omalley: rev
9cf40547ce5a2aafb9cd7da8a0b614e1168ab068)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/storage/ContainerProtocolCalls.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClient.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientHandler.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientRatis.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientSpi.java
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16450928#comment-16450928
]
Hudson commented on HDFS-11580:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14057/])
HDFS-11580. Ozone: Support asynchronus client API for SCM and (aengineer: rev
92945d01b236f32ba59789d6c022c030df0deb34)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/storage/ContainerProtocolCalls.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientRatis.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClient.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientHandler.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientSpi.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Anu Engineer
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109541#comment-16109541
]
Chen Liang commented on HDFS-11580:
---
Thanks [~anu] for the follow-up! makes to me.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109526#comment-16109526
]
Anu Engineer commented on HDFS-11580:
-
[~vagarychen] The following tests will fail if we enforce non-null or empty
trace ID check. I am going to pass this for time being. We can always file an
independent JIRA to fix this if needed. From the point of view of this JIRA, an
empty or null trace ID is valid.
{code}
TestContainerMetrics.testContainerMetrics
TestOzoneContainer.testBothGetandPutSmallFile
TestOzoneContainer.testCloseContainer
TestOzoneContainer.testOzoneContainerViaDataNode
TestOzoneContainer.testXcieverClientAsync
TestOzoneContainer.testCreateOzoneContainer
TestOzoneContainer.testDeleteContainer
TestContainerServer.testClientServer
TestContainerServer.testClientServerWithContainerDispatcher
TestKeys.testPutAndGetKeyWithDnRestart
{code}
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109426#comment-16109426
]
Anu Engineer commented on HDFS-11580:
-
[~vagarychen] Thanks for your comment. It is very insightful. Let me share my
experience trying to solve this issue.
bq. if it does not have a requestID set, before making the async call, the code
should either throw an exception,
I did that change while I was working on this patch and realized something
funny. +An empty trace id is a valid key+. Here is the issue we are trying to
solve, if you send 2 outstanding requests with the same trace ID we will fail.
That is, we cannot keep 2 requests with the same ID in flight *concurrently*.
But, you can certainly send the same request ID twice, as soon as the first one
is finished processing. That is, Two trace IDs which are same is valid if
processed *serially*.
So in reality, even though we can seem to care for a "valid" trace ID, we just
care that no duplicate trace IDs exist at the same time. I agree forcing the
trace IDs to non-null does reduce the probability of having "duplicate" trace
IDs in flight, but it will not fix it.
I am going to make the change that you proposed, but if too many tests are
failing I would like to postpone that change to a different JIRA Since this
patch just cares about the uniqueness of the trace ID, not validity. In other
words, right now, Empty or Null trace ID is a valid trace ID as long as they
are not concurrently executing.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109394#comment-16109394
]
Anu Engineer commented on HDFS-11580:
-
[~nandakumar131] Thanks for the comments. I completely agree with your comment.
bq. default asynchronous and we can make XceiverClient#sendCommand call into
XceiverClient#sendCommandAsync(request).get() which will
I would like to do that in some later patch since if I do that now, I have to
make corresponding changes in ContainerProtocolCalls, which will make this
patch larger and we will we have to rework changes in ContainerProtocolCalls
later. My aim was to illustrate the idea of making the async patch first class,
we need a series of patches on top of this to get all the way to Corona. So I
took the path of leaving the signatures as-is for time being.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109385#comment-16109385
]
Chen Liang commented on HDFS-11580:
---
Thanks [~anu] for the explanation and the patch! LGTM, only one thing, since we
are largely relying on requestID, I think we should do some checking when a
message gets passed in, i.e. if it does not have a requestID set, before making
the async call, the code should either throw an exception, or quietly pick up a
random requestID itself.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108675#comment-16108675
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 16 new
+ 1 unchanged - 0 fixed = 17 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.ksm.TestKSMMetrcis |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
| | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879787/HDFS-11580-HDFS-7240.012.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7ff660be66c2 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchproces
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108555#comment-16108555
]
Nandakumar commented on HDFS-11580:
---
Thanks [~anu] for the explanation and patch.
The patch looks good to me, below are some minor comments
Instead of having two calls {{sendCommand}} and {{sendCommandAsync}} in
{{XceiverClientHandler}}, we can have single
{{XceiverClientHandler#sendCommand}} which is by default asynchronous and we
can make {{XceiverClient#sendCommand}} call into
{{XceiverClient#sendCommandAsync(request).get()}} which will make the call
synchronous.
NIT: XceiverClientHandler:120 typo in "completed"
NIT: XceiverClient :126-130 indentation
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108229#comment-16108229
]
Anu Engineer commented on HDFS-11580:
-
[~vagarychen], [~linyiqun], [~nanda], [~msingh] Thanks for the patches and code
reviews. I think one of my old changes is creating a road block in this work.
Allow me to explain.
# When I first wrote this code, I wanted to offer a synchronous interface to
make it easy to write clients. So I painfully wrapped the Async interface
offered by Netty into a Sync one -- Probably due to a mistaken impression that
it will make the life of programmers easy.
# Then when we came to do the async work -- what we should have done in the
first place is to make the underlying netty's async interface first class,
which is currently hidden under the {{sendCommand}} function.
# Instead, since the code already offers a synchronous interface, this code
patch tries to create a thread pool and attempts to call into the "Sync"
interface offered by the XcieverClient. I propose that we *don't* use a thread
pool to make the "sync" -> "async", instead, we expose the async interface
as-is.
# Since I wrote the original code, I propose that I post a patch with the idea
that I am talking about so that we can see what I am proposing. Please let me
know if you have any concerns or questions.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108049#comment-16108049
]
Chen Liang commented on HDFS-11580:
---
Failed tests are unrelated.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108002#comment-16108002
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | hadoop.ozone.web.client.TestKeys |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879691/HDFS-11580-HDFS-7240.011.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1eab87e93add 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 1f5353d |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20502/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://build
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103518#comment-16103518
]
Chen Liang commented on HDFS-11580:
---
Thanks [~nandakumar131] and [~linyiqun] for the review! Interesting founding
about the {{OverWriteRequested}} flag, I'll look into it. Thanks for the catch.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103029#comment-16103029
]
Yiqun Lin commented on HDFS-11580:
--
Thanks [~nandakumar131] and [~vagarychen] for looking into this. I totally
agree with what you said. Seems I wrote some duplicate async call codes, :P.
Hi [~vagarychen], the latest patch almost looks good to me. Just caught one
thing:
{code}
-KeyValue keyValue = KeyValue.newBuilder()
-.setKey("OverWriteRequested").setValue("true").build();
ChunkInfo chunk = ChunkInfo
.newBuilder()
.setChunkName(key + "_chunk")
.setOffset(0)
.setLen(data.length)
-.addMetadata(keyValue)
.build();
{code}
Here we remove {{OverWriteRequested}} flag, that will lead a different
behaviour I think. That is say if we write a exist key name file. That will be
rejected and throws IOException.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102649#comment-16102649
]
Nandakumar commented on HDFS-11580:
---
Thanks [~vagarychen] for updating the patch.
+1 for v010 patch, LGTM.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102391#comment-16102391
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.web.client.TestKeys |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879039/HDFS-11580-HDFS-7240.010.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7356df870f9c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 04b3906 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20419/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.a
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102178#comment-16102178
]
Chen Liang commented on HDFS-11580:
---
Thanks for the reply [~nandakumar131]. I did not realize {{sendCommand}} is
calling {{waitForResponse}} too. In this case then, I think it should be right
to make the change like you suggested... Any comments [~linyiqun]?
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102132#comment-16102132
]
Nandakumar commented on HDFS-11580:
---
I totally agree with you [~vagarychen].
But in our case, since we are using the same {{waitForResponse}} logic in
{{XceiverClientHandler#sendCommand}}, we can be sure that the below code will
return proper response
{code:title=ContainerProtocolCalls.java}
public static CompletableFuture readSmallFileAsync(
XceiverClientSpi client, String containerName, String key, String traceID)
throws IOException {
return CompletableFuture
.supplyAsync(() -> {
try {
return readSmallFile(client, containerName, key, traceID);
} catch (IOException e) {
LOG.error("read small file async call failed.", e);
}
return null;
});
}
{code}
since {{XceiverClientHandler#sendCommand}} uses {{waitForResponse}} and gets
the response using traceID we will get correct response through
{{CompletableFuture}}.
The problem you explained is not only for {{CompletableFuture}} but we will
also get it if same {{XceiverClient}} is used by multiple threads.
I faced this issue while testing Corona, which used multiple threads to create
containers and put data into ozone with same client, {{waitForResponse}} logic
will solve this issue.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101998#comment-16101998
]
Chen Liang commented on HDFS-11580:
---
Thanks [~nandakumar131] for the comments!
As far as I remember how this JIRA has come along, the potential issue here was
that it was not clear whether the response will match the request. For example,
say, the async readfile is calling readfilesync directly, i.e.
{code}
Data readFileAsync(..) {
return readFileSync(...)
}
{code}
And we are calling readSmallFile twice, to read file1 and file2.
{code}
readFileAsync(file1)
readFileAsync(file2)
{code}
There is no guarantee which request arrives and returns first. So the
readFileSync(file2) can potentially return earlier, if so, can we be sure it's
returned gets delivered to second call definitely, instead of the first?
Essentially, the issue is, if there are two simultaneous calls of
{{supplyAsync}} with the same supplier, and the supplier returns one time,
which of the two {{supplyAsync}} will return?
This is related to {{CompletableFuture}}'s feature I haven't digged into or
tested. Totally possible that it be will handled well in some way. If you (or
anyone looking at this) have anything to add, please feel free to share. But as
I follow this JIRA, it was because of this, the implementation is that
{{sendCommandAsync}} is calling {{waitForResponse}}, which has an internal map
that matches the response to the request.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101624#comment-16101624
]
Nandakumar commented on HDFS-11580:
---
Thanks [~linyiqun] for working on this.
Correct me if my understanding is wrong.
{{ContainerProtocolCalls}} has {{*Async}} methods which is used for async calls
(we have corresponding synchronized calls for all of the operations). Inside
{{ContainerProtocolCalls#*Async}} methods we call
{{XceiverClient#sendCommandAsync}}, and use {{Future#thenApply}} for
{{validateContainerResponse}}.
Since we are creating {{CompletableFuture}} in
{{ContainerProtocolCalls#*Async}} why can't we use
{{XceiverClient#sendCommand}} directly instead of defining new
{{sendCommandAsync}} in {{XceiverClient}} and {{XceiverClientHandler}}.
Take the below {{createContainerAsync}} for example
{code}
public static CompletableFuture createContainerAsync(
XceiverClientSpi client, String traceID) throws IOException {
return CompletableFuture
.supplyAsync(() -> {
try {
createContainer(client, traceID);
} catch (IOException e) {
LOG.error("create container async call failed.", e);
}
return null;
});
}
{code}
won't the above change make {{createContainerAsync}} an asynchronous call? And
we don't have to repeat request, and response builder code in many places.
{{*Async}} can internally call synchronized calls inside a CompletableFuture
which will make it async.
Am I missing something here?
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058819#comment-16058819
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
0 unchanged - 1 fixed = 7 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| Timed out junit tests |
org.apache.hadoop.ozone.container.ozoneimpl.TestRatisManager |
| | org.apache.hadoop.hdfs.TestDFSClientRetries |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874001/HDFS-11580-HDFS-7240.009.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5dba7bd13738 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / c77d7f9 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058472#comment-16058472
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 0 unchanged - 1 fixed = 10 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
20s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.ozone.scm.TestXceiverClientManager |
| Timed out junit tests | org.apache.hadoop.hdfs.TestDFSClientRetries |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12873952/HDFS-11580-HDFS-7240.008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0984c894f70b 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b9a6441 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058466#comment-16058466
]
Chen Liang commented on HDFS-11580:
---
To [~linyiqun] and anyone reviewing this. There are several places with
{code}
try {
validateContainerResponse(response);
} catch (StorageContainerException ignored) {
}
{code}
The {{StorageContainerException}} contains an error code which I think we
should expose to caller somehow. I spent quite some time looking at how to
expose exception from callbacks (e.g. {{thenApply}} we use here) but found
nothing yet...alternatively we may be able to return the error code but not in
an exception, but still not sure how to. Any thoughts?
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996507#comment-15996507
]
Yiqun Lin commented on HDFS-11580:
--
I took some time to look into the failure test
{{TestContainerSmallFile#testCorrectnessWithMultipleAsyncCalls}}. The test is
not always failed, when I reduced the value {{numRequest}} to a smaller value.
It can pass in my local. I'm not so sure if the running test used out of my
local memory and then leads the failure.
In addition, the existing codes can also ensure the read small file commands
are enqueued in parallel, right? How do you think, [~msingh]? Could you share
your thoughts on this?
{code}
+// send 20 read small file requests and add response future into a list
+for (int i = 0; i < numRequest; i++) {
+ returnFuture3 = ContainerProtocolCalls.readSmallFileAsync(client,
+ containerName, "key" + i, traceID + i);
+ futureList.add(returnFuture3);
+}
+
+// Invoke future.get() to get the response and check
+// if the content returned is correct.
+for (int i = 0; i < numRequest; i++) {
+ returnFuture3 = futureList.get(i);
{code}
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994693#comment-15994693
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
3s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 1 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.cblock.TestLocalBlockCache |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| Timed out junit tests | org.apache.hadoop.ozone.scm.TestContainerSmallFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866151/HDFS-11580-HDFS-7240.007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux eaaf45ace647 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 6def74c |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/19294/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994392#comment-15994392
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
0 unchanged - 1 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
36s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Possible null pointer dereference of response in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtos$ContainerCommandResponseProto)
Dereferenced at ContainerProtocolCalls.java:response in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtos$ContainerCommandResponseProto)
Dereferenced at ContainerProtocolCalls.java:[line 752] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.cblock.TestLocalBlockCache |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| Timed out junit tests | org.apache.hadoop.ozone.scm.TestContainerSmallFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12866122/HDFS-11580-HDFS-7240.006.patch
|
| Op
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994323#comment-15994323
]
Yiqun Lin commented on HDFS-11580:
--
{quote}
It's not a good way to generate id everywhere. It would be better to use a
special id generator to help us generate unique id. I will file a new ticket to
for this.
{quote}
Had filed JIRA HDFS-11744 for this. Any further comments are welcomed there.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992773#comment-15992773
]
Mukul Kumar Singh commented on HDFS-11580:
--
Thanks Yiqun Lin for the updated patch. Following are my comments
1) Thanks for dividing the apis into sync and async. I feel that for sync apis,
lets use the previous function without any change. This is to ensure that
current behavior doesn't break.
2) Also can we add an extra capability to {{ XceiverClientHandler }} which
initializes it as either sync or async. Only a sync client will be able to send
sync command and vice versa.
3) in {{ testCorrectnessWithMultipleAsyncCalls }}, I feel that the return
future should be accepted in an array and then they can waited upon in another
loop.This will ensure that commands are enqueued in parallel
{code}
traceID = UUID.randomUUID().toString();
for(int i = 0; i < numRequest; i++) {
CompletableFuture returnFuture2 =
ContainerProtocolCalls.writeSmallFileAsync(client, containerName,
"key" + i, ("data" + i).getBytes(), traceID + i);
returnFuture2.get();
}
{code}
4) Another comment which has earlier been noted, this code will not work with
cblock correctly. as there are cases where TraceID is not set correctly.
example from BlockWriterTask
{code}
ContainerProtocolCalls.writeSmallFile(client, containerName,
Long.toString(block.getBlockID()), data, "");
{code}
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984615#comment-15984615
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
52s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
0 unchanged - 1 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Possible null pointer dereference of response in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtos$ContainerCommandResponseProto)
Dereferenced at ContainerProtocolCalls.java:response in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtos$ContainerCommandResponseProto)
Dereferenced at ContainerProtocolCalls.java:[line 652] |
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12865092/HDFS-11580-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 436f38fb59
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984356#comment-15984356
]
Yiqun Lin commented on HDFS-11580:
--
Thanks [~vagarychen], [~msingh] for the quick review!
{quote}
will supplyAsync be called by multiple threads at the same time? how about
waitForResponse? If so, is this method thread-safe?
{quote}
Each supplyAsync call will be running in a thread pool inside. So there is a
chance that multiple threads will do the get/remove operations in pending map.
So I used {{ConcurrentHashMap}} here and it can ensure this method thread-safe.
{quote}
This can be somewhat tricky, but can we add a unit test to verify this?
{quote}
Had addressed this in the latest patch.
{quote}
Make this one line?
{quote}
Fixed.
{quote}
Let's either file another JIRA to follow up, or throw an
UnsupportedOperationException..
{quote}
I think here we would be better to throw an UnsupportedOperationException.
{quote}
However, with Cblock, there are cases where TraceID is not set correctly. I
feel that TraceID should not be used to match a response to its corresponding
request. Would a counter be a better parameter to use here ?
{quote}
I'm not so familiar with Cblock currently. But I think the way of using a
counter is also a optional way. We can use the existing class
{{SequentialNumber}} to generate the unique id. In addition, I see
{{SequentialNumber}} has already used to generated blockId and blockgroupId in
HDFS.
Attach the new patch to mainly address [~vagarychen]'s comments.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984221#comment-15984221
]
Mukul Kumar Singh commented on HDFS-11580:
--
Thanks [~linyiqun] for the updated patch.
In the latest patch, Trace ID is being used to match a response to its
corresponding request.
{code}
if (request.getTraceID().equals(curResponse.getTraceID())) {
response = curResponse;
} else {
pendingResponses.put(curResponse.getTraceID(), curResponse);
// Try to get response from pending responses map and remove the
// response in map.
response = pendingResponses.remove(request.getTraceID());
}
{code}
However, with Cblock, there are cases where TraceID is not set correctly. I
feel that TraceID should not be used to match a response to its corresponding
request. Would a counter be a better parameter to use here ?
{code}
BlockWriterTask
---
ContainerProtocolCalls.writeSmallFile(client, containerName,
Long.toString(block.getBlockID()), data, "");
{code}
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984042#comment-15984042
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
0 unchanged - 1 fixed = 7 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
8s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Possible null pointer dereference of response in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtos$ContainerCommandResponseProto)
Dereferenced at ContainerProtocolCalls.java:response in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtos$ContainerCommandResponseProto)
Dereferenced at ContainerProtocolCalls.java:[line 650] |
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.server.namenode.TestStartup |
| Timed out junit tests |
org.apache.hadoop
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983505#comment-15983505
]
Chen Liang commented on HDFS-11580:
---
Thanks [~linyiqun] for updating the patch!
in {{XceiverClientHandler.java}}
1. I'm not sure how {{CompletableFuture}} works here, so this is more like a
question and everyone's thoughts are welcome: In {{sendCommandAsync}}, will
{{supplyAsync}} be called by multiple threads at the same time? how about
{{waitForResponse}}? If so, is this method thread-safe? (e.g. do we need
protection for {{pendingResponses}}?)
in {{ContainerProtocolCalls.java}}
2. Actually I think [~msingh] brought up a critical point, which is that we
need to be certain that the responses match the requests. This can be somewhat
tricky, but can we add a unit test to verify this? I'm thinking of may be
having a test that sends a number of async requests and see if they all got
properly responded.
3. Make this one line?
{code}
ContainerCommandResponseProto response;
response = xceiverClient.sendCommand(request);
{code}
in {{XceiverClientRatis.java}}
4. {{// TODO: Implement the async interface.}}
Let's either file another JIRA to follow up, or throw an
{{UnsupportedOperationException}} if we will not support async calls with
Ratis. [~anu]?
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982375#comment-15982375
]
Mukul Kumar Singh commented on HDFS-11580:
--
Thanks for working on this [~linyiqun],
1) As [~vagarychen] pointed out, the current implementation will keep the code
synchronous.
2) Also because of the async nature of the interface responses, need not be in
the same order as the requests. We will need a method to match the response to
the replies.
{code}
boolean interrupted = false;
try {
return responses.take();
{code}
3) With an async interface, we will always need to keep an eye on the queue
depth, and it would be desirable to have some sort of metrics for the number of
pending commands at any instance.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982055#comment-15982055
]
Chen Liang commented on HDFS-11580:
---
Hi [~linyiqun],
Thanks for working on this!
But I'm not sure this is the right way to make the calls asynchronous.
Specifically, looks like the changes are like the following:
{code}
CompletableFuture returnFuture =
client.sendCommandAsync(request);
try {
response = returnFuture.get();
} catch (InterruptedException | ExecutionException e) {
...
}
{code}
But this does not appear to me as asynchronous. The behaviour of this code
seems to be that: it sends a command asynchronously, which is good, but it then
calls {{returnFuture.get();}} this is a blocking call and it waits until the
result is returned. Which is the same behaviour as with synchronous calls.
I think it is fair to say that as long as we try to get the result of a
command, after we send the command in the same thread, it will be, by
definition, synchronous. We probably need to either register a callback, or
call {{get()}} from some other dedicated thread.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981902#comment-15981902
]
Anu Engineer commented on HDFS-11580:
-
[~linyiqun] Thanks for the patch. I am still reading thru the patch. I had a
style question, Do you think we should have 2 functions, like readChunk and
readChunkAsync so that it looks more like java 8-ish ? rather than a boolean
flag ?
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980074#comment-15980074
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} HDFS-7240 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
43s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
8s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 8s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
1 unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
45s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
43s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 3 new
+ 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 45s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Nullcheck of response at line 115 of value previously dereferenced in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.getKey(XceiverClientSpi,
ContainerProtos$KeyData, String, boolean) At ContainerProtocolCalls.java:115
of value previously dereferenced in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.getKey(XceiverClientSpi,
ContainerProtos$KeyData, String, boolean) At ContainerProtocolCalls.java:[line
114] |
| | Nullcheck of response at line 215 of value previously dereferenced in
org.apache.hadoop.scm.storage.ContainerProtocolCalls.readChunk(XceiverClientSpi,
ContainerProtos$ChunkInfo, String, String, boolean) At
ContainerProtocolCalls.java:215 of value previously dereferenced in
org.apache.hadoop.scm.storage.Cont
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973110#comment-15973110
]
Anu Engineer commented on HDFS-11580:
-
[~linyiqun] Thanks for the update. I was hoping that we would support the
standard java 8 interface specified here.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
A CompletableFuture future will allow us to write async code using standard
java 8 patterns.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969046#comment-15969046
]
Hadoop QA commented on HDFS-11580:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
7s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
14s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
31s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-client generated 1 new
+ 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HDFS-11580 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863460/HDFS-11580-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 09097b87fdb6 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 1a418ef |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/19091/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/19091/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/19091/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/19091/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/19091/artifact/patc
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968964#comment-15968964
]
Yiqun Lin commented on HDFS-11580:
--
Attach the initial patch. In the first patch, I implemented the prototype of
async interface.
Based on this, I'd like to implement some async APIs for the following contain
related operation:
* putKey
* writeChunk
* writeSmallFile
* createContainer
Hi [~anu], is anything I am missing on this?
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960174#comment-15960174
]
Yiqun Lin commented on HDFS-11580:
--
HI [~anu], I'd like to work on this. I have taken a quick look on this. I am
still having something to confirm.
* If we use async interface to send request command, that means we don't return
the response to the client, right? And we will not do the
{{validateContainerResponse}} operation. As I see the current logic, it will do
the response checking.
* SCM API will also invoke {{XceiverClient#sendCommand}}, so is that mean I
should only refactor the sendCommand? Is that anything else should I do for SCM
API?
Please just let me know if I am not correct. Thanks!
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
