[jira] [Comment Edited] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109541#comment-16109541
]
Chen Liang edited comment on HDFS-11580 at 8/1/17 7:17 PM:
---
Thanks [~anu] for the follow-up! makes to me. +1 to v012 patch.
was (Author: vagarychen):
Thanks [~anu] for the follow-up! makes to me.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch,
> HDFS-11580-HDFS-7240.012.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16103029#comment-16103029
]
Yiqun Lin edited comment on HDFS-11580 at 7/27/17 11:08 AM:
Thanks [~nandakumar131] and [~vagarychen] for looking into this. I totally
agree with what you said. Seems I wrote some duplicate async call codes, :P.
Hi [~vagarychen], the latest patch almost looks good to me. Just caught one
thing:
{code}
-KeyValue keyValue = KeyValue.newBuilder()
-.setKey("OverWriteRequested").setValue("true").build();
ChunkInfo chunk = ChunkInfo
.newBuilder()
.setChunkName(key + "_chunk")
.setOffset(0)
.setLen(data.length)
-.addMetadata(keyValue)
.build();
{code}
Here we remove {{OverWriteRequested}} flag, that will lead a different
behaviour I think. That is say if we write a exist key name file. That will be
rejected and throws IOException. But I think this should intend to be the right
way here. So, also +1 from me, :).
was (Author: linyiqun):
Thanks [~nandakumar131] and [~vagarychen] for looking into this. I totally
agree with what you said. Seems I wrote some duplicate async call codes, :P.
Hi [~vagarychen], the latest patch almost looks good to me. Just caught one
thing:
{code}
-KeyValue keyValue = KeyValue.newBuilder()
-.setKey("OverWriteRequested").setValue("true").build();
ChunkInfo chunk = ChunkInfo
.newBuilder()
.setChunkName(key + "_chunk")
.setOffset(0)
.setLen(data.length)
-.addMetadata(keyValue)
.build();
{code}
Here we remove {{OverWriteRequested}} flag, that will lead a different
behaviour I think. That is say if we write a exist key name file. That will be
rejected and throws IOException.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch,
> HDFS-11580-HDFS-7240.010.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101998#comment-16101998
]
Chen Liang edited comment on HDFS-11580 at 7/26/17 5:47 PM:
Thanks [~nandakumar131] for the comments!
As far as I remember how this JIRA has come along, the potential issue here was
that it was not clear whether the response will match the request. For example,
say, the async readfile is calling readfilesync directly, i.e.
{code}
Data readFileAsync(..) {
return readFileSync(...)
}
{code}
And we are calling readSmallFile twice, to read file1 and file2.
{code}
readFileAsync(file1)
readFileAsync(file2)
{code}
There is no guarantee which request arrives and returns first. So the
readFileSync(file2) can potentially return earlier, if so, can we be sure it's
returned gets delivered to second call definitely, instead of the first?
Essentially, the issue is, if there are two simultaneous calls of
{{supplyAsync}} with the same supplier, and the supplier returns one time,
which of the two {{supplyAsync}} will return?
This is related to {{CompletableFuture}}'s feature I haven't digged into or
tested. Totally possible that it is handled properly in some way. If you (or
anyone looking at this) have anything to add, please feel free to share. But as
I follow this JIRA, it was because of this, the implementation is that
{{sendCommandAsync}} is calling {{waitForResponse}}, which has an internal map
that matches the response to the request.
was (Author: vagarychen):
Thanks [~nandakumar131] for the comments!
As far as I remember how this JIRA has come along, the potential issue here was
that it was not clear whether the response will match the request. For example,
say, the async readfile is calling readfilesync directly, i.e.
{code}
Data readFileAsync(..) {
return readFileSync(...)
}
{code}
And we are calling readSmallFile twice, to read file1 and file2.
{code}
readFileAsync(file1)
readFileAsync(file2)
{code}
There is no guarantee which request arrives and returns first. So the
readFileSync(file2) can potentially return earlier, if so, can we be sure it's
returned gets delivered to second call definitely, instead of the first?
Essentially, the issue is, if there are two simultaneous calls of
{{supplyAsync}} with the same supplier, and the supplier returns one time,
which of the two {{supplyAsync}} will return?
This is related to {{CompletableFuture}}'s feature I haven't digged into or
tested. Totally possible that it be will handled well in some way. If you (or
anyone looking at this) have anything to add, please feel free to share. But as
I follow this JIRA, it was because of this, the implementation is that
{{sendCommandAsync}} is calling {{waitForResponse}}, which has an internal map
that matches the response to the request.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch,
> HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch,
> HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[
https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984356#comment-15984356
]
Yiqun Lin edited comment on HDFS-11580 at 4/26/17 8:31 AM:
---
Thanks [~vagarychen], [~msingh] for the quick review!
{quote}
will supplyAsync be called by multiple threads at the same time? how about
waitForResponse? If so, is this method thread-safe?
{quote}
Each supplyAsync call will be running in a thread pool inside. So there is a
chance that multiple threads will do the get/remove operations in pending map.
So I used {{ConcurrentHashMap}} here and it can ensure this method thread-safe.
{quote}
This can be somewhat tricky, but can we add a unit test to verify this?
{quote}
Had addressed this in the latest patch.
{quote}
Make this one line?
{quote}
Fixed.
{quote}
Let's either file another JIRA to follow up, or throw an
UnsupportedOperationException..
{quote}
I think here we would be better to throw an UnsupportedOperationException.
{quote}
However, with Cblock, there are cases where TraceID is not set correctly. I
feel that TraceID should not be used to match a response to its corresponding
request. Would a counter be a better parameter to use here ?
{quote}
I'm not so familiar with Cblock currently. But I think the way of using a
counter is also a optional way. We can use the existing class
{{SequentialNumber}} to generate the unique id. In addition, I see
{{SequentialNumber}} has already used to generated blockId and blockgroupId in
HDFS. Before doing this, I suppose that we need more discussion on this.
Attach the new patch to mainly address [~vagarychen]'s comments.
was (Author: linyiqun):
Thanks [~vagarychen], [~msingh] for the quick review!
{quote}
will supplyAsync be called by multiple threads at the same time? how about
waitForResponse? If so, is this method thread-safe?
{quote}
Each supplyAsync call will be running in a thread pool inside. So there is a
chance that multiple threads will do the get/remove operations in pending map.
So I used {{ConcurrentHashMap}} here and it can ensure this method thread-safe.
{quote}
This can be somewhat tricky, but can we add a unit test to verify this?
{quote}
Had addressed this in the latest patch.
{quote}
Make this one line?
{quote}
Fixed.
{quote}
Let's either file another JIRA to follow up, or throw an
UnsupportedOperationException..
{quote}
I think here we would be better to throw an UnsupportedOperationException.
{quote}
However, with Cblock, there are cases where TraceID is not set correctly. I
feel that TraceID should not be used to match a response to its corresponding
request. Would a counter be a better parameter to use here ?
{quote}
I'm not so familiar with Cblock currently. But I think the way of using a
counter is also a optional way. We can use the existing class
{{SequentialNumber}} to generate the unique id. In addition, I see
{{SequentialNumber}} has already used to generated blockId and blockgroupId in
HDFS.
Attach the new patch to mainly address [~vagarychen]'s comments.
> Ozone: Support asynchronus client API for SCM and containers
>
>
> Key: HDFS-11580
> URL: https://issues.apache.org/jira/browse/HDFS-11580
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Anu Engineer
>Assignee: Yiqun Lin
> Attachments: HDFS-11580-HDFS-7240.001.patch,
> HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch,
> HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch
>
>
> This is an umbrella JIRA that needs to support a set of APIs in Asynchronous
> form.
> For containers -- or the datanode API currently supports a call
> {{sendCommand}}. we need to build proper programming interface and support an
> async interface.
> There is also a set of SCM API that clients can call, it would be nice to
> support Async interface for those too.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
