[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413277#comment-16413277
]
Rushabh S Shah edited comment on HDFS-11156 at 3/26/18 2:17 AM:
My opinion would be revert this patch from branch-2.
We can commit HDFS-12459 in all the branches.
{quote}Rushabh S Shah did the review and gave his +1, but he did not commit
{quote}
Unfortunately thats all I can do. :( It was +1 (non-binding).
I can request Kihwal/Daryn for additional reviews. Will ping in that jira.
Hope it helps.
was (Author: shahrs87):
My opinion would be revert this patch from branch-2.
We can commit HDFS-12459 in all the branches.
bq. Rushabh S Shah did the review and gave his +1, but he did not commit
Unfortunately thats all I can do. :( It was +1 (non-binding).
I can request Kihwal/Daryn for reviews. Will ping in that jira. Hope it helps.
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156-branch-2.01.patch, HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch, HDFS-11156.04.patch, HDFS-11156.05.patch,
> HDFS-11156.06.patch, HDFS-11156.07.patch, HDFS-11156.08.patch,
> HDFS-11156.09.patch, HDFS-11156.10.patch, HDFS-11156.11.patch,
> HDFS-11156.12.patch, HDFS-11156.13.patch, HDFS-11156.14.patch,
> HDFS-11156.15.patch, HDFS-11156.16.patch, Output_JSON_format_v10.jpg,
> SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413277#comment-16413277
]
Rushabh S Shah edited comment on HDFS-11156 at 3/26/18 2:35 AM:
My opinion would be revert this patch from branch-2/branch-2.8.
We can commit HDFS-12459 in all the branches.
{quote}Rushabh S Shah did the review and gave his +1, but he did not commit
{quote}
Unfortunately thats all I can do. :( It was +1 (non-binding).
I can request Kihwal/Daryn for additional reviews. Will ping in that jira.
Hope it helps.
was (Author: shahrs87):
My opinion would be revert this patch from branch-2.
We can commit HDFS-12459 in all the branches.
{quote}Rushabh S Shah did the review and gave his +1, but he did not commit
{quote}
Unfortunately thats all I can do. :( It was +1 (non-binding).
I can request Kihwal/Daryn for additional reviews. Will ping in that jira.
Hope it helps.
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156-branch-2.01.patch, HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch, HDFS-11156.04.patch, HDFS-11156.05.patch,
> HDFS-11156.06.patch, HDFS-11156.07.patch, HDFS-11156.08.patch,
> HDFS-11156.09.patch, HDFS-11156.10.patch, HDFS-11156.11.patch,
> HDFS-11156.12.patch, HDFS-11156.13.patch, HDFS-11156.14.patch,
> HDFS-11156.15.patch, HDFS-11156.16.patch, Output_JSON_format_v10.jpg,
> SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166424#comment-16166424
]
Rushabh S Shah edited comment on HDFS-11156 at 9/14/17 3:07 PM:
bq. Current approach was trying to maintain compatibility as much as possible.
I am unable to understand your point.
{{DistributedFileSystem#getFileBlockLocations()}} after going through bunch of
DFSClient related code calls {{ClientProtocol#getBlockLocations}}.
{{NamenodeWebHdfsMethods#getRPCServer()}} returns {{NamenodeProtocols}} which
in turn implements {{ClientProtocol}} also.
So you can directly call {{np.getBlockLocations(src, offset, length)}} instead
of {{fs.getFileBlockLocations(...)}} and then convert {{LocatedBlocks}} to
{{BlockLocation[]}} using {{DFSUtilClient.locatedBlocks2Locations()}}
Hope it makes sense.
If not, I can write a small patch illustrating my idea.
was (Author: shahrs87):
bq. Current approach was trying to maintain compatibility as much as possible.
I am unable to understand your point.
{{DistributedFileSystem#getFileBlockLocations()}} after going through bunch of
DFSClient related code calls {{ClientProtocol#getBlockLocations}}.
{{NamenodeWebHdfsMethods#getRPCServer()}} returns {{NamenodeProtocols}} which
in turn implements {{ClientProtocol}} also.
So you can directly call {{np.np.getBlockLocations(src, offset, length)}}
instead of {{fs.getFileBlockLocations(...)}} and then convert {{LocatedBlocks}}
to {{BlockLocation[]}} using {{DFSUtilClient.locatedBlocks2Locations()}}
Hope it makes sense.
If not, I can write a small patch illustrating my idea.
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
> HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch,
> HDFS-11156.07.patch, HDFS-11156.08.patch, HDFS-11156.09.patch,
> HDFS-11156.10.patch, HDFS-11156.11.patch, HDFS-11156.12.patch,
> HDFS-11156.13.patch, HDFS-11156.14.patch, HDFS-11156.15.patch,
> HDFS-11156.16.patch, HDFS-11156-branch-2.01.patch,
> Output_JSON_format_v10.jpg, SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15724468#comment-15724468
]
Weiwei Yang edited comment on HDFS-11156 at 12/6/16 6:00 AM:
-
Hi [~andrew.wang]
In that case client can still call GET_BLOCK_LOCATIONS, that is still working.
Adding a new API should not break compatibility, this one is jus like
HDFS-10756 which adds "GETTRASHROOT" to webhdfs. What do you think?
was (Author: cheersyang):
Hi [~andrew.wang]
In that case client can still call GET_BLOCK_LOCATIONS, that is still working.
Adding a new API should not break compatibility, this one now is jus like
HDFS-10756 which adds "GETTRASHROOT" to webhdfs.
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch, HDFS-11156.04.patch, HDFS-11156.05.patch,
> HDFS-11156.06.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728894#comment-15728894
]
Weiwei Yang edited comment on HDFS-11156 at 12/7/16 2:38 PM:
-
Hello [~andrew.wang]
Thanks for your comments. I just uploaded v8 patch with following changes
bq. unit test for the fallback behavior
Added an unit test for the fallback code. Mocked an old version of server,
tested 3 scenarios, 1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send
back an error. 2) Server handles GET_BLOCK_LOCATIONS and returns a sample of
data. 3) Server sends out a random exception.
bq. separate functions
I am still using single function to handle that, but I added more java doc
ahead. The reason is 1) Separate them would have majority of code similar 2) I
checked the times of this function gets called while doing the fallback UT,
this is easier with single method.
bq. The example curl and response are indented by 9 instead of 8
I am following the indents format with rest of the document. They are also 9.
Anyhow I checked the indents again in the doc I added, it should be OK now.
bq. provide an example response for a file with more than one block
Done
bq. I don't think this schema works
I verified the schema again with this [online
tool|http://www.jsoneditoronline.org/], and compared with
FileStatuses/FileStatus schemas (they are similar). See [Sample
response|^SampleResponse_JSON.jpg], [BlockLocations JSON
schema|^BlockLocations_JSON_Schema.jpg] which looks just like [FileStatuses
JSON schema|^FileStatuses_JSON_Schema.jpg], and [BlockLocation Properties
Schema |^BlockLocationProperties_JSON_Schema.jpg]. Let me if it looks good now.
Thank you and let me know if you have any further comments.
was (Author: cheersyang):
Hello [~andrew.wang]
Thanks for your comments. I just uploaded v8 patch with following changes
bq. unit test for the fallback behavior
Added an unit test for the fallback code. Mocked an old version of server,
tested 3 scenarios, 1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send
back an error. 2) Server handles GET_BLOCK_LOCATIONS and returns a sample of
data. 3) Server sends out a random exception.
bq. separate functions
I am still using single function to handle that, but I added more java doc
ahead. The reason is 1) Separate them would have majority of code similar 2) I
checked the times of this function gets called while doing the fallback UT,
this is easier with single method.
bq. The example curl and response are indented by 9 instead of 8
I am following the indents format with rest of the document. They are also 9.
Anyhow I checked the indents again in the doc I added, it should be OK now.
bq. provide an example response for a file with more than one block
Done
bq. I don't think this schema works
I verified the schema again with this [online
tool|http://www.jsoneditoronline.org/], and compared with
FileStatuses/FileStatus schemas (they are similar). See [Sample
response|^SampleResponse_JSON.jpg], [BlockLocations JSON
schema|^BlockLocations_JSON_Schema.jpg] which looks just like [FileStatuses
JSON schema|^FileStatuses_JSON_Schema.jpg], and [BlockLocation Properties
Schema |^BlockLocationProperties_JSON_Schema.jpg]It looks good to me now.
Thank you and let me know if you have any further comments.
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
> HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch,
> HDFS-11156.07.patch, HDFS-11156.08.patch, SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728894#comment-15728894
]
Weiwei Yang edited comment on HDFS-11156 at 12/7/16 2:38 PM:
-
Hello [~andrew.wang]
Thanks for your comments. I just uploaded v8 patch with following changes
bq. unit test for the fallback behavior
Added an unit test for the fallback code. Mocked an old version of server,
tested 3 scenarios, 1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send
back an error. 2) Server handles GET_BLOCK_LOCATIONS and returns a sample of
data. 3) Server sends out a random exception.
bq. separate functions
I am still using single function to handle that, but I added more java doc
ahead. The reason is 1) Separate them would have majority of code similar 2) I
checked the times of this function gets called while doing the fallback UT,
this is easier with single method.
bq. The example curl and response are indented by 9 instead of 8
I am following the indents format with rest of the document. They are also 9.
Anyhow I checked the indents again in the doc I added, it should be OK now.
bq. provide an example response for a file with more than one block
Done
bq. I don't think this schema works
I verified the schema again with this [online
tool|http://www.jsoneditoronline.org/], and compared with
FileStatuses/FileStatus schemas (they are similar). See [Sample
response|^SampleResponse_JSON.jpg], [BlockLocations JSON
schema|^BlockLocations_JSON_Schema.jpg] which looks just like [FileStatuses
JSON schema|^FileStatuses_JSON_Schema.jpg], and [BlockLocation Properties
Schema |^BlockLocationProperties_JSON_Schema.jpg]It looks good to me now.
Thank you and let me know if you have any further comments.
was (Author: cheersyang):
Hello [~andrew.wang]
Thanks for your comments. I just uploaded v8 patch with following changes
bq. unit test for the fallback behavior
Added an unit test for the fallback code. Mocked an old version of server,
tested 3 scenarios, 1) Server doesn't recognize GETFILEBLOCKLOCATIONS and send
back an error. 2) Server handles GET_BLOCK_LOCATIONS and returns a sample of
data. 3) Server sends out a random exception.
bq. separate functions
I am still using single function to handle that, but I added more java doc
ahead. The reason is 1) Separate them would have majority of code similar 2) I
checked the times of this function gets called while doing the fallback UT,
this is easier with single method.
bq. The example curl and response are indented by 9 instead of 8
I am following the indents format with rest of the document. They are also 9.
Anyhow I checked the indents again in the doc I added, it should be OK now.
bq. provide an example response for a file with more than one block
Done
bq. I don't think this schema works
I verified the schema again with this [online
tool|http://www.jsoneditoronline.org/], and compared with
FileStatuses/FileStatus schemas (they are similar). It looks good to me now.
Thank you and let me know if you have any further comments.
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
> HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch,
> HDFS-11156.07.patch, HDFS-11156.08.patch, SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743813#comment-15743813
]
Weiwei Yang edited comment on HDFS-11156 at 12/13/16 1:52 AM:
--
Hello [~andrew.wang]
Please help to review v14 patch when you get a chance, it addressed the
comments from you with the changes I described
[here|https://issues.apache.org/jira/browse/HDFS-11156?focusedCommentId=15739324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15739324].
The UT failure was not related, and the 2 checkstyle issues were not new as
well (I explained ealier).
Thank you
was (Author: cheersyang):
Hello [~andrew.wang]
Please help to review v14 patch when you get a chance, it addressed the
comments from you with the changes I described
[here|https://issues.apache.org/jira/browse/HDFS-11156?focusedCommentId=15739324&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15739324].
Thank you
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
> HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch,
> HDFS-11156.07.patch, HDFS-11156.08.patch, HDFS-11156.09.patch,
> HDFS-11156.10.patch, HDFS-11156.11.patch, HDFS-11156.12.patch,
> HDFS-11156.13.patch, HDFS-11156.14.patch, Output_JSON_format_v10.jpg,
> SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753223#comment-15753223
]
Weiwei Yang edited comment on HDFS-11156 at 12/16/16 2:39 AM:
--
Thanks [~andrew.wang] for the review. Just uploaded v15 patch including
following changes
# Completed 4 cases in {{TestWebHDFS}} to cover all combinations of offset and
length parameters
# Added 1 test case to verify that rest call returns an empty array of
BlockLoaction when offset is bigger than the file length
was (Author: cheersyang):
Uploaded v15 patch including following changes
# Completed 4 cases in {{TestWebHDFS}} to cover all combinations of offset and
length parameters
# Added 1 test case to verify that rest call returns an empty array of
BlockLoaction when offset is bigger than the file length
> Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: BlockLocationProperties_JSON_Schema.jpg,
> BlockLocations_JSON_Schema.jpg, FileStatuses_JSON_Schema.jpg,
> HDFS-11156.01.patch, HDFS-11156.02.patch, HDFS-11156.03.patch,
> HDFS-11156.04.patch, HDFS-11156.05.patch, HDFS-11156.06.patch,
> HDFS-11156.07.patch, HDFS-11156.08.patch, HDFS-11156.09.patch,
> HDFS-11156.10.patch, HDFS-11156.11.patch, HDFS-11156.12.patch,
> HDFS-11156.13.patch, HDFS-11156.14.patch, HDFS-11156.15.patch,
> Output_JSON_format_v10.jpg, SampleResponse_JSON.jpg
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS&offset=0&length=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
