[jira] [Comment Edited] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API

2016-11-30 Thread Weiwei Yang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711204#comment-15711204
 ] 

Weiwei Yang edited comment on HDFS-11156 at 12/1/16 7:50 AM:
-

Regarding to the jenkins job result

# The findbugs error was from a recent commit HDFS-10930, not from this patch.
# The checkstyle issues were not caused by this patch either, 1st issue was the 
method too long, 2nd the code style was using nested blocks in switch-case 
block. 
# The 2 UT failures also not related, I have run them locally and they all 
succeed. They seemed to be intermittent failures,  HDFS-9599 and HDFS-11030 
might be related. 

I've also tested this with latest trunk code on a small cluster,  both
{noformat}
http://:/webhdfs/v1/tmp/test.log?op=GETFILEBLOCKLOCATIONS=100=0
{noformat}

and 
{noformat}
http://:/webhdfs/v1/tmp/test.log?op=GET_BLOCK_LOCATIONS=100=0
{noformat}

works and returned expected value. [~liuml07] Would you please help to review 
the patch?

Thanks a lot.


was (Author: cheersyang):
Regarding to the jenkins job result

# The findbugs error was from a recent commit HDFS-10930, not from this patch.
# The checkstyle issues were not caused by this patch either, 1st issue was the 
method too long, 2nd the code style was using nested blocks in switch-case 
block. 
# The 2 UT failures also not related, I have run them locally and they all 
succeed. They seemed to be intermittent failures,  HDFS-9599 and HDFS-11030 
might be related. 

Hello [~liuml07] Would you please help to review the patch ? Thanks a lot.

> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch, 
> HDFS-11156.03.patch, HDFS-11156.04.patch, HDFS-11156.05.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to 
> *FileSystem* API, 
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be 
> fixed. Marked as Incompatible change as this will change the output of the 
> GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API

2016-11-29 Thread Weiwei Yang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704644#comment-15704644
 ] 

Weiwei Yang edited comment on HDFS-11156 at 11/29/16 8:28 AM:
--

Hello [~andrew.wang]

Thanks for providing your thought.
>From the [webhdfs 
>document|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html]
> 

{noformat}
The HTTP REST API supports the complete FileSystem/FileContext interface for 
HDFS
{noformat} 

user would expect the BlockLocation[] instead of LocatedBlocks according to 
[FileSystem#getFileBlockLocations|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations-org.apache.hadoop.fs.Path-long-long-]
 interface. I checked rest of APIs, they seemed good like

{noformat}
GETFILESTATUS (see FileSystem.getFileStatus)
{noformat}

Isn't this a bug worth to fix? I am hoping you can reconsider this.

Thank you






was (Author: cheersyang):
Hello [~andrew.wang]

Thanks for providing your thought.
>From the [webhdfs 
>document|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html]
> 

{{The HTTP REST API supports the complete FileSystem/FileContext interface for 
HDFS}} 

user would expect the BlockLocation[] instead of LocatedBlocks according to 
[FileSystem#getFileBlockLocations|https://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations-org.apache.hadoop.fs.Path-long-long-]
 interface. I checked rest of APIs, they seemed good like

{{GETFILESTATUS (see FileSystem.getFileStatus)}}

Isn't this a bug worth to fix? I am hoping you can reconsider this.

Thank you





> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch, 
> HDFS-11156.03.patch, HDFS-11156.04.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to 
> *FileSystem* API, 
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be 
> fixed. Marked as Incompatible change as this will change the output of the 
> GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API

2016-11-22 Thread Weiwei Yang (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688666#comment-15688666
 ] 

Weiwei Yang edited comment on HDFS-11156 at 11/23/16 2:37 AM:
--

Hello [~liuml07]

Would you help to review this patch ? 
I have built latest trunk and tested with this patch, use

{code}
http://:/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS=0=1=yangww
{code}

it returns output

{code}
{"BlockLocations":[{"cachedHosts":[],"corrupt":false,"hosts":["deale1.fyre.ibm.com"],"length":134217728,"names":["9.30.167.122:9866"],"offset":0,"storageIds":["DS-2c136bcd-d658-4853-8b7d-9ecb354e6093"],"storageTypes":["DISK"],"topologyPaths":["/default-rack/9.30.167.122:9866"]}]}
{code}

client would easily parse this to BlockLocation array using json parser or 
object mapper, just like the test case code I added to {{TestWebHDFS}}.


was (Author: cheersyang):
Hello Mingliang Liu
Would you help to review this patch ? 
I have built latest trunk and tested with this patch, use

{code}
http://:/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS=0=1=yangww
{code}

it returns output

{code}
{"BlockLocations":[{"cachedHosts":[],"corrupt":false,"hosts":["deale1.fyre.ibm.com"],"length":134217728,"names":["9.30.167.122:9866"],"offset":0,"storageIds":["DS-2c136bcd-d658-4853-8b7d-9ecb354e6093"],"storageTypes":["DISK"],"topologyPaths":["/default-rack/9.30.167.122:9866"]}]}
{code}

client would easily parse this to BlockLocation array using json parser or 
object mapper, just like the test case code I added to {{TestWebHDFS}}.

> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch, 
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
>   "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
>   }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to 
> *FileSystem* API, 
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be 
> fixed. Marked as Incompatible change as this will change the output of the 
> GET_BLOCK_LOCATIONS API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org