Hi all,
I'm writing up a Google Summer of Code proposal to add HDFS support to
Disco, an Erlang MapReduce framework.
We're interested in using WebHDFS. I have two questions:
1) Does WebHDFS allow querying data locality information?
2) If the data locality information is known, can data on spec
According to this page:
http://hortonworks.com/blog/webhdfs-%E2%80%93-http-rest-access-to-hdfs/
> *Data Locality*: The file read and file write calls are redirected to the
> corresponding datanodes. It uses the full bandwidth of the Hadoop cluster
> for streaming data.
>
> *A HDFS Built-in Compone
well, this is for the first block of the file, the rest of the file (blocks
being local or not) are streamed out by the same datanode. for small files (one
block) you'll get locality, for large files only the first block, and by chance
if other blocks are local to that datanode.
Alejandro
(ph
Thank you, Mingjiang and Alejandro.
This is interesting. Since we will use the data locality information for
scheduling, we could "hack" this to get the data locality information, at
least for the first block. As Alejandro says, we'd have to test what
happens for other data blocks -- e.g., what
I may have expressed myself wrong. You don't need to do any test to see how
locality works with files of multiple blocks. If you are accessing a file
of more than one block over webhdfs, you only have assured locality for the
first block of the file.
Thanks.
On Sun, Mar 16, 2014 at 9:18 PM, RJ N
Hi Alejandro,
The WebHDFS API allows specifying an offset and length for the request. If
I specify an offset that start in the second block for a file (thus
skipping the first block all together), will the namenode still direct me
to a datanode with the first block or will it direct me to a namen
dont recall how skips are handled in webhdfs, but i would assume that you'll
get to the first block As usual, and the skip is handled by the DN serving the
file (as webhdfs doesnot know at open that you'll skip)
Alejandro
(phone typing)
> On Mar 17, 2014, at 9:47, RJ Nowling wrote:
>
> Hi Ale
actually, i am wrong, the webhdfs rest call has an offset.
Alejandro
(phone typing)
> On Mar 17, 2014, at 10:07, Alejandro Abdelnur wrote:
>
> dont recall how skips are handled in webhdfs, but i would assume that you'll
> get to the first block As usual, and the skip is handled by the DN serv
The file offset is considered in WebHDFS redirection. It redirects to a
datanode with the first block the client going to read, not the first block of
the file.
Hope it helps.
Tsz-Wo
On Monday, March 17, 2014 10:09 AM, Alejandro Abdelnur
wrote:
actually, i am wrong, the webhdfs rest call
Thank you, Tsz. That helps!
On Mon, Mar 17, 2014 at 2:30 PM, Tsz Wo Sze wrote:
> The file offset is considered in WebHDFS redirection. It redirects to a
> datanode with the first block the client going to read, not the first block
> of the file.
>
> Hope it helps.
> Tsz-Wo
>
>
> On Monday,
10 matches
Mail list logo
