Re: Data Locality and WebHDFS

2014-03-17 Thread RJ Nowling
assured locality for the first block of the file. Thanks. On Sun, Mar 16, 2014 at 9:18 PM, RJ Nowling rnowl...@gmail.com wrote: Thank you, Mingjiang and Alejandro. This is interesting. Since we will use the data locality information for scheduling, we could hack this to get the data locality

Re: Data Locality and WebHDFS

2014-03-17 Thread RJ Nowling
would assume that you'll get to the first block As usual, and the skip is handled by the DN serving the file (as webhdfs doesnot know at open that you'll skip) Alejandro (phone typing) On Mar 17, 2014, at 9:47, RJ Nowling rnowl...@gmail.com wrote: Hi Alejandro, The WebHDFS API allows

Data Locality and WebHDFS

2014-03-16 Thread RJ Nowling
Hi all, I'm writing up a Google Summer of Code proposal to add HDFS support to Disco, an Erlang MapReduce framework. We're interested in using WebHDFS. I have two questions: 1) Does WebHDFS allow querying data locality information? 2) If the data locality information is known, can data on

Re: Data Locality and WebHDFS

2014-03-16 Thread RJ Nowling
will be redirected to the data node automatically. On Mon, Mar 17, 2014 at 6:07 AM, RJ Nowling rnowl...@gmail.com wrote: Hi all, I'm writing up a Google Summer of Code proposal to add HDFS support to Disco, an Erlang MapReduce framework. We're interested in using WebHDFS. I have two questions: 1