On 6 Apr 2015, at 23:05, Patrick Young 
<[email protected]<mailto:[email protected]>> 
wrote:

does anyone have any thoughts on storing a really large raster in HDFS?  Seems 
like if I just dump the image into HDFS as it, it'll get stored in blocks all 
across the system and when I go to read it, there will be a ton of network 
traffic from all the blocks to the reading node!

It get splt into blocks scattered (at default 3x replication) to: 1 on current 
host, 2 elsewhere.

I'd recommend you look @ Russell Perry's @ HPLabs's 2009 paper, High Speed 
Raster Image Streaming For Digital Presses Using the Hadoop File System, which 
was about using HDFS/MapReduce to render images, rather than analyse them. 
Similar to things like tile generation of google/open-street/apple maps:

http://www.hpl.hp.com/techreports/2009/HPL-2009-345.pdf

Russ modified the HDFS client so that rather than have it pick a block for the 
app to read from, the app got to make the decision itself. Then running code on 
the server hooked straight up to the "line rate" printing press, fetched data 
from different racks so as to get max bandwidth out of each host, and of each 
rack switch; 4 Gb/s overall. I don' think that patch was ever contributed back 
—or at least got in.

Reply via email to