On 6 Apr 2015, at 23:05, Patrick Young <[email protected]<mailto:[email protected]>> wrote:
does anyone have any thoughts on storing a really large raster in HDFS? Seems like if I just dump the image into HDFS as it, it'll get stored in blocks all across the system and when I go to read it, there will be a ton of network traffic from all the blocks to the reading node! It get splt into blocks scattered (at default 3x replication) to: 1 on current host, 2 elsewhere. I'd recommend you look @ Russell Perry's @ HPLabs's 2009 paper, High Speed Raster Image Streaming For Digital Presses Using the Hadoop File System, which was about using HDFS/MapReduce to render images, rather than analyse them. Similar to things like tile generation of google/open-street/apple maps: http://www.hpl.hp.com/techreports/2009/HPL-2009-345.pdf Russ modified the HDFS client so that rather than have it pick a block for the app to read from, the app got to make the decision itself. Then running code on the server hooked straight up to the "line rate" printing press, fetched data from different racks so as to get max bandwidth out of each host, and of each rack switch; 4 Gb/s overall. I don' think that patch was ever contributed back —or at least got in.
