Heya,

You might be interesting at looking at GeoTrellis
They use RDDs of Tiles to process big images like Landsat ones can be
(specially 8).

However, I see you have only 1G per file, so I guess you only care of a
single band? Or is it a reboxed pic?

Note: I think the GeoTrellis image format is still single band, although
it's highly optimized for distributed geoprocessing

my2ยข
andy


On Tue, Apr 7, 2015 at 12:06 AM Patrick Young <
[email protected]> wrote:

> Hi all,
>
> I'm new to Spark and wondering if it's appropriate to use for some image
> processing tasks on pretty sizable (~1 GB) images.
>
> Here is an example use case.  Amazon recently put the entire Landsat8
> archive in S3:
>
> http://aws.amazon.com/public-data-sets/landsat/
>
> I have a bunch of GDAL based (a C library for geospatial raster I/O)
> Python scripts that take a collection of Landsat images and mash them into
> a single mosaic.  This works great for little mosaics, but if I wanted to
> do the entire world, I need more horse power!  The scripts do the following:
>
>    1. Copy the selected rasters down from S3 to the local file system
>    2. Read each image into memory as numpy arrays (a big 3D array), do
>    some image processing using various Python libs, and write the result out
>    to the local file system
>    3. Blast all the processed imagery back to S3, and hooks up MapServer
>    for viewing
>
> Step 2 takes a long time; this is what I'd like to leverage Spark for.
> Each image, if you stack all the bands together, can be ~1 GB in size.
>
> So here are a couple of questions:
>
>
>    1. If I have a large image/array, what's a good way of getting it into
>    an RDD?  I've seen some stuff about folks tiling up imagery into little
>    chunks and storing it in HBase.  I imagine I would want an image chunk in
>    each partition of the RDD.  If I wanted to apply something like a gaussian
>    filter I'd need each chunk to to overlap a bit.
>    2. In a similar vain, does anyone have any thoughts on storing a
>    really large raster in HDFS?  Seems like if I just dump the image into HDFS
>    as it, it'll get stored in blocks all across the system and when I go to
>    read it, there will be a ton of network traffic from all the blocks to the
>    reading node!
>    3. How is the numpy's ndarray support in Spark?  For instance, if I do
>    a map on my theoretical chunked image RDD, can I easily realize the image
>    chunk as a numpy array inside the function?  Most of the Python algorithms
>    I use take in and return a numpy array.
>
> I saw some discussion in the past on image processing:
>
> These threads talk about processing lots of little images, but this isn't
> really my situation as I've got one very large image:
>
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Better-way-to-process-large-image-data-set-td14533.html
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Processing-audio-video-images-td6752.html
>
> Further, I'd like to have the imagery in HDFS rather than on the file
> system to avoid I/O bottlenecks if possible!
>
> Thanks for any ideas and advice!
> -Patrick
>
>
>

Reply via email to