The one thing the comes to my mind when we talk about large datasets,
large processing, large bandwidth, etc is building something that is
distributed across a large set of servers and is redundant because
pieces are on multiple servers.
Think peer-to-peer, s...@home, etc. This just seems like a better more
scalable solution. OAM might then be the central directory, task
scheduler, whatever.
This way no one company/org needs to bear the heavy lifting. And with
some advanced planning if someone needs to leave the network that has a
lot of data, it could be scheduled to be offloaded to other boxes.
Something to think about.
-Steve
Mikel Maron wrote:
Seems to me that there are a number of distinct problems to hurdle in
the OAM domain.
-There are several very large raster data sets in the public domain
(Landsat, USGS aerials) that need processing in a more accesible form.
-There are myriad smaller raster data sets, like pict'earth/diydrones,
and municipal imagery, etc.
-Mapping APIs want to hook into these data sets in a relatively seemless
way through tiles.
Solving all these problems on one monolithic server, or set of servers,
doesn't seem scalable. Unlike OSM with vector data, rasters simply take
too much space and too much processing.
The large datasets are a special case, and there's a public interest in
making these widely available. For landsat at least,
http://onearth.jpl.nasa.gov/tiled.html is almost there. Perhaps
Telascience, or Amazon Public Data Sets, or OSGeo data committee, or
some other body could be persuaded to help with hosting. Processing and
setup is not a recurring problem.
The smaller data sets need a more interactive tool set. I haven't looked
at the current OAM code, but I take it it's not all the way there. The
need puts me in mind of the Map Warper
[http://wrp.geothings.net/frontpage], a soon to be open source tool
modeled on the MetaCarta rectifier. Warper is forming the basis of the
NYPL's historic NYC map project [http://dev.maps.nypl.org/]. This tool
handles manual rectification of smallish uploaded images, then produces
a catalog and tiles. It doesn't seem like too much of a stretch to build
another uploading tool into this system, accepting already georeferenced
imagery, and converting it to whatever format Warper uses. This tool can
be deployed in a number of different places, to handle different data
sets. State of California, diydrones, OSGeo Italia could have their own
instances.
Linking numerous tile sources together is the last problem. Perhaps to
start, all of these service just serve tiles for their own areas
directly, with OpenLayers and whatever other client handling requesting
the proper tile for the area configured for that application. The TMS
spec [http://wiki.osgeo.org/wiki/Tile_Map_Service_Specification] is
designed for advertising tiles in any arbitrary scheme. There's been
proposals like distributed tile caching
[http://wiki.osgeo.org/wiki/Distributed_Tile_Caching] to integrate
distributed sources. I reckon that these ideas are a bit too baroque --
we can pretty much assume clients will want tiles in the standard G-Y-M
tiling scheme, i.e. TMS could be simplified. The logic for requesting a
tile in a particular location from the appropriate server is pushed to
the client.
I'm just thinking out loud here, haven't looked so much into the
technicals or implications. But generally seems to me that something
like this approach gives OAM next steps and scalability.
-Mikel
*From:* Brent Pedersen <[email protected]>
*To:* [email protected]
*Sent:* Saturday, January 3, 2009 11:41:04 AM
*Subject:* [OAM-talk] where to start helping...
hi, of the hurdles mentioned in the "state of openaerialmap"
(http://openaerialmap.org/pipermail/talk_openaerialmap.org/2008-December/000055.html),
i'd like to understand how to help with the programming stuff. chris
has said this:
09:25 < crschmidt> The thing that doesn't exist that should
09:26 < crschmidt> is a chunk of code that can take any set of georectified
images
09:26 < crschmidt> in any projection
09:26 < crschmidt> and make them into a set of 'happy' images
09:26 < crschmidt> tiled, overviewed, in the right projection
so, what's required to do that? and what needs to be determined to start?
what is the happy projection (4326?)
what is the happy format (geo-tiff?)
is it safe to assume it'll be using python-gdal to do this?
at this point, i dont even know enough to know the proper questions to ask.
i have messed with the code a bit, and updated to use django > 1.0, but not
addressed any of the real problems.
-brentp
_______________________________________________
talk mailing list
[email protected] <mailto:[email protected]>
http://openaerialmap.org/mailman/listinfo/talk_openaerialmap.org
------------------------------------------------------------------------
_______________________________________________
talk mailing list
[email protected]
http://openaerialmap.org/mailman/listinfo/talk_openaerialmap.org
_______________________________________________
talk mailing list
[email protected]
http://openaerialmap.org/mailman/listinfo/talk_openaerialmap.org
