Dane,
Thank you very much for the thoughtful response. My team has discovered
much of this on their own but I think you've definitely got some "experience
only" nuggets that we'd have to figure out the hard way otherwise without
your reply so it's much appreciated. I've dealt with creating tile servers
in the past so have some strong opinions on optimal ways of pre-caching. If
the proof of concept goes well and our client funds the full follow-up
project that will definitely be in scope and we will contribute some of
these general capabilities to the GIS community. Thanx for the great work
and general helpful attitude!
-- Ben Scherrey
On Fri, Nov 19, 2010 at 11:35 PM, Dane Springmeyer <[email protected]> wrote:
>
> On Nov 16, 2010, at 12:23 AM, Benjamin Scherrey wrote:
>
> > Just recently stumbled upon MapNik
>
> Ben, welcome to Mapnik.
>
> > and would like to use it in our upcoming project as it appears to fit the
> exact technologies we're looking for. We are a python/django/postgis/C++
> shop so it was great to see all these things seemingly brought together for
> us.
>
> Yes, several of the lead developers of Mapnik are also longtime django
> users.
>
> > There's so much stuff that MapNik connects to, however,
>
> > that I'm not sure how much of the capabilities of the underlying
> libraries are well supported or easily accessed.
>
> > Would appreciate if anyone could give me any pointers in MapNik "best
> practices" or, even better, sample code out there that is doing something
> very similar to our fairly common use case.
> >
> > The proof-of-concept is a simple zoomable map with a vector street
> map layer + superimposed raster imagery layer (orthorectified to fit the
> prior layer correctly) + another superimposed grid of some specified
> length/width rectangles which will all be managed in a PostGIS DB and viewed
> from a web browser. The follow up would be to tile the imagery (if this
> isn't done automatically) and support multiple zoom levels of imagery ala
> Google Maps. Our challenge, beyond getting the base functionality working,
> is to also make it highly responsive to the map viewing user.
>
> Yes, Mapnik can be the rendering engine that handles reading in data from
> those variety of datasources (either raster imagery from the file system
> with the GDAL plugin [1] or vector data from PostGIS[2]) and sending along
> to a web client or desktop app.
>
> The Mapnik python bindings can also be used to programmatically inspect
> your data, like grabbing the extent or envelope:
>
> import mapnik
> bbox = mapnik.Gdal(file="/path/to/geo.tiff").envelope()
>
> Ultimately, you'll need to put a server in front of Mapnik that meets the
> demands of your use case.
>
> mod_tile, TileStash, and tirex are the three tile-based servers I am most
> familiar with that excel with Mapnik. TileCache also works well but must be
> used with mod_wsgi daemon mode with threads=1 (e.g NOT mod_python otherwise
> you'll hit thread safety issues).
>
> > So my questions are:
> >
> > 1. What aspects of the above are already well supported by MapNik?
>
> Mapnik is just the backend library that handles turning abstract
> datasources into either rendered tiles or styled svg/pdf (or in trunk also
> geojson [3]).
>
> Combine it with a tile server (as listed above), a database (like PostGIS),
> and a front end UI (like OpenLayers or Polymaps), and you have a full app.
>
> > 2. What aspects of the above maybe aren't "built-in" to MapNik directly
> but would be fairly easily implemented on top of it and that you feel MapNik
> is an appropriate platform to build upon?
>
> Application logic like permissions to certain datasets, custom styles of
> certain users, etc. Stuff like django might handle. I've done some R&D on
> deeper integration between GeoDjango spatial types and Mapnik (see [4]), so
> you can pass - in pure python - spatial querysets to be rendered by mapnik.
> But, in general if you want highly scalable and performant tile services
> then you likely DONT want to mix your tiles with your application, so that
> they can be separate concerns, on separate machines.
>
> > 3. What aspect of the above does MapNik currently have no support for and
> should be done using other tools?
>
> Wrangling raw data. Use the GDAL/OGR toolset (command line utilities) for
> prepping your GIS data, like georeferenceing and reprojecting raster
> datasets to your target projection.
>
> > 4. Is there any open source examples of doing major aspects of the above
> that we should build upon for our initial proof of concept?
>
> Yes, but your use case to me is still pretty abstract so I can't recommend
> anything specific.
>
> > 5. Regarding user responsiveness, has anyone tried various techniques of
> pre-loading tiles behind the scenes in AJAX,
>
> I've not messed with that - bigger moles to wack usually for performance
> and pre-caching commonly loaded tiles is way faster than trying to guess
> what direction someone is going to zoom at any second and does not cost
> anything in terms of http requests.
>
> That said, if you come up with a good algorithm it would be cool to check
> out.
>
> > storing tile URIs in a manner where memcache or an independent media
> server can serve them up interdependently of the app server, etc..?
>
> Basically that is 101 tile serving. All the servers listed above use
> cacheable schemes. If your data does not change and you have the disk space
> precaching and serving statically through apache, lighttpd, nginx is the
> fastest method around. Just be aware that if you need zoom levels past
> around 16-17 and continental or global extents then that is WAY more tiles
> that you want to precache, so caching on-demand is what most people do,
> while precaching at zooms 0-5,6...
>
> > Be curious to hear any recommendations for techniques or experience as to
> what is typically the bottleneck.
> >
>
> Depends on the data of course. How big it is and how frequently it changes,
> and how complex or dynamic your styling needs are.
>
> Generally however with big datasets - common in mapping - in the many GB's
> range, disk-boundness is a common mole.
>
> > I would expect what we're doing has been done by many before us so
> would like to avoid reinventing the wheel and get a chance to spend some of
> this effort making the wheel better. :-)
>
> :)
>
> > We will certainly be willing to share the results of anything we come up
> with that may be useful.
> >
>
> "willing" ;) Actions speak louder than words.
>
> > Thanks for any pointers/advice/interest!
>
> [1] http://trac.mapnik.org/wiki/GDAL
> [2] http://trac.mapnik.org/wiki/PostGIS
> [3] http://trac.mapnik.org/wiki/MetaWriter
> [4] http://bitbucket.org/springmeyer/djmapnik
>
> >
> > -- Ben Scherrey
> > _______________________________________________
> > Mapnik-users mailing list
> > [email protected]
> > https://lists.berlios.de/mailman/listinfo/mapnik-users
>
>
_______________________________________________
Mapnik-users mailing list
[email protected]
https://lists.berlios.de/mailman/listinfo/mapnik-users
