I appreciate the guidance. The performance does seem to be on par with mapserver now with teh google_300 shapefile. I did notice some strange artifacts but i didn't try to troubleshoot it... (http://dl.dropbox.com/u/203098/artifacts.PNG)
I think unfortunately this doesn't help me for the high zoom scenario, where i need the detailed shapes. Low zoom scenarios can be pre-rendered, while the high zoom needs to be rendered on demand. I may see how far I can get with mapserver for the time being. Please let me know if you think i should try a different method. On Wed, Jul 14, 2010 at 7:49 PM, Jon Burgess <[email protected]> wrote: > On Wed, 2010-07-14 at 17:52 -0400, joel collins wrote: >> http://dl.dropbox.com/u/203098/mapnik.7z >> >> File is 30mb compressed, 80mb uncompressed. Today i setup mapnik on >> apache via fastcgi, and the performance is still about 10x slower than >> mapserver. I'd really like to know if i'm doing something dumb... >> >> The .index file may not have found its way into the 7z file, but i did >> generate it using shapeindex. Also, just as a sanity check, here are >> 2 example urls i'm using to query the data: >> >> Mapserver: (response time 60ms) >> http://servername/cgi-bin/mapserv.exe?map=C%3A%2FOSGeo4W%2Fapps%2Fmapserv-demo%2Fdistricts_googproj.map&transparent=true&mode=map&map_imagetype=png&mapext=-10018754.1704+4383204.9484+-9705668.1026+4696291.0162&imgext=-10018754.1704+4383204.9484+-9705668.1026+4696291.0162&map_size=256+256&imgx=128&imgy=128&imgxy=256+256 >> >> Mapnik: >> http://servername/fcgi-bin/wms?LAYERS=dist_google&VERSION=1.1.1&UNITS=meters&SERVICE=WMS&REQUEST=GetMap&STYLES=&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&FORMAT=image%2Fjpeg&SRS=EPSG%3A900913&BBOX=-10018754.1704,4383204.9484,-9705668.1026,4696291.0162&WIDTH=256&HEIGHT=256 > > With your shapefile & WMS setup I get about 6 requests per second using > Mapnik which seems to roughly match your performance. > > It looks to me that the slow performance may be because the geometries > in your shapefiles are quite detailed. If I run the commands below to > simplify the shapefile from 80MB to 2.5MB then the rendering using > google_300.shp is about 10 times faster with no obvious degradation in > the output quality. > > $ shp2pgsql dist_google.shp | psql gis > $ pgsql2shp -f google_300 gis "select simplify(the_geom, 300) as > the_geom, color from dist_google where area(simplify(the_geom,300)) > 0" > > I don't know whether Mapserver is better at deciding how much data > detail of the geometries to render or if it has a simpler rendering > model which is faster at filling complex polygons. > > At higher zooms you may need to go back to rendering from the more > detailed shapefiles again. > > Jon > > >> On Wed, Jul 14, 2010 at 4:16 AM, Artem Pavlenko <[email protected]> wrote: >> > Hi Joel, >> > >> > Could you post that shapefile somewhere or provide a link, pls. >> > >> > Artem >> > >> > On 14 July 2010 00:11, joel collins <[email protected]> wrote: >> >> Sorry guys, i know i've sent a lot of mail about this. I managed to >> >> get my shapefile loaded into mapserver to try to compare performance, >> >> and the difference is night and day. There is essentially no >> >> noticable lag when viewing the shapefile via mapserver, regardless of >> >> zoom level. >> >> >> >> Do you think i'm doing something wrong that is making mapnik a lot >> >> slower than mapserver? My layer is very simple, just some borders >> >> essentially. Am I better off using a tool like mapserver or geoserver >> >> rather than mapnik for simple layers without much visual pizzaz? >> >> >> >> I am frustrated because the toolsets around mapnik seem much more >> >> modern and easy for me to comprehend (cascadenick, etc), while >> >> mapserver seems more difficult to grasp. >> >> >> >> On Mon, Jul 12, 2010 at 9:42 PM, Dane Springmeyer <[email protected]> wrote: >> >>> >> >>> On Jul 12, 2010, at 6:14 PM, joel collins wrote: >> >>> >> >>>> I realize that my performance will improve using apache / mod_tile, >> >>>> but 0.5 cpu seconds to render a tiny tile is a lot! Even with a 4 >> >>>> core machine this would mean that 8 uncached tile requests per second >> >>>> would max out the CPU. I have kept experimenting and found >> >>>> (obviously, in hindsight), that since my shapefile was not in the >> >>>> google projection, this conversion was being done at runtime. >> >>> >> >>> Yes, reprojection on the fly is very costly, so ensuring the layer srs >> >>> exactly matches the map srs is a critical thing. >> >>> >> >>>> after >> >>>> running ogr2ogr my render time is below 0.2 seconds for zoom level 10. >> >>>> This may have to do... >> >>> >> >>> nice. >> >>> >> >>>> >> >>>> An interesting thing i noticed was when running on windows (osgeo4w >> >>>> installation), mapnik was 2x as slow as mapnik running on an ubuntu VM >> >>>> on the same physical machine! >> >>> >> >>> Make sure you rebuild any indexes for shapefiles on each platform. If >> >>> that does not bring the speeds more closely inline then it would be >> >>> worthwhile to do a bit more digging as they should be comparable. >> >>> >> >>>> >> >>>> On Thu, Jul 8, 2010 at 6:37 PM, Dane Springmeyer <[email protected]> >> >>>> wrote: >> >>>>> >> >>>>> On Jul 7, 2010, at 5:16 PM, joel collins wrote: >> >>>>> >> >>>>>> New mapnik user here, i've made a lot of progress in the last few >> >>>>>> weeks but its been slow going. I am trying to create a tile overlay >> >>>>>> on google maps of some census bureau tiger data (congressional >> >>>>>> district maps in particular) and i have downloaded the associated >> >>>>>> shapefiles (containing polygons only, no markers or other data types). >> >>>>>> >> >>>>>> For a web server I am currently just using Tilelight (mapnik-utils) >> >>>>>> for development, and although i'm planning on using a different >> >>>>>> configuration in production, i'm seeing very poor performance during >> >>>>>> tile generation. >> >>>>> >> >>>>> That's because the 'liteserv.py' script is only for development, and >> >>>>> deployment targets multicore architectures only (mod_wsgi with N >> >>>>> processes = number of cores). >> >>>>> >> >>>>> Although I too find that python's wsgiref server implementation (that >> >>>>> liteserv.py) uses is very slow. You can get about double the speed >> >>>>> with other development servers that implement the WSGI standard, one >> >>>>> being werkzeug. In fact, just the other day I added support for >> >>>>> werkzeug: >> >>>>> >> >>>>> http://bitbucket.org/springmeyer/tilelite/changeset/e2335783f42c >> >>>>> >> >>>>> But, again the faster deployment for TileLite is with mod_wsgi inside >> >>>>> Apache. >> >>>>> >> >>>>> And faster again will be a multithreaded server like mod_tile. >> >>>>> >> >>>>>> On my google maps, for example, when i scroll to a >> >>>>>> new area or change zoom levels it can often take 15 seconds for the >> >>>>>> new tiles to appear (this is especially slow because there are about a >> >>>>>> dozen simultaneous tile requests that queue up). >> >>>>> >> >>>>> That's because you are running a single-threaded, single-process >> >>>>> development server that is only for development. >> >>>>> >> >>>>>> I realize that on a >> >>>>>> multi-core production machine this wont be so bad but it still seems >> >>>>>> very slow. >> >>>>>> >> >>>>>> When I render a single tile, it still takes between 0.5-3 seconds. I >> >>>>>> have tried loading the shapes into postgres but that actually resulted >> >>>>>> in worse performance (5 seconds), although I may have configured >> >>>>>> something wrong. Some rudimentary math tells me that I wont be able >> >>>>>> to pre-render all tiles (we would like to be able to show street-level >> >>>>>> zooming in google maps, around 16 or 17), so we will need to have the >> >>>>>> ability to render tiles that haven't been pre-rendered fast enough for >> >>>>>> real time browsing. Also, using something like TileCache will help of >> >>>>>> course but at the end of the day new tiles still need to be rendered. >> >>>>>> >> >>>>>> My questions are: >> >>>>>> What kind of response times should I expect from mapnik for creating >> >>>>>> tiles from shapefiles? >> >>>>>> >> >>>>>> What can be done to speed up the rendering? Should I be downsampling >> >>>>>> the number of data points in the shape files? We like the high level >> >>>>>> of accuracy that we see in the tiger files but maybe thats what is >> >>>>>> slowing this down? >> >>>>>> >> >>>>>> Is leaving the data in shapefiles the appropriate approach? (I was >> >>>>>> planning on using postgis so i could do other queries like see which >> >>>>>> polygons touch, find what polygons are near a lat/long, etc) >> >>>>>> >> >>>>>> Thanks! >> >>>>>> _______________________________________________ >> >>>>>> Mapnik-users mailing list >> >>>>>> [email protected] >> >>>>>> https://lists.berlios.de/mailman/listinfo/mapnik-users >> >>>>> >> >>>>> >> >>> >> >>> >> >> _______________________________________________ >> >> Mapnik-users mailing list >> >> [email protected] >> >> https://lists.berlios.de/mailman/listinfo/mapnik-users >> >> >> > >> _______________________________________________ >> Mapnik-users mailing list >> [email protected] >> https://lists.berlios.de/mailman/listinfo/mapnik-users > > > _______________________________________________ Mapnik-users mailing list [email protected] https://lists.berlios.de/mailman/listinfo/mapnik-users
