Re: [OSM-talk] Slow TileMill rendering - Postgres using 1 core?
As far as I know, Tilemill is using mapnik which is querying postgres. Plain vanilla Mapnik is not doing more than one postgres query at a time (not multithreading queries). A patch made by mappy allows mapnik to multithread its pg queries. Are you using the exact same version of Mapnik as before ? 2013/8/25 Steve Bennett stevag...@gmail.com Hi all, I'm running TileMill on an 8 core Ubuntu VM with 32GB of memory, on an OpenStack cloud. Recently, my VM was destroyed, and I rebuilt it (identically, I thought) on slightly different hardware (same cloud, but different physical infrastructure). The new build is much slower at rendering - a screen worth of tiles at zoom 13 can take around a minute. That is, with virtually the same setup, same data, same styles. You can see some slow tiles here: http://emscycletours.site44.com/mel.html While panning around, the 'top' command shows mostly Postgres processes (different from last time I had performance problems[1], when the bottleneck was in Mapnik). Total CPU usage hangs around 12%: ie, exactly 1 out of 8 cores is being used. https://dl.dropboxusercontent.com/u/767553/GIS/Screen%20shot%202013-08-25%20at%2011.15.01%20AM.png top - 11:10:32 up 3 days, 36 min, 1 user, load average: 0.06, 0.17, 0.22 Tasks: 133 total, 4 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.5 us, 0.1 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32950396 total, 7150132 used, 25800264 free, 117864 buffers KiB Swap:0 total,0 used,0 free, 5221356 cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2353 postgres 20 0 8510m 640m 635m S 41.2 2.0 23:36.57 postgres 2354 postgres 20 0 8510m 644m 639m S 40.2 2.0 23:24.26 postgres 2350 postgres 20 0 8510m 642m 638m S 14.0 2.0 23:19.19 postgres 2375 postgres 20 0 8510m 643m 639m S 14.0 2.0 23:17.80 postgres 13102 postgres 20 0 8508m 531m 527m S 13.6 1.7 13:03.21 postgres 2355 postgres 20 0 8508m 531m 526m S 13.3 1.7 13:45.15 postgres 2352 postgres 20 0 8510m 640m 636m S 10.0 2.0 23:31.17 postgres 2348 postgres 20 0 8510m 644m 639m S 9.3 2.0 23:41.88 postgres 12420 mapbox20 0 3818m 1.0g 755m S 9.3 3.2 36:48.39 nodejs 2357 postgres 20 0 8508m 530m 526m S 7.3 1.7 13:38.57 postgres 2356 postgres 20 0 8508m 531m 526m R 6.3 1.7 13:42.52 postgres 2376 postgres 20 0 8508m 531m 527m S 6.0 1.7 13:35.51 postgres 13195 postgres 20 0 8508m 531m 527m S 5.3 1.7 12:33.65 postgres 3027 postgres 20 0 8508m 531m 527m R 3.3 1.7 13:29.06 postgres 2349 postgres 20 0 8508m 530m 526m S 3.0 1.6 13:38.19 postgres 2358 postgres 20 0 8508m 531m 527m S 3.0 1.7 13:44.59 postgres 26 root 20 0 000 S 0.3 0.0 0:08.64 ksoftirqd/5 2335 postgres 20 0 8489m 2732 1340 S 0.3 0.0 1:00.48 postgres So, wondering if anyone has any suggestions what the problem is, or how to fix it? Why is Postgres apparently using only one core, even though it has many processes? What tools could I use to further diagnose? My changed Postgres settings are as follows: shared_buffers = 8GB autovacuum = on effective_cache_size = 8GB work_mem = 128MB maintenance_work_mem = 64MB wal_buffers = 1MB checkpoint_segments = 10 The server is set up as described here: http://steveko.wordpress.com/2013/05/08/tilemill-server/ I'm not yet using any tile cache. I will do that next, but the problem I'm trying to solve at the moment is very slow tile generation, not slow serving of rendered tiles. Many thanks in advance, Steve [1] http://gis.19327.n5.nabble.com/TileMill-performance-td5751158.html ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Christian Quest - OpenStreetMap France Un nouveau serveur pour OSM... http://donate.osm.org/server2013/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Slow TileMill rendering - Postgres using 1 core?
Hi Christian, Thanks for the reply. After further investigation the actual problem was a missing index on planet_osm_polygon. (I'm really not sure why.) Steve On Sun, Aug 25, 2013 at 9:00 PM, Christian Quest cqu...@openstreetmap.frwrote: As far as I know, Tilemill is using mapnik which is querying postgres. Plain vanilla Mapnik is not doing more than one postgres query at a time (not multithreading queries). A patch made by mappy allows mapnik to multithread its pg queries. Are you using the exact same version of Mapnik as before ? 2013/8/25 Steve Bennett stevag...@gmail.com Hi all, I'm running TileMill on an 8 core Ubuntu VM with 32GB of memory, on an OpenStack cloud. Recently, my VM was destroyed, and I rebuilt it (identically, I thought) on slightly different hardware (same cloud, but different physical infrastructure). The new build is much slower at rendering - a screen worth of tiles at zoom 13 can take around a minute. That is, with virtually the same setup, same data, same styles. You can see some slow tiles here: http://emscycletours.site44.com/mel.html While panning around, the 'top' command shows mostly Postgres processes (different from last time I had performance problems[1], when the bottleneck was in Mapnik). Total CPU usage hangs around 12%: ie, exactly 1 out of 8 cores is being used. https://dl.dropboxusercontent.com/u/767553/GIS/Screen%20shot%202013-08-25%20at%2011.15.01%20AM.png top - 11:10:32 up 3 days, 36 min, 1 user, load average: 0.06, 0.17, 0.22 Tasks: 133 total, 4 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.5 us, 0.1 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32950396 total, 7150132 used, 25800264 free, 117864 buffers KiB Swap:0 total,0 used,0 free, 5221356 cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2353 postgres 20 0 8510m 640m 635m S 41.2 2.0 23:36.57 postgres 2354 postgres 20 0 8510m 644m 639m S 40.2 2.0 23:24.26 postgres 2350 postgres 20 0 8510m 642m 638m S 14.0 2.0 23:19.19 postgres 2375 postgres 20 0 8510m 643m 639m S 14.0 2.0 23:17.80 postgres 13102 postgres 20 0 8508m 531m 527m S 13.6 1.7 13:03.21 postgres 2355 postgres 20 0 8508m 531m 526m S 13.3 1.7 13:45.15 postgres 2352 postgres 20 0 8510m 640m 636m S 10.0 2.0 23:31.17 postgres 2348 postgres 20 0 8510m 644m 639m S 9.3 2.0 23:41.88 postgres 12420 mapbox20 0 3818m 1.0g 755m S 9.3 3.2 36:48.39 nodejs 2357 postgres 20 0 8508m 530m 526m S 7.3 1.7 13:38.57 postgres 2356 postgres 20 0 8508m 531m 526m R 6.3 1.7 13:42.52 postgres 2376 postgres 20 0 8508m 531m 527m S 6.0 1.7 13:35.51 postgres 13195 postgres 20 0 8508m 531m 527m S 5.3 1.7 12:33.65 postgres 3027 postgres 20 0 8508m 531m 527m R 3.3 1.7 13:29.06 postgres 2349 postgres 20 0 8508m 530m 526m S 3.0 1.6 13:38.19 postgres 2358 postgres 20 0 8508m 531m 527m S 3.0 1.7 13:44.59 postgres 26 root 20 0 000 S 0.3 0.0 0:08.64 ksoftirqd/5 2335 postgres 20 0 8489m 2732 1340 S 0.3 0.0 1:00.48 postgres So, wondering if anyone has any suggestions what the problem is, or how to fix it? Why is Postgres apparently using only one core, even though it has many processes? What tools could I use to further diagnose? My changed Postgres settings are as follows: shared_buffers = 8GB autovacuum = on effective_cache_size = 8GB work_mem = 128MB maintenance_work_mem = 64MB wal_buffers = 1MB checkpoint_segments = 10 The server is set up as described here: http://steveko.wordpress.com/2013/05/08/tilemill-server/ I'm not yet using any tile cache. I will do that next, but the problem I'm trying to solve at the moment is very slow tile generation, not slow serving of rendered tiles. Many thanks in advance, Steve [1] http://gis.19327.n5.nabble.com/TileMill-performance-td5751158.html ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Christian Quest - OpenStreetMap France Un nouveau serveur pour OSM... http://donate.osm.org/server2013/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Slow TileMill rendering - Postgres using 1 core?
Hi all, I'm running TileMill on an 8 core Ubuntu VM with 32GB of memory, on an OpenStack cloud. Recently, my VM was destroyed, and I rebuilt it (identically, I thought) on slightly different hardware (same cloud, but different physical infrastructure). The new build is much slower at rendering - a screen worth of tiles at zoom 13 can take around a minute. That is, with virtually the same setup, same data, same styles. You can see some slow tiles here: http://emscycletours.site44.com/mel.html While panning around, the 'top' command shows mostly Postgres processes (different from last time I had performance problems[1], when the bottleneck was in Mapnik). Total CPU usage hangs around 12%: ie, exactly 1 out of 8 cores is being used. https://dl.dropboxusercontent.com/u/767553/GIS/Screen%20shot%202013-08-25%20at%2011.15.01%20AM.png top - 11:10:32 up 3 days, 36 min, 1 user, load average: 0.06, 0.17, 0.22 Tasks: 133 total, 4 running, 129 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.5 us, 0.1 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 32950396 total, 7150132 used, 25800264 free, 117864 buffers KiB Swap:0 total,0 used,0 free, 5221356 cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2353 postgres 20 0 8510m 640m 635m S 41.2 2.0 23:36.57 postgres 2354 postgres 20 0 8510m 644m 639m S 40.2 2.0 23:24.26 postgres 2350 postgres 20 0 8510m 642m 638m S 14.0 2.0 23:19.19 postgres 2375 postgres 20 0 8510m 643m 639m S 14.0 2.0 23:17.80 postgres 13102 postgres 20 0 8508m 531m 527m S 13.6 1.7 13:03.21 postgres 2355 postgres 20 0 8508m 531m 526m S 13.3 1.7 13:45.15 postgres 2352 postgres 20 0 8510m 640m 636m S 10.0 2.0 23:31.17 postgres 2348 postgres 20 0 8510m 644m 639m S 9.3 2.0 23:41.88 postgres 12420 mapbox20 0 3818m 1.0g 755m S 9.3 3.2 36:48.39 nodejs 2357 postgres 20 0 8508m 530m 526m S 7.3 1.7 13:38.57 postgres 2356 postgres 20 0 8508m 531m 526m R 6.3 1.7 13:42.52 postgres 2376 postgres 20 0 8508m 531m 527m S 6.0 1.7 13:35.51 postgres 13195 postgres 20 0 8508m 531m 527m S 5.3 1.7 12:33.65 postgres 3027 postgres 20 0 8508m 531m 527m R 3.3 1.7 13:29.06 postgres 2349 postgres 20 0 8508m 530m 526m S 3.0 1.6 13:38.19 postgres 2358 postgres 20 0 8508m 531m 527m S 3.0 1.7 13:44.59 postgres 26 root 20 0 000 S 0.3 0.0 0:08.64 ksoftirqd/5 2335 postgres 20 0 8489m 2732 1340 S 0.3 0.0 1:00.48 postgres So, wondering if anyone has any suggestions what the problem is, or how to fix it? Why is Postgres apparently using only one core, even though it has many processes? What tools could I use to further diagnose? My changed Postgres settings are as follows: shared_buffers = 8GB autovacuum = on effective_cache_size = 8GB work_mem = 128MB maintenance_work_mem = 64MB wal_buffers = 1MB checkpoint_segments = 10 The server is set up as described here: http://steveko.wordpress.com/2013/05/08/tilemill-server/ I'm not yet using any tile cache. I will do that next, but the problem I'm trying to solve at the moment is very slow tile generation, not slow serving of rendered tiles. Many thanks in advance, Steve [1] http://gis.19327.n5.nabble.com/TileMill-performance-td5751158.html ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Slow?
Is it just me, or are the website and API wicked slow right now? ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Slow?
On Tue, 16 Nov 2010 21:23:58 -0500, Nathan Edgars II nerou...@gmail.com wrote: Is it just me, or are the website and API wicked slow right now? It's not just you. I've had waits for minutes uploading 10-part chunks. Maarten ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
