Re: [OSM-talk] Building a tile-server
On Mon, Jul 6, 2020 at 17:52 Tom Browder wrote: > On Fri, Jul 3, 2020 at 12:56 Tom Browder wrote: > >> On Wed, Jul 1, 2020 at 15:18 Tom Browder wrote: >> > Yesterday I took my first try at using osm2pgsql on a pbm file and, >> > too late, I realized I didn't take full advantage of the server's >> > large RAM and multi-core capability. >> ... >> Okay, I'm making progress, thanks to you all. > > > But got stopped again running out of space trying to import > north-america-latest.osm.pbf (approx 10 Gb pbm file). > > I had a 256 Gg SSD added and moved both the osm user and the postgresql > data files to it. The failure was at approx 147 Gb pgsql and 60 Gb osm. > > So, which should I move away? If I had my choice I would keep the pg on > SSD for serving, but is there enough space? > It looks like I may have hit a sweet spot in my blundering (fingers crossed). The process is still running. The Pg on the 240 Gb SSD is taking 150 Gb and, finally, relations are being processed. -Tom > ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Mon, Jul 6, 2020 at 23:52 José Juan Montes wrote: > > Entire Spain or Germany, imported into PostgreSQL with carto style, > without history, range between 30-60 GB after import (the corresponding > PBF files are 800MB-3GB). I'm speaking from the top of my head, but maybe > that helps you figuring out your disk space needs. > > I seem to recall some wiki mentions that a full planet import will > take around 2TB, but I can't find that now. This page has some figures: > https://wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks#Planet_import_on_custom_E5-1650_.2832GB_RAM_-_6_cores.29 > , > and as you can see, the full planet import took already hundreds of GB > several years ago. > Thank you, Jose, that's very helpful! -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Fri, Jul 3, 2020 at 12:56 Tom Browder wrote: > On Wed, Jul 1, 2020 at 15:18 Tom Browder wrote: > > Yesterday I took my first try at using osm2pgsql on a pbm file and, > > too late, I realized I didn't take full advantage of the server's > > large RAM and multi-core capability. > ... > > Okay, I'm making progress, thanks to you all. But got stopped again running out of space trying to import north-america-latest.osm.pbf (approx 10 Gb pbm file). I had a 256 Gg SSD added and moved both the osm user and the postgresql data files to it. The failure was at approx 147 Gb pgsql and 60 Gb osm. So, which should I move away? If I had my choice I would keep the pg on SSD for serving, but is there enough space? I can get a larger single SSD up to 1 Tb but my budget is not unlimited. Does anyone have ball-park figures available on space really needed during and after import? Thanks for any hints. -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Fri, Jul 3, 2020 at 1:44 PM Andy Townsend wrote:... ... > If you get to the point where you've got something that could be turned > into a Debian version of > https://switch2osm.org/serving-tiles/manually-building-a-tile-server-20-04-lts/ I'll be glad to, Andy. You can follow my progress somewhat by checking my github repo at: https://github.com/tbrowder/config-scripts/tree/master/Openstreetmap Can you recommend a good link on the workflow for building vector tiles? Julien has done it with another language and I'm not sure I will be able to follow it well enough. Best regards, -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On 03/07/2020 18:56, Tom Browder wrote: On Wed, Jul 1, 2020 at 15:18 Tom Browder wrote: ... Okay, I'm making progress, thanks to you all. First, lesson learned (as suggested by all): I used Antarctica pbm as a test bed for import which showed an immediate problem with osm2pgsql. Thanks to Sarah about the requirement for a current osm2pgsql. I wound up compiling and installing from source which made the import process work. Strangely, that version shows as 1.2.0 while the Buster backports shows version 1.21! If you get to the point where you've got something that could be turned into a Debian version of https://switch2osm.org/serving-tiles/manually-building-a-tile-server-20-04-lts/ that'd be great - either the whole new page or just a set of notes that someone could add some text around. Also (not necessarily the same page) it'd be good to have a Tirex version of those notes for https://switch2osm.org too. The source for the pages is over at https://github.com/switch2osm/switch2osm.github.io . Best Regards, Andy ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Wed, Jul 1, 2020 at 15:18 Tom Browder wrote: > Yesterday I took my first try at using osm2pgsql on a pbm file and, > too late, I realized I didn't take full advantage of the server's > large RAM and multi-core capability. ... Okay, I'm making progress, thanks to you all. First, lesson learned (as suggested by all): I used Antarctica pbm as a test bed for import which showed an immediate problem with osm2pgsql. Thanks to Sarah about the requirement for a current osm2pgsql. I wound up compiling and installing from source which made the import process work. Strangely, that version shows as 1.2.0 while the Buster backports shows version 1.21! Thanks to Paul for the postgresql settings I made for work_mem and maintenance_work_mem (and the size catch by Martin!). As to suggestions to change other postgresql settings I got mixed signals. One I didn't ask about was autovacuum on or off? I plan to try Frederik's tirex at some point after I get a working server. Finally, thanks to Julien for the links to his vector tile work. I will have to study that a bit more later. Cheers! -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Thu, Jul 2, 2020 at 02:41 Sarah Hoffmann wrote: > On Wed, Jul 01, 2020 at 03:18:30PM -0500, Tom Browder wrote: ... > > Should I compile those from source? > > Make sure to use osm2pgsql from backports. This always has > the newest released version. Thanks, Sarah! -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Thu, Jul 2, 2020 at 02:01 Julien djakk wrote: > Hello Tom, > > About vector tiles : I have managed to render some vector tiles on > heroku : https://github.com/djakk/openstreetmap-on-heroku > The database is populated thanks to osm2pgsql. Rendering is done by mapnik. > Maybe you can adapt it to your server ? Thanks, Julien! -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Wed, Jul 01, 2020 at 03:18:30PM -0500, Tom Browder wrote: > I am running Debian Buster on a bare-iron dedicated server with 32 Gb > RAM, 32 cores, and a single 4 TB spinning disk. I am running Pg 12 > (the version maintained by the Pg developers) with its default > configuration. > > My PostGIS is the latest Debian package version (version ?). osm2pgsql > is also a package (version ?). > > Should I compile those from source? Make sure to use osm2pgsql from backports. This always has the newest released version. Be aware that you'll likely need osm2pgsql 1.2.2 at the moment to import a planet. If your import gets stuck when reading relations then your osm2pgsql is too old. osm2pgsql 1.2.2 should be available in backports in a couple of days. Sarah ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
Hello Tom, About vector tiles : I have managed to render some vector tiles on heroku : https://github.com/djakk/openstreetmap-on-heroku The database is populated thanks to osm2pgsql. Rendering is done by mapnik. Maybe you can adapt it to your server ? Julien "djakk" J Le jeu. 2 juil. 2020 à 02:44, Paul Norman via talk a écrit : > > On 2020-07-01 3:28 p.m., Martin Koppenhoefer wrote: > > > > sent from a phone > > > >> On 1. Jul 2020, at 23:26, Paul Norman via talk > >> wrote: > >> > >> In general, work_mem=128GB is good with most styles. > > > > > > Paul, he wrote he had 32GB of RAM, should one assign more work_mem than > > there physically is on the machine? > > > > I am asking because I thought that the value of work_mem could be used > > multiple times and the sum should not exceed the actual RAM, but maybe this > > isn’t how it works? > > > > Cheers Martin > > > Whoops, that should be 128MB. > > > ___ > talk mailing list > talk@openstreetmap.org > https://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On 2020-07-01 3:28 p.m., Martin Koppenhoefer wrote: sent from a phone On 1. Jul 2020, at 23:26, Paul Norman via talk wrote: In general, work_mem=128GB is good with most styles. Paul, he wrote he had 32GB of RAM, should one assign more work_mem than there physically is on the machine? I am asking because I thought that the value of work_mem could be used multiple times and the sum should not exceed the actual RAM, but maybe this isn’t how it works? Cheers Martin Whoops, that should be 128MB. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
sent from a phone > On 1. Jul 2020, at 23:26, Paul Norman via talk wrote: > > In general, work_mem=128GB is good with most styles. Paul, he wrote he had 32GB of RAM, should one assign more work_mem than there physically is on the machine? I am asking because I thought that the value of work_mem could be used multiple times and the sum should not exceed the actual RAM, but maybe this isn’t how it works? Cheers Martin ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On Wed, Jul 1, 2020 at 16:26 Paul Norman via talk wrote: ... Thank you, Paul. -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
On 2020-07-01 1:42 p.m., Frederik Ramm wrote: + I have seen a couple of different postgresql config suggestions. Is there a one-size fits all or should I tailor it more to my server's configuration. Most configs you see will be for earlier Postgres versions and hence not necessarily valid for Pg 12. Switching off everything that has to do with "safe writing", e.g. fsync, certainly makes sense (you have to restart the import anyway if you suffer from power failure mid-way, no use in setting up Postgres to avoid data loss when writing). The two settings you should adjust are work_mem and maintenance_work_mem. In general, work_mem=128GB is good with most styles. I haven't experiment with more. If you have the RAM, maintenance_work_mem=4GB would be plenty. Do not adjust synchronous_commit. osm2pgsql turns it off for the import, and this will override your setting. Do not adjust fsync. synchronous_commit gets you the performance gains. You may want to increase checkpoint_timeout and checkpoint_completion_target, but these offer minor gains at mosts. ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Building a tile-server
Hi, On 7/1/20 22:18, Tom Browder wrote: > Yesterday I took my first try at using osm2pgsql on a pbm file and, > too late, I realized I didn't take full advantage of the server's > large RAM and multi-core capability. That's because the task is immensely disk intensive, and all the CPU and RAM sit there bored waiting for the disk to finish, no matter how many cores you try to utilize. If you want to import anything larger than a medium sized country, you will have to upgrade to a SSD or NVMe disk. > + should I dump all the tables created before I start anew? No, osm2pgsl will truncate them on import unless you explicitly ask for an "append". > + I have seen a couple of different postgresql config suggestions. Is > there a one-size fits all or should I tailor it more to my server's > configuration. Most configs you see will be for earlier Postgres versions and hence not necessarily valid for Pg 12. Switching off everything that has to do with "safe writing", e.g. fsync, certainly makes sense (you have to restart the import anyway if you suffer from power failure mid-way, no use in setting up Postgres to avoid data loss when writing). > My PostGIS is the latest Debian package version (version ?). osm2pgsql > is also a package (version ?). > > Should I compile those from source? Not necessary IMHO. > Of course any other advice is welcomed, especially as to which other > pieces are recommended downstream for the actual tile server. And I > would really like to have vector tiles, Vector tiles are produced with a wholly different toolchain, sometimes not even using osm2pgsql at all. Even if you have a toolchain that uses osm2pgsql, it might come with a specific style file it expects you to use on osm2pgsql import... > Oh, for the actual rendering: > > + prefer renderd or tirex? These are raster tile only. Personally I prefer tirex because I am one of its authors. I think it is superior to renderd in every aspect, except that renderd has the larger user base and hence it will be easier to find someone to help you when there's a problem. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Building a tile-server
Yesterday I took my first try at using osm2pgsql on a pbm file and, too late, I realized I didn't take full advantage of the server's large RAM and multi-core capability. This morning I killed the import process and want to start anew. Some questions: + should I dump all the tables created before I start anew? + when I import pbfs in + I have seen a couple of different postgresql config suggestions. Is there a one-size fits all or should I tailor it more to my server's configuration. I am running Debian Buster on a bare-iron dedicated server with 32 Gb RAM, 32 cores, and a single 4 TB spinning disk. I am running Pg 12 (the version maintained by the Pg developers) with its default configuration. My PostGIS is the latest Debian package version (version ?). osm2pgsql is also a package (version ?). Should I compile those from source? Of course any other advice is welcomed, especially as to which other pieces are recommended downstream for the actual tile server. And I would really like to have vector tiles, so recommendations in that direction are needed since it looks like not all the pieces can handle vectors. Oh, for the actual rendering: + prefer renderd or tirex? Thank you very much. Cheers! -Tom ___ talk mailing list talk@openstreetmap.org https://lists.openstreetmap.org/listinfo/talk
