Re: [OSM-dev] PostgreSQL 11 - osm2pgsql performance problems during --append?
Sounds similar to this issue. How is your RAM usage? Perhaps the new -C0 flag will help, I dunno. https://github.com/openstreetmap/osm2pgsql/issues/946 j On Thu, 12 Sep 2019 17:59:06 +0200 Michael Kussmaul wrote: > Hi > > I'm running my osm2pgsql setup for a couple of years now by > processing the whole planet and doing "--append" updates on a weekly > basis. I have a Debian server and 1.5 TB SSD storage for the planet > and run with flat-nodes configuration. > > I keep my system up-to-date and recently switched to PostgreSQL 11 > and added a SSD disk to my SSD-RAID. Since this change I see a large > drop on nodes processing by approx a factor of 50x (initial import is > doing fine - just when I apply a diff with --append, I now see a > processing rate of about 0.1k/s, it was in the 5k/s before), way and > relation processing is still ok, faster even! > > Before: > Processing: Node(20644k 4.7k/s) Way(2943k 0.59k/s) Relation(65680 > 29.95/s) After: > Processing: Node(30299k 0.1k/s) Way(4609k 1.15k/s) Relation(81710 > 40.57/s) > > So now the whole update process takes longer as if I drop the > complete planet database and start a fresh import, which is > sub-optimal :-) > > It probably boils down to this: > > - Either my new SSD has some performance problem? > - Or PostgreSQL 11 has some regressions? > - osm2pgsql does not work well with PostgreSQL 11? > > I also don't see any I/O bottleneck (iowait is very low <1%), CPU > overall is steady at 25% (12% user, 12% system) > > I also updated osm2pgsql to the newest 1.0 release, but it did not > change anything. > > Now, before I downgrade the DB or replace the disk: Does anybody else > run PostgreSQL 11 and sees similar problems? > > kind regards > Michael > ___ > dev mailing list > dev@openstreetmap.org > https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
On Tue, 4 Jun 2019 11:13:29 + (UTC) Sven Geggus wrote: > As I am about to setup two machines for this purpose right now I > reworked the scritps a bit. They are available here: > https://github.com/giggls/tileserver-scripts I'm afraid that these scripts will not work in our rendering environment. For more flexibility with render chains, I have created a small utility that works with the 0.92 output and expands expiration to include parent tiles. It takes an osm2pgsql expiration file as input and streams the expanded tile list to stdout in the same format. Code is currently untested at any scale. Feel free to use in any manner whatsoever. This is not meant as a long term fix, merely a kludge. https://intergalacticdata.com/public_domain/expandExpire.c j ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
> My own test hit 22GB before I ^C'd. 21GB more than required w/o > expiration. I have no idea where that 22GB number came from. My notes actually indicate >50GB memory usage. Caffeine deficiency? j ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
On Tue, 4 Jun 2019 14:28:38 +0200 Michael Reichert wrote: > Just to ensure that all important test conditions are known. > > Do you mean the weekly changesets dump from > https://planet.openstreetmap.org/planet/changesets-latest.osm.bz2? Or > how did you create changes.osc or where did you download it from? > changesets-latest.osm.bz contains changeset metadata only. changes.osc was created via osmosis. 0.92 was running w/ a max interval of 7 days. 0.96 was running w/ a max interval of 1 day. For reference, this is the command used to create the changeset: osmosis --read-replication-interval workingDirectory=$WORKDIR_OSM --simplify-change --write-xml-change changes.osc.gz Replication url from osmosis configuration.txt: https://planet.openstreetmap.org/replication/day/ > The tests were done with Europe only which is roughly 2/3 of the whole > planet. I did not realize Europe accounted for such a large portion of the OSM data! I did notice that your test reported a 10GB memory usage for the import. I did not pay attention to the memory usage for 0.92 imports, but tile expiration seems to be requiring significant memory as of 0.96. My own test hit 22GB before I ^C'd. 21GB more than required w/o expiration. > I will have a deeper look into this but my test machine is currently > busy doing other work. My own dev box is very busy for the next couple of weeks, but I can spin something up for testing purposes if needed. j ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql tile expiry performance
> To be fair, I am not sure this is even related to tile expiration. I > have not tried 0.96 updates without tile expiration as a baseline. After 3 hours, the same update has reached the same state of progress. osm2pgsql (debian backport) 0.96 maintains < 1GB of memory usage without expiring tiles. While expiring tiles, it ballooned to >22GB before being killed. Seems the issue is specific to tile-expiration overhaul. I note that initial testing of this code used an extract of Europe: https://github.com/openstreetmap/osm2pgsql/pull/747 I did not note it earlier, but I am working with the planet. I doubt these issues would present themselves on a modestly sized machine working with Europe only. j ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] osm2pgsql tile expiry performance
I'm beginning to implement a tile expiration solution and have run into some issues with the new tile expiry expansion. I'm afraid I do not have precise timings, but I'm seeing what appears to be at least an order of magnitude performance penalty, probably due to memory exhaustion. Test machine is Debian stretch 4CPUs, 26GB RAM, SSD Array. Initial osm import was performed with osm2pgsql 0.92 and finished in under 48 hours. Under 0.92 I was running multiple render chains while importing changesets. Tests w/ 0.96 have been run against an otherwise idle machine. Database & flatnodes file were restored to initial state between each round of testing. WEEKLY changeset using 0.92: -=-=-=- time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U \ $PGUSER -r xml changes.osc \ --flat-nodes /database/postgresql/OSM-FLATNODES --slim \ --number-processes 4 --style openstreetmap-carto.style \ --tag-transform-script openstreetmap-carto.lua -e19 -o \ $WORKDIR_OSM/expired-tiles With -e19, I was able to import a weekly changeset in roughly 24 hours. Using git repository (5/27? pull): -=-=-=- time $OSM2PGSQL --append -s -C 3000 -G --hstore -d gis -H $PGHOST -U $PGUSER -r xml changes.osc \ --flat-nodes /database/postgresql/OSM-FLATNODES --slim \ --number-processes 4 --style openstreetmap-carto.style \ --tag-transform-script openstreetmap-carto.lua -e10-16 -o \ $WORKDIR_OSM/expired-tiles Consistently crashed w/ a bad alloc(). I didn't note any unusual output in the compile. Crashes even with a 24 hour changeset. DAILY changeset using Debian backport to stretch (0.96): -=-=-=- As above, using -e10-16. 22 hours spent processing a 24 hour changeset and still importing new relations. It's 35GB into swap, with osm2pgsql claiming 89% of the memory usage. --- I realize there is a large difference between zoom level 19 and 10-16, but I assume it should take significantly less RAM/CPU for 10-16. Please feel free to point out any stupidity I've generated here and/or recommend a better way to generate a list of dirty tiles at lower zoom levels based on the 0.92 output. To be fair, I am not sure this is even related to tile expiration. I have not tried 0.96 updates without tile expiration as a baseline. j ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] HTTPS support for osmchanges url
Hi, Since the OSM updates diff support only http, I cannot update the OSM database in my production environment due to security issue. It is vulnerable to MITM(Man-In-The-Middle) attack. When I tried https url, it redirects to http url. https://planet.openstreetmap.org/replication/day/000/000/ Is there any specific reason for not having https support ? It would be great, if we give support https. And also we may add md5check sum details in the xx_state.txt files or a separate file. I am ready to take this work and contribute it. Please guide me that where can I start this work ? Thanks, Ramesh ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] HTTPS support for osmchanges url
Hi Grant, Amazing...What a speed of giving support ? Thank you very much for very quick fix. Special Thanks to Tom Hughes !!! Regards, Ramesh On Tue, Apr 7, 2015 at 5:57 PM, Grant Slater openstreet...@firefishy.com wrote: Hi Ramesh, https:// has now been enabled on planet.openstreetmap.org. https://git.openstreetmap.org/chef.git/commitdiff/5f2de3518b17b1fb18294649f6c238e289ca7835 Kind regards, Grant On 7 April 2015 at 08:26, J Ramesh Kumar rameshj1...@gmail.com wrote: Hi, Since the OSM updates diff support only http, I cannot update the OSM database in my production environment due to security issue. It is vulnerable to MITM(Man-In-The-Middle) attack. When I tried https url, it redirects to http url. https://planet.openstreetmap.org/replication/day/000/000/ Is there any specific reason for not having https support ? It would be great, if we give support https. And also we may add md5check sum details in the xx_state.txt files or a separate file. I am ready to take this work and contribute it. Please guide me that where can I start this work ? Thanks, Ramesh ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] PBF: add a package keyword to the .proto for CPP namespace
On Tue, Nov 16, 2010 at 12:19 PM, Chris Browet c...@semperpax.com wrote: There is something I don't understand, though... I cloned pbf2osm which has OSM-binary as submodule. When I do a git submodule init + git submodule update as instructed, I get the .proto dated sep 15th (thus without the package) There must be some magic in git submodules that evades me... Git submodules are different from svn externals in that git submodules refer to a specific commit in the submodule rather than whatever is the latest in the branch. -- Jeff Ollie ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
