Re: [OSM-dev] Slow osmosis import

2019-10-05 Thread merspieler
I did some more testing. I've taken a smaller area and put everything into a tmpfs but even with the .pbf as well as the tmp files of osmosis both being stored in ram, the performance isn't too good. It improved to about 1500 objects/second but this would still means that all ways (according to thi

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
The same problem applies to a 3.4 GB .pbf file. The nodes were done quickly but as soon as it started processing the ways, it got super slow. merspieler: > I've imported small extracts in the past but I've never actually > monitored the performance of these as they were done in

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
I've imported small extracts in the past but I've never actually monitored the performance of these as they were done in reasonable time. I'll try a smaller area again... Yves: > No, Imposm as it's own schema. > I never used Osmosis to import a complete planet file, but I would find > reasonable

Re: [OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
I've wanted to use osm2pgsql but the schema is a different one. The software [1] I'm going to use the db with only supports the osmosis one. As for the hardware: 2x Xeon E5 8 cores/16 threads 96GB ram 5x 4TB HDD in a RAIDZ2 I've done some benchmarking of the raid and osmosis doesn't even reach 5%

[OSM-dev] Slow osmosis import

2019-10-04 Thread merspieler
I'm currently trying to import the planet.osm.pbf file with osmosis. While it's quite fast with nodes (took about 7h) it massively lows down when it comes to the ways. INFO: Processing Node 6814667967, 307145.5708858228 objects/second. Oct 04, 2019 9:13:37 AM org.openstreetmap.osmosis.core.progres