Hi Kieran,
we run an OSRM update (planet) once a week on a central instance and copy the
generated files to diffrent environments.
osrm@box:~$ ./osrm-update-planet-files.sh
Checking for md5sum [OK]
Checking for osrm-extract [OK]
Checking for osrm-prepare [OK]
Checking for tar [OK]
Checking for wget [OK]
Downloading planet-latest.osm.pbf.md5 ... [OK]
Downloading http://planet.osm.org/pbf/planet-latest.osm.pbf ... [OK]
Verifying md5 checksum of planet-latest.osm.pbf ... [OK]
Starting osrm-extract at Wed Mar 2 11:57:41 CET 2016...
Finished osrm-extract at Thu Mar 3 00:21:34 CET 2016!
Starting osrm-prepare at Thu Mar 3 00:21:34 CET 2016...
Finished osrm-prepare at Thu Mar 3 09:21:23 CET 2016!
Removing old extracts from /data/current ... empty [OK]
Copying new generated files to /data/current ... [OK]
Renaming files in /data/current with Prefix 201609 ... [OK]
Creating md5 checksum over all 201609_planet-latest* ... [OK]
Compressing 201609_planet-latest* to 201609_planet-latest.tar.gz ... [OK]
Determining if test or prod env is the target ... TEST [OK]
Copying new generated files to /mnt/osrm-extract (TEST) ... [OK]
Cleaning up /mnt/osrm-extract ... [OK]
Cleanup /data/planet-latest.osm.pbf ... [OK]
On a VM with 96GB RAM, 4 Cores and a RAID5 (HDD) it took about 12,5 hours to
extract and 9 hours to prepare.
SWAP is 100GB, stxxl=disk=/data/stxxl,250000,syscall
We currently use Version 4.9.0 of osrm-backend.
BR
Björn
________________________________________
Von: Kieran Caplice <kieran.capl...@temetra.com>
Gesendet: Mittwoch, 2. März 2016 18:23
An: osrm-talk@openstreetmap.org
Betreff: Re: [OSRM-talk] osrm-extract taking hours to complete
Hi Patrick,
That makes sense then. It's obvious the process is just going to take
upwards of 8-10 hours for us in that case.
Thanks for the help.
Kind regards,
Kieran Caplice
On 02/03/16 17:01, Patrick Niklaus wrote:
> Hey Kieran,
>
>
> there have been a lot of structural changes (e.g. moving code from
> osrm-prepare into osrm-extract) that probably invalidate that numbers.
> Also we support 64bit OSM ids now, which sadly uses a lot more disk
> space. I think stxxl need like 200GB. I think on our setup we have a
> turn-around of 6 hours for the planet dataset on an SSD setup (car
> profile, any other profile needs significantly longer). You should
> probably think about updating your hard drives as this is IO bound. At
> your current read/write speed it will already take more than an hour
> to just write 200GB of data once. We scan it at least twice just for
> pre-processing.
>
> Cheers,
> Patrick
>
>
> On Wed, Mar 2, 2016 at 5:51 PM, Kieran Caplice
> <kieran.capl...@temetra.com> wrote:
>> Hello,
>>
>> I'm currently extracting the planet PBF (~31 GB), and it's been running for
>> hours. I notice in the "Running OSRM" wiki page, it says " On a Core i7 with
>> 8GB RAM and (slow) 5400 RPM Samsung SATA hard disks it took about 65 minutes
>> to do so from a PBF formatted planet", which is making me wonder why it's
>> taking so long on our server. Below are some example output messages:
>>
>> [info] Parsing finished after 3584.35 seconds
>> [extractor] Erasing duplicate nodes ... ok, after 319.091s
>> [extractor] Sorting all nodes ... ok, after 3632.87s
>> [extractor] Building node id map ... ok, after 2025.29s
>> [extractor] Confirming/Writing used nodes ... ok, after 1096.24s
>> [extractor] Sorting edges by start ... ok, after 2000.08s
>>
>> Some stxxl errors were outputted as I set the disk size to 100GB thinking it
>> was enough - but I didn't think it would cause such slowdowns as this,
>> considering extracting the Europe PBF takes hours also without the stxxl
>> errors.
>>
>> Server specs:
>> Ubuntu 14.04
>> Intel Xeon CPU E5-1650 v3 @ 3.50GHz (hex-core with HT)
>> 64 GB RAM @ 2133 MHz
>> 2 TB Western Digital Enterprise 7200 RPM hard drive
>>
>> At the moment, disk IO is averaging around 35-40 MB/s R/W (~90%).
>>
>> Anyone have any ideas as to what might be going on? Or is it normal to take
>> this long without an SSD?
>>
>> Thanks in advance.
>>
>> Kind regards,
>> Kieran Caplice
>>
>>
>> _______________________________________________
>> OSRM-talk mailing list
>> OSRM-talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/osrm-talk
>>
> _______________________________________________
> OSRM-talk mailing list
> OSRM-talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osrm-talk
_______________________________________________
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk
_______________________________________________
OSRM-talk mailing list
OSRM-talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osrm-talk
