Thanks Gerd! This is valuable information for those of us processing large areas of the planet.

Unfortunately there is no additional speedup for me because I already use o5m because of osmupdate (to keep a local planet copy up-to-date).

On 07/05/2014 11:59, Gerd Petermann wrote:
Hi Felix,

try o5m for both input and output, it is much faster.
The command
osmcovert --drop-version europe-latest.osm.pbf -o=europe.o5m
runs quite fast (~70 seconds on my machine),
the o5m file is ~2.430 MB, the pbf file has 2.104 MB.

Splitter is much faster reading from o5m when
the keep-complete option is in use.
(210 secs for the o5m, 441 for pbf)

With --output=pbf both are slower, and mkgmap is also much slower.

All times with I/O on one normal hard disk. Even better results if you have
two different disks for reading and writing.

Gerd

------------------------------------------------------------------------
Date: Wed, 7 May 2014 11:37:58 +0200
From: extremecar...@gmail.com
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new option

Well I still use pbf and not o5m.
First pbf is smaller..
Second - Geofabrik only offers pbf - that's why I stayed with it.

I don't think I can cut a lot of time by first converting to 05m, then
hand it over to splitter...
Actually I also let splitter output pbf... Maybe I could change that in
future to 05m..
On 07.05.2014 11:36, Gerd Petermann wrote:

    Hi Felix,

    well, nowadays splitter performance mostly depends on I/O if you use
    o5m format
    for input and output and give enough heap.

    Reg. mkgmap performance improvements: yes, that's what I expected.
    In short, the branch improved the evaluation of tags and the
    creation of the NOD file.

    Gerd


    ------------------------------------------------------------------------
    Date: Wed, 7 May 2014 11:29:10 +0200
    From: extremecar...@gmail.com <mailto:extremecar...@gmail.com>
    To: mkgmap-dev@lists.mkgmap.org.uk
    <mailto:mkgmap-dev@lists.mkgmap.org.uk>
    Subject: Re: [mkgmap-dev] splitter r325: improved split algo and new
    option

    Well - I'll update all my maps on Thursday again, to recheck. Maybe
    it has to do with increasing-maxnodes? Though I thought the higher
    the max-nodes, the faster...
    And I only meant splitter. I upgraded mkgmap at the same time (now
    integrating performance branch changes) - so mkgmap by itself got
    faster (though it depends on the country - seems like well mapped
    countries profit a lot more (e.g. Austria like 30% time off), than
    countries where few continue commands will be in action cause their
    mapping is basic like Asia).

    I'm not using any pre-split files or cached files of any sort either...
    On 07.05.2014 06:49, Gerd Petermann wrote:

        Hi Felix,

        reg. speed: I can't reproduce that. I compared a split of Germany,
        both versions (r321 and r325) are more or less running the same
        time.
        (I've executed both programs two times to make sure that disk
        caches
        are not causing big differences)

        Or did you mean the combination of splitter + mkgmap to process
        e.g. Asia?

        Gerd

        ------------------------------------------------------------------------
        Date: Tue, 6 May 2014 18:22:00 +0200
        From: extremecar...@gmail.com <mailto:extremecar...@gmail.com>
        To: mkgmap-dev@lists.mkgmap.org.uk
        <mailto:mkgmap-dev@lists.mkgmap.org.uk>
        Subject: Re: [mkgmap-dev] splitter r325: improved split algo and
        new option

        Seems to be much better now. I don't think I can increase the
        max-nodes value though, but for most maps the new algo creates
        less tiles for the same max-nodes value (e.g. Austria from 43
        down to 35 for me, with the smallest tile now around 5MB instead
        of 2.8, and the biggest 12MB instead of 11MB, for Asia I
        simultaneously increased max-nodes from 800k to 900k- so I'm
        down from 624 tiles to 493.... and size from 970KB-16MB to now
        ). So it still seems to depend on the country, but it's already
        a lot better...
        It's a bit slower (about 10% more time)

        On 06.05.2014 13:56, Gerd Petermann wrote:

            Hi all,

            I've applied num-tiles-v1.patch  and improved the split
            algo, see
            
http://gis.19327.n5.nabble.com/mkgmap-ToDo-list-tp5803388p5805165.html


            It is now less likely that splitter creates tiles with a low
            number of
            nodes, it is more likely that all tiles have nearly the same
            number of nodes,
            and typically you will see fewer tiles.
            Maybe this also means that you can increase the max-nodes value.

            I hope this also reduces the need for complex interactions
            between
            spltter and mkgmap.



            Gerd


            _______________________________________________
            mkgmap-dev mailing list
            mkgmap-dev@lists.mkgmap.org.uk  
<mailto:mkgmap-dev@lists.mkgmap.org.uk>
            http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


        --
        keep on biking and discovering new trails

        Felix
        openmtbmap.org &www.velomap.org  <http://www.velomap.org>


        _______________________________________________ mkgmap-dev
        mailing list mkgmap-dev@lists.mkgmap.org.uk
        <mailto:mkgmap-dev@lists.mkgmap.org.uk>
        http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


        _______________________________________________
        mkgmap-dev mailing list
        mkgmap-dev@lists.mkgmap.org.uk  <mailto:mkgmap-dev@lists.mkgmap.org.uk>
        http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


    --
    keep on biking and discovering new trails

    Felix
    openmtbmap.org &www.velomap.org  <http://www.velomap.org>


    _______________________________________________ mkgmap-dev mailing
    list mkgmap-dev@lists.mkgmap.org.uk
    <mailto:mkgmap-dev@lists.mkgmap.org.uk>
    http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


    _______________________________________________
    mkgmap-dev mailing list
    mkgmap-dev@lists.mkgmap.org.uk  <mailto:mkgmap-dev@lists.mkgmap.org.uk>
    http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


--
keep on biking and discovering new trails

Felix
openmtbmap.org &www.velomap.org  <http://www.velomap.org>


_______________________________________________ mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Reply via email to