I share the same issue exposed by Felix, but not only for contour lines
but also for DEM, which I have precompiled as *.img for many countries
and just combine with the updated OSM tiles.
El 12/9/21 a las 19:10, Felix Hartmann escribió:
And well - it is burning SSD for the contourlines currently even if
not calling multiple times. 130GB minimum per worldwide map update for
all geofabrik extracts at 20m equidistance. 260GB at 10m. With calling
multiple times and 10m and 20m I had about 2TB of useless writes per
weekly map update. I've got rid of all of them with my uncommenting
the line, plus saved about 500GB of writes by now doing all the
gmapsupp.img and gmap stuff in Ramdisk So now instead of 4TB of disk
writes per map update I'm down to 1TB.. (and yeah the resplitting of
tiles added another 5TB of writes - but that could have been fixed
easily by changing order of commands too).
On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecar...@gmail.com
<mailto:extremecar...@gmail.com>> wrote:
because you need to do it if you want to show contourlines. Now
compiling the worldwide contourlines each time again - would burn
SSD as well - besides taking longer than just compiling the maps.
So everyone who offers maps for many countries as download puts
the contourlines separate. If you want to offer the choice of
showing contourlines or not - mkmgap will write the gmap
contourlines once not needed, and once the maps not needed. If you
don't want to offer that option - it's only the contourlines being
written without need. Contourlines for all geofabrik extracts 20m
equidistance are about 130GB, 10m would be 260GB.
If you offer 10m and 20m it will be even more not needed writes.
The only solution is to go for a Ramdisk instead - However you
likely will need 128GB of RAM to do that - because for Asia or
Europe you need a 64GB Ramdisk. Same for North America extract
(but few people offer that and just have Canada and the US divided
into the 4-5 zones). For other maps you will get by with a 15-20GB
Ramdisk (which I have resorted to now for all but the windows
installers because I don't want to let the server wait for ages
for the single thread NSIS wrapping up the data and instead start
the next country in parallel). And yes going RAMDISK already using
my patch and symlinking those files so mkgmap cannot overwrite
them (would still be faster if mkgmap didn't try in first place I
think). If you want to write out the contourlines as well besides
the additional time - for Asia continent you may then go for a
90GB Ramdisk minimum if offering windows and gmap installers. In
this case gmapsupp.img donwnloads aren't possible anyhow due to
the huge data amounts. For smaller countries I have them too....
I coded around this now by using symlinks - but yeah that will be
quite a lot of work and is prone to break - while I guess it's
only 10 lines or so to add to mkgmap code to have a mode that does
not write them out if they are present - or if you tell on
commandline they are present already. For .img files they aren't
overwritten either...
On Sun, 12 Sept 2021 at 18:40, Gerd Petermann
<gpetermann_muenc...@hotmail.com
<mailto:gpetermann_muenc...@hotmail.com>> wrote:
Hi Felix,
did not read all the posts in detail. I understand that mkgmap
is neither burning SSDs nor doing any excessive writing unless
you call it multiple times for the same input files. So the
question is: Why do you do that?
Gerd
________________________________________
Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk
<mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag
von Felix Hartmann <extremecar...@gmail.com
<mailto:extremecar...@gmail.com>>
Gesendet: Freitag, 10. September 2021 00:02
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and
converting to disk if used with --gmapi option
Well I think the .tmp files are just building up - and the
renamed. So they are not causing any actual excessive write.
As for the gmap - it would be cool if there is a mode to not
write them.
Actually it would be great if mkgmap could write all in one
go. Because the thing that takes so much time - is the address
search - and that is always the same. The differences are tiny
(just because MapInstall is crashing when files are missing)
you need to compile them separately.
but maybe there could be a mode where mkgamp writes all in one go.
So family-name / family-name1 / family-name2
description / description1 / description2
input input1 input2
family-name..
show-profiles
overview-mapname
product-id
(and maybe I missed some options are those that would need to
be given for each set of input tiles. And then just an option
where you tell mkgmap files starting with which first 4
numbers are relevant for address search. No need to analyze if
those other supplied .img (e.g. buildings or contourlines)
need to be added to address search.). I know coded around the
problem of the gmap files causing excessive writes. But yeah
that is actually really complicated be it on windows or linux...
On Wed, 8 Sept 2021 at 17:07, Felix Hartmann
<extremecar...@gmail.com
<mailto:extremecar...@gmail.com><mailto:extremecar...@gmail.com
<mailto:extremecar...@gmail.com>>> wrote:
Well I could give it 20 GB ram disk, maybe 32 but then I need
to render on less than 12 processes 64GB ram available). But
that is not enough for Asia continent map, and I guess super
tight for Europe...
Mkgmap could definitely keep those .tmp files in memory. But
the important bit is the gmap files not needeed to be
written.... Also would save quite some CPU time.
On Wed, 8 Sep 2021, 14:31 ael <witwa...@disroot.org
<mailto:witwa...@disroot.org><mailto:witwa...@disroot.org
<mailto:witwa...@disroot.org>>> wrote:
On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote:
>
> Yes on an nvme disk you barely notice the conversion - it's
really quick.
> BUT it is not needed if you have the files and even more -
it burns your
> NVME SSD disk.
+1. I always use an old spinning rust disk when using mkgmap
to save ssd
write cycles, even without contours and such. It seems to be
profligate
in its use of disk cycles. I did try using RAM disk, but even
with 16GB
on a laptop, that was soon exhausted.
ael
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk><mailto:mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk>>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
<https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk>
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
<https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
--
Felix Hartman - Openmtbmap.org & VeloMap.org
--
Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
