Thanks Gerd, but that is just removing the warning if it cannot overwrite, correct? If it can overwrite the gmap file it will still overwrite it as I see.. So in essence just a bit more intelligent then my disabling that line.
I think it should not overwrite at all if present and --x-gmapi-minimal (then you don't have to move away the files and link them back to the original folder). On Mon, 13 Sept 2021 at 10:43, Gerd Petermann < gpetermann_muenc...@hotmail.com> wrote: > Hi all, > > attached is a quick patch that implements experimaltal option > --x-gmapi-minimal. > > If used instead of --gmapi mkgmap will not fail if a write-protected > output file exists in the gmapi output folder and mkgmap copies data from > *.img. It should still crash when other files like Info.xml are written. > > BTW: no conversion is done when those files are written. I think mkgmap > simply copies data from sub files in *.img into single files. So, the same > sequence of Bytes exists two or more times on the Computer. Windows Server > seems to support automatic data deduplication (1). I am sure Linux offers > similar support. No idea how effective or reliable it is, but it might be > worth trying. > > Gerd > (1) > > https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enable > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von > Carlos Dávila <car...@alternativaslibres.org> > Gesendet: Sonntag, 12. September 2021 22:45 > An: mkgmap-dev@lists.mkgmap.org.uk > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting to > disk if used with --gmapi option > > I think it's a good idea if mkgmap checks the required files are present. > > El 12/9/21 a las 21:16, Gerd Petermann escribió: > > Hi Felix, > > > > so you just want a new option so that mkgmap doesn't fail if it cannot > overwrite (write-protected) files in the output directory, right? No need > to verify the content of the exiting file(s)? > > > > Gerd > > > > ________________________________________ > > Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag von > Felix Hartmann <extremecar...@gmail.com> > > Gesendet: Sonntag, 12. September 2021 19:10 > > An: Development list for mkgmap > > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting > to disk if used with --gmapi option > > > > And well - it is burning SSD for the contourlines currently even if not > calling multiple times. 130GB minimum per worldwide map update for all > geofabrik extracts at 20m equidistance. 260GB at 10m. With calling multiple > times and 10m and 20m I had about 2TB of useless writes per weekly map > update. I've got rid of all of them with my uncommenting the line, plus > saved about 500GB of writes by now doing all the gmapsupp.img and gmap > stuff in Ramdisk So now instead of 4TB of disk writes per map update I'm > down to 1TB.. (and yeah the resplitting of tiles added another 5TB of > writes - but that could have been fixed easily by changing order of > commands too). > > > > On Sun, 12 Sept 2021 at 20:04, Felix Hartmann <extremecar...@gmail.com > <mailto:extremecar...@gmail.com>> wrote: > > because you need to do it if you want to show contourlines. Now > compiling the worldwide contourlines each time again - would burn SSD as > well - besides taking longer than just compiling the maps. So everyone who > offers maps for many countries as download puts the contourlines separate. > If you want to offer the choice of showing contourlines or not - mkmgap > will write the gmap contourlines once not needed, and once the maps not > needed. If you don't want to offer that option - it's only the contourlines > being written without need. Contourlines for all geofabrik extracts 20m > equidistance are about 130GB, 10m would be 260GB. > > If you offer 10m and 20m it will be even more not needed writes. > > > > The only solution is to go for a Ramdisk instead - However you likely > will need 128GB of RAM to do that - because for Asia or Europe you need a > 64GB Ramdisk. Same for North America extract (but few people offer that and > just have Canada and the US divided into the 4-5 zones). For other maps you > will get by with a 15-20GB Ramdisk (which I have resorted to now for all > but the windows installers because I don't want to let the server wait for > ages for the single thread NSIS wrapping up the data and instead start the > next country in parallel). And yes going RAMDISK already using my patch and > symlinking those files so mkgmap cannot overwrite them (would still be > faster if mkgmap didn't try in first place I think). If you want to write > out the contourlines as well besides the additional time - for Asia > continent you may then go for a 90GB Ramdisk minimum if offering windows > and gmap installers. In this case gmapsupp.img donwnloads aren't possible > anyhow due to the huge data amounts. For sm > > aller countries I have them too.... > > > > I coded around this now by using symlinks - but yeah that will be quite > a lot of work and is prone to break - while I guess it's only 10 lines or > so to add to mkgmap code to have a mode that does not write them out if > they are present - or if you tell on commandline they are present already. > For .img files they aren't overwritten either... > > > > On Sun, 12 Sept 2021 at 18:40, Gerd Petermann < > gpetermann_muenc...@hotmail.com<mailto:gpetermann_muenc...@hotmail.com>> > wrote: > > Hi Felix, > > > > did not read all the posts in detail. I understand that mkgmap is > neither burning SSDs nor doing any excessive writing unless you call it > multiple times for the same input files. So the question is: Why do you do > that? > > > > Gerd > > > > ________________________________________ > > Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk<mailto: > mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag von Felix Hartmann < > extremecar...@gmail.com<mailto:extremecar...@gmail.com>> > > Gesendet: Freitag, 10. September 2021 00:02 > > An: Development list for mkgmap > > Betreff: Re: [mkgmap-dev] mkgmap doing excessive writing and converting > to disk if used with --gmapi option > > > > Well I think the .tmp files are just building up - and the renamed. So > they are not causing any actual excessive write. > > As for the gmap - it would be cool if there is a mode to not write them. > > > > Actually it would be great if mkgmap could write all in one go. Because > the thing that takes so much time - is the address search - and that is > always the same. The differences are tiny (just because MapInstall is > crashing when files are missing) you need to compile them separately. > > > > but maybe there could be a mode where mkgamp writes all in one go. > > So family-name / family-name1 / family-name2 > > description / description1 / description2 > > input input1 input2 > > family-name.. > > show-profiles > > overview-mapname > > product-id > > (and maybe I missed some options are those that would need to be given > for each set of input tiles. And then just an option where you tell mkgmap > files starting with which first 4 numbers are relevant for address search. > No need to analyze if those other supplied .img (e.g. buildings or > contourlines) need to be added to address search.). I know coded around the > problem of the gmap files causing excessive writes. But yeah that is > actually really complicated be it on windows or linux... > > > > On Wed, 8 Sept 2021 at 17:07, Felix Hartmann <extremecar...@gmail.com > <mailto:extremecar...@gmail.com><mailto:extremecar...@gmail.com<mailto: > extremecar...@gmail.com>>> wrote: > > Well I could give it 20 GB ram disk, maybe 32 but then I need to render > on less than 12 processes 64GB ram available). But that is not enough for > Asia continent map, and I guess super tight for Europe... > > > > Mkgmap could definitely keep those .tmp files in memory. But the > important bit is the gmap files not needeed to be written.... Also would > save quite some CPU time. > > > > On Wed, 8 Sep 2021, 14:31 ael <witwa...@disroot.org<mailto: > witwa...@disroot.org><mailto:witwa...@disroot.org<mailto: > witwa...@disroot.org>>> wrote: > > On Wed, Sep 08, 2021 at 02:10:52PM +0300, Felix Hartmann wrote: > >> Yes on an nvme disk you barely notice the conversion - it's really > quick. > >> BUT it is not needed if you have the files and even more - it burns your > >> NVME SSD disk. > > +1. I always use an old spinning rust disk when using mkgmap to save ssd > > write cycles, even without contours and such. It seems to be profligate > > in its use of disk cycles. I did try using RAM disk, but even with 16GB > > on a laptop, that was soon exhausted. > > > > ael > > > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk > ><mailto:mkgmap-dev@lists.mkgmap.org.uk<mailto: > mkgmap-dev@lists.mkgmap.org.uk>> > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev@lists.mkgmap.org.uk<mailto:mkgmap-dev@lists.mkgmap.org.uk> > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > > > > -- > > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > > > > > -- > > Felix Hartman - Openmtbmap.org & VeloMap.org > > > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev@lists.mkgmap.org.uk > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > > > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Felix Hartman - Openmtbmap.org & VeloMap.org
_______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
