date:20180108

Hi all,

I see no way to improve the handling of zipped files much more. We have two 
conflicting optimization goals here: 
1) reduce heap memory by storing less data in buffers (risking additional 
rather slow unzip actions)
2) reduce run time be storing more data in buffers (risking 
OutOfMemoryException)
The problem is that the DEM algo needs more or less random access to the data 
of one or more *.hgt.zip files,
and I see no simple way to change that now.  

I did a few tests now with r4036 and a map for scandinavia and hgt files in 1'' 
resolution.
The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed 
folder the required disk space is ~ 37%,
when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means ~14%.

I've used a simple poly file to cut scandinavia out of an older europe.o5m with 
osmconvert and used splitter with max-nodes=100.
This gave me 174 tiles.
I used the following command to create a map: 
java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt 
--output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 
--dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   
--dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args

I repeated this command two times, once with the uncompressed files, once with 
the NTFS compressed files.
All worked fine, java heap memory was no problem, but run time was different:
953 secs for the uncompressed files
1084 secs for the NTFS compressed files
1538 secs for the zipped files

I think NTFS compressed folder is a good compromise if disk space matters. I 
hope the numbers are similar for compressing Linux file systems.
I plan to implement a new dem-temp=[dir] option. If given, mkgmap could create 
temp files for the unzipped hgt data.  

Gerd
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Hi Nick,

thanks for reporting. I'll try to reproduce the problem. Please note that
overview-dem-dist=88
is probably too large, 8 would be a "normal" value.

Gerd


Von: mkgmap-dev  im Auftrag von 
nwillink 
Gesendet: Montag, 8. Januar 2018 11:01:46
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: [mkgmap-dev] Error when not using dem-poly

Hi Gerd

I'm using latest r4036

I'm applying what you suggested to Minko to avoid contours been given a dem.

In my dem.args I have

dem=e:\dem\europe\
overview-dem-dist=88
dem-dists=3312,13248,26512,53024

and get the following error:

Number of MapFailedExceptions: 0
Exception in thread "main" java.lang.NullPointerException
at
uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getElevation(HGTConvert
er.java:89)
at
uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getHeights(HGTConverter
.java:287)
at
uk.me.parabola.imgfmt.app.dem.DEMSection.calcTiles(DEMSection.java:10
5)
at
uk.me.parabola.imgfmt.app.dem.DEMSection.(DEMSection.java:78)
at uk.me.parabola.imgfmt.app.dem.DEMFile.calc(DEMFile.java:60)
at
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:340)
at
uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over
viewBuilder.java:191)
at
uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil
der.java:103)
at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:617)
at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja
va:128)
at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:136)
at uk.me.parabola.mkgmap.main.Main.main(Main.java:107)

1) When I add dem-poly=areas.poly the error disappears.

2) I also don't get the error if I replace -c dem-args with
--dem=e:\dem\europe\ --overview-dem-dist=88
--dem-dists=3312,13248,26512,53024

3) This only seems to affect certain pbfs.

file:  http://files.mkgmap.org.uk/download/385/12345678.osm.pbf

r
Nick








--
Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

2018-01-08 Thread osm@pinns


Thanks Gerd


On 08/01/2018 10:21, Gerd Petermann wrote:

Hi Nick,

thanks for reporting. I'll try to reproduce the problem. Please note that
overview-dem-dist=88
is probably too large, 8 would be a "normal" value.

Gerd


Von: mkgmap-dev  im Auftrag von nwillink 

Gesendet: Montag, 8. Januar 2018 11:01:46
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: [mkgmap-dev] Error when not using dem-poly

Hi Gerd

I'm using latest r4036

I'm applying what you suggested to Minko to avoid contours been given a dem.

In my dem.args I have

dem=e:\dem\europe\
overview-dem-dist=88
dem-dists=3312,13248,26512,53024

and get the following error:

Number of MapFailedExceptions: 0
Exception in thread "main" java.lang.NullPointerException
 at
uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getElevation(HGTConvert
er.java:89)
 at
uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getHeights(HGTConverter
.java:287)
 at
uk.me.parabola.imgfmt.app.dem.DEMSection.calcTiles(DEMSection.java:10
5)
 at
uk.me.parabola.imgfmt.app.dem.DEMSection.(DEMSection.java:78)
 at uk.me.parabola.imgfmt.app.dem.DEMFile.calc(DEMFile.java:60)
 at
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:340)
 at
uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over
viewBuilder.java:191)
 at
uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil
der.java:103)
 at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:617)
 at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja
va:128)
 at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:136)
 at uk.me.parabola.mkgmap.main.Main.main(Main.java:107)

1) When I add dem-poly=areas.poly the error disappears.

2) I also don't get the error if I replace -c dem-args with
--dem=e:\dem\europe\ --overview-dem-dist=88
--dem-dists=3312,13248,26512,53024

3) This only seems to affect certain pbfs.

file:  http://files.mkgmap.org.uk/download/385/12345678.osm.pbf

r
Nick








--
Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files


Hi Gerd,

> 2) reduce run time be storing more data in buffers

This is what I would chose. I think java x64 can use swap to increase 
heap size. And RAM memory is a relatively cheap upgrade.


> I think NTFS compressed folder is a good compromise

I did it long time ago. Whole set of SRTM3 is about 38.3GB and takes 
23.1G on HDD. It is even better for Viewfinder Panoramas files, where 
44.3GB od HGT takes 13.8GB of drive space.


Another question, does mkgmap support non-standard HGT size? With 
BuildDemFile I use HGT with raster 76x76 for overview maps. It is faster.


--
Best regards,
Andrzej


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Hi Nck,

I was not able to reproduce the problem with that file, but I think I fixed the 
problem with r4037:
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4037

Please try it.

Gerd



Von: mkgmap-dev  im Auftrag von 
osm@pinns 
Gesendet: Montag, 8. Januar 2018 11:28:00
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Error when not using dem-poly

Thanks Gerd


On 08/01/2018 10:21, Gerd Petermann wrote:
> Hi Nick,
>
> thanks for reporting. I'll try to reproduce the problem. Please note that
> overview-dem-dist=88
> is probably too large, 8 would be a "normal" value.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von 
> nwillink 
> Gesendet: Montag, 8. Januar 2018 11:01:46
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: [mkgmap-dev] Error when not using dem-poly
>
> Hi Gerd
>
> I'm using latest r4036
>
> I'm applying what you suggested to Minko to avoid contours been given a dem.
>
> In my dem.args I have
>
> dem=e:\dem\europe\
> overview-dem-dist=88
> dem-dists=3312,13248,26512,53024
>
> and get the following error:
>
> Number of MapFailedExceptions: 0
> Exception in thread "main" java.lang.NullPointerException
>  at
> uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getElevation(HGTConvert
> er.java:89)
>  at
> uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getHeights(HGTConverter
> .java:287)
>  at
> uk.me.parabola.imgfmt.app.dem.DEMSection.calcTiles(DEMSection.java:10
> 5)
>  at
> uk.me.parabola.imgfmt.app.dem.DEMSection.(DEMSection.java:78)
>  at uk.me.parabola.imgfmt.app.dem.DEMFile.calc(DEMFile.java:60)
>  at
> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:340)
>  at
> uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over
> viewBuilder.java:191)
>  at
> uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil
> der.java:103)
>  at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:617)
>  at
> uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja
> va:128)
>  at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:136)
>  at uk.me.parabola.mkgmap.main.Main.main(Main.java:107)
>
> 1) When I add dem-poly=areas.poly the error disappears.
>
> 2) I also don't get the error if I replace -c dem-args with
> --dem=e:\dem\europe\ --overview-dem-dist=88
> --dem-dists=3312,13248,26512,53024
>
> 3) This only seems to affect certain pbfs.
>
> file:  http://files.mkgmap.org.uk/download/385/12345678.osm.pbf
>
> r
> Nick
>
>
>
>
>
>
>
>
> --
> Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Andrzej,

thanks for the input.
The current implemention with zip files fills a buffer for each hgt file when 
it is needed (when the first elevation data is needed).
The buffer is freed if the DEM calculation for the tile is done or when mkgmap 
is sure that the buffer is no longer needed for the current tile.

I did not try it but mkgmap should support 76x76 tiles, but I see no need to 
use that with mkgmap, at least not with unzipped hgt files.

Gerd


Von: mkgmap-dev  im Auftrag von Andrzej 
Popowski 
Gesendet: Montag, 8. Januar 2018 11:50:13
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,

 > 2) reduce run time be storing more data in buffers

This is what I would chose. I think java x64 can use swap to increase
heap size. And RAM memory is a relatively cheap upgrade.

 > I think NTFS compressed folder is a good compromise

I did it long time ago. Whole set of SRTM3 is about 38.3GB and takes
23.1G on HDD. It is even better for Viewfinder Panoramas files, where
44.3GB od HGT takes 13.8GB of drive space.

Another question, does mkgmap support non-standard HGT size? With
BuildDemFile I use HGT with raster 76x76 for overview maps. It is faster.

--
Best regards,
Andrzej


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

2018-01-08 Thread osm@pinns


Hi Gerd,

Whatever you did, it seems to work! ;Many thanks will do some more testing

r


Nick


On 08/01/2018 10:51, Gerd Petermann wrote:

Hi Nck,

I was not able to reproduce the problem with that file, but I think I fixed the 
problem with r4037:
http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap&rev=4037

Please try it.

Gerd



Von: mkgmap-dev  im Auftrag von osm@pinns 

Gesendet: Montag, 8. Januar 2018 11:28:00
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Error when not using dem-poly

Thanks Gerd


On 08/01/2018 10:21, Gerd Petermann wrote:

Hi Nick,

thanks for reporting. I'll try to reproduce the problem. Please note that
overview-dem-dist=88
is probably too large, 8 would be a "normal" value.

Gerd


Von: mkgmap-dev  im Auftrag von nwillink 

Gesendet: Montag, 8. Januar 2018 11:01:46
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: [mkgmap-dev] Error when not using dem-poly

Hi Gerd

I'm using latest r4036

I'm applying what you suggested to Minko to avoid contours been given a dem.

In my dem.args I have

dem=e:\dem\europe\
overview-dem-dist=88
dem-dists=3312,13248,26512,53024

and get the following error:

Number of MapFailedExceptions: 0
Exception in thread "main" java.lang.NullPointerException
  at
uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getElevation(HGTConvert
er.java:89)
  at
uk.me.parabola.mkgmap.reader.hgt.HGTConverter.getHeights(HGTConverter
.java:287)
  at
uk.me.parabola.imgfmt.app.dem.DEMSection.calcTiles(DEMSection.java:10
5)
  at
uk.me.parabola.imgfmt.app.dem.DEMSection.(DEMSection.java:78)
  at uk.me.parabola.imgfmt.app.dem.DEMFile.calc(DEMFile.java:60)
  at
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:340)
  at
uk.me.parabola.mkgmap.combiners.OverviewBuilder.writeOverviewMap(Over
viewBuilder.java:191)
  at
uk.me.parabola.mkgmap.combiners.OverviewBuilder.onFinish(OverviewBuil
der.java:103)
  at uk.me.parabola.mkgmap.main.Main.endOptions(Main.java:617)
  at
uk.me.parabola.mkgmap.CommandArgsReader.readArgs(CommandArgsReader.ja
va:128)
  at uk.me.parabola.mkgmap.main.Main.mainStart(Main.java:136)
  at uk.me.parabola.mkgmap.main.Main.main(Main.java:107)

1) When I add dem-poly=areas.poly the error disappears.

2) I also don't get the error if I replace -c dem-args with
--dem=e:\dem\europe\ --overview-dem-dist=88
--dem-dists=3312,13248,26512,53024

3) This only seems to affect certain pbfs.

file:  http://files.mkgmap.org.uk/download/385/12345678.osm.pbf

r
Nick








--
Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,
how about using a compressed temp-format for DEM information, which is
more suitable for DEM-calculation?
Or even store DEM-information in temp folder /DEM and just
reuse them next time if possible. I think (at least for me) they don't
change that often. So this will have a high impact and should be quite
easy to implement.

These DEM-precompiled data mkgmap could calculate in a separate run
after user changed map areas.

I guess whole DEM needs to be recalculated if areas change, am I right?
That's why we can't store pre-calculated DEM-1°-Tiles.

Henning

On 08.01.2018 18:19, Gerd Petermann wrote:
> Hi all,
>
> I see no way to improve the handling of zipped files much more. We have two 
> conflicting optimization goals here: 
> 1) reduce heap memory by storing less data in buffers (risking additional 
> rather slow unzip actions)
> 2) reduce run time be storing more data in buffers (risking 
> OutOfMemoryException)
> The problem is that the DEM algo needs more or less random access to the data 
> of one or more *.hgt.zip files,
> and I see no simple way to change that now.  
>
> I did a few tests now with r4036 and a map for scandinavia and hgt files in 
> 1'' resolution.
> The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed 
> folder the required disk space is ~ 37%,
> when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means 
> ~14%.
>
> I've used a simple poly file to cut scandinavia out of an older europe.o5m 
> with osmconvert and used splitter with max-nodes=100.
> This gave me 174 tiles.
> I used the following command to create a map: 
> java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt 
> --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 
> --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   
> --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
>
> I repeated this command two times, once with the uncompressed files, once 
> with the NTFS compressed files.
> All worked fine, java heap memory was no problem, but run time was different:
> 953 secs for the uncompressed files
> 1084 secs for the NTFS compressed files
> 1538 secs for the zipped files
>
> I think NTFS compressed folder is a good compromise if disk space matters. I 
> hope the numbers are similar for compressing Linux file systems.
> I plan to implement a new dem-temp=[dir] option. If given, mkgmap could 
> create temp files for the unzipped hgt data.  
>
> Gerd
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Hi Gerd,
one general question: Do you see any advantage in using 'rounded' binary
values or doesn't it matter at all?
Henning

On 08.01.2018 18:21, Gerd Petermann wrote:
> Hi Nick,
>
> thanks for reporting. I'll try to reproduce the problem. Please note that
> overview-dem-dist=88
> is probably too large, 8 would be a "normal" value.
>
> Gerd
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Hi Henning,

no idea what that means. Where would you use these values?
The data in hgt files is stored as 16 bit signed integer value in metres.

Gerd

Von: mkgmap-dev  im Auftrag von Henning 
Scholland 
Gesendet: Montag, 8. Januar 2018 12:56:18
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Error when not using dem-poly

Hi Gerd,
one general question: Do you see any advantage in using 'rounded' binary
values or doesn't it matter at all?
Henning

On 08.01.2018 18:21, Gerd Petermann wrote:
> Hi Nick,
>
> thanks for reporting. I'll try to reproduce the problem. Please note that
> overview-dem-dist=88
> is probably too large, 8 would be a "normal" value.
>
> Gerd
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Hi Gerd,
I was thinking about the distance-values for DEM.

Henning

On 08.01.2018 20:35, Gerd Petermann wrote:
> Hi Henning,
>
> no idea what that means. Where would you use these values?
> The data in hgt files is stored as 16 bit signed integer value in metres.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von 
> Henning Scholland 
> Gesendet: Montag, 8. Januar 2018 12:56:18
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Error when not using dem-poly
>
> Hi Gerd,
> one general question: Do you see any advantage in using 'rounded' binary
> values or doesn't it matter at all?
> Henning
>
> On 08.01.2018 18:21, Gerd Petermann wrote:
>> Hi Nick,
>>
>> thanks for reporting. I'll try to reproduce the problem. Please note that
>> overview-dem-dist=88
>> is probably too large, 8 would be a "normal" value.
>>
>> Gerd
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Henning,
I also thought about that and I think it could work using a similar algo as for 
the DEM format in IMG files.
We may have an index as with precomp sea data and rather small files compared 
to zip. On the other hand that format
would not be used by any other application, so in the end you might  need the 
data twice.

Gerd

Von: mkgmap-dev  im Auftrag von Henning 
Scholland 
Gesendet: Montag, 8. Januar 2018 12:52:05
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,
how about using a compressed temp-format for DEM information, which is
more suitable for DEM-calculation?
Or even store DEM-information in temp folder /DEM and just
reuse them next time if possible. I think (at least for me) they don't
change that often. So this will have a high impact and should be quite
easy to implement.

These DEM-precompiled data mkgmap could calculate in a separate run
after user changed map areas.

I guess whole DEM needs to be recalculated if areas change, am I right?
That's why we can't store pre-calculated DEM-1°-Tiles.

Henning

On 08.01.2018 18:19, Gerd Petermann wrote:
> Hi all,
>
> I see no way to improve the handling of zipped files much more. We have two 
> conflicting optimization goals here:
> 1) reduce heap memory by storing less data in buffers (risking additional 
> rather slow unzip actions)
> 2) reduce run time be storing more data in buffers (risking 
> OutOfMemoryException)
> The problem is that the DEM algo needs more or less random access to the data 
> of one or more *.hgt.zip files,
> and I see no simple way to change that now.
>
> I did a few tests now with r4036 and a map for scandinavia and hgt files in 
> 1'' resolution.
> The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS compressed 
> folder the required disk space is ~ 37%,
> when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means 
> ~14%.
>
> I've used a simple poly file to cut scandinavia out of an older europe.o5m 
> with osmconvert and used splitter with max-nodes=100.
> This gave me 174 tiles.
> I used the following command to create a map:
> java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt 
> --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 
> --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   
> --dem-dists=3312,5846,8948,12646,20282 --gmapi -c e:\scand\tiles\template.args
>
> I repeated this command two times, once with the uncompressed files, once 
> with the NTFS compressed files.
> All worked fine, java heap memory was no problem, but run time was different:
> 953 secs for the uncompressed files
> 1084 secs for the NTFS compressed files
> 1538 secs for the zipped files
>
> I think NTFS compressed folder is a good compromise if disk space matters. I 
> hope the numbers are similar for compressing Linux file systems.
> I plan to implement a new dem-temp=[dir] option. If given, mkgmap could 
> create temp files for the unzipped hgt data.
>
> Gerd
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Hi Henning,

Do you think that a certain value would allow faster processing of zipped files?
Well, we probably always have to unzip most of the file, but with a clever 
algorithm we may be able to store only
needed values. For dem-dists below 1 this is probably nearly every value, 
for much higher values it might be possible
to omit some. This gets more complicated when you take into account that img 
tiles don't fully overlap hgt bounds
and even more complex with --dem-poly.
I think the code would be quite complex and I don't expect much improvement 
reg. heap memory.

Before coding that I'd prefer to extract zip files into a temp file directory 
and use the MappedByteBuffer class as with normal hgt files.
This seems to manage memory very well for our purpose, at least with Oracles 
JRE.

Gerd

Von: mkgmap-dev  im Auftrag von Henning 
Scholland 
Gesendet: Montag, 8. Januar 2018 13:36:46
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Error when not using dem-poly

Hi Gerd,
I was thinking about the distance-values for DEM.

Henning

On 08.01.2018 20:35, Gerd Petermann wrote:
> Hi Henning,
>
> no idea what that means. Where would you use these values?
> The data in hgt files is stored as 16 bit signed integer value in metres.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von 
> Henning Scholland 
> Gesendet: Montag, 8. Januar 2018 12:56:18
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Error when not using dem-poly
>
> Hi Gerd,
> one general question: Do you see any advantage in using 'rounded' binary
> values or doesn't it matter at all?
> Henning
>
> On 08.01.2018 18:21, Gerd Petermann wrote:
>> Hi Nick,
>>
>> thanks for reporting. I'll try to reproduce the problem. Please note that
>> overview-dem-dist=88
>> is probably too large, 8 would be a "normal" value.
>>
>> Gerd
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,
I was thinking of writing the 'final' DEM not only in img-container but
also in a temp-file. This temp file can be used for all updates of map
data without changes in areas.list.

More or less the work flow can be: splitter calculate areas.list, mkgmap
can calculate temporary DEM for areas.list. As long as I use the same
areas.list for the map, I can reuse the finished DEM. If I need to
change areas.list of course I need to let mkgmap calculate a new DEM.
Overview-DEM will only change, if total map area is changing. So
probably only needs to be calculated once per map. If we work like that
at least for me compilation time wouldn't be that important.

Basically the idea is to keep the static map-data and only update the
dynamic part of the data. The other advantage would be, that we can use
full amount of RAM during DEM-calculation, as there is no RAM used for
map- data and so on...

I don't know if it's really possible to store the 'final' DEM but at
least some pre-compiled file should be possible I assume. I'm not sure,
how img-files are written.

Henning

On 08.01.2018 20:42, Gerd Petermann wrote:
> Hi Henning,
> I also thought about that and I think it could work using a similar algo as 
> for the DEM format in IMG files.
> We may have an index as with precomp sea data and rather small files compared 
> to zip. On the other hand that format
> would not be used by any other application, so in the end you might  need the 
> data twice.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von 
> Henning Scholland 
> Gesendet: Montag, 8. Januar 2018 12:52:05
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Performance with zipped hgt files
>
> Hi Gerd,
> how about using a compressed temp-format for DEM information, which is
> more suitable for DEM-calculation?
> Or even store DEM-information in temp folder /DEM and just
> reuse them next time if possible. I think (at least for me) they don't
> change that often. So this will have a high impact and should be quite
> easy to implement.
>
> These DEM-precompiled data mkgmap could calculate in a separate run
> after user changed map areas.
>
> I guess whole DEM needs to be recalculated if areas change, am I right?
> That's why we can't store pre-calculated DEM-1°-Tiles.
>
> Henning
>
> On 08.01.2018 18:19, Gerd Petermann wrote:
>> Hi all,
>>
>> I see no way to improve the handling of zipped files much more. We have two 
>> conflicting optimization goals here:
>> 1) reduce heap memory by storing less data in buffers (risking additional 
>> rather slow unzip actions)
>> 2) reduce run time be storing more data in buffers (risking 
>> OutOfMemoryException)
>> The problem is that the DEM algo needs more or less random access to the 
>> data of one or more *.hgt.zip files,
>> and I see no simple way to change that now.
>>
>> I did a few tests now with r4036 and a map for scandinavia and hgt files in 
>> 1'' resolution.
>> The 380 unzipped hgt files require ~ 9.4 GB. When stored in a NTFS 
>> compressed folder the required disk space is ~ 37%,
>> when stored in single zipped files (*.hgt.zip) the size is ~1.3 GB, means 
>> ~14%.
>>
>> I've used a simple poly file to cut scandinavia out of an older europe.o5m 
>> with osmconvert and used splitter with max-nodes=100.
>> This gave me 174 tiles.
>> I used the following command to create a map:
>> java -Xmx6800m -jar d:\mkgmap\dist\mkgmap.jar -c d:\dbg\dev-addr.opt 
>> --output-dir=.\map --block-size=2048 --max-jobs=4 --overview-dem-dist=55000 
>> --dem-poly=f:\osm\scandinavia.poly --dem=f:\srtm3_1_zip   
>> --dem-dists=3312,5846,8948,12646,20282 --gmapi -c 
>> e:\scand\tiles\template.args
>>
>> I repeated this command two times, once with the uncompressed files, once 
>> with the NTFS compressed files.
>> All worked fine, java heap memory was no problem, but run time was different:
>> 953 secs for the uncompressed files
>> 1084 secs for the NTFS compressed files
>> 1538 secs for the zipped files
>>
>> I think NTFS compressed folder is a good compromise if disk space matters. I 
>> hope the numbers are similar for compressing Linux file systems.
>> I plan to implement a new dem-temp=[dir] option. If given, mkgmap could 
>> create temp files for the unzipped hgt data.
>>
>> Gerd
>> ___
>> mkgmap-dev mailing list
>> mkgmap-dev@lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Error when not using dem-poly

Basically my idea was, if I always double the zoom-level (dist = 4000,
8000, ...) then I just need to calculate the highest resolution and all
others can be produced by just throwing out every 2nd value. It's not
related to zipped input. It would be faster I think, but definitely less
flexible. So I'm not sure, if it's worth thinking about it.

Henning

On 08.01.2018 20:59, Gerd Petermann wrote:
> Hi Henning,
>
> Do you think that a certain value would allow faster processing of zipped 
> files?
> Well, we probably always have to unzip most of the file, but with a clever 
> algorithm we may be able to store only
> needed values. For dem-dists below 1 this is probably nearly every value, 
> for much higher values it might be possible
> to omit some. This gets more complicated when you take into account that img 
> tiles don't fully overlap hgt bounds
> and even more complex with --dem-poly.
> I think the code would be quite complex and I don't expect much improvement 
> reg. heap memory.
>
> Before coding that I'd prefer to extract zip files into a temp file directory 
> and use the MappedByteBuffer class as with normal hgt files.
> This seems to manage memory very well for our purpose, at least with Oracles 
> JRE.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von 
> Henning Scholland 
> Gesendet: Montag, 8. Januar 2018 13:36:46
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Error when not using dem-poly
>
> Hi Gerd,
> I was thinking about the distance-values for DEM.
>
> Henning
>
> On 08.01.2018 20:35, Gerd Petermann wrote:
>> Hi Henning,
>>
>> no idea what that means. Where would you use these values?
>> The data in hgt files is stored as 16 bit signed integer value in metres.
>>
>> Gerd
>>
>> 
>> Von: mkgmap-dev  im Auftrag von 
>> Henning Scholland 
>> Gesendet: Montag, 8. Januar 2018 12:56:18
>> An: mkgmap-dev@lists.mkgmap.org.uk
>> Betreff: Re: [mkgmap-dev] Error when not using dem-poly
>>
>> Hi Gerd,
>> one general question: Do you see any advantage in using 'rounded' binary
>> values or doesn't it matter at all?
>> Henning
>>
>> On 08.01.2018 18:21, Gerd Petermann wrote:
>>> Hi Nick,
>>>
>>> thanks for reporting. I'll try to reproduce the problem. Please note that
>>> overview-dem-dist=88
>>> is probably too large, 8 would be a "normal" value.
>>>
>>> Gerd
>> ___
>> mkgmap-dev mailing list
>> mkgmap-dev@lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> ___
>> mkgmap-dev mailing list
>> mkgmap-dev@lists.mkgmap.org.uk
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Henning,

I think that works already. I know for sure that there is no link between the 
other files (RGN,NOD etc) and DEM data,
only the *.tdb contains a hint that the sub file exists.
You can replace a *.DEM file by another with different resolutions , at least 
if they cover the same boundaries.
So, if you use the gmap folder (--gmapi) you can already do what you want with 
a script.
Example: Calculate the DEM with the wanted resolution, store the *.DEM files. 
Next time use options which calculate
an almost DEM file (one level with very high dem-dist) and finally replace the 
*.DEM files with the stored ones.
You probably need another tool to combine those files into a gmapsupp.img.

So, possible workflow with r4037 looks like this:
1) run splitter with real OSM file to calculate areas.list
2) run splitter with this areas.list and an dummy osm file containing no data, 
this will produce osm.pbf files with the wanted boundaries
3) run mkgmap for those empty osm files with the wanted DEM options to create 
the good DEM files
4) run mkgmap for the files created in 1) with fake DEM options , e.g. 
--overview-dem-dist=276000 --dem-dists=5 --dem=
5) replace *.DEM files created in 4) by *.DEM created in 3)
6) use a tool to combine the files into *.img

Next time when you update the osm data you split with the areas.list created in 
step 1)
and repeat steps 3-6.
I've just tried that with a map for nepal and it worked fine (steps 1-5) , at 
least MapSource is happy with the resulting gmap folder.
Don't know what tool and options you need to combine that folder back into img 
format.
I've attached the dummy.osm file used in 2)

Please let me know if that works for you.
The only problem that I see is in combination with option --dem-poly. If you 
want to reuse tiles for different maps with different polygons (e.g.
calculate DEM for Europe once and create maps for singles countries using 
specific *.poly files) you need a new method to filter the preprocessed DEM 
data.

Gerd

Von: mkgmap-dev  im Auftrag von Henning 
Scholland 
Gesendet: Montag, 8. Januar 2018 14:16:31
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,
I was thinking of writing the 'final' DEM not only in img-container but
also in a temp-file. This temp file can be used for all updates of map
data without changes in areas.list.

More or less the work flow can be: splitter calculate areas.list, mkgmap
can calculate temporary DEM for areas.list. As long as I use the same
areas.list for the map, I can reuse the finished DEM. If I need to
change areas.list of course I need to let mkgmap calculate a new DEM.
Overview-DEM will only change, if total map area is changing. So
probably only needs to be calculated once per map. If we work like that
at least for me compilation time wouldn't be that important.

Basically the idea is to keep the static map-data and only update the
dynamic part of the data. The other advantage would be, that we can use
full amount of RAM during DEM-calculation, as there is no RAM used for
map- data and so on...

I don't know if it's really possible to store the 'final' DEM but at
least some pre-compiled file should be possible I assume. I'm not sure,
how img-files are written.

Henning

On 08.01.2018 20:42, Gerd Petermann wrote:
> Hi Henning,
> I also thought about that and I think it could work using a similar algo as 
> for the DEM format in IMG files.
> We may have an index as with precomp sea data and rather small files compared 
> to zip. On the other hand that format
> would not be used by any other application, so in the end you might  need the 
> data twice.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag von 
> Henning Scholland 
> Gesendet: Montag, 8. Januar 2018 12:52:05
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] Performance with zipped hgt files
>
> Hi Gerd,
> how about using a compressed temp-format for DEM information, which is
> more suitable for DEM-calculation?
> Or even store DEM-information in temp folder /DEM and just
> reuse them next time if possible. I think (at least for me) they don't
> change that often. So this will have a high impact and should be quite
> easy to implement.
>
> These DEM-precompiled data mkgmap could calculate in a separate run
> after user changed map areas.
>
> I guess whole DEM needs to be recalculated if areas change, am I right?
> That's why we can't store pre-calculated DEM-1°-Tiles.
>
> Henning
>
> On 08.01.2018 18:19, Gerd Petermann wrote:
>> Hi all,
>>
>> I see no way to improve the handling of zipped files much more. We have two 
>> conflicting optimization goals here:
>> 1) reduce heap memory by storing less data in buffers (risking additional 
>> rather slow unzip actions)
>> 2) reduce run time be storing more data in buffers (risking 
>> OutOfMemoryException)
>> The problem is that the DEM algo needs more or less r

Re: [mkgmap-dev] Performance with zipped hgt files

2018-01-08 Thread Frank Stinner

Hi Gerd,

at last you need the data uncompressed in a buffer, right? That's why i 
see no difference between using compressed or uncompressed hgt files. 
Compressed files need a little bit more time for uncompressing, but after 
that the data need the same size in memory. That's why i see no vantage 
for mkgmap if the data are stored in an other form. It can only save disc 
space, no heap space.

What speaks against a limitation for the extent of a maptile? This limited 
the count of hgt's for one maptile too.

You have the option --max-areas for splitter. We need only an additional 
option like --maxextent or whatever. Perhaps we have then maptiles without 
points, lines or areas (on oceans, deserts or so), but why not?


Frank___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Frank,

the current problem with zipped files is that I have to extract them into a
buffer that is kept in Java Heap (managed by Garbage Collection)
while uncompressed files are manged by class MappedByteBuffer. These are not
managed as Java Heap, they are somewhat virtual and
typically only require a small portion of the available memory, see
https://howtodoinjava.com/java-7/nio/java-nio-2-0-memory-mapped-files-mappedbytebuffer-tutorial/

I don't yet see a need to change splitter like that but I can do that as well.
I am now playing with code to extract zipped files into a temp dir.

Gerd

Von: mkgmap-dev im Auftrag von Frank
Stinner
Gesendet: Montag, 8. Januar 2018 15:30:08
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,

at last you need the data uncompressed in a buffer, right? That's why i see no
difference between using compressed or uncompressed hgt files. Compressed files
need a little bit more time for uncompressing, but after that the data need the
same size in memory. That's why i see no vantage for mkgmap if the data are
stored in an other form. It can only save disc space, no heap space.

What speaks against a limitation for the extent of a maptile? This limited the
count of hgt's for one maptile too.

You have the option --max-areas for splitter. We need only an additional option
like --maxextent or whatever. Perhaps we have then maptiles without points,
lines or areas (on oceans, deserts or so), but why not?

Frank
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files


Hi Gerd,

> I did not try it but mkgmap should support 76x76 tiles, but I see no
> need to use that with mkgmap, at least not with unzipped hgt files.

Yes, but since I already have created small version, I can save several 
minutes of reading HGT files form drive, when creating a big overview map.


The real use for non standard resolutions would be when dealing with 
data sources other than SRTM, like for example EU-DEM.


--
Best regards,
Andrzej
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files


Hi Gerd,


I don't yet see a need to change splitter like that but I can do that
as well.


I would use this option too. Whenever I split data for Europe, I get a 
tile which includes Azores and south shore of Island. I split it 
manually then.


For maps with DEM I use an awk script, which process areas.list. I set 
maximum extent to 11.25 degree.


--
Best regards,
Andrzej
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,
basically this describes my idea, just less quick&dirty ;-) I will give
it a try next weekend.

Of course for every map, you need a individual temp-DEM. So I don't seen
any issues with poly-cutted-tiles.

Henning
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Andrzej,

okay, I'll add an option for that. At the moment, we have these hard coded 
limits in SplittableDensityArea:
private static final int MAX_LAT_DEGREES = 85;
private static final int MAX_LON_DEGREES = 90;

I have to find out how a change in these limits affects the search for the best 
split, esp. when the splitting planet.
I think one has to accept empty tiles with a max of 11.25 degrees. I think the 
last changes in the split algo were done to avoid this.

Gerd


Von: mkgmap-dev  im Auftrag von Andrzej 
Popowski 
Gesendet: Montag, 8. Januar 2018 18:43:33
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,

> I don't yet see a need to change splitter like that but I can do that
> as well.

I would use this option too. Whenever I split data for Europe, I get a
tile which includes Azores and south shore of Island. I split it
manually then.

For maps with DEM I use an awk script, which process areas.list. I set
maximum extent to 11.25 degree.

--
Best regards,
Andrzej
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files


Hi Gerd,

I decided for 11.25 (=360/32) because of memory limits in early 
BuildDemFile. Maybe this is not a problem for mkgmap?


--
Best regards,
Andrzej

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files

Hi Andrzej,

I think with unzipped hgt files the only limit is the memory that is needed to 
store the data for the tiles before they are written to img.
So, with --max-jobs=4 or higher  and all jobs with large tiles containing large 
amounts of data for RGN,NOD,NET etc. it is much more likely that the additional 
data for DEM causes trouble. I've not yet tried to reduce the memory for this 
temp. data, a possible alternative is to write it to temp files instead of 
storing it in buffers. This is esp. true for large tiles were DEM can be > 
100MB while the other files cannot grow > 16MB.

Gerd


Von: mkgmap-dev  im Auftrag von Andrzej 
Popowski 
Gesendet: Montag, 8. Januar 2018 21:04:48
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] Performance with zipped hgt files

Hi Gerd,

I decided for 11.25 (=360/32) because of memory limits in early
BuildDemFile. Maybe this is not a problem for mkgmap?

--
Best regards,
Andrzej

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] Performance with zipped hgt files


Hi Gerd,

I can confirm, that mkgmap has no memory problems with DEM for big 
tiles. I can compile map without limiting tile sizes.


--
Best regards,
Andrzej
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] DEM Noddy Guide