--
Date: Tue, 13 Nov 2012 18:15:43 +0100
From: ulib...@gmail.com
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: [mkgmap-dev] splitter improvements
I think the memory requirements are more dependent on the data than on the
splitter version:
I have a self created data set
Hi Uli,
please tell me also how you create the file containing iceland.
If you have kept the log files, please send them also.
Gerd
--
View this message in context:
http://gis.19327.n5.nabble.com/splitter-improvements-tp5735718p5735727.html
Sent from the Mkgmap Development mailing list
@lists.mkgmap.org.uk
Subject: Re: [mkgmap-dev] splitter improvements
Hi,
in case you have trouble with reproducing the problem, there is java option
-XX:-HeapDumpOnOutOfMemoryError. Uli can run splitter with this option and then
send you produced hprof file which you can open in MemoryAnalyzer
...@gmail.com
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: [mkgmap-dev] splitter improvements
I think the memory requirements are more dependent on the data than on the
splitter version:
I have a self created data set consisting of germany + ~50 kilometers plus the
data from iceland cut with osmconvert.exe
Subject: Re: [mkgmap-dev] splitter improvements
Hi,
in case you have trouble with reproducing the problem, there is java
option -XX:-HeapDumpOnOutOfMemoryError. Uli can run splitter with this
option and then send you produced hprof file which you can open in
MemoryAnalyzer or jhat. There you
Hi Gerd,
this is the way i create the germany + 50km + iceland .osm.pbf-file:
First step is download from geofabrik (only about once a month):
..\bin\wget --no-cache http://download.geofabrik.de/openstreetmap/europe.osm.pbf
Then i cut the interesting area(s), i'll attach the corresponding
Hi Gerd,
i just tried it once again using splitter-r231.
This is the error, that occurs:
23.000.000 ways processed... id=175802045
24.000.000 ways processed... id=184296766
Stats for MultiTileProcessor pass 2 endMap Pass2 end
SparseBitSet problemRels 173576 (917000 bytes)
SparseBitSet
UliBär (Gmail) wrote
Hi Gerd,
i just tried it once again using splitter-r231.
Hi Uli,
I fixed the error with r232 which I've just committed.
Thanks for reporting !
Gerd
--
View this message in context:
http://gis.19327.n5.nabble.com/splitter-improvements-tp5735785p5735791.html
Sent
Hi Uli,
I fixed the error with r232 which I've just committed.
Thanks for reporting !
Gerd
Hi Gerd,
you're welcome! :)
Only minutes ago splitter-r232 just flew thru the germany+ dataset on
a 32-bit-environment using -Xmx1376m without a single problem!
The problem-candidates-list sizes
Thorsten Kukuk wrote
Correct, I all do is:
osmconvert planet.osm.pbf -B=osmmaps/scripts/poly/DACH.poly --drop-author
--drop-version --drop-broken-refs --out-pbf -o=data/osm/DACH.osm.pbf
Ok. First test with r224 crashed with NPE . I did not use the
-drop-broken-refs parm, so I used r225 to
On Tue, Nov 13, GerdP wrote:
Ahh, didn't I write that r229 uses less memory? Yes, I did. The explanation
is here: r225 detected fewer problem cases:
Ok, thanks for the analysis!
Besides that I have an idea how to reduce the memory needs in the critical
pass :-)
Good to hear :-)
Thorsten
Thorsten Kukuk wrote
Besides that I have an idea how to reduce the memory needs in the
critical
pass :-)
Good to hear :-)
One drawback: This better solution will not work if input is mixed (ids not
strictly ordered).
I see two options:
1) use two alternative methods to store data. The
I think the memory requirements are more dependent on the data than on the
splitter version:
I have a self created data set consisting of germany + ~50 kilometers plus
the data from iceland cut with osmconvert.exe from the europe.osm.pbf from
geofabrik, which is processed without errors with
Hi,
I want to reproduce the problem, but I don't see a file with 1.9GB at this
place: http://140.78.94.22/osm/
Where am I wrong?
Gerd
Date: Tue, 13 Nov 2012 18:15:43 +0100
From: ulib...@gmail.com
To: mkgmap-dev@lists.mkgmap.org.uk
Subject: [mkgmap-dev] splitter improvements
I think
Hi Gerd,
take th germany+.osm.xz
Henning
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Hi,
I want to reproduce the problem, but I don't see a file with 1.9GB at this
place: http://140.78.94.22/osm/ Where am I wrong?
GerdHi Gerd,sorry, i forgot:I take the germany+.osn.xz as input to xz.exe and
pipe the outputto osmconvert.exe and convert the
stdin to germany+.osm.pbf.This
Hi,
I want to reproduce the problem, but I don't see a file with 1.9GB at
this place: http://140.78.94.22/osm/
Where am I wrong?
Gerd
Just saw, my email program messed up the format, so here once more in
better readable form:
Hi Gerd,
sorry, i forgot to mention: I take the germany+.osm.xz
Hi Uli,
can please also tell me the splitter parms that you use?
The speed and memory needs depends directly on the number and the position
of the tiles.
Gerd
UliBär (Gmail) wrote
Hi,
I want to reproduce the problem, but I don't see a file with 1.9GB at
this place:
Hi,
On Sat, Nov 10, GerdP wrote:
I've just committed r229.
I've fixed all reported errors regarding --keep-complete and found some nice
performance improvements for Hennings special case (planet as input,
split-file with 800 tiles, many of them overlapping each other)
and some general
Hi Thorsten,
that's strange, the newer version should require less memory, esp. in that
part
of the program. Did you use exactly the same input files and parameters?
I will check if D A CH is somehow special, please let me know your
parameters.
Please try again with r231, which I've just
Hi,
On Mon, Nov 12, GerdP wrote:
Hi Thorsten,
that's strange, the newer version should require less memory, esp. in that
part
of the program. Did you use exactly the same input files and parameters?
Yes, I did, and verfied it two times.
I will check if D A CH is somehow special, please
Hi Thorsten,
java -jar /usr/share/java/splitter-r224.jar --mapid=7121
--max-nodes=100 --overlap=0 --keep-complete=true
--geonames-file=osmmaps/scripts/cities/DACH.txt --description=TK-DACH-Tile
--output=pbf --output-dir=build/DACH/tiles data/osm/DACH.osm.pbf
if you really
Hi Thorsten,
no problem. Just to make sure: I want to reproduce the problem with an
extact of europe using the DACH.poly from your package
http://osm.thkukuk.de/tk-osm.tar.bz2
I did not try to understand or run all the scripts. Is it correct that you
do not merge other data into this file?
On Mon, Nov 12, GerdP wrote:
Hi Thorsten,
no problem. Just to make sure: I want to reproduce the problem with an
extact of europe using the DACH.poly from your package
http://osm.thkukuk.de/tk-osm.tar.bz2
I did not try to understand or run all the scripts. Is it correct that you
do not
Henning Scholland wrote
Hi Gerd,
it tooks a lot of RAM, but it works now for me. It took 2:30 h so its
ok. Thanks a lot!
Great! Please send me the log, I like to find out where it needs most
memory.
Today I've created a version of SparseLong2ShortMap that requires ~ 10% less
memory, but is a
Hi Gerd,
log is available here: http://www.aighes.de/data/mkgmap.zip
Henning
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
26 matches
Mail list logo
