Re: [mkgmap-dev] [locator] Separate boundary files
The European boundaries compiled by WanMil are available for download:
http://www.navmaps.eu/wanmil/europe_bounds_20110429.zip
Johan
On Fri, 29 Apr 2011 22:49:54 +0200, WanMil wmgc...@web.de wrote:
Great! That was a silly bug...
I think I found another big issue that affects quite a lot of
boundaries. My osmosis command line was not complete. Ways without
boundary=administrative were not contained in the output file. I
think
for this the --used-way parameter must be added. It is new in osmosis
0.39.
I started osmosis with the europe dump from today. Hopefully that
works
and then I will upload the precompiled tiles. Details where you can
download them will be posted afterwards.
WanMil
It seems r1929 did the trick. Now I get 44 matches for Calle
Calvario
(vs 46 with trunk), all of them but two with complete city, region,
country information. Also most of the States/Provinces are now
correct.
El 28/04/11 23:23, Carlos Dávila escribió:
El 28/04/11 21:49, WanMil escribió:
Carlos,
thanks for your reports. I cannot check your examples in detail
now but
I can give you some general hints.
The current locator branch relies on complete boundaries. The word
complete is important. The boundary precompilation algorithm does
not
automatically close multipolygons which are open and which has a
loose
endpoints out of the tile borders. This is necessary because one
major
reason for using precompiled data is that I want to get rid of
wrong
autoclosed boundaries. (Please read
http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q1/010797.html
where I
tried to explain the problem more in detail).
I know the importance of complete boundaries. That's the reason I'm
trying to fix all incomplete ones detected by mkgmap.
If you use a dump of spain only you might loose some of the
boundaries
that are located on the border. I would recommend to download a
larger
dump and cut out a larger area than spain for boundary
precompilation
(e.g. spain + 10km around).
I modified the cut polygon used by Geofabrik for the Spain dump a
few
weeks ago, so that it fits exactly Spanish border+ a few meters, so
no
boundary near the border should be cut. In fact, I have not faced
any
one detected by mp processing. Only a couple of boundaries are cut
due
to some changes I did yesterday in a portion of the border.
Of course the changes are not tested very well and there are tons
of
bugs in it. I'll have to implement some checks to see if the
precompilation really works well.
WanMil
These are the results of my first (limited) tests:
The list of places under State/Province field in MapSource
Search
places includes a lot more wrong places than without the
boundary
precompilation (locator r1922). For example I get State/Province
La
Zubia (from relation 347255) and place Cumbres Verdes (node
1107558640), LA ZUBIA, ESP under it.
In the last days I've been correcting incomplete boundary
polygons from
mkgmap log. Searching for a random street (Calle Calvario) that
exists
within one of these multipolygons I fixed yesterday (relation
346527) I
get the following:
Trunk+spain.osm.pbf 28/4- 46 Calle Calvario matches with
complete
city, region, country information (e.g. way 62120822: Calle
Calvario,
Alburquerque, EXTREMADURA, ESP)
Locator r1925+spain.osm.pbf 27/4 - 39 matches most of them
with
incomplete or even wrong city, region, country information (e.g.
way
62120822: Calle Calvario, CÁCERES (should be BADAJOZ), ESP).
Cáceres is
relation 349018 and Badajoz 348994.
Locator r1925+spain.osm.pbf 28/4 - same result. Note mkgmap
didn't
complain about mp 346527.
The boundary tiles are extracted daily from the same pbf file
than map.
My locator related styles:
mkgmap:country!=* addr:country=* { set
mkgmap:country='${addr:country}' }
mkgmap:country!=* is_in:country=* { set
mkgmap:country='${is_in:country}' }
mkgmap:country!=* mkgmap:admin_level2=* { set
mkgmap:country='${mkgmap:admin_level2}' }
mkgmap:region!=* is_in:province=* { set
mkgmap:region='${is_in:province}' }
mkgmap:region!=* mkgmap:admin_level6=* { set
mkgmap:region='${mkgmap:admin_level6}' }
mkgmap:region!=* mkgmap:admin_level5=* { set
mkgmap:region='${mkgmap:admin_level5}' }
mkgmap:region!=* mkgmap:admin_level4=* { set
mkgmap:region='${mkgmap:admin_level4}' }
mkgmap:region!=* mkgmap:admin_level3=* { set
mkgmap:region='${mkgmap:admin_level3}' }
mkgmap:city!=* openGeoDB:name=* { set
mkgmap:city='${openGeoDB:name}' }
mkgmap:city!=* is_in:city=* { set
mkgmap:city='${is_in:city}' }
mkgmap:city!=* addr:city=* { set mkgmap:city='${addr:city}'
}
mkgmap:city!=* mkgmap:admin_level8=* { set
mkgmap:city='${mkgmap:admin_level8}' }
mkgmap:city!=* mkgmap:admin_level7=* { set
mkgmap:city='${mkgmap:admin_level7}' }
mkgmap:city!=* mkgmap:admin_level9=* { set
Re: [mkgmap-dev] [locator] Separate boundary files
On Apr 30, 2011, at 8:51, navmaps wrote: The European boundaries compiled by WanMil are available for download: http://www.navmaps.eu/wanmil/europe_bounds_20110429.zip Thanks for uploading the file. Regarding the boundary function, when I compile, I notice a few errors in the log file for bounds_205_10.bnd: 2011/04/30 10:29:20 SEVERE (BoundaryUtil): 9401.osm.gz: Cannot load boundary file bounds/bounds_205_10.bnd: java.io.UTFDataFormatException: malformed input around byte 250 Is there a bad character compiled into this file? It seems to function correctly otherwise. Cheers. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
I have tried locator v1928 with the uploaded europe-boundaries but still no success. In Mapsource about 45/80 of the random streets that I tested had no placename. In all those cases mkgmap-locator-r1915.jar could find a matching placename. If I look to those places that could be matched with a placename, and I enter the placename on the GPS, the streets show up in a list but if I choose that street no location can be found. I can only find that location by skipping the placename and enter the streetname directly. With the other 50% (streets with no location in Mapsource) they are not searchable at all on the GPS. With r1915 most of the streets were found, even on the GPS. I also examined the logfile. This street couldn't be found with a matching place (Tilburg) http://www.openstreetmap.org/browse/way/77208790 (locator 1928 can't find not just one street in Tilburg, a big city, locator 1915 finds many streets there) In the logfile I found these messages: Added tag mkgmap:admin_level2 = NLD to [name=Cyprespad,highway=pedestrian,mkgmap:admin_level2=NLD Added tag mkgmap:admin_level4 = Noord-Brabant to [name=Cyprespad,highway=pedestrian,mkgmap:admin_level2=NLD,mkgmap:admin_level4=Noord-Brabant Nothing more, no admin_level 8 and 10 are matched. One example of a street that could be matched in Mapsource with a placename: Added tag mkgmap:admin_level8 = Gilze en Rijen to [name=Gerardus Majellastraat,mkgmap:admin_level8=Gilze en Rijen,highway=unclassified Added tag mkgmap:admin_level10 = Hulten to [name=Gerardus Majellastraat,mkgmap:admin_level8=Gilze en Rijen,highway=unclassified,mkgmap:admin_level10=Hulten Added tag mkgmap:admin_level2 = NLD to [name=Gerardus Majellastraat,highway=unclassified,mkgmap:admin_level2=NLD Added tag mkgmap:admin_level4 = Noord-Brabant to [name=Gerardus Majellastraat,highway=unclassified,mkgmap:admin_level2=NLD,mkgmap:admin_level4=Noord-Brabant If I enter Hulten on the GPS, the streetname Gerardus Majellastraat shows up in the list of streetnames, but if I click on it, no location can be found. If I skip the placename, and enter directly the name Gerardus Majellastraat, the location can be found. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
El 30/04/11 00:59, WanMil escribió: I have the problem that some ways are missing in the osmosis filtered dump. Here is my osmosis call: osmosis.bat --rb europe.osm.pbf --tf accept-ways boundary=administrative --tf accept-relations boundary=administrative --used-way --used-node --wx europe-boundaries.osm.gz One example what is missing so far: Relation 7374 (http://www.openstreetmap.org/browse/relation/7374) Missing way 41119221 (http://www.openstreetmap.org/browse/way/41119221) The way is not tagged with boundary=administrative but is a member of the boundary=administrative tagged relation 7374. Is that the only way missing in this relation? Ways 40404314 and 37208295 are not tagged boundary=administrative, are they present? I have the same problem with relation 342099: ways 85765542 and 85765550 are missing. I tried several ways of running osmosis but didn't succeed. I think the accept-ways part of the osmosis run is discarding those ways not tagged administrative, although they are part of a boundary=administrative mp. I tried to extract ways and mp in separate osmosis runs, but I get a java heap space error with any of the commands below: osmosis --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way --tf reject-ways --tf reject-nodes --write-xml file=spain-boundaries3.osm osmosis --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way --write-xml file=spain-boundaries3.osm Adding --used-way discards some ways tagged boundary=administrative which are not members of multipolygons. I'm currently checking those discarded from the spain-boundaries.osm (38) and some of them were wrongly tagged as administrative or islands missing in the continental admin_level=8 multipolygon, but others seem to be correct (e.g. ways 30108987, 32892880 and 31408486) ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
Try: osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm I have't tried it, because I haven't now access to my desktop , but I think, this should catch all needed data. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
Am 30.04.2011 12:58, schrieb Henning Scholland: osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
On Apr 30, 2011, at 8:51, navmaps wrote: The European boundaries compiled by WanMil are available for download: http://www.navmaps.eu/wanmil/europe_bounds_20110429.zip Thanks for uploading the file. Regarding the boundary function, when I compile, I notice a few errors in the log file for bounds_205_10.bnd: 2011/04/30 10:29:20 SEVERE (BoundaryUtil): 9401.osm.gz: Cannot load boundary file bounds/bounds_205_10.bnd: java.io.UTFDataFormatException: malformed input around byte 250 Is there a bad character compiled into this file? It seems to function correctly otherwise. Cheers. I've seen such an error too but I couldn't find out what's the real problem. When I run the BoundaryFile2Gpx on this file everythings ok. I have two ideas: 1. Maybe the boundary utils are not thread safe. Do you use more than one thread? 2. The file format for each boundary starts with the boundary rect and then the size that must be skipped to reach the next boundary. This is used to load only those boundaries which intersects the tile. Maybe there is a problem in this size information. I will check that. WanMil ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
On Apr 30, 2011, at 13:43, WanMil wrote: 1. Maybe the boundary utils are not thread safe. Do you use more than one thread? Yes, I do. I'll try it in single threaded mode. Cheers ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
On Apr 30, 2011, at 13:43, WanMil wrote: 1. Maybe the boundary utils are not thread safe. Do you use more than one thread? I still get the error with only one thread. By the way, I'm on Mac OS with Java version 1.6.0_24 (64 bit), if that's any help. Cheers. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
El 30/04/11 13:00, Henning Scholland escribió: Am 30.04.2011 12:58, schrieb Henning Scholland: osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
Am 30.04.2011 14:19, schrieb Carlos Dávila: El 30/04/11 13:00, Henning Scholland escribió: Am 30.04.2011 12:58, schrieb Henning Scholland: osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --write-xml boundary.osm Sorry, I've forgotten an --used-node osmosis --read-pbf europe.osm.pbf --tf accept-ways boundary=administrative --used-node --tf accept-relations boundary=administrative --used-way --used-node --write-xml boundary.osm I get exactly the same file than without the extra --used-node in the ways --tf task. I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
Am 30.04.2011 13:58, schrieb Clinton Gladstone: On Apr 30, 2011, at 13:43, WanMil wrote: 1. Maybe the boundary utils are not thread safe. Do you use more than one thread? I still get the error with only one thread. By the way, I'm on Mac OS with Java version 1.6.0_24 (64 bit), if that's any help. Cheers. Could you please try to run the BoundaryFile2Gpx converter for this boundary file? java -cp mkgmap.jar uk.me.parabola.mkgmap.reader.osm.boundary.BoundaryFile2Gpx bounds/bounds_205_10.bnd Does it throw the same error? I am not sure if I there is a problem with the big- and little-endian order on Mac versus PC? WanMil ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
Minko, please try the latest r1930. Carlos reported that r1929 did the trick for him (http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q2/011245.html). WanMil I have tried locator v1928 with the uploaded europe-boundaries but still no success. In Mapsource about 45/80 of the random streets that I tested had no placename. In all those cases mkgmap-locator-r1915.jar could find a matching placename. If I look to those places that could be matched with a placename, and I enter the placename on the GPS, the streets show up in a list but if I choose that street no location can be found. I can only find that location by skipping the placename and enter the streetname directly. With the other 50% (streets with no location in Mapsource) they are not searchable at all on the GPS. With r1915 most of the streets were found, even on the GPS. I also examined the logfile. This street couldn't be found with a matching place (Tilburg) http://www.openstreetmap.org/browse/way/77208790 (locator 1928 can't find not just one street in Tilburg, a big city, locator 1915 finds many streets there) In the logfile I found these messages: Added tag mkgmap:admin_level2 = NLD to [name=Cyprespad,highway=pedestrian,mkgmap:admin_level2=NLD Added tag mkgmap:admin_level4 = Noord-Brabant to [name=Cyprespad,highway=pedestrian,mkgmap:admin_level2=NLD,mkgmap:admin_level4=Noord-Brabant Nothing more, no admin_level 8 and 10 are matched. One example of a street that could be matched in Mapsource with a placename: Added tag mkgmap:admin_level8 = Gilze en Rijen to [name=Gerardus Majellastraat,mkgmap:admin_level8=Gilze en Rijen,highway=unclassified Added tag mkgmap:admin_level10 = Hulten to [name=Gerardus Majellastraat,mkgmap:admin_level8=Gilze en Rijen,highway=unclassified,mkgmap:admin_level10=Hulten Added tag mkgmap:admin_level2 = NLD to [name=Gerardus Majellastraat,highway=unclassified,mkgmap:admin_level2=NLD Added tag mkgmap:admin_level4 = Noord-Brabant to [name=Gerardus Majellastraat,highway=unclassified,mkgmap:admin_level2=NLD,mkgmap:admin_level4=Noord-Brabant If I enter Hulten on the GPS, the streetname Gerardus Majellastraat shows up in the list of streetnames, but if I click on it, no location can be found. If I skip the placename, and enter directly the name Gerardus Majellastraat, the location can be found. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
I'm sorry, it didnt work either. In short, a) entering a placename NEVER gives a matching location (tested on a nüvi) b) skipping the placename, roughly 50% of the streetnames have matching places and a location can be found. tested this on two tiles: 10010037: 2394112,225280 to 2410496,241664 # : 51.372070,4.833984 to 51.723633,5.185547 10010049: 2390016,204800 to 2410496,225280 # : 51.284180,4.394531 to 51.723633,4.833984 Another tile had better results (all streets seemed to have corresponding places) but on the GPS searching via placenames didnt work either. mkgmap settings: family-id: 30010 product-id: 1 draw-priority: 20 description: ofm_test country-name: BNL country-abbr: #style-file: Styles (custom on or off didnt seem to have any effect) generate-sea: extend-sea-sectors,close-gaps=6000,floodblocker,land-tag=natural=background #createboundsfile: c:\downloads\europe-boundaries.osm boundsdirectory: c:\downloads\bounds latin1 code-page: 1252 location-autofill: 0 show-profiles=1 ignore-maxspeeds remove-short-arcs merge-lines add-pois-to-areas min-size-polygon: 4 preserve-element-order keep-going net route index nsis ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Separate boundary files
On Apr 30, 2011, at 15:13, WanMil wrote: Could you please try to run the BoundaryFile2Gpx converter for this boundary file? Does it throw the same error? I am not sure if I there is a problem with the big- and little-endian order on Mac versus PC? The converter does not produce an error. I get the following output: true false bounds_205_10.bnd Start converting bounds_205_10.bnd 318 boundaries loaded Finished bounds_205_10.bnd (Since I am using an Intel Mac there should not be an endian problem.) I'll see if I can narrow down the problem on my side. Cheers. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
El 30/04/11 15:07, WanMil escribió: I've posted a question to the osmosis mailing list: http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001019.html According to the answer in osmosis mailing list [1] I run osmosis --rb spain.osm.pbf --tf accept-ways boundary=administrative --used-node --tf reject-relations outPipe.0=ways --rb spain.osm.pbf --tf accept-relations boundary=administrative --used-node --used-way outPipe.0=relations --merge inPipe.0=ways inPipe.1=relations --write-xml file=spain-boundaries.osm Although I get a file with all the desired ways and relations it doesn't seem to be the optimal osmosis call, as the output file is only ~2/3 of the uncompressed spain.osm (2.9 GB vs 4.5 GB), so a lot of unnecessary elements seems to be there. Anyway, now at first sight result is great: all streets have complete city, region, country information (and apparently correct) and some of the wrong State/Province places have gone (remaining ones could come from is_in information). Spain bounds are available at http://mapas.alternativaslibres.es/bounds.tar.gz [1] http://lists.openstreetmap.org/pipermail/osmosis-dev/2011-April/001020.html ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [locator] Osmosis parameters required
You could solve this with multiple runs of osmosis (but it might take some time). The idea is that on the first run, you retain just the relations tagged with boundary=administrative, the ways referred to in those relations, and the nodes used in those ways and relations. This gives you everything you want except for ways tagged with boundary=administrative, that are not referred to in any relation. So you do a second run on the Europe extract and retain ways tagged boundary=administrative. On the third run, you merge the two sets of results. This is similar to the advice from the osmosis mailing list, but I have filled in some details. Suggested commands (I haven't tried this): First create named pipes rel and way using mkfifo. You also need to create a file dummy.osm.pbf which contains the bounding box you want, to work round the bounding box problem in the merge function. See http://www.mkgmap.org.uk/pipermail/mkgmap-dev/2011q2/011229.html osmosis --rb europe.osm.pbf \ --tf accept-relations boundary=administrative \ --used-way idTrackerType=BitSet \ --used-node outPipe.0=rel idTrackerType=BitSet \ \ --rb europe.osm.pbf \ --tf accept-ways boundary=administrative \ --tf reject-relations \ --used-node outPipe.0=way idTrackerType=BitSet \ \ --rb dummy.osm.pbf \ --merge inPipe.1=rel \ --merge inPipe.1=way \ --wb boundary.osm.pbf An alternative that does not use named pipes (may be quicker because it only reads the input file once): osmosis --rb europe.osm.pbf \ --tee \ --tf accept-relations boundary=administrative \ --used-way idTrackerType=BitSet \ --used-node idTrackerType=BitSet \ --wb rel.osm.pbf \ --tf accept-ways boundary=administrative \ --tf reject-relations \ --used-node idTrackerType=BitSet \ --wb way.osm.pbf osmosis --rb dummy.osm.pbf \ --rb rel.osm.pbf \ --rb way.osm.pbf \ --merge \ --merge \ --wb boundary.osm.pbf ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
