Re: [mkgmap-dev] Splitter PBF support patch
On Thu, May 19, 2011 at 01:12:06PM -0400, Francisco Moraes wrote: >After much delay, here's the splitter patch. > >Enjoy. I would love to, but: >+ currentWriters[j] = pbfOutput ? new >BinaryMapWriter(area, fileOutputDir) : new OSMXMLWriter(area, >fileOutputDir); It appears that you forgot to include BinaryMapWriter.java in the patch. Please execute "svn status" in your working directory, and execute "svn add" for every file that is listed with a ? next to it. After that, "svn diff" should include all new files. (You would have to remember the "svn add" also before "svn commit".) Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Thu, May 19, 2011 at 03:37:19PM -0400, Francisco Moraes wrote: >Ok, take 2 on the splitter PBF patch. All new files should be present. I >also fixed the EOL mode, so the diffs should be better. That would explain why the new diff is shorter. I guess we should always "svn propset svn:eol-style native" on all non-binary files. >To enable the PBF output, I added the option --pbf. If omitted, it >should output the normal .osm.gz format. Doh, I missed this. I had the broken *.osm.pbf files lying around from a previous broken run, and was just about to complain that the patch fails to close the polygons, as your initial splitter.jar release did. I will try this one more time on today's finland.osm.pbf, but I will report the results tomorrow. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Thu, May 19, 2011 at 11:46:57PM +0300, Marko Mäkelä wrote: >Doh, I missed this. I had the broken *.osm.pbf files lying around from >a previous broken run, and was just about to complain that the patch >fails to close the polygons, as your initial splitter.jar release did. >I will try this one more time on today's finland.osm.pbf The split time was reduced from 140ish (IIRC) to 102 seconds. There seems to be something wrong with the pbf output, because mkgmap is outputting this kind of messages: 2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way http://www.openstreetmap.org/browse/way/107168753 references undefined node 1231941341 2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way http://www.openstreetmap.org/browse/way/107168753 references undefined node 1232127462 2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way http://www.openstreetmap.org/browse/way/107168753 references undefined node 1232017145 Could it be that when you delete nodes outside the output tile, you forget to delete the node references from the way? I have not seen these messages before. Or is the osm.gz output of the splitter equally broken, but the mkgmap XML parser is not complaining about ways referencing deleted nodes? Apart from these warnings, I am still getting the warnings about unclosed polygons, like this: 2011/05/19 23:54:39 WARNING (StyledConverter): 63240008.osm.pbf: Unclosed way http://www.openstreetmap.org/browse/way/23679990 [natural=water] should be converted as shape but the start or end point lies inside the bbox. Skip it. For some reason, the pbf splitter workflow failed to issue the data error (one dead-end oneway) that I got reported for the same finland.osm.pbf when splitting it to osm.gz. Can you fix these issues? Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On 5/19/2011 5:09 PM, mkgmap-dev-requ...@lists.mkgmap.org.uk wrote: > 2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way > http://www.openstreetmap.org/browse/way/107168753 references undefined > node 1231941341 > 2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way > http://www.openstreetmap.org/browse/way/107168753 references undefined > node 1232127462 > 2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way > http://www.openstreetmap.org/browse/way/107168753 references undefined > node 1232017145 > > Could it be that when you delete nodes outside the output tile, you > forget to delete the node references from the way? I have not seen > these messages before. Or is the osm.gz output of the splitter equally > broken, but the mkgmap XML parser is not complaining about ways > referencing deleted nodes? Can you check if the regular OSM XML path still works without these messages? > Apart from these warnings, I am still getting the warnings about > unclosed polygons, like this: > > 2011/05/19 23:54:39 WARNING (StyledConverter): 63240008.osm.pbf: > Unclosed way http://www.openstreetmap.org/browse/way/23679990 > [natural=water] should be converted as shape but the start or end > point lies inside the bbox. Skip it. > > For some reason, the pbf splitter workflow failed to issue the data > error (one dead-end oneway) that I got reported for the same > finland.osm.pbf when splitting it to osm.gz. > > Can you fix these issues? Can you tell me exactly what file you used for input and the settings you used? I can give it a try but I would like first to know if my patch broke anything on the old XML path. I could try to make the selection of XML or PBF output based on the input instead of the current default of XML. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hi Francisco,
On Thu, May 19, 2011 at 10:24:08PM -0400, Francisco Moraes wrote:
Can you check if the regular OSM XML path still works without these
messages?
I am checking that now, but with today's dump instead of yesterday's.
The splitting time is 199s for osm.gz vs 102s for osm.pbf.
The XML path seems to be unaffected by your patch. My filter only passed
two errors, which I will soon correct (so that they should not be in
tomorrow's finland.osm.pbf).
2011/05/20 08:27:10 WARNING (SeaGenerator): 63240001.osm.gz: Converting
anti-island starting at
http://www.openstreetmap.org/?mlat=59.96130&mlon=19.95804&zoom=17 into
an island as it is surrounded by water
2011/05/20 08:31:39 WARNING (RouteNode): 63240005.osm.gz: Oneway road
(http://www.openstreetmap.org/browse/way/36917351) goes nowhere at
http://www.openstreetmap.org/?mlat=62.27274&mlon=25.2&zoom=17
Can you tell me exactly what file you used for input and the settings
you used?
You can get my scripts at http://www.polkupyoraily.net/osm/. A patch
against my osm2img.sh is attached.
I can give it a try but I would like first to know if my patch
broke anything on the old XML path.
Could it make sense to use Osmosis for testing? Convert the split tiles
with Osmosis to a "canonical format", whatever that might be, and ensure
that the XML and Protobuf paths yield the same results. Steve should
have good ideas from the times when he implemented the osm.pbf parser in
mkgmap. One option would be to process the tiles with mkgmap and then
compare the mkgmap log output (after stripping timestamps and sorting
the messages, so that the different ordering of the messages does not
matter).
I could try to make the selection of XML or PBF output based on the
input instead of the current default of XML.
That is a good idea.
Best regards,
Marko
--- osm2imgpbf.sh 2011-05-20 08:29:56.0 +0300
+++ osm2img.sh 2011-04-20 11:39:18.0 +0300
@@ -31,7 +31,7 @@
wget -N http://download.geofabrik.de/osm/europe/"$OSM";
if [ -f "$ZIP" -a ! "$OSM" -nt "$ZIP" ]; then exit; fi
-$JAVACMD $JAVACMD_OPTIONS -jar splitter.jar --pbf --split-file=areas.list "$OSM"
+$JAVACMD $JAVACMD_OPTIONS -jar splitter.jar --split-file=areas.list "$OSM"
rm -f 6324*.img
mkgmap() {
@@ -44,23 +44,23 @@
mkgmap_tiles() {
mkgmap "$@" \
--description='L\304ntinen Suomenlahti' \
- --mapname=6324${L}001 --input-file=63240001.osm.pbf \
+ --mapname=6324${L}001 --input-file=63240001.osm.gz \
--description='Helsinki' \
- --mapname=6324${L}002 --input-file=63240002.osm.pbf \
+ --mapname=6324${L}002 --input-file=63240002.osm.gz \
--description='It\304inen Suomenlahti' \
- --mapname=6324${L}003 --input-file=63240003.osm.pbf \
+ --mapname=6324${L}003 --input-file=63240003.osm.gz \
--description='Lounainen Suomi' \
- --mapname=6324${L}004 --input-file=63240004.osm.pbf \
+ --mapname=6324${L}004 --input-file=63240004.osm.gz \
--description='Etel\304inen Suomi' \
- --mapname=6324${L}005 --input-file=63240005.osm.pbf \
+ --mapname=6324${L}005 --input-file=63240005.osm.gz \
--description='Kaakkois-Suomi' \
- --mapname=6324${L}006 --input-file=63240006.osm.pbf \
+ --mapname=6324${L}006 --input-file=63240006.osm.gz \
--description='Keskinen Suomi' \
- --mapname=6324${L}007 --input-file=63240007.osm.pbf \
+ --mapname=6324${L}007 --input-file=63240007.osm.gz \
--description='Oulu' \
- --mapname=6324${L}008 --input-file=63240008.osm.pbf \
+ --mapname=6324${L}008 --input-file=63240008.osm.gz \
--description='Lappi' $GL \
- --mapname=6324${L}009 --input-file=63240009.osm.pbf
+ --mapname=6324${L}009 --input-file=63240009.osm.gz
}
L1="--family-id=1 --family-name=$(date -r"$OSM" +"OSM-%x")"
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Am 20.05.2011 07:37, schrieb Marko Mäkelä: > >> I could try to make the selection of XML or PBF output based on the >> input instead of the current default of XML. > > That is a good idea. I think this is a bad idea, because it will confuse the users. I would name the parameter --output=pbf or --output=xml. If no parameter --output is specified, splitter should write xml. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On 5/20/2011 3:59 AM, mkgmap-dev-requ...@lists.mkgmap.org.uk wrote: > I am checking that now, but with today's dump instead of yesterday's. > The splitting time is 199s for osm.gz vs 102s for osm.pbf. > > The XML path seems to be unaffected by your patch. My filter only > passed two errors, which I will soon correct (so that they should not > be in tomorrow's finland.osm.pbf). I compared the splitter output from both the XML and PBF with your script. I took one of the missing node messages (node 1217412566 referenced from way 105721933 and compared the XML and PBF splitter outputs. As far as I can tell, the node is not present on the tile 63240006 in either the XML or PBF output. It is present on tile 63240007 though and that matches the areas.list you are using. Is this a case of the PBF parser in mkgmap being more chatty than the XML parser? I also noticed that the splitter outputs XML at version 0.5. That probably should be fixed. Finally, I don't know if this is an issue, but my PBF patch doesn't output any metadata as I didn't see that in the data structure used in the splitter. Let me know if I am off base, but that's seems to be what I found so far. I won't be able to look at it again until next week now. Francisco ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Am 20.05.2011 16:45, schrieb Francisco Moraes: > I also noticed that the splitter outputs XML at version 0.5. That > probably should be fixed. Finally, I don't know if this is an issue, but > my PBF patch doesn't output any metadata as I didn't see that in the > data structure used in the splitter. Metadata is not needed at all, because mkgmap doesn't need them. Thats why xml is used in version 0.5. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hello, I've downloaded the splitter from trunk, but how can I splitt the files into pbf-format. --pbf or output=pbf didn't work. And does the locator-branch already can handle pbf-files? Cheers Martin Am 20.05.2011 um 18:42 schrieb Henning Scholland: > Am 20.05.2011 16:45, schrieb Francisco Moraes: >> I also noticed that the splitter outputs XML at version 0.5. That >> probably should be fixed. Finally, I don't know if this is an issue, but >> my PBF patch doesn't output any metadata as I didn't see that in the >> data structure used in the splitter. > Metadata is not needed at all, because mkgmap doesn't need them. Thats > why xml is used in version 0.5. > > Henning > > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On 5/22/2011 4:35 PM, mkgmap-dev-requ...@lists.mkgmap.org.uk wrote: > I've downloaded the splitter from trunk, but how can I splitt the files into > pbf-format. --pbf or output=pbf didn't work. > And does the locator-branch already can handle pbf-files? The code has not been committed to SVN yet. You need to apply the patch on top of the trunk code. The patch currently uses --pbf option to enable PBF output. I will change it to --output=pbf as suggested on the next patch once I hear confirmation about the unreferenced node issues that have been reported as I think that also happens on the XML output path. Francisco ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
how is the syntax in mkgmap for pbf-split prozessing ? ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Mon, May 23, 2011 at 09:46:49AM +0200, fla...@googlemail.com wrote: >how is the syntax in mkgmap for pbf-split prozessing ? Mkgmap eats pbf files just fine. It detects the input file format automatically. Just replace the .osm or .osm.gz file name suffix with .osm.pbf. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hi > Ok, take 2 on the splitter PBF patch. All new files should be present. I > also fixed the EOL mode, so the diffs should be better. I don't have > commit access and not looking for it at this point. I have committed this on a new pbf-write branch. I've left out the removal of the @Override annotations however. I'm not against removing them (we don't use them in mkgmap) but we should do it as a separate checkin if that is what we decide to do. I'm also in favour of making the default output type PBF, or at the very least having the output type be the same as the input type. Thanks ..Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
> > Hi > >> Ok, take 2 on the splitter PBF patch. All new files should be present. I >> also fixed the EOL mode, so the diffs should be better. I don't have >> commit access and not looking for it at this point. > > I have committed this on a new pbf-write branch. I've left out the > removal of the @Override annotations however. > > I'm not against removing them (we don't use them in mkgmap) but we > should do it as a separate checkin if that is what we decide to do. > > I'm also in favour of making the default output type PBF, or at the very > least having the output type be the same as the input type. +1 for pbf as default output type WanMil > > Thanks > > ..Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Am 23.05.2011 um 18:32 schrieb WanMil: >> >> Hi >> >>> Ok, take 2 on the splitter PBF patch. All new files should be present. I >>> also fixed the EOL mode, so the diffs should be better. I don't have >>> commit access and not looking for it at this point. >> >> I have committed this on a new pbf-write branch. I've left out the >> removal of the @Override annotations however. >> >> I'm not against removing them (we don't use them in mkgmap) but we >> should do it as a separate checkin if that is what we decide to do. >> >> I'm also in favour of making the default output type PBF, or at the very >> least having the output type be the same as the input type. > > +1 for pbf as default output type > > WanMil Also +1 for pbf as default output type Short question: Can the locatorbranch handle pbf-files? Martin > >> >> Thanks >> >> ..Steve > > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On 2:59 PM, Steve Ratcliffe wrote: > I'm not against removing them (we don't use them in mkgmap) but we > should do it as a separate checkin if that is what we decide to do. Well, the only reason I removed them is that Eclipse will not compile without. As per Javadoc, it is an error to place @Override on a method that's not overriding a base method. A question on general splitter operation: is it supposed to remove node references from ways if the node is outside the area bounding box? ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hi > Well, the only reason I removed them is that Eclipse will not compile > without. As per Javadoc, it is an error to place @Override on a method > that's not overriding a base method. That is a difference between java 1.5 and 1.6. In 1.5 it was invalid to place @Override on method which is implementing a method of an interface, whereas in 1.6 it is allowed. ..Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Fri, May 20, 2011 at 12:00:58AM +0300, Marko Mäkelä wrote: >There seems to be something wrong with the pbf output, because mkgmap >is outputting this kind of messages: > >2011/05/19 23:53:11 WARNING (OsmBinHandler): 63240005.osm.pbf: Way >http://www.openstreetmap.org/browse/way/107168753 references undefined >node 1231941341 This turns out to be a chattiness difference in the mkgmap parsers. The XML parser does not seem to complain when ways are referencing undefined nodes. I tested by checking the nodes of way 57746082, the role=outer way of relation 900367, in my tile 63240006.osm.gz. I searched for '2442732[01345789][0-9]' and did not find any matches apart from the nd ref in the way. >Apart from these warnings, I am still getting the warnings about >unclosed polygons, like this: > >2011/05/19 23:54:39 WARNING (StyledConverter): 63240008.osm.pbf: >Unclosed way http://www.openstreetmap.org/browse/way/23679990 >[natural=water] should be converted as shape but the start or end point >lies inside the bbox. Skip it. > >For some reason, the pbf splitter workflow failed to issue the data >error (one dead-end oneway) that I got reported for the same >finland.osm.pbf when splitting it to osm.gz. These two issues are still unexplained and need to be fixed in my opinion. Best regards, Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On 2:59 PM, Marko Mäkelä wrote: >> Apart from these warnings, I am still getting the warnings about >> unclosed polygons, like this: >> >> 2011/05/19 23:54:39 WARNING (StyledConverter): 63240008.osm.pbf: >> Unclosed way http://www.openstreetmap.org/browse/way/23679990 >> [natural=water] should be converted as shape but the start or end >> point lies inside the bbox. Skip it. >> >> For some reason, the pbf splitter workflow failed to issue the data >> error (one dead-end oneway) that I got reported for the same >> finland.osm.pbf when splitting it to osm.gz. > > These two issues are still unexplained and need to be fixed in my > opinion. So, are you saying that the splitter itself should have done something different on that tile with that way? I briefly compared both the XML and PBF->XML output and they seemed to both have the same node refs. Francisco ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Wed, May 25, 2011 at 01:36:07PM -0400, Francisco Moraes wrote: >On 2:59 PM, Marko Mäkelä wrote: >>> Apart from these warnings, I am still getting the warnings about >>> unclosed polygons, like this: >>> >>> 2011/05/19 23:54:39 WARNING (StyledConverter): 63240008.osm.pbf: >>> Unclosed way http://www.openstreetmap.org/browse/way/23679990 >>> [natural=water] should be converted as shape but the start or end >>> point lies inside the bbox. Skip it. >>> >>> For some reason, the pbf splitter workflow failed to issue the data >>> error (one dead-end oneway) that I got reported for the same >>> finland.osm.pbf when splitting it to osm.gz. >> >> These two issues are still unexplained and need to be fixed in my >> opinion. >So, are you saying that the splitter itself should have done something >different on that tile with that way? I briefly compared both the XML >and PBF->XML output and they seemed to both have the same node refs. Yes, I suspect that the PBF splitter output or the mkgmap PBF parser is broken or different from the XML output or parser in this aspect. I fed the same finland.osm.pbf file to the toolchain. The only difference was the output format of splitter (osm.pbf instead of osm.gz). As far as I understand, it is a little challenging to compare the XML and PBF with each other, because splitter outputs 0.5 XML, which is no longer supported by Osmosis. I may look at the code in some days if needed. Sorry, I get an overdosis of software development at work, and I try to avoid doing too much of it in my spare time. Best regards, Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Tue, May 24, 2011 at 03:36:31PM -0400, Francisco Moraes wrote: >A question on general splitter operation: is it supposed to remove node >references from ways if the node is outside the area bounding box? All I can tell is that Osmosis does so. If it is too hard to remove the references to omitted nodes from the ways, I guess that we could tone down the mkgmap parser, so that it does not complain about this when reading osm.pbf. The XML parser already does not complain when reading. I tested the splitter osm.pbf output again on yesterday's dump. The resulting img files were smaller than the ones produced from the osm.gz output. Thus, I would believe that there is something wrong with the splitter osm.pbf output that causes mkgmap to think that polygons are no longer closed and must be omitted. Best regards, Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hi
I compared the pbf and xml formats as output from a small file which
only produced one output tile.
I converted the pbf to xml with osmosis. Its tricky to compare them, but
by counting elements it seemed that the main differences was that there
were fewer relations in the pbf output. Further more it just seemed to
be the final few that were missing.
Comparing with the osmosis code I think there should be a switchTypes()
call to ensure that the last relations are written out (see attached patch).
After the patch, I can't see any further substantial difference between
the outputs. Compiled img files are the same size although not entirely
identical. I'm not sure if the differences are significant or not yet.
There could still be bugs in the mkgmap pbf reader code. Although it has
been in for a while, its probably not been used much until now.
..Steve
Index: src/uk/me/parabola/splitter/BinaryMapWriter.java
===
--- src/uk/me/parabola/splitter/BinaryMapWriter.java (revision 173)
+++ src/uk/me/parabola/splitter/BinaryMapWriter.java (revision )
@@ -498,8 +498,9 @@
public void finishWrite()
{
try {
+ serializer.switchTypes();
- serializer.processBatch();
- serializer.close();
+ serializer.processBatch();
+ serializer.close();
}
catch(IOException e) {
System.out.println("Could not write end of file: " + e);
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Fri, May 27, 2011 at 11:29:15AM +0100, Steve Ratcliffe wrote: >There could still be bugs in the mkgmap pbf reader code. Although it >has been in for a while, its probably not been used much until now. I routinely use the PBF input format for creating my --style=routes-* layers. If there are bugs in the PBF input, they could be in the parsing of polygons. I guess I could find this out easily: create a style that translates only natural=water polygons, and feed finland.osm.pbf to it. If it complains about areas that the default style does not complain about, then the PBF parser is to blame. Some time ago, I complained that the PBF code path did not produce some warnings about oneways coming from or going to nowhere, while the XML path did. I just realized that the head of the mkgmap.log.* was most likely discarded because of the numerous warnings about unreferenced nodes in ways. I only had configured 4 log files of at most 5MB each. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On Fri, May 27, 2011 at 06:36:32PM +0300, Marko Mäkelä wrote: >If there are bugs in the PBF input, they could be in the parsing of >polygons. I guess I could find this out easily: create a style that >translates only natural=water polygons, and feed finland.osm.pbf to it. >If it complains about areas that the default style does not complain >about, then the PBF parser is to blame. The result is somewhat inconclusive. I copied the default style, discarded all points, lines and relations rules, and kept only the natural=* polygons rules. The map generation was finally interrupted by this: uk.me.parabola.imgfmt.MapFailedException: There is not enough room in a single garmin map for all the input data Not much error log was produced. All messages about unclosed polygons seemed legitimate. There were 104 such messages. 101 of them were on my blacklist (polygons severed by Geofabrik's cutting polygon). It seems to me that the mkgmap PBF parser works flawlessly. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hi Steve,
On Fri, May 27, 2011 at 11:29:15AM +0100, Steve Ratcliffe wrote:
There could still be bugs in the mkgmap pbf reader code. Although it
has been in for a while, its probably not been used much until now.
The attached patch zaps the mkgmap warnings about unreferenced nodes.
With your patch applied on top of the splitter-pbf branch, I got exactly
the same warnings through my filter that I got through the XML toolchain
today. The gmapsupp.img size is identical, and the warnings about
unclosed polygons are gone.
Can you please commit your patch to the splitter-pbf branch? Is it OK to
commit the warning-zapping patch to mkgmap, or would it be feasible to
do something similar to Osmosis --used-node in splitter?
Marko
Index: src/uk/me/parabola/mkgmap/reader/osm/bin/OsmBinHandler.java
===
--- src/uk/me/parabola/mkgmap/reader/osm/bin/OsmBinHandler.java (revision 1955)
+++ src/uk/me/parabola/mkgmap/reader/osm/bin/OsmBinHandler.java (working copy)
@@ -152,7 +152,7 @@ public class OsmBinHandler extends OsmHa
// nodes (way joins) will have highwayCount > 1
co.incHighwayCount();
} else {
- log.warn("Way", way.toBrowseURL(), "references undefined node", nid);
+ log.info("Way", way.toBrowseURL(), "references undefined node", nid);
}
}
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Hi Steve, Thanks for committing the patches. Is there a downloadable .jar of splitter-pbf-write r175 available somewhere for download, or would it be possible to merge the pbf-write branch to trunk now? I have built the map a few times with it, and have not experienced anything unusual. Best regards, Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
On 09/06/11 07:02, Marko Mäkelä wrote: > Hi Steve, > > Thanks for committing the patches. No problem. > Is there a downloadable .jar of splitter-pbf-write r175 available > somewhere for download, or would it be possible to merge the pbf-write > branch to trunk now? I have built the map a few times with it, and have > not experienced anything unusual. I've put one here URL: http://files.mkgmap.org.uk/download/28/splitter-r175.zip Description: Splitter with the pbf write patch (by Francisco Moraes). But I do think it is ready to merge, I am sure that it now gives results identical to XML. The only thing is changing the option name. ..Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Here's the latest patch with the output type option instead of --pbf.
Use --output=pbf to select PBF output or omit it or specify --output=xml
to use the old XML output. Default is still XML.
Index: src/uk/me/parabola/splitter/args/SplitterParams.java
===
--- src/uk/me/parabola/splitter/args/SplitterParams.java(revision 175)
+++ src/uk/me/parabola/splitter/args/SplitterParams.java(working copy)
@@ -70,6 +70,6 @@
@Option(defaultValue = "auto", description = "The maximum number of
threads used by splitter.")
ThreadCount getMaxThreads();
- @Option(defaultValue = "false", description = "Enable PBF output.")
- boolean isPbf();
+ @Option(defaultValue = "xml", description = "The output type, either
xml or pbf.")
+ String getOutput();
}
Index: src/uk/me/parabola/splitter/Main.java
===
--- src/uk/me/parabola/splitter/Main.java (revision 175)
+++ src/uk/me/parabola/splitter/Main.java (working copy)
@@ -226,7 +226,11 @@
geoNamesFile = params.getGeonamesFile();
resolution = params.getResolution();
trim = !params.isNoTrim();
- pbfOutput = params.isPbf();
+ String output = params.getOutput();
+ if(!output.equals("xml") && !output.equals("pbf")) {
+ System.err.println("The --output parameter must be
either xml or pbf. Resetting to xml.");
+ }
+ pbfOutput = "pbf".equals(output);
if (resolution < 1 || resolution > 24) {
System.err.println("The --resolution parameter must be
a value between 1 and 24. Resetting to 13.");
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
> Here's the latest patch with the output type option instead of --pbf. > Use --output=pbf to select PBF output or omit it or specify --output=xml > to use the old XML output. Default is still XML. > Francisco, thanks a lot for your splitter pbf support. That's great! Just my two cents: I would like to have pbf enabled as default because it is the 'better' format regarding speed and disk usage. To avoid problems a hint could be added to the download page. WanMil ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter PBF support patch
Push push. Still not commited... Would be nice if the splitter trunk would get the PBF support implemented. I used: http://files.mkgmap.org.uk/download/28/splitter-r175.zip without any probs for my latest round of map updates (and due to my overly extensive style, I'm usually one of the first to run into bugs). Also +1 for making pbf the default output. On 24.05.2011 12:58, Martin wrote: > Am 23.05.2011 um 18:32 schrieb WanMil: > >>> Hi >>> Ok, take 2 on the splitter PBF patch. All new files should be present. I also fixed the EOL mode, so the diffs should be better. I don't have commit access and not looking for it at this point. >>> I have committed this on a new pbf-write branch. I've left out the >>> removal of the @Override annotations however. >>> >>> I'm not against removing them (we don't use them in mkgmap) but we >>> should do it as a separate checkin if that is what we decide to do. >>> >>> I'm also in favour of making the default output type PBF, or at the very >>> least having the output type be the same as the input type. >> +1 for pbf as default output type >> >> WanMil > Also +1 for pbf as default output type > > Short question: Can the locatorbranch handle pbf-files? > > Martin > >>> Thanks >>> >>> ..Steve >> ___ >> mkgmap-dev mailing list >> mkgmap-dev@lists.mkgmap.org.uk >> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev >> > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
