Re: [mkgmap-dev] Still problems with lakes
> Thanks for the feedback. I'll try coding a few changes (e.g. filtering by tags), if > that doesn't work out, I'll code the handling of a external list and we'll see what it helps. > > Gerd Thank you so much! ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Hello Henning, > Sonds like an easy workaround without to much data, which have to > cached. A manual list would be a good thing, because mainly you as a > user know, which polygons are problematic. I like the idea of a list, however it is produced. I wanted to create the list automatically, but it was too big because of false candidates. As long as no one finds an effecient and correct algorithm to create the list in splitter, we can at least use it to produce correct output in splitter and to verify that mkgmap produces good maps with it. The last point is quite important: mkgmap contains a lot of routines to handle incomplete or wrong or redundant data. That makes it very difficult to verify wether a change in splitter is producing better maps. > > Maybe mkgmap could throw out a csv-list of each polygon and > multipolygon, which is causing problems. I am not sure if mkgmap really detects all ids that cause problems. I think it can't do that if the problem is a missing relation (in one tile), but maybe mkgmap sees the same relation id again in other tiles and reports it then. > > r, (for mp) > w, (for p) > > So the user could have a view to the polygon and could start after > checking the list a second run. Well, dependent on the input I expect the list can contain thousands of ids, so I doubt that you will want to look at that. > > Of course it won't help, if you don't use a fix areas.list Hmm, as long as the input files dont change, splitter will always produce the same areas.list, so I see no big problem here. It doesn't harm if the list of problematic ids contains a few ids that are no longer problematic, as long as it is complete. The automatic algorithm that I implemented creates a complete list, but that list contains too many false entries (I guess the ratio is something like 1:100 (100 false entries for one really problematic id) > > Also wo could start a wiki-page and collect problematic objects. > > Henning Thanks for the feedback. I'll try coding a few changes (e.g. filtering by tags), if that doesn't work out, I'll code the handling of a external list and we'll see what it helps. Gerd > Am 21.09.2012 14:59, schrieb toc-rox: > > If a list with the IDs of all huge polygons is helpful, such a list could > > perhaps created by > > - user (manually) > > - mkgmap (automatic) > > > > Regards Klaus > > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Sonds like an easy workaround without to much data, which have to cached. A manual list would be a good thing, because mainly you as a user know, which polygons are problematic. Maybe mkgmap could throw out a csv-list of each polygon and multipolygon, which is causing problems. r, (for mp) w, (for p) So the user could have a view to the polygon and could start after checking the list a second run. Of course it won't help, if you don't use a fix areas.list Also wo could start a wiki-page and collect problematic objects. Henning Am 21.09.2012 14:59, schrieb toc-rox: > If a list with the IDs of all huge polygons is helpful, such a list could > perhaps created by > - user (manually) > - mkgmap (automatic) > > Regards Klaus ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
If a list with the IDs of all huge polygons is helpful, such a list could perhaps created by - user (manually) - mkgmap (automatic) Regards Klaus -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5726700.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Hi Henning, yes, I think looking at the tags is needed and the most promising approach. This was one of the reasons why I stopped: Up to now, splitter doesn't contain much logic regarding the tags, most of that is only in mkgmap. The same goes for the handling of multi-polygon-relations. I did not want to double all the program code, and I did not want to invent completely new code. Splitter and mkgmap use totally different data structures, so it is not easy to share code. Regarding the memory needs: afaik splitter r200 is able to split europe on a 32 bit system (with a max. heap of about 1.5 GB) with three or four read passes (plus one for the calc. of the areas.list) . Planet will require about 10 or more passes. On 64 bit system with eg 15 GB available heap I guess that even planet can be done in two or three passes. I think these are already gigantic requirements, so I hate the idea of adding anything that could double these needs, not talking about a factor of 10. Ciao, Gerd > Date: Thu, 20 Sep 2012 10:45:34 +0200 > From: o...@aighes.de > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] Still problems with lakes > > Hi > do you thought about filtering by a given tag-list? Mainly there is > water- or forest-polygons causing visual problems. Buildings look mostly > corrupt because of lower precision of Garmin-coords. > > About how much memory we are talking about for splitting a planet or > Europe or Germany? It would be great, if user could decide, if splitter > should filter polygons or not. Eg. if I would have a machine with 16gb > RAM it doesn't harm if splitter need more RAM to generate better maps. > Of course also it should be possible to split the planet with smaller > machines. > > Henning > > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Hi do you thought about filtering by a given tag-list? Mainly there is water- or forest-polygons causing visual problems. Buildings look mostly corrupt because of lower precision of Garmin-coords. About how much memory we are talking about for splitting a planet or Europe or Germany? It would be great, if user could decide, if splitter should filter polygons or not. Eg. if I would have a machine with 16gb RAM it doesn't harm if splitter need more RAM to generate better maps. Of course also it should be possible to split the planet with smaller machines. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Hi, I did never think about calculating the size of the area. My understanding of the problem is that 1) mkgmap sometimes fails to handle incomplete (closed) polygons 2) splitter sometimes causes incomplete polygons, esp. for large polygons that cross tile borders My approach to solve the problem: 1) detect polygons that cross tile borders (I call them multi-tile-polygons mtp). This is not so difficult and is already implemented in my version: you just have to note polygons that have points in more than one tile. 2) detect those tiles that are (or maybe) crossed or enclosed by an mtp, because mkgmap should see the complete data of them, but it doesn't with splitters (r200) implementation. This is were I got stuck. I tried to keep the precise coordinates of all points belonging to all mtp, but that is far too much data, so splitter would no longer work on 32 bit machines, and I wanted to avoid that. So, I started to implement filters for mtp that are not able to cause problems, which reduced the amount of data to maybe 50%, but still my algorithm is keeping too many false candidates and therefor requires too much heap. Just to make this clear: beause of the structure of the OSM data, splitter first sees all points, than all ways, than the relation definitions. Up to now, splitter can keep the information to what tiles a given point was written, but it can't save details about coordinates. It is not possible to calculate the size of an area without having the coordinates (no matter how precise), so this can't help. If you open the kml file produced by splitter (using the --write-kml parm) with e.g. google earth, you see the tiles. Any polygon which is near the border of two tiles is likely to be a mtp, but obviously most of them are houses or small areas, and only a very small amount is interesting for us. The problem is: You can't tell before you know the coordinates, because splitter just knows that a mtp contains points in two or more tiles, it doesn't know anything about the position of these points whithin the tiles. Even this bit of information requires a lot of heap (r200 keeps more or less the same data and is already highly optimized) So, what I have until now is a splitter program that doesn't write incomplete polygon data to the tiles (if the source data is complete), but it still doesn't write mtp data to tiles that do not contain any point of the polygon. If input is in o5m format, this version performs quite well and doesn't require much more heap memory, the values are almost equal if I remember correctly. Ciao, Gerd > Date: Wed, 19 Sep 2012 07:25:10 -0700 > From: easyclassp...@googlemail.com > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] Still problems with lakes > > @GerdP: > > Some ideas concerning the "huge polygon problem": > - huge polygons are rare > - a broken polygon leads to an ugly (amateur like) map > - it's hard to find all broken polygons in a map manually > > Some questions concerning a solution: > - is it possible to implement a "huge polygon" parameter ? > - the parameter acts as threshold and triggers splitter for extra processing > - the parameter could be the area (e.g. 2000 sqkm) or circumference (e.g. > 1000 km) of an "huge" polygon > > If the calculating of all polygons has a large negative performance impact: > - is it possible to identify the huge polygons in an extra (on demand) > splitter run ? > - the result could be a list with all identified huge polygons (generated > only once) > - this list is than the input for the essential splitter run > > What do you think ? > > Regards Klaus > > PS: What is the exactly influence of the overlapping parameter concerning > the splitting of (huge) polyons ? > > > > -- > View this message in context: > http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5726209.html > Sent from the Mkgmap Development mailing list archive at Nabble.com. > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
@GerdP: Some ideas concerning the "huge polygon problem": - huge polygons are rare - a broken polygon leads to an ugly (amateur like) map - it's hard to find all broken polygons in a map manually Some questions concerning a solution: - is it possible to implement a "huge polygon" parameter ? - the parameter acts as threshold and triggers splitter for extra processing - the parameter could be the area (e.g. 2000 sqkm) or circumference (e.g. 1000 km) of an "huge" polygon If the calculating of all polygons has a large negative performance impact: - is it possible to identify the huge polygons in an extra (on demand) splitter run ? - the result could be a list with all identified huge polygons (generated only once) - this list is than the input for the essential splitter run What do you think ? Regards Klaus PS: What is the exactly influence of the overlapping parameter concerning the splitting of (huge) polyons ? -- View this message in context: http://gis.19327.n5.nabble.com/Still-problems-with-lakes-tp5725668p5726209.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
> This need to be fixed in the splitter. Sometimes it helps making the overlap > higher (5000 or more). > But this is just a workaround... > > > Regards, Geoff. > Even at an overlap of 12000 Vänern is still corrupted. Tweaking the areas.list is also not very satisfying as there may be more corrupted lakes and forests which I did not find, yet. And for Saimaa it would be a lot of work as there are very many tile borders to be shifted out of the lake. I really hope that GerdP or someone else can provide a new splitter to handle this problem automatically. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Hi, just to let everybody know: I began development on an improved splitter to handle this problem some months ago, but I got stuck after my system crashed when installing an SSD (my fault). After reinstalling around 10 times I was really pissed with computers and than I started planning (and doing) two longer bike rides. I am not sure if I find the energy to start again with splitter, my solution is somewhat half done. It turned out to be quite complex to collect all needed information without requiring huge amounts of memory or writing large temp files when splitting e.g. Europe . As a first work aproach I decided to implement multiple read passes, which is quite slow with pbf format, so I also implemented reading (and writing o5m format), which makes splitter much faster. The major open problem: Many polygons cross tiles, but most of them do not need special handling (think of all the houses near tile borders) You cannot decide that before you know all nodes (not just the ids, but the position) and tags of the polygon. It is impossible to collect all needed infomation for all polygons in heap, so one needs clever filters and effective data structures to filter only the really problematic polygons. I must confess that I did not find these clever filters after weeks of thinking, so maybe I don't see the forest because of the trees, maybe it is really too complex for me. A completely different approach came into my mind today: Instead of implementing a lot of logic to find the polygons, it might already help if we implement a function that allows to specify polygon ids that cause problems if they are incomplete, and splitter would simply make sure that those polygons are completely written to all related tiles? If someone is willing to help or continue, I can create a patch for a new branch, else maybe the winter will bring me back to the computer ;) Ciao, GerdP > Date: Tue, 18 Sep 2012 08:26:08 +0200 > From: ligfiet...@online.nl > To: mkgmap-dev@lists.mkgmap.org.uk > Subject: Re: [mkgmap-dev] Still problems with lakes > > It is not only large lakes that are affected, also large forest or other > multipolygon areas are sometimes broken at the tile borders. > This need to be fixed in the splitter. Sometimes it helps making the overlap > higher (5000 or more). > But this is just a workaround... > > > Would it be difficult to have precompiled water rather than sea? > > > > It would save messing about with area files. > > > > Regards, Geoff. > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
It is not only large lakes that are affected, also large forest or other multipolygon areas are sometimes broken at the tile borders. This need to be fixed in the splitter. Sometimes it helps making the overlap higher (5000 or more). But this is just a workaround... > Would it be difficult to have precompiled water rather than sea? > > It would save messing about with area files. > > Regards, Geoff. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Would it be difficult to have precompiled water rather than sea? It would save messing about with area files. Regards, Geoff. From: aighes Sent: Monday, September 17, 2012 5:08 PM To: Development list for mkgmap Subject: Re: [mkgmap-dev] Still problems with lakes Am 17.09.2012 17:42, schrieb RheinSkipper: The problem still exists using the latest precompiled sea from Wanmil. This is completely independent, because they only create sea, no lakes. Using one of my older areas.list Vänern is OK but Saimaa is not. Letting the splitter create its own areas.list Saimaa is OK but then southern Vänern is dried out. You have to tweak your areas.list manually, so that the hole multipolygon is in one maptile. You could also take a look at my areas.list file (http://www.aighes.de/OSM/data/style.zip) (cc-by). It's in data-directory and starting with 1200. There are only two errors in Saimaa. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Still problems with lakes
Am 17.09.2012 17:42, schrieb RheinSkipper: The problem still exists using the latest precompiled sea from Wanmil. This is completely independent, because they only create sea, no lakes. Using one of my older areas.list Vänern is OK but Saimaa is not. Letting the splitter create its own areas.list Saimaa is OK but then southern Vänern is dried out. You have to tweak your areas.list manually, so that the hole multipolygon is in one maptile. You could also take a look at my areas.list file (http://www.aighes.de/OSM/data/style.zip) (cc-by). It's in data-directory and starting with 1200. There are only two errors in Saimaa. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
