[mkgmap-dev] Commit: r2019: Implement --ignore-builtin-relations.
Version 2019 was commited by marko on 2011-08-29 08:02:35 +0100 (Mon, 29 Aug 2011) Implement --ignore-builtin-relations. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> geofabrik extracts are to blame - they're too "tight" in some places, so > the coastline breaks in the extraction process already In a sense, this is true. However, *no* polygon can ever be guaranteed to contain enough data. Even if Geofabrik used a bounding box instead of a bounding polygon so that data for the entire region containing the extract was available, coastlines would still break. It is the rounding of tile boundaries in the splitter that pushes them beyond the region of the original extract. The only robust solution is to extract coastlines separately, for a larger region of the map. > I generated poly files for these countries which simply extend roughly > 10 km into the neighbouring countries and the broken coastlines vanish, > even if the splitter splits the coastline in these places. Do you use these poly files to extract coastlines only or do you use them for all your data? In the former case, you are doing what I found to be a good workaround. In the latter case, you are increasing the region for which coastlines are available while also increasing the region for which tiles are built. If the coastlines happen to work with the new regions, that is great - but it is coincidental, not guaranteed :(. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> It still takes a little tweaking to get the coastlines right. I have > manually chosen the tile borders so that the coastline will end outside > the tile border. Only in the Swedish/Finnish border I am using > extend-sea-sectors to make up some coastline in Sweden. This is why I collected the coastlines for all of Europe in a single file. When all European coastlines are available, you no longer have to tweak tile boundaries. > I think that it could be useful to have the tile-splitter support a set > of fixed tiles that it would split itself further as needed. I presume you would then use this so that you extract a larger region and then let the splitter generate tiles for a subset of this region? With a sufficient margin of safety between the two regions, this should work by providing the splitter with coastline data even when tiles grow due to rounding. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> I use my own polygon file to extract germany from the european extract. It should be sufficient to extract coastlines using that larger polygon only. Of course once you are extracting, you can just use your own extracts throughout. But a combination of the Geofabrik extract for map data and your larger poly for coastlines should work equally well. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v1] Check size of polygons to avoid exceeding subdivision size
> 2011/08/29 09:46:03 WARNING (Subdivision): 63240010.osm.pbf: Subdivision > width is 36627 at 3231057/1236133 > > I guess it is time to split my tile further. But, this is not the > biggest tile: The Subdivision class is complaining about the width of an area in Garmin units. The area may not contain much data but it is too large. I am not sure this is directly related to the issue being addressed by this patch. The patch deals with MapAreas, the warning above is about Subdivisions. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
fixed tiles may produce better coastlines, but you'll get "map too big" at some point as the map grows. so you could either set a large safety margin and get a lot more tiles or you'd have to fix the tiles anew more often. Am 29.08.2011 09:08, schrieb Bartosz Fabianowski: It still takes a little tweaking to get the coastlines right. I have manually chosen the tile borders so that the coastline will end outside the tile border. Only in the Swedish/Finnish border I am using extend-sea-sectors to make up some coastline in Sweden. This is why I collected the coastlines for all of Europe in a single file. When all European coastlines are available, you no longer have to tweak tile boundaries. I think that it could be useful to have the tile-splitter support a set of fixed tiles that it would split itself further as needed. I presume you would then use this so that you extract a larger region and then let the splitter generate tiles for a subset of this region? With a sufficient margin of safety between the two regions, this should work by providing the splitter with coastline data even when tiles grow due to rounding. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
[mkgmap-dev] Memory Consumption on Index Creation
1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem. How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore? 2. Also as running out of memory primarily happens on index creation, it would be great if mkgmap writes out the index under a temporary name, and once it has passed correctly, renames the file. Else it is not possible to easily assess in a bash/batch script whether mkgmap completed the map correctly or not (and hence one uploads broken maps if everything automated). Or at least I don't see any way to find out whether it passed correctly or not. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> fixed tiles may produce better coastlines, but you'll get "map too big" > at some point as the map grows. The way I understand Marko's suggestion, you would give the splitter an initial list of tiles (or just a region) and it would then subdivide that into smaller tiles as it does today. The difference would be that you could provide the splitter with a large dataset (say all of Europe) while telling it to cut out tiles for a small region (say the Netherlands) only. This would essentially integrate the osmosis cutting step into the splitter. The advantage of this is that the splitter has data for the tiles it cuts out, including ones that expand when rounding. The disadvantages are a need to push around larger datasets, to need to specify tile boundaries each time the splitter is invoked and a loss of the UNIX principle that each tool handles one small thing and handles it well. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
On Mon, Aug 29, 2011 at 10:08:26AM +0300, Bartosz Fabianowski wrote: >This is why I collected the coastlines for all of Europe in a single >file. When all European coastlines are available, you no longer have to >tweak tile boundaries. There are a few drawbacks to that approach. First, you will have to download and process much more data than just the country extract. Second, if the continent extract was generated at a different time than the country extract and there have been edits to the coastline, your map might not correspond to any snapshot of the OSM database. >>I think that it could be useful to have the tile-splitter support a >>set of fixed tiles that it would split itself further as needed. > >I presume you would then use this so that you extract a larger region >and then let the splitter generate tiles for a subset of this region? Right, the idea is to make the combined bounding box of the splitter-generated tiles something else than a rectangle. If we ensure that all coastlines that were "cut open" in the Geofabrik extract will have their endpoints outside the splitter-generated coastline, the flooding should be gone. >With a sufficient margin of safety between the two regions, this should >work by providing the splitter with coastline data even when tiles grow >due to rounding. Right. I wonder if Geofabrik or some other entity could start providing a little wider coastline extracts that are in sync with the Geofabrik country extracts. Boundary extracts could be nice too, but I suppose that the Geofabrik country extracts already include all the boundary=administrative lines of the country. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter and o5m
Am 28.08.2011 17:15, schrieb WanMil: Splitter is generating more than 255 tiles. But there is (or was?) a limit of 255 tiles per pass. I am not sure if this limit is still valid. The webpage (http://www.mkgmap.org.uk/page/tile-splitter) is a bit out of date, e.g. does not tell about support for pbf files. >>> Hi WanMil >>> >>> Current behaviour is, that max. 255 tiles are generated out of an >>> existing areas.list-file. If there are 256 tiles listed, just the first >>> 128 tiles were generated. Maybe there is a bug in reading >>> aeras.list-file with more than 255 entries? >>> >>> Henning >>> >> Yes, sounds like a bug. >> >> WanMil > Henning, > > I have tried to reproduce the problem but everything was ok. > I used an areas.list with 256 tiles. Splitter uses two passes with each > 128 tiles. After the first 128 tiles there are some messages that some > worker threads have finished and it takes some time between finishing > the first 128 tiles and starting the other 128 tiles. > > Can you please recheck if you really have the problem? > > WanMil Hi WanMil and a big sorry. I never thought about a second run of splitter. I just noticed that splitter just processes half of the tiles. It works fine in two runs and seems to be faster than two separate runs. Do you have a guess why splitter needs a second run? Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
i'm starting to wonder if we're talking about 2 separate issues that lead to similar outcomes. coastlines break because: 1. they get broken in the extraction process because the poly is too tight 2. they get broken in the splitting process a separate coastline file would quite likely mend both issues in the same go by creating the coastline out of external data. only drawback: if the poly for extracting the country is too tight, then there'd be an empty area between the coast and the rest of the map data, so all the beach bars are missing. Am 29.08.2011 09:03, schrieb Bartosz Fabianowski: geofabrik extracts are to blame - they're too "tight" in some places, so the coastline breaks in the extraction process already In a sense, this is true. However, *no* polygon can ever be guaranteed to contain enough data. Even if Geofabrik used a bounding box instead of a bounding polygon so that data for the entire region containing the extract was available, coastlines would still break. It is the rounding of tile boundaries in the splitter that pushes them beyond the region of the original extract. The only robust solution is to extract coastlines separately, for a larger region of the map. I generated poly files for these countries which simply extend roughly 10 km into the neighbouring countries and the broken coastlines vanish, even if the splitter splits the coastline in these places. Do you use these poly files to extract coastlines only or do you use them for all your data? In the former case, you are doing what I found to be a good workaround. In the latter case, you are increasing the region for which coastlines are available while also increasing the region for which tiles are built. If the coastlines happen to work with the new regions, that is great - but it is coincidental, not guaranteed :(. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter and o5m
Am 27.08.2011 13:36, schrieb Henning Scholland: >> Splitter is generating more than 255 tiles. But there is (or was?) a >> limit of 255 tiles per pass. I am not sure if this limit is still >> valid. The webpage (http://www.mkgmap.org.uk/page/tile-splitter) is a >> bit out of date, e.g. does not tell about support for pbf files. > Hi WanMil > > Current behaviour is, that max. 255 tiles are generated out of an > existing areas.list-file. If there are 256 tiles listed, just the first > 128 tiles were generated. Maybe there is a bug in reading > aeras.list-file with more than 255 entries? Hi, I can't confirm. I've had 280 tiles in my areas.list and splitter processed them in 1 pass. Chris ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH] Splitter area rounding seems wrong
Hi > This looks like a bug to me. The attached patch rounds the minimum > longitude/latitude down and the maximum longitude/latitude up, making > sure that the rounded area always fully contains the original area. Not sure that I agree. Wouldn't this mean that tile boundaries would overlap if they were originally touching but not aligned? There is a case for expanding the dimensions of the outside edges of a complete tile set. ..Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter and o5m
Hi Chris, do you have set --max-areas. I found out, that it is set by default to 255, so that's why splitter uses a second run. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
Hi, Am 29.08.2011 09:09, schrieb Bartosz Fabianowski: >> I use my own polygon file to extract germany from the european extract. > > It should be sufficient to extract coastlines using that larger polygon > only. Of course once you are extracting, you can just use your own > extracts throughout. But a combination of the Geofabrik extract for map > data and your larger poly for coastlines should work equally well. That's great. With this procedure I can use the german extract from geofabrik without the missing sea near Emden. Thanks for the hint. Greetings -- PGP Schlüssel: 311D1055 http://keyserver.pgp.com ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
Hi, Am 29.08.2011 01:45, schrieb Bartosz Fabianowski: > Right now, the files were generated once-off. If they are found to be > useful by others, I am happy to automate the process and make updated > coastline files available on a regular basis. Good idea. Josef -- PGP Schlüssel: 311D1055 http://keyserver.pgp.com ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> i'm starting to wonder if we're talking about 2 separate issues Indeed, it seems so :). > 1. they get broken in the extraction process because the poly is too tight You mean the bounding polygon is so tight that it actually clips away the coastlines? If so, the bounding polygon is simply wrong. It is no longer *bounding* but cutting into the country. This would definitely damage coastlines. And yes, you are right - this is not the problem I was referring to. > 2. they get broken in the splitting process This is what I meant. More specifically, the coastlines stay as they are but the map area increases so that there are suddenly regions in the map for which no coastline data is available, breaking the subsequent sea generation. > a separate coastline file would quite likely mend both issues in the > same go by creating the coastline out of external data. only drawback: > if the poly for extracting the country is too tight, then there'd be an > empty area between the coast and the rest of the map data, so all the > beach bars are missing. Issue 1 described by you is one of missing data. A separate coastline file would be able to restore coastline data but would be unable to fill in the gaps, just as you say. I think that the only way to address issue 1 is by fixing the bounding polygon. If issue 1 happens with Geofabrik extracts, I am rather sure Fred would like to hear so he can fix the polygons used. For issue 2, which I found to be quite common throughout Europe, separate coastlines are a correct solution. This is what I suggested they be used for. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH] Splitter area rounding seems wrong
> Wouldn't this mean that tile boundaries would overlap if they were > originally touching but not aligned? Yes, you are right. The splitter code is not sufficiently documented to make it clear what it is trying to achieve. After looking at the code, it seemed to me that the goal was to expand the boundary of *each tile separately* so that it falls onto a coarse grid while still bounding the original tile. Having read your comment, I see now that this was an incorrect interpretation. The goal is to shift the boundaries *between tiles* so that they fall onto a coarse grid without introducing overlap. In this case, my patch is not correct. However, I have three observations/questions: 1. If the goal is to shift the boundaries onto a coarse grid, why round *down* for longitudes and *up* for latitudes? Why not round in the same direction for both? 2. After rounding boundaries, the code may shift some of them to ensure that the tile sizes are even multiples of the grid spacing. Since this is done for each tile separately, I think there is a potential to introduce overlap here. If overlap really is a problem (does it break drawing and/or routing?), the decision which boundaries to shift should be coordinated among the tiles. > There is a case for expanding the dimensions of the outside edges of > a complete tile set. 3. My last point is related to this comment. The current rounding code does not distinguish between internal subdivisions and outside boundaries. Thus, it will shift the area covered by the tiles so that it no longer include the complete region originally requested. To fix this, as you say, the outside boundaries should be expanded onto the coarse grid. Finally, I am wondering how important the rounding is. It seems that mkgmap will happily process arbitrary .osm files, including those that never passed through the splitter and thus do not have rounded boundaries. Would it be possible to leave out the rounding? Or is it needed for inter-tile routing for example? - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> That's great. With this procedure I can use the german extract from > geofabrik without the missing sea near Emden. You will still have to download all of Europe to extract the boundaries for your larger polygon. Or alternatively you can use the extracted boundaries I provided. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> First, you will have to download and process much more data than just > the country extract. This is why I shared my extracted coastlines. The extraction has to be done only once. Everyone else can just download the (relatively small) coastline files. > Second, if the continent extract was generated at a different time than > the country extract and there have been edits to the coastline, your map > might not correspond to any snapshot of the OSM database. Absolutely. What I uploaded yesterday were snapshots based on the most recent Geofabrik Europe extract and the most recent full planet. If these prove useful, I will make the generation a batch job so that the coastlines updated whenever a new Geofabrik Europe extract or full planet are made. > Right, the idea is to make the combined bounding box of the > splitter-generated tiles something else than a rectangle. If we ensure > that all coastlines that were "cut open" in the Geofabrik extract will > have their endpoints outside the splitter-generated coastline, the > flooding should be gone. There is no need to make the combined bounds non-rectangular. The simplest solution actually is a rectangle, one large enough to cover all splitter-generated tiles plus the padding added to these by boundary rounding. > I wonder if Geofabrik or some other entity could start providing a > little wider coastline extracts that are in sync with the Geofabrik > country extracts. This is exactly what I volunteered to do. The wider regions I would extract coastlines for are Geofabrik Europe (daily) and planet (weekly). We have the server capacity and are happy to do it. Our website is not set up for it yet but I can use my private hosting for now. I am trying to gauge interest. If people are interested in such regular extracts, I will be happy to provide them. > Boundary extracts could be nice too, but I suppose that the Geofabrik > country extracts already include all the boundary=administrative > lines of the country. This is an interesting point. The boundary files seem to be updated about once a month right now. If there is interest in such boundary extracts, I could see what I can do on our servers. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
Hi, Am 29.08.2011 17:16, schrieb Bartosz Fabianowski: >> That's great. With this procedure I can use the german extract from >> geofabrik without the missing sea near Emden. > > You will still have to download all of Europe to extract the boundaries > for your larger polygon. Or alternatively you can use the extracted > boundaries I provided. I would prefer the last one. Josef -- PGP Schlüssel: 311D1055 http://keyserver.pgp.com ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
Thanks, that's one person interested in automatically updated coastlines. I will watch the download logs over the next few days to see whether I should start generating coastlines daily. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Further possible removals from builtin-tag-list
>> Not sure about addr:phone though - isn't this deprecated and instead >> phone=* is used to fill the telephone field in the address form? > > I believe that phone=* is being replaced with contact:phone, just like > url is being replaced with contact:website. I guess we should support > both. > > Marko Ok, I will commit phone=* only. If you think that contact:phone should additionally used that you can add that later on to the style. WanMil ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Memory Consumption on Index Creation
> 1. I noticed that (at least for my style-file) 7500MB of RAM available > does not suffice to compile a map of Europe including index! However if > I recreate the index in another go, then it is no problem. > > How comes? The index creation seems to be completly independent of the > maptiles creation. Is mkgmap leaving stuff inside the memory, that is in > reality not needed anymore? I think you have listed your style and your mkgmap parameters on your website. Can you give a link to them? That makes it easier to find the problem. And important: Do you use any patches? WanMil > > 2. Also as running out of memory primarily happens on index creation, it > would be great if mkgmap writes out the index under a temporary name, > and once it has passed correctly, renames the file. Else it is not > possible to easily assess in a bash/batch script whether mkgmap > completed the map correctly or not (and hence one uploads broken maps if > everything automated). Or at least I don't see any way to find out > whether it passed correctly or not. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
On Mon, Aug 29, 2011 at 07:08:44PM +0300, Bartosz Fabianowski wrote: >Absolutely. What I uploaded yesterday were snapshots based on the most >recent Geofabrik Europe extract and the most recent full planet. If >these prove useful, I will make the generation a batch job so that the >coastlines updated whenever a new Geofabrik Europe extract or full >planet are made. This would be great. If you have the disk space, you might also keep N latest coastline extracts, so that when the coastline gets broken somewhere, a work-around of using an older extract exists. >>Boundary extracts could be nice too, but I suppose that the Geofabrik >>country extracts already include all the boundary=administrative lines >>of the country. > >This is an interesting point. The boundary files seem to be updated >about once a month right now. If there is interest in such boundary >extracts, I could see what I can do on our servers. I will probably change my workflow so that I will generate the boundary data directly from the Geofabrik finland.osm.pbf, but I suppose it could be very useful for others. Right now, many boundaries inside Finland are either non-existent or broken. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Routing - Documentation and Best Practice
On 08/28/2011 06:33 PM, Minko wrote: > Sounds interesting, but how will you do that? Can you give an example > which I can download? Sure. I first run mkgmap on the OSM files to generate the IMGs and the gmapsupp.img. The config file looks somewhat like this: gmapsupp [more options ...] family-id=1234 family-name=Layer1 input-file: 1001.osm.gz input-file: 1002.osm.gz family-id=5678 family-name=Layer2 input-file: 1003.osm.gz input-file: 1004.osm.gz [possibly more input files ...] This produces the IMG files and a gmapsupp.img for the GPS unit which contains 2 layers which can be switched on and off in the unit. If you use a custom typ file you'll need two, one for each layer. Mapsource can't do this, so I need a different setup: tdbfile=yes family-id=1234 family-name=AllLayers input-file: 1001.img input-file: 1002.img input-file: 1003.img input-file: 1004.img This will make a TDB file and an overview map for mapsource in which both layers show up. For mapsource you need a typ file with all types in it. An example of such a map is my topo map of Germany, which contains an OSM layer and three SRTM layers: http://www.kleineisel.de/blogs/index.php/osmmap/ ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
>> Boundary extracts could be nice too, but I suppose that the Geofabrik >> country extracts already include all the boundary=administrative >> lines of the country. > > This is an interesting point. The boundary files seem to be updated > about once a month right now. If there is interest in such boundary > extracts, I could see what I can do on our servers. > > - Bartosz Up to now I have created and uploaded the boundary files manually once a month. An "automatic" service for this would be better. But: Each time I have created these extracts I have found several country borders that have been broken. So I repaired them before creating the same stuff again and ensured that the boundary extracts were uploaded only if the country information was complete. Regaring Geofabriks dumps: The europe dump does not contain the complete spanish border because some parts are in africa and they wanted to have continent specific dumps. Therefore I create my boundary extracts from a planet file, split it into 5 areas (america (north & south), africa, europe asia and australia-oceania), compile the boundary extracts and merge them afterwards. If you are interested I can send you the steps more in detail (with .poly files). WanMil ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
I have just uploaded today's coastlines_europe.osm.pbf. The time stamp matches the europe.osm.pbf file that this was created from. Right now, I still need to manually upload and fiddle with the timestamps. I am switching to a better hosting setup in a couple of days that will allow me to fully automate this. > If you have the disk space, you might also keep N latest coastline > extracts, so that when the coastline gets broken somewhere, a > work-around of using an older extract exists. Good idea. I renamed the old extract and kept it around. I have 50GB of disk space, so I can keep a lot of extracts around. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> But: Each time I have created these extracts I have found several > country borders that have been broken. So I repaired them before > creating the same stuff again and ensured that the boundary extracts > were uploaded only if the country information was complete. I am happy to devote server capacity and bandwidth to this but I fear I would not have the time to maintain things manually. So I will stay away from boundaries for now. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
Bartosz, thanks for you detailed analysis!! I have never seen the splitter problem you've described. Maybe the overlap parameter of splitter prevented that because splitter puts all points in the tile that is either contained in the bounding box or contained in a overlap region with width "overlap" garmin units. If overlap is larger than the expansion you don't see this problem. If you go in such details I propose to make use of the GpxCreator class in the mkgmap source code. It has helped me very much to visualize the different steps during processing. You have to add some code to the lines where the GPX files should be created and with which data but that's not a big deal. The --coastlinefile option is a good way to be 100% sure that you get valid coastlines and no flooded tiles. But only if you use "confirmed" coastlines (so a file that is confirmed to be error free). You also pointed to a very big disadvantage: the memory usage is very HIGH, because the coastline file needs to be completey loaded in parallel. This might be fixed by adapting the bounds file algorithm to the coastline processing. A precompiler (that can be run on a system with very much memory) can compile coast tiles. These tiles can be used by mkgmap to fill the complete OSM tiles with this coastline information. The memory requirements and the processing time are low for this step. I don't have time to implement that but if someone likes to do that I can give hints where to start and which part of the boundary precompiler can be reused (with small changes). WanMil > Hi list > > I spent the last 12 hours debugging sea generation problems in Europe. > After digging through a lot of mkgmap and splitter code, I believe I > understand the source of the issues now. > > Since data is processed in tiles, the sea generator will often encounter > coastlines clipped at the tile boundaries. There are heuristics in the > code that should produce valid sea multipolygons for tiles with > incomplete coastlines as well. However, the heuristics fail for the > Geofabrik extracts of several European countries. Here is what I believe > the problem to be: > > The sea generator correctly identifies incomplete coastlines. It then > attempts to determine for each such coastline whether it was clipped by > a tile boundary. It is this test that fails for some countries, leading > to missing or inverted sea. The source of this issue can be traced back > to the splitter. Here, tile boundaries are expanded to coarse multiples > of Garmin map units. Tiles therefore become larger than originally > requested. If map data for the entire expanded area is available, this > is OK. But when dealing with country extracts, there may be no data > available for some of the expanded regions. > > Here is an ASCII art example, a rectangular extract with a coastline > inside, shown as a double line: > >+-+ >| | >| | >|=| >| | >+-+ > > If this data is fed to mkgmap directly, the sea generator will correctly > determine that the coastline was clipped by the left and right tile > boundaries. > > If the data is passed through the splitter first, even if no splitting > into multiple tiles should be needed, the tile boundaries are rounded to > the multiples of Garmin map units mentioned above. The tile may thus grow: > > +---+ > | | > | | > | | > | = | > | | > | | > +---+ > > The problem is that now, the coastline no longer touches the tile > boundaries. The mkgmap sea generator is confused by this and produces > invalid or no sea at all. > > An obvious but incorrect solution is for the sea generator to check > whether the coastline reaches the *original* tile boundary, not the > rounded one. This would require the splitter to pass the original > boundary to mkgmap along with each tile. The real issue with this > solution would be that even if mkgmap correctly detects a clipped > coastline, there is no valid data between the original and rounded > boundaries. To construct a proper sea multipolygon, mkgmap would have to > synthesize a coastline in that space, likely introducing ugly artifacts. > > A correct solution, already implemented in mkgmap today, is to read > coastlines from a separate file. The file should contain coastlines that > extend at least as far as the rounded tile boundaries. From the mail > archives, it appears that WanMil has identified the same issue before > and proposed this very solution [1]. Coastlines for a larger area can be > extracted using: > > osmosis \ > --rb larger_area.osm.pbf \ > --tf accept-ways natural=coastline \ > --tf reject-relations \ > --used-node \ > --wb coastlines.osm.pbf omitmetadata=true > > The coaslines.osm.pbf file can then be fed to mkgmap via the > --coastlinefile option. > > While WanMil described this solution in December 2010 already, it seems
Re: [mkgmap-dev] Coastline issues - analysis and possible solution
> Maybe the overlap parameter of splitter prevented that because splitter > puts all points in the tile that is either contained in the bounding box > or contained in a overlap region with width "overlap" garmin units. If > overlap is larger than the expansion you don't see this problem. I investigated whether the overlap parameter could fix this. The answer, unfortunately, is no. This problem occurs whenever a tile is expanded, due to rounding, beyond the boundary of the extract being used as input data. Even with the overlap parameter set to a large value, there is simply no data in the extract for the offending region. The only way to get this data is to take it from somewhere else, such as a separate coastline file covering a larger area. > If you go in such details I propose to make use of the GpxCreator class > in the mkgmap source code. Thanks for the pointer. I will use it in future investigations. > The --coastlinefile option is a good way to be 100% sure that you get > valid coastlines and no flooded tiles. But only if you use "confirmed" > coastlines (so a file that is confirmed to be error free). I will be making available daily coastline extracts from now on. I will not be verifying that these are error-free. They will simply be coastlines extracted by osmosis. If the coastlines in Geofabrik's europe.osm.pbf are broken, so will be those in coastlines_europe.osm.pbf. My aim is to make the complete European coastlines available as an easy download. Fixing broken coastlines is another issue which I am not trying to address at this time. > You also pointed to a very big disadvantage: the memory usage is very > HIGH, because the coastline file needs to be completey loaded in parallel. > > This might be fixed by adapting the bounds file algorithm to the > coastline processing. I have an alternative idea: As I described, the problem is that coastlines for part of a tile may be missing. To fix this, there is no need to load a complete coastline file possibly covering the entire planet. Only the part of the coastline file overlapping the tile is needed. Memory consumption could be reduced to exactly that of normal tile processing by parsing the input data for a tile first and then loading only the ways and nodes from the coastline file that fall within the tile's boundary. This would require a reload of the coastline data for each tile. Another option would be to preload the coastline data for the combined bounding box of all tiles to be processed. - Bartosz ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
[mkgmap-dev] Commit: r2020: Cleanup builtin-tag-list to reduce number of tags loaded by sparse styles
Version 2020 was commited by wanmil on 2011-08-29 20:46:37 +0100 (Mon, 29 Aug 2011) Cleanup builtin-tag-list to reduce number of tags loaded by sparse styles There are three categories of changes: 1. Address tags are now assigned dynamically by the style file. This enhances flexibility to assign the address fields. addr:housename addr:housenumber addr:street phone is_in 2. Remove tags that are used only if the route option is set. These tags are now added dynamically by the new RoutingHook class. access bicycle carpool delivery emergency except exception foot goods hgv motorcar motorcycle psv restriction route taxi toll 3. Remove unused tags: osm:id (only used internally by mkgmap) addr:phone openGeoDB:postal_codes ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v1] Check size of polygons to avoid exceeding subdivision size
>> 2011/08/29 09:46:03 WARNING (Subdivision): 63240010.osm.pbf: Subdivision >> width is 36627 at 3231057/1236133 >> >> I guess it is time to split my tile further. But, this is not the >> biggest tile: > > The Subdivision class is complaining about the width of an area in > Garmin units. The area may not contain much data but it is too large. I > am not sure this is directly related to the issue being addressed by > this patch. The patch deals with MapAreas, the warning above is about > Subdivisions. > > - Bartosz I'll have to check that. I remember that the patch was not complete so it cannot be commited yet. It was just a first test case if the main idea of the solution was ok. In the end the patch won't fix all problems (this would require a larger rewrite of the subdivision section). But it should avoid some of the known problems. I will try to finish the patch within the next days and then I can also say more to the "Subdivison width" error message. WanMil ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Memory Consumption on Index Creation
On 29.08.2011 20:52, WanMil wrote: 1. I noticed that (at least for my style-file) 7500MB of RAM available does not suffice to compile a map of Europe including index! However if I recreate the index in another go, then it is no problem. How comes? The index creation seems to be completly independent of the maptiles creation. Is mkgmap leaving stuff inside the memory, that is in reality not needed anymore? I think you have listed your style and your mkgmap parameters on your website. Can you give a link to them? That makes it easier to find the problem. Subversion * Path: https://svn.origo.ethz.ch/openmtbmap/ * Web SVN: http://svn.origo.ethz.ch/wsvn/openmtbmap/ * ViewVC: http://svn.origo.ethz.ch/viewvc/openmtbmap/ * Nightly SVN Dump: http://svn.origo.ethz.ch/dump/openmtbmap/latest.tar.gz And here is the specific options on full creation (crashes low on java heap space, passes if Xmx11000M is given, but I only have 12GB RAM on a testmachine on my build sever there is only about 7800M before it seriously slows down -- this is running against Europe extract from geofabrik): c:\OpenMTBMap\maps>start /low /b /wait java -ea -jar -Xmx7500M c:\openmtbmap\mkgmap.jar --max-jobs=4 --generate-sea=extend-sea-sectors,close-gaps=6000,floodblocker,fbgap=60,fbthres=200,fbratio=0.6 --latin1 "--style-file=c:\openmtbmap\new4" --reduce-point-density=4 --nsis --index --transparent --adjust-turn-headings --add-pois-to-areas --ignore-maxspeeds --x-reduce-point-density-polygon=8 --link-pois-to-ways --ignore-turn-restrictions --min-size-polygon=15 --remove-short-arcs=4 --description=openmtbmap_eu --merge-l ines --location-autofill=1 --route --country-abbr=eu --country-name=europe --mapname=6550 --family-id=6550 --product-id=1 --series-name=openmtbmap_europe_29.08.2011 --family-name=mtbmap_eu_29.08.2011 --tdbfile --overview-mapname=mapset --keep-going --area-name="europe_29.08.2011_openmtbmap.org" -c c:\openmtbmap\maps\template.europe 1>NUL versus the options for index only creation (works well, I can even reduce Xmx to about 4500M): c:\OpenMTBMap\maps>start /low /b /wait java -ea -jar -Xmx7500M c:\openmtbmap\mkgmap.jar --nsis --index --transparent --description=openmtbmap_eu --location-autofill=1 --country-abbr=eu --country-name=europe --mapname=6550 --family-id=6550 --product-id=1 --series-name=openmtbmap_europe_29.08.2011 --family-name=mtbmap_eu_29.08.2011 --tdbfile --overview-mapname=mapset --keep-going --area-name="europe_29.08.2011_openmtbmap.org" 6550*.img And important: Do you use any patches? yes listed here: http://svn.origo.ethz.ch/wsvn/openmtbmap/mkgmap_patches.patch but they shouldn't affect the index creation. Maybe it's again an issue of assigning names like ft to unnamed footways. However there must still be some memory leak if creating the index standalone it needs only about 50% of memory. Actually it would be really great if the index creation could be lighter on memory, maybe using a temporary file for buffer? WanMil 2. Also as running out of memory primarily happens on index creation, it would be great if mkgmap writes out the index under a temporary name, and once it has passed correctly, renames the file. Else it is not possible to easily assess in a bash/batch script whether mkgmap completed the map correctly or not (and hence one uploads broken maps if everything automated). Or at least I don't see any way to find out whether it passed correctly or not. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
[mkgmap-dev] !!!IMPORTANT!!! To all style implementors
Release 2020 makes it necessary to add some rules to your style if you want to use the address search. From now on the address fields are composed by the mgkmap tags: mkgmap:country mkgmap:region mkgmap:city mkgmap:postal_code (NEW) mkgmap:street (NEW) mkgmap:housenumber (NEW) mkgmap:phone Please have a look to the default style how to use them. If you use the --location-autofill=is_in please also add a rule for mkgmap:is_in. Before complaining about this change first think about your new flexibility to define which tags should be used to fill the given address fields. Now it's up to you to define them. They are no longer coded statically. WanMil > > Version 2020 was commited by wanmil on 2011-08-29 20:46:37 +0100 (Mon, 29 Aug > 2011) > > Cleanup builtin-tag-list to reduce number of tags loaded by sparse styles > > There are three categories of changes: > 1. Address tags are now assigned dynamically by the style file. This enhances > flexibility to assign the address fields. > addr:housename > addr:housenumber > addr:street > phone > is_in > > 2. Remove tags that are used only if the route option is set. These tags are > now added dynamically by the new RoutingHook class. > access > bicycle > carpool > delivery > emergency > except > exception > foot > goods > hgv > motorcar > motorcycle > psv > restriction > route > taxi > toll > > 3. Remove unused tags: > osm:id (only used internally by mkgmap) > addr:phone > openGeoDB:postal_codes > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] !!!IMPORTANT!!! To all style implementors
On 29.08.2011 22:21, WanMil wrote: > Release 2020 makes it necessary to add some rules to your style if you > want to use the address search. > > From now on the address fields are composed by the mgkmap tags: > mkgmap:country > mkgmap:region > mkgmap:city > mkgmap:postal_code > (NEW) mkgmap:street > (NEW) mkgmap:housenumber > (NEW) mkgmap:phone > > Please have a look to the default style how to use them. > > If you use the --location-autofill=is_in please also add a rule for > mkgmap:is_in. what would be a rule to get back the --location-autofill=1 behavior (or better, but no bounds analysis at all, I want ise tags plus whatever tags to be analysed, plus if that fails to search for nearest city/region)? Also for rules on Polygons, I assume they are only used IF the polygon is converted to a POI??? I need no address information for any kind of end result area/polygon (but I do want address information for any POI that is created by the area-pois option). It's good that It's inside the style-file now though (just don't know how to use it best without using any of the bounds file that for me does not improve the address search) (BTW I think dropping location-autofill=0,1,2 was never announced on the list either explicitely ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
