Re: [OSM-talk] 'wget'ing largish portion of planetOSM
On Wed, 19 Oct 2011 22:58:41 -0700 Paul Norman wrote: > > From: Mick [mailto:bare...@tpg.com.au] > > Subject: Re: [OSM-talk] 'wget'ing largish portion of planetOSM > > > > On Wed, 19 Oct 2011 22:58:28 -0400 > > Richard Weait wrote: > > > > > On Wed, Oct 19, 2011 at 10:51 PM, Mick wrote: > > > > I have been struggling to get a largish chunk of open street map > > > > covering an area from the Isles of Scilly in the south west to > > > > Bristol [ ... ] > > > > > > The Planet page points to a number of planet extracts. Will one > > > of the UK-like extracts work for you? > > > > > Sadly, no, the area is split between southwest, south, swales & > > oxford extracts with 1 to 5 mile gaps between. > > > > I have tried to fill the gaps using the export to xml feature of the > > online map but that requires a (for me at least) mind bending array > > of inconsistently sized bits. > > Would the united_kingdom extract at > http://downloads.cloudmade.com/europe/northern_europe/united_kingdom > work? > > If not, you could try the jxapi. It is more reliable than the old > xapi, although the osm.org hosted one is currently down for > maintenance. Mapquest runs one, the details are at > http://wiki.openstreetmap.org/wiki/Xapi#Java > I've downloaded all of UK from nick.dev.openstreetmap.org and extracted the bit I want, its a bit old (28 Oct 2009) but mayhap I can update it without massive downloads. I've got most of the area current to last week. Now for a couple of weeks 'normalizing' the tags and load it into postgis. Thanks to all who replied mick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] 'wget'ing largish portion of planetOSM
> From: Mick [mailto:bare...@tpg.com.au] > Subject: Re: [OSM-talk] 'wget'ing largish portion of planetOSM > > On Wed, 19 Oct 2011 22:58:28 -0400 > Richard Weait wrote: > > > On Wed, Oct 19, 2011 at 10:51 PM, Mick wrote: > > > I have been struggling to get a largish chunk of open street map > > > covering an area from the Isles of Scilly in the south west to > > > Bristol [ ... ] > > > > The Planet page points to a number of planet extracts. Will one of > > the UK-like extracts work for you? > > > Sadly, no, the area is split between southwest, south, swales & oxford > extracts with 1 to 5 mile gaps between. > > I have tried to fill the gaps using the export to xml feature of the > online map but that requires a (for me at least) mind bending array of > inconsistently sized bits. Would the united_kingdom extract at http://downloads.cloudmade.com/europe/northern_europe/united_kingdom work? If not, you could try the jxapi. It is more reliable than the old xapi, although the osm.org hosted one is currently down for maintenance. Mapquest runs one, the details are at http://wiki.openstreetmap.org/wiki/Xapi#Java ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] 'wget'ing largish portion of planetOSM
On Wed, 19 Oct 2011 22:58:28 -0400 Richard Weait wrote: > On Wed, Oct 19, 2011 at 10:51 PM, Mick wrote: > > I have been struggling to get a largish chunk of open street map > > covering an area from the Isles of Scilly in the south west to > > Bristol [ ... ] > > The Planet page points to a number of planet extracts. Will one of > the UK-like extracts work for you? > Sadly, no, the area is split between southwest, south, swales & oxford extracts with 1 to 5 mile gaps between. I have tried to fill the gaps using the export to xml feature of the online map but that requires a (for me at least) mind bending array of inconsistently sized bits. mick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] 'wget'ing largish portion of planetOSM
On 19 Oct 2011, at 19:58, Richard Weait wrote: > On Wed, Oct 19, 2011 at 10:51 PM, Mick wrote: >> I have been struggling to get a largish chunk of open street map >> covering an area from the Isles of Scilly in the south west to Bristol [ ... >> ] > > The Planet page points to a number of planet extracts. Will one of > the UK-like extracts work for you? http://wiki.openstreetmap.org/wiki/Extract If the UK is too large use Osmosis or another similar tool to extract a smaller bbox http://wiki.openstreetmap.org/wiki/Osmosis Shaun > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] 'wget'ing largish portion of planetOSM
If you are needing that much data I would strongly recommend starting off with an extract of the whole country and then trim out what you need using osmosis. Extract providers can be found here: http://wiki.openstreetmap.org/wiki/Planet#Country_and_area_extracts And osmosis docs on trimming an extract are here: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#Area_Filtering_Tasks Toby On Wed, Oct 19, 2011 at 9:51 PM, Mick wrote: > I have been struggling to get a largish chunk of open street map > covering an area from the Isles of Scilly in the south west to Bristol > in the north east using commands like: > wget -t 0 > http://xapi.openstreetmap.org/api/xapi?map[bbox=-6.41,49.85,-1.68,51.56] > -O souwest.osm > > but all I can get is an error - 'connection reset by peer' or 'timeout' > > I also tried: > wget -t 0 > http://www.overpass-api.de/api/xapi?relation[bbox=-6,49.8,-1.0,51.6][natural=*] > -O souwest-relation-natural.osm > > but this required dozens of individual transaction to get results > osmosis refused to read. > > am I biting off more than the server can chew or using the wrong > procedure? > > could some kind soul point me in the right direction > > mick > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] 'wget'ing largish portion of planetOSM
On Wed, Oct 19, 2011 at 10:51 PM, Mick wrote: > I have been struggling to get a largish chunk of open street map > covering an area from the Isles of Scilly in the south west to Bristol [ ... ] The Planet page points to a number of planet extracts. Will one of the UK-like extracts work for you? ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] 'wget'ing largish portion of planetOSM
I have been struggling to get a largish chunk of open street map covering an area from the Isles of Scilly in the south west to Bristol in the north east using commands like: wget -t 0 http://xapi.openstreetmap.org/api/xapi?map[bbox=-6.41,49.85,-1.68,51.56] -O souwest.osm but all I can get is an error - 'connection reset by peer' or 'timeout' I also tried: wget -t 0 http://www.overpass-api.de/api/xapi?relation[bbox=-6,49.8,-1.0,51.6][natural=*] -O souwest-relation-natural.osm but this required dozens of individual transaction to get results osmosis refused to read. am I biting off more than the server can chew or using the wrong procedure? could some kind soul point me in the right direction mick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Installing your own tileserver on Ubuntu
Hi, Thanks for that, loads of good info in there! >Playing with indexes, especially partial indexes on the ways columns, >reflecting the where condition of the most expensive queries may help. I'm told that the easiest way to start on this is with pgAdmin, which raises a very basic question - what's the username and password I can connect to the gis database with? I presume it's listed somewhere in the setup scripts, but I've not had the chance to go through them. Cheers, Joseph On 19 October 2011 14:57, Kai Krueger wrote: > Hi, > > On 10/19/2011 05:01 AM, Joseph Reeves wrote: >> Hi Kai, >> >>> The default Ram cache size is 800Mb. You can increase it with the -C >>> parameter of osm2pgsql. But given that your netbook probably doesn't >>> have that much ram, I am not sure increasing that option is such a good >>> idea. >> >> -C only has an effect in slim mode, unless I'm wrong? > > That also changed with the commit a few days ago. As again for smaller > extracts, who have a sparse distribution of node IDs, the cache was very > inefficient, I reused the improved node cache from the --slim mode. > Perhaps unfortunately, that also incorporated the limit and effect of > the -C parameter. On 64 bit machines at least, you can simply set the -C > parameter very high as it only reserves virtual memory. Only the amount > actually used will result in physical ram allocation. It is perhaps a > little bit more problematic on 32 bit Operating Systems though, as > virtual memory is also fairly limited. > > Setting a very high -C parameter and --cache-strategy chunked basically > gets you back to the old behavior though. > > As I was hoping >> to append data to my db, I'm running osm2pgsql in full fat mode. >> Thanks for the --cache-strategy tip - I got the import working with >> the sparse option. It seems to be working surprisingly quickly in >> fact. > > Fat mode definitely has its advantage in speed, perhaps especially on > slow disk systems like a netbook. This is perhaps why it was a bit > unfortunate that non-slim mode previously was (and for ways and > relations it still is) very wasteful with ram for extracts. > >> >> Of course, as you said, getting the data into the db is one thing, but >> actually using it is another matter. The netbook is now rendering >> tiles for a large strip across Europe - this often takes a while to >> get tiles created, but I think it should work for what I need it for. >> If the load gets too high I can always empty the db and add extracts >> as I need them. Before that, however, I think I'll try and find any >> database optimisations that might exist. > > Playing with indexes, especially partial indexes on the ways columns, > reflecting the where condition of the most expensive queries may help. > > If you do come up with good optimizations, it may be good to collect > them somewhere to build up a knowledge base for optimization tips. > > Kai > >> >> Thanks again for all this, >> >> Cheers, Joseph >> >> >> >> On 18 October 2011 18:34, Kai Krueger wrote: >>> On 10/18/2011 10:48 AM, Joseph Reeves wrote: > It is possible that one could catch the errors in slim mode and then only > do the expensive diff processing for those node / ways that are > >duplicate in the extracts. Interesting, although I think this is beyond the limits of my OSM skills. >>> >>> Yes, that comment was more directed at my self that I should look into >>> that, or if any other dev of osm2pgsql gets to it first... ;-) >>> >>> Unfortunately Austria seems to be beyond the capabilities of my netbook; an import without --slim gives the error: Node cache size is too small to fit all nodes. Please increase cache size Presumably a slim import would help, but this would then fail because of overlapping ways... I can't Google up anyone suffering that error message before; I guess nobody else is trying to get a number of European countries into a db on their netbook... >>> >>> You will likely not find that error message on google yet, as if I am >>> not mistaken, I only commited that error message two days ago. >>> >>> The default Ram cache size is 800Mb. You can increase it with the -C >>> parameter of osm2pgsql. But given that your netbook probably doesn't >>> have that much ram, I am not sure increasing that option is such a good >>> idea. >>> >>> Given that you can't fit Austria into 800Mb, I suspect you are using the >>> 32bit version of osm2pgsql. It defaults to the old (for extracts >>> inefficient) cache allocation strategy. You can try using the >>> "--cache-strategy" option and set it to either optimized or sparse. That >>> should be more efficient for extracts than the default method. In the >>> optimized option, you might run out of virtual memory on 32 bit though. >>> >>> But you might have troubles with Austria on a netbook anyway, as it >>> might still be too resource intense. >>> >>> Kai >>> Thanks again, Josep
Re: [OSM-talk] Installing your own tileserver on Ubuntu
Hi, On 10/19/2011 05:01 AM, Joseph Reeves wrote: > Hi Kai, > >> The default Ram cache size is 800Mb. You can increase it with the -C >> parameter of osm2pgsql. But given that your netbook probably doesn't >> have that much ram, I am not sure increasing that option is such a good >> idea. > > -C only has an effect in slim mode, unless I'm wrong? That also changed with the commit a few days ago. As again for smaller extracts, who have a sparse distribution of node IDs, the cache was very inefficient, I reused the improved node cache from the --slim mode. Perhaps unfortunately, that also incorporated the limit and effect of the -C parameter. On 64 bit machines at least, you can simply set the -C parameter very high as it only reserves virtual memory. Only the amount actually used will result in physical ram allocation. It is perhaps a little bit more problematic on 32 bit Operating Systems though, as virtual memory is also fairly limited. Setting a very high -C parameter and --cache-strategy chunked basically gets you back to the old behavior though. As I was hoping > to append data to my db, I'm running osm2pgsql in full fat mode. > Thanks for the --cache-strategy tip - I got the import working with > the sparse option. It seems to be working surprisingly quickly in > fact. Fat mode definitely has its advantage in speed, perhaps especially on slow disk systems like a netbook. This is perhaps why it was a bit unfortunate that non-slim mode previously was (and for ways and relations it still is) very wasteful with ram for extracts. > > Of course, as you said, getting the data into the db is one thing, but > actually using it is another matter. The netbook is now rendering > tiles for a large strip across Europe - this often takes a while to > get tiles created, but I think it should work for what I need it for. > If the load gets too high I can always empty the db and add extracts > as I need them. Before that, however, I think I'll try and find any > database optimisations that might exist. Playing with indexes, especially partial indexes on the ways columns, reflecting the where condition of the most expensive queries may help. If you do come up with good optimizations, it may be good to collect them somewhere to build up a knowledge base for optimization tips. Kai > > Thanks again for all this, > > Cheers, Joseph > > > > On 18 October 2011 18:34, Kai Krueger wrote: >> On 10/18/2011 10:48 AM, Joseph Reeves wrote: It is possible that one could catch the errors in slim mode and then only do the expensive diff processing for those node / ways that are >duplicate in the extracts. >>> >>> Interesting, although I think this is beyond the limits of my OSM >>> skills. >> >> Yes, that comment was more directed at my self that I should look into >> that, or if any other dev of osm2pgsql gets to it first... ;-) >> >> Unfortunately Austria seems to be beyond the capabilities of >>> my netbook; an import without --slim gives the error: >>> >>> Node cache size is too small to fit all nodes. Please increase cache size >>> >>> Presumably a slim import would help, but this would then fail because >>> of overlapping ways... I can't Google up anyone suffering that error >>> message before; I guess nobody else is trying to get a number of >>> European countries into a db on their netbook... >> >> You will likely not find that error message on google yet, as if I am >> not mistaken, I only commited that error message two days ago. >> >> The default Ram cache size is 800Mb. You can increase it with the -C >> parameter of osm2pgsql. But given that your netbook probably doesn't >> have that much ram, I am not sure increasing that option is such a good >> idea. >> >> Given that you can't fit Austria into 800Mb, I suspect you are using the >> 32bit version of osm2pgsql. It defaults to the old (for extracts >> inefficient) cache allocation strategy. You can try using the >> "--cache-strategy" option and set it to either optimized or sparse. That >> should be more efficient for extracts than the default method. In the >> optimized option, you might run out of virtual memory on 32 bit though. >> >> But you might have troubles with Austria on a netbook anyway, as it >> might still be too resource intense. >> >> Kai >> >>> >>> Thanks again, >>> >>> Joseph >>> >>> >>> >>> >>> On 18 October 2011 16:59, Kai Krueger wrote: On 10/18/11 9:31 AM, Joseph Reeves wrote: > > Hi Kai, > >> The pre-rendered tiles are stored in /var/lib/mod_tile/default. You can >> simply delete those files and they will automatically get rerendered the >> next time you view them. > > Great, thanks, that's working great. > >> I have seen that you appear to need to restart renderd (sudo >> /etc/init.d/renderd restart) after a new import, as it otherwise appears >> to >> still use old data (It is kind of odd, so I might have the wrong >> impression >> here). >
Re: [OSM-talk] Yahoo Imagery
does the permission to use the Yahoo imagery in OSM include aerial images that are only visible on their website maps.yahoo.com? I noticed that there is an area which I am interested in that is not covered in Bing and in Potlatch/JOSM when choosing Yahoo. But at their website they have a high resolution image. -- View this message in context: http://gis.638310.n2.nabble.com/Yahoo-Imagery-tp6907652p6908151.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Installing your own tileserver on Ubuntu
Hi Kai, >The default Ram cache size is 800Mb. You can increase it with the -C >parameter of osm2pgsql. But given that your netbook probably doesn't >have that much ram, I am not sure increasing that option is such a good >idea. -C only has an effect in slim mode, unless I'm wrong? As I was hoping to append data to my db, I'm running osm2pgsql in full fat mode. Thanks for the --cache-strategy tip - I got the import working with the sparse option. It seems to be working surprisingly quickly in fact. Of course, as you said, getting the data into the db is one thing, but actually using it is another matter. The netbook is now rendering tiles for a large strip across Europe - this often takes a while to get tiles created, but I think it should work for what I need it for. If the load gets too high I can always empty the db and add extracts as I need them. Before that, however, I think I'll try and find any database optimisations that might exist. Thanks again for all this, Cheers, Joseph On 18 October 2011 18:34, Kai Krueger wrote: > On 10/18/2011 10:48 AM, Joseph Reeves wrote: >>> It is possible that one could catch the errors in slim mode and then only >>> do the expensive diff processing for those node / ways that are >duplicate >>> in the extracts. >> >> Interesting, although I think this is beyond the limits of my OSM >> skills. > > Yes, that comment was more directed at my self that I should look into > that, or if any other dev of osm2pgsql gets to it first... ;-) > > Unfortunately Austria seems to be beyond the capabilities of >> my netbook; an import without --slim gives the error: >> >> Node cache size is too small to fit all nodes. Please increase cache size >> >> Presumably a slim import would help, but this would then fail because >> of overlapping ways... I can't Google up anyone suffering that error >> message before; I guess nobody else is trying to get a number of >> European countries into a db on their netbook... > > You will likely not find that error message on google yet, as if I am > not mistaken, I only commited that error message two days ago. > > The default Ram cache size is 800Mb. You can increase it with the -C > parameter of osm2pgsql. But given that your netbook probably doesn't > have that much ram, I am not sure increasing that option is such a good > idea. > > Given that you can't fit Austria into 800Mb, I suspect you are using the > 32bit version of osm2pgsql. It defaults to the old (for extracts > inefficient) cache allocation strategy. You can try using the > "--cache-strategy" option and set it to either optimized or sparse. That > should be more efficient for extracts than the default method. In the > optimized option, you might run out of virtual memory on 32 bit though. > > But you might have troubles with Austria on a netbook anyway, as it > might still be too resource intense. > > Kai > >> >> Thanks again, >> >> Joseph >> >> >> >> >> On 18 October 2011 16:59, Kai Krueger wrote: >>> On 10/18/11 9:31 AM, Joseph Reeves wrote: Hi Kai, > The pre-rendered tiles are stored in /var/lib/mod_tile/default. You can > simply delete those files and they will automatically get rerendered the > next time you view them. Great, thanks, that's working great. > I have seen that you appear to need to restart renderd (sudo > /etc/init.d/renderd restart) after a new import, as it otherwise appears > to > still use old data (It is kind of odd, so I might have the wrong > impression > here). Sorry, I should've said in my previous email - I was assuming this to be the case. Out of interest, is there any log output from renderd? >>> >>> sudo tail -f /var/log/syslog >>> >>> should give you some output of what renderd is doing, including which tiles >>> it is rendering and when it has completed them I'm running this on a netbook so tiles take a while to be produced; >>> >>> I'd imagine a netbook might struggle a little with rendering tiles on the >>> fly... ;-) But eventually they should be done. it would be interesting to see some logs so that I know it's all working as I expect. > However, what you are trying to do is as far as I know not supported by > osm2pgsql. Although it seems to be a much requested feature, I don't > think > osm2pgsql currently handles importing of multiple extracts. The --append > option doesn't really do what you would think it does. I tried the append flag and got an error about an already existing way - it would be good if osm2pgsql would simply ignore any ways that already exist in the database. I re-ran osm2pgsql without the --slim option, however, and the import was successful. I currently have Bulgaria and Romania working on my netbook :) >>> >>> Interesting that the non-slim mode works with appending multiple extracts. >>> >>> It is possible that one could catch the errors in slim mode
Re: [OSM-talk] Jerusalem name tag - Mediation
> I'm not arguing > there shouldn't be a standard, but I am pointing out OSM is hardly > consistent. Question is, can we remove some of the inconsistency by introducing new tags? If no, then Serge's suggestion (without the additional tags) seems to be in line with current OSM practice. Dmitry ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Yahoo Imagery
Richard Fairhurst wrote: Or we switch P1 off. ;) Eeek! Don't go saying things like that... ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Yahoo Imagery
Ilya Zverev wrote: > Hi! A month ago we said bye to Yahoo imagery due to the shutting > down of some of their services. But it is still available in both > Potlatch versions. As Tom says, the situation with permission has not changed either way. Yahoo have slightly extended the switch-off period because some other users of their Maps API requested it: "A few remaining customers have requested additional time to switch over to new APIs and then the service will be shut down... We anticipate this will be completed sometime later this year." (http://blog.programmableweb.com/2011/09/22/yahoo-maps-watch-life-support-may-continue-through-2011/) Given that there are a few small areas of the world covered by Yahoo imagery but not Bing, I've not been in a hurry to remove it from P2. This may still happen before the eventual closure but, regardless, I don't have any plans to alter the P1 code, so that at least will still be running until they finally switch the Yahoo servers off. Or we switch P1 off. ;) cheers Richard -- View this message in context: http://gis.638310.n2.nabble.com/Yahoo-Imagery-tp6907652p6907856.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Yahoo Imagery
On 19/10/11 09:25, SomeoneElse wrote: What Yahoo actually said (http://developer.yahoo.com/blogs/ydn/posts/2011/06/yahoo-maps-apis-service-closure-announcement-new-maps-offerings-coming-soon/) was that they would "no longer support [these] Maps APIs" on September 13th. I never saw an announcement saying that permission had been withdrawn; merely that the thing that we trace from might not be there any more. If a particular editor misunderstood that announcement, perhaps log a bug in their trac? I don't think anybody misunderstood anything, except that we thought they were actually going to turn them off on that date so that they would no longer work. Our permission to trace from them was never an issue, and nobody thought that it was. Tom -- Tom Hughes (t...@compton.nu) http://compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] XML database VS PostgreSQL for OSM
Hi, Considering an OSM data fragment representing a town as Paris, London, etc, I wonder assets and drawbacks of using a XML file (myTown.osm) against tools as pgRouting or another PostgreSQL based database for routing, visualization, updates from main server? Thanks___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Yahoo Imagery
Ilya Zverev wrote: Hi! A month ago we said bye to Yahoo imagery due to the shutting down of some of their services. But it is still available in both Potlatch versions. Did the permission to trace mention the services that can be used for tracing (and how is it available in Potlatch then), or it is possible to get Yahoo imagery back in other editors, like JOSM? What Yahoo actually said (http://developer.yahoo.com/blogs/ydn/posts/2011/06/yahoo-maps-apis-service-closure-announcement-new-maps-offerings-coming-soon/) was that they would "no longer support [these] Maps APIs" on September 13th. I never saw an announcement saying that permission had been withdrawn; merely that the thing that we trace from might not be there any more. If a particular editor misunderstood that announcement, perhaps log a bug in their trac? Cheers, Andy ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] Yahoo Imagery
Hi! A month ago we said bye to Yahoo imagery due to the shutting down of some of their services. But it is still available in both Potlatch versions. Did the permission to trace mention the services that can be used for tracing (and how is it available in Potlatch then), or it is possible to get Yahoo imagery back in other editors, like JOSM? IZ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
