Re: [OSM-talk] does one need to download the planet to use osmosis
>> wget http://planet.openstreetmap.org/planet-latest.osm.bz2 -O - | >> bzcat >>osmosis --read-xml "/dev/stdin" --bounding-box top=49.5138 >>left=10.9351 bottom=49.3866 right=11.201 --write-xml "extract.xml" > > Isn't this bbox only a few times larger then what the API will download > in a single call? The bbox is just a sample from the osmisis wiki page. I just wanted to outline the process of working with the streaming download. You can add any command to the pipe, e.g. a sax-parser in php: wget http://planet.openstreetmap.org/planet-latest.osm.bz2 -O - -q | \ bzcat | php parser.php > data.csv where parser.php creates a sax-parser reading from STDIN. Peter ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
On 16 Nov 2009, at 00:10, Kenneth Gonsalves wrote: > On Monday 16 Nov 2009 3:27:05 am Jon Burgess wrote: >> Isn't this bbox only a few times larger then what the API will download >> in a single call? >> >> IMO you'd be crazy to download nearly 8GB of data just to get hold of >> what probably amounts to a few 10's of MB. At the very least you should >> start with an extract for something smaller, like a single country. >> >> Using JOSM to selectively download all the pieces to cover your area >> seems a much better idea to me. I normally would endorse this if you >> were attempting to download something of many degrees in size, but for a >> few tenths of a degree you are probably OK. >> > > I was talking about India - it would be impossible to to this with josm. > However the problem has gone away as Geofabrik asia extract is now up to > date. > Although cloudmade India data is still stuck at 4/11 You could take the CloudMade extract, then use Osmosis to apply the diffs. You can thereafter clip the area that is in the DB (Osmosis can do this in one command line). Shaun ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
On Monday 16 Nov 2009 4:08:45 am Frederik Ramm wrote: > I lost track of who wrote what in this thread, but: > > Although geofabrik says the extracts are done every > > night, the extract I got yesterday was atleast 2 weeks out of date for my > > area. > > We generate all extracts every day, with the sole exception of the North > America extract which is really only a relic and should be deleted from > the download server (it is currently done once a week). The NA extract > doesn't help anybody a lot because it is so big that you might as well > download the planet. > > Back to the original question; if you get yourself an account on the OSM > dev server then you have NFS access to the planet file and you could run > Osmosis on it directly. This is not recommended for regular use as it > creates considerable load on the machine, but if you are in a situation > where download bandwidth is limited, that's an option. But bear in mind > that the planet is only updated once each week, so from Friday to > Wednesday you'll probably get a more current snapshot if you use one of > the Geofabrik files (except of course the North America one). > it was some temporary problem with geofabrik files - for a few days they did not contain the up to date data - it is ok now. Although there is no 'india' file, asia is small enough for use. Thanks for the service. -- regards Kenneth Gonsalves Senior Project Officer NRC-FOSS http://nrcfosshelpline.in/web/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
On Monday 16 Nov 2009 3:27:05 am Jon Burgess wrote: > Isn't this bbox only a few times larger then what the API will download > in a single call? > > IMO you'd be crazy to download nearly 8GB of data just to get hold of > what probably amounts to a few 10's of MB. At the very least you should > start with an extract for something smaller, like a single country. > > Using JOSM to selectively download all the pieces to cover your area > seems a much better idea to me. I normally would endorse this if you > were attempting to download something of many degrees in size, but for a > few tenths of a degree you are probably OK. > I was talking about India - it would be impossible to to this with josm. However the problem has gone away as Geofabrik asia extract is now up to date. Although cloudmade India data is still stuck at 4/11 -- regards Kenneth Gonsalves Senior Project Officer NRC-FOSS http://nrcfosshelpline.in/web/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
On Sun, 2009-11-15 at 22:04 +0100, Peter Körner wrote: > > > good idea - but can you confirm that it is impossible to extract it > from the > > remote file? > > you could do it with good ol' shell pipes: > > wget http://planet.openstreetmap.org/planet-latest.osm.bz2 -O - | > bzcat >osmosis --read-xml "/dev/stdin" --bounding-box top=49.5138 >left=10.9351 bottom=49.3866 right=11.201 --write-xml "extract.xml" Isn't this bbox only a few times larger then what the API will download in a single call? IMO you'd be crazy to download nearly 8GB of data just to get hold of what probably amounts to a few 10's of MB. At the very least you should start with an extract for something smaller, like a single country. Using JOSM to selectively download all the pieces to cover your area seems a much better idea to me. I normally would endorse this if you were attempting to download something of many degrees in size, but for a few tenths of a degree you are probably OK. Jon ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
Hi, I lost track of who wrote what in this thread, but: > Although geofabrik says the extracts are done every > night, the extract I got yesterday was atleast 2 weeks out of date for my > area. We generate all extracts every day, with the sole exception of the North America extract which is really only a relic and should be deleted from the download server (it is currently done once a week). The NA extract doesn't help anybody a lot because it is so big that you might as well download the planet. Back to the original question; if you get yourself an account on the OSM dev server then you have NFS access to the planet file and you could run Osmosis on it directly. This is not recommended for regular use as it creates considerable load on the machine, but if you are in a situation where download bandwidth is limited, that's an option. But bear in mind that the planet is only updated once each week, so from Friday to Wednesday you'll probably get a more current snapshot if you use one of the Geofabrik files (except of course the North America one). Bye Frederik ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
> good idea - but can you confirm that it is impossible to extract it from the > remote file? you could do it with good ol' shell pipes: wget http://planet.openstreetmap.org/planet-latest.osm.bz2 -O - | bzcat osmosis --read-xml "/dev/stdin" --bounding-box top=49.5138 left=10.9351 bottom=49.3866 right=11.201 --write-xml "extract.xml" similar you can use /dev/stdout as output-file and pipe another command at the end, eg. bzip to comress the data again or a script that parses the data from stdin. Peter ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
On Saturday 14 Nov 2009 10:17:06 am you wrote: > >> Or use the extracts geofabrik publishes. > >> > >> http://download.geofabrik.de/osm/ > > > > I usually use either geofabrik or cloudmade. Cloudmade is supposed to > > update once a week, but it is saturday and the latest extract is dated > > thursday the 4th. Although geofabrik says the extracts are done every > > night, the extract I got yesterday was atleast 2 weeks out of date for my > > area. I am keen on seeing some changes I made a few days back. > > Downloading the planet takes 20 minutes, and it is nearly an hour since > > bunzip2 started running and only half the file has been unzipped. > > You can bzcat the file and then pipe it to osmosis rather than trying > to uncompress it first. > good idea - but can you confirm that it is impossible to extract it from the remote file? -- regards Kenneth Gonsalves Senior Project Officer NRC-FOSS http://nrcfosshelpline.in/web/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] does one need to download the planet to use osmosis
2009/11/14 Kenneth Gonsalves : > hi, > > I want to use osmosis to download and extract from the planet. I tried giving > the web address of the planet latest file, but get an error saying > 'http:/openstre../planet_latest.. not found'. Note that the error gives the > address as http:/ with only one forward slash. The question is, do I need to > have the whole planet file on my local machine in order to extract a part of > it? Or use the extracts geofabrik publishes. http://download.geofabrik.de/osm/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] does one need to download the planet to use osmosis
hi, I want to use osmosis to download and extract from the planet. I tried giving the web address of the planet latest file, but get an error saying 'http:/openstre../planet_latest.. not found'. Note that the error gives the address as http:/ with only one forward slash. The question is, do I need to have the whole planet file on my local machine in order to extract a part of it? -- regards Kenneth Gonsalves Senior Project Officer NRC-FOSS http://nrcfosshelpline.in/web/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
