[OSM-talk] Downloading compressed data from API
Hi all, I was wondering if it's possible to download data from the API in a (bz2/g)zipped format? This would be much more kind to bandwidth since a 10MB file for the plain data can be easily compressed to under 1MB. Greetings Ben ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
Ben Laenen wrote: I was wondering if it's possible to download data from the API in a (bz2/g)zipped format? This would be much more kind to bandwidth since a 10MB file for the plain data can be easily compressed to under 1MB. It is compressed if your user agent indicates that it supports compressed transfers. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
Am So, 10.08.2008, 10:35, schrieb Ben Laenen: Hi all, I was wondering if it's possible to download data from the API in a (bz2/g)zipped format? This would be much more kind to bandwidth since a 10MB file for the plain data can be easily compressed to under 1MB. Actually http supports compression itself. It may be not as efficient as bz2 (or others) but I don't think that you gain much benefit when the servers additionally have to gzip the data (or use another compression algo). -- -m*sh- ___ |harry w. graner |mail: hy [_at_] sha-mash [_dot_] de |--- [public gpg-key on request] take a look at my blogs: http://sha-mash.blog.de ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
Hi, m*sh wrote: Actually http supports compression itself. That's what Ben and Tom were talking about. One party sets a header that says I would also accept compressed format if you are capable to send that and the other party then sends compressed data. Which compression methods are available can also be specified in the headers. This has nothing to do with calling an external gzip/bzip2 program. Bye Frederik -- Frederik Ramm ## eMail [EMAIL PROTECTED] ## N49°00'09 E008°23'33 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
On Sunday 10 August 2008, Tom Hughes wrote: Ben Laenen wrote: I was wondering if it's possible to download data from the API in a (bz2/g)zipped format? This would be much more kind to bandwidth since a 10MB file for the plain data can be easily compressed to under 1MB. It is compressed if your user agent indicates that it supports compressed transfers. Hmm, I was hoping more for some extra tag attached to the url... So, how should I do that when I use wget to download the data? I have shell scripts to download data automatically and wget is by far the easiest way to do that, but can't find any documentation for that. Ben ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
On Sun, Aug 10, 2008 at 09:35:35PM +0200, Ben Laenen wrote: On Sunday 10 August 2008, Tom Hughes wrote: Ben Laenen wrote: I was wondering if it's possible to download data from the API in a (bz2/g)zipped format? This would be much more kind to bandwidth since a 10MB file for the plain data can be easily compressed to under 1MB. It is compressed if your user agent indicates that it supports compressed transfers. Hmm, I was hoping more for some extra tag attached to the url... So, how should I do that when I use wget to download the data? I have shell scripts to download data automatically and wget is by far the easiest way to do that, but can't find any documentation for that. You could do something with the --header=header-line option to wget or you can switch to curl which has a --compressed option. Jochen -- Jochen Topf [EMAIL PROTECTED] http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
Hi, So, how should I do that when I use wget to download the data? I have shell scripts to download data automatically and wget is by far the easiest way to do that, but can't find any documentation for that. Use the --header='accept-encoding: gzip' flag: [EMAIL PROTECTED]:/tmp/wget$ wget -Otest1.osm http://www.openstreetmap.org/api/0.5/way/4290149/history --21:49:04-- http://www.openstreetmap.org/api/0.5/way/4290149/history = `test1.osm' Resolving www.openstreetmap.org... 128.40.58.203 Connecting to www.openstreetmap.org|128.40.58.203|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 470 [text/xml] 100%[=] 470 --.--K/s 21:49:04 (58.88 MB/s) - `test1.osm' saved [470/470] [EMAIL PROTECTED]:/tmp/wget$ wget -Otest2.osm.gz --header=accept-encoding: gzip http://www.openstreetmap.org/api/0.5/way/4290149 /history --21:49:31-- http://www.openstreetmap.org/api/0.5/way/4290149/history = `test2.osm.gz' Resolving www.openstreetmap.org... 128.40.58.203 Connecting to www.openstreetmap.org|128.40.58.203|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 262 [text/xml] 100%[=] 262 --.--K/s 21:49:31 (38.05 MB/s) - `test2.osm.gz' saved [262/262] [EMAIL PROTECTED]:/tmp/wget$ gunzip test2.osm [EMAIL PROTECTED]:/tmp/wget$ md5sum test*osm b66910215e75f774e31166d4b34b9290 test1.osm b66910215e75f774e31166d4b34b9290 test2.osm [EMAIL PROTECTED]:/tmp/wget$ Bye Frederik -- Frederik Ramm ## eMail [EMAIL PROTECTED] ## N49°00'09 E008°23'33 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Downloading compressed data from API
Ben Laenen wrote: So, how should I do that when I use wget to download the data? I have shell scripts to download data automatically and wget is by far the easiest way to do that, but can't find any documentation for that. I'd be quite surprised if a popular client like wget didn't already do it, but it will also decompress it automatically so the fact that the network traffic is compressed will be transparent. Tom -- Tom Hughes ([EMAIL PROTECTED]) http://www.compton.nu/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
