Re: [OSM-talk] short links
Hi, There is some info in a blog post I made a few months back: http://blog.shaunmcdonald.me.uk/2010/01/openstreetmap-shortlinks/ Shaun On 13 Mar 2010, at 21:35, Vincent Pottier wrote: > Hi, > One question : > What is the algorithm used to convert the short OSM link from/to lat/lon > > One sugestion : > It sshould be nice to increase the potientiality of the osm.org. I explain : > http://osm.org/go/0CUlOS6B- > redirects to > http://www.openstreetmap.org/?lat=47.2553&lon=6.0121&zoom=14&layers=B000FTF > > > http://osm.org/at/0CUlOS6B- > could redirect to > http://www.openstreetmap.org/?mlat=47.2553&mlon=6.0121&zoom=14&layers=B000FTF > > Other shortcuts can be suggested... > like > > http://osm.org/car/0CUlOS6B-/0CUpGUF for car routing > http://osm.org/hike/0CUlOS6B-/0CUpGUF > http://osm.org/bike/0CUlOS6B-/0CUpGUF > > and, i'm shure you have understoud the principle so you have understoud > what could be > > http://osm.org/car/0A75uZ/0CUlOS6B-/0CUpGUF > -- > FrViPofm > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] short links
Hi, One question : What is the algorithm used to convert the short OSM link from/to lat/lon One sugestion : It sshould be nice to increase the potientiality of the osm.org. I explain : http://osm.org/go/0CUlOS6B- redirects to http://www.openstreetmap.org/?lat=47.2553&lon=6.0121&zoom=14&layers=B000FTF http://osm.org/at/0CUlOS6B- could redirect to http://www.openstreetmap.org/?mlat=47.2553&mlon=6.0121&zoom=14&layers=B000FTF Other shortcuts can be suggested... like http://osm.org/car/0CUlOS6B-/0CUpGUF for car routing http://osm.org/hike/0CUlOS6B-/0CUpGUF http://osm.org/bike/0CUlOS6B-/0CUpGUF and, i'm shure you have understoud the principle so you have understoud what could be http://osm.org/car/0A75uZ/0CUlOS6B-/0CUpGUF -- FrViPofm ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
No. It runs on the uncompressed planet, like this : bzcat /osm/planet-10*.osm.bz2 | /osm/gosmore/bboxSplit \ -85.05113 73.125009.44906 180.0 gzip 0720048510241024.osm.gz \ -25.48295 120.58594 72.91964 180.0 gzip 0855020310240587.osm.gz \ -85.05113 98.43750 13.23995 172.61719 gzip 0792047410031024.osm.gz \ ... I'm not too worried about further optimizations: Unlike wikipedia, there isn't the same urgency to have up-to-date. Except for disaster relief. On Sat, Mar 13, 2010 at 10:42 PM, jamesmikedup...@googlemail.com wrote: > you are bunziping the code ? you are scanning the bzip blocks? > it is faster than the bunzip. But maybe you mean that it is very fast. > > I have experimented with bziprecover to extract blocks on their own, > i made a perl script to extract blocks from a wikipedia file that can be > used to run the processing of the huge file by many people in parallel. > > https://code.launchpad.net/~jamesmikedupont/+junk/openstreetmap-wikipedia > > It is a tool to extract lat/long coords from the wikipedia articles. > > Such a processing of the large files would allow us to team up and all help. > We really need to just have an index file of all the blocks so that we can > find the ones that we need. Imagine being able to process the bzip file > directly! > > mike > > On Sat, Mar 13, 2010 at 9:31 PM, Nic Roets wrote: >> >> Hello James, >> >> I wanted to split the planet into overlapping bboxes like this (click >> to see actual size): >> http://dev.openstreetmap.de/gosmore/ >> >> On talk I described how I was dissatisfied with osmosis's memory >> consumption. So I came up with this observation: Most entities will >> end up in one or two extracts. And when it's two, it's in a pattern >> that is often repeated, say Africa bbox and Middle East bbox. Never >> Africa and Canada. So of the 2^168 possible combinations only around >> 3000 is actually used. >> >> So bboxSplit allocates 16 bits for each entity. Those are then indexes >> into the array of 'youniouns'. If a new node comes along, I check it >> against list of bboxes and it typically matches 1 or 2. So to find out >> quickly if I already have that combination of bboxes, I also have an >> STL map on the array of younions. A hashtable would have been faster. >> >> Ways and relations also trigger the code that merge younions. >> >> bboxSplit is faster than the corresponding bunzip and any program that >> uses libxml, i.e. very fast. >> >> Regards, >> Nic >> >> On Sat, Mar 13, 2010 at 10:03 PM, jamesmikedup...@googlemail.com >> wrote: >> > That is very deep c++ code! >> > care to comment on how it works? >> > would be very interested to understand its performance ! looks very >> > fast. >> > mike >> > >> > On Sat, Mar 13, 2010 at 7:06 PM, Nic Roets wrote: >> >> >> >> My understanding is that all Xml compliant* parsers will abort at the >> >> file offsets that Frederik mentions. >> >> My advice is to use the egrep filter when in doubt, because you will >> >> loose no more than a dozen lines in a planet file of billions of >> >> lines. >> >> >> >> *: (My split program is not compliant and will happily ignore these >> >> errors: >> >> >> >> >> >> http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp) >> >> >> >> On Sat, Mar 13, 2010 at 7:44 PM, John Mitchell >> >> wrote: >> >> > Will this also be a problem if you try to import via osm2pgsql into >> >> > postgres? >> >> > >> >> > Thanks, >> >> > >> >> > John >> >> > >> >> > On 3/13/10, hbogner wrote: >> >> >> Thx for help, I'll try it. >> >> >> >> >> >> Now I have to follow 'dev' too :D >> >> >> >> >> >> Nic Roets wrote: >> >> >>> There's a bug in the code that generated this week's planet. You >> >> >>> should either wait until next week or filter the planet with the >> >> >>> following command: >> >> >>> bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... >> >> >>> >> >> >>> There has been a long discussion on 'dev', mentioning other >> >> >>> remedies. >> >> >>> >> >> >> >> >> >> >> >> >> ___ >> >> >> talk mailing list >> >> >> talk@openstreetmap.org >> >> >> http://lists.openstreetmap.org/listinfo/talk >> >> >> >> >> > >> >> > >> >> > -- >> >> > John J. Mitchell >> >> > >> >> > ___ >> >> > talk mailing list >> >> > talk@openstreetmap.org >> >> > http://lists.openstreetmap.org/listinfo/talk >> >> > >> >> >> >> ___ >> >> talk mailing list >> >> talk@openstreetmap.org >> >> http://lists.openstreetmap.org/listinfo/talk >> > >> > > > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
you are bunziping the code ? you are scanning the bzip blocks? it is faster than the bunzip. But maybe you mean that it is very fast. I have experimented with bziprecover to extract blocks on their own, i made a perl script to extract blocks from a wikipedia file that can be used to run the processing of the huge file by many people in parallel. https://code.launchpad.net/~jamesmikedupont/+junk/openstreetmap-wikipedia It is a tool to extract lat/long coords from the wikipedia articles. Such a processing of the large files would allow us to team up and all help. We really need to just have an index file of all the blocks so that we can find the ones that we need. Imagine being able to process the bzip file directly! mike On Sat, Mar 13, 2010 at 9:31 PM, Nic Roets wrote: > Hello James, > > I wanted to split the planet into overlapping bboxes like this (click > to see actual size): > http://dev.openstreetmap.de/gosmore/ > > On talk I described how I was dissatisfied with osmosis's memory > consumption. So I came up with this observation: Most entities will > end up in one or two extracts. And when it's two, it's in a pattern > that is often repeated, say Africa bbox and Middle East bbox. Never > Africa and Canada. So of the 2^168 possible combinations only around > 3000 is actually used. > > So bboxSplit allocates 16 bits for each entity. Those are then indexes > into the array of 'youniouns'. If a new node comes along, I check it > against list of bboxes and it typically matches 1 or 2. So to find out > quickly if I already have that combination of bboxes, I also have an > STL map on the array of younions. A hashtable would have been faster. > > Ways and relations also trigger the code that merge younions. > > bboxSplit is faster than the corresponding bunzip and any program that > uses libxml, i.e. very fast. > > Regards, > Nic > > On Sat, Mar 13, 2010 at 10:03 PM, jamesmikedup...@googlemail.com > wrote: > > That is very deep c++ code! > > care to comment on how it works? > > would be very interested to understand its performance ! looks very fast. > > mike > > > > On Sat, Mar 13, 2010 at 7:06 PM, Nic Roets wrote: > >> > >> My understanding is that all Xml compliant* parsers will abort at the > >> file offsets that Frederik mentions. > >> My advice is to use the egrep filter when in doubt, because you will > >> loose no more than a dozen lines in a planet file of billions of > >> lines. > >> > >> *: (My split program is not compliant and will happily ignore these > >> errors: > >> > >> > http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp > ) > >> > >> On Sat, Mar 13, 2010 at 7:44 PM, John Mitchell > >> wrote: > >> > Will this also be a problem if you try to import via osm2pgsql into > >> > postgres? > >> > > >> > Thanks, > >> > > >> > John > >> > > >> > On 3/13/10, hbogner wrote: > >> >> Thx for help, I'll try it. > >> >> > >> >> Now I have to follow 'dev' too :D > >> >> > >> >> Nic Roets wrote: > >> >>> There's a bug in the code that generated this week's planet. You > >> >>> should either wait until next week or filter the planet with the > >> >>> following command: > >> >>> bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... > >> >>> > >> >>> There has been a long discussion on 'dev', mentioning other > remedies. > >> >>> > >> >> > >> >> > >> >> ___ > >> >> talk mailing list > >> >> talk@openstreetmap.org > >> >> http://lists.openstreetmap.org/listinfo/talk > >> >> > >> > > >> > > >> > -- > >> > John J. Mitchell > >> > > >> > ___ > >> > talk mailing list > >> > talk@openstreetmap.org > >> > http://lists.openstreetmap.org/listinfo/talk > >> > > >> > >> ___ > >> talk mailing list > >> talk@openstreetmap.org > >> http://lists.openstreetmap.org/listinfo/talk > > > > > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
Hello James, I wanted to split the planet into overlapping bboxes like this (click to see actual size): http://dev.openstreetmap.de/gosmore/ On talk I described how I was dissatisfied with osmosis's memory consumption. So I came up with this observation: Most entities will end up in one or two extracts. And when it's two, it's in a pattern that is often repeated, say Africa bbox and Middle East bbox. Never Africa and Canada. So of the 2^168 possible combinations only around 3000 is actually used. So bboxSplit allocates 16 bits for each entity. Those are then indexes into the array of 'youniouns'. If a new node comes along, I check it against list of bboxes and it typically matches 1 or 2. So to find out quickly if I already have that combination of bboxes, I also have an STL map on the array of younions. A hashtable would have been faster. Ways and relations also trigger the code that merge younions. bboxSplit is faster than the corresponding bunzip and any program that uses libxml, i.e. very fast. Regards, Nic On Sat, Mar 13, 2010 at 10:03 PM, jamesmikedup...@googlemail.com wrote: > That is very deep c++ code! > care to comment on how it works? > would be very interested to understand its performance ! looks very fast. > mike > > On Sat, Mar 13, 2010 at 7:06 PM, Nic Roets wrote: >> >> My understanding is that all Xml compliant* parsers will abort at the >> file offsets that Frederik mentions. >> My advice is to use the egrep filter when in doubt, because you will >> loose no more than a dozen lines in a planet file of billions of >> lines. >> >> *: (My split program is not compliant and will happily ignore these >> errors: >> >> http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp) >> >> On Sat, Mar 13, 2010 at 7:44 PM, John Mitchell >> wrote: >> > Will this also be a problem if you try to import via osm2pgsql into >> > postgres? >> > >> > Thanks, >> > >> > John >> > >> > On 3/13/10, hbogner wrote: >> >> Thx for help, I'll try it. >> >> >> >> Now I have to follow 'dev' too :D >> >> >> >> Nic Roets wrote: >> >>> There's a bug in the code that generated this week's planet. You >> >>> should either wait until next week or filter the planet with the >> >>> following command: >> >>> bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... >> >>> >> >>> There has been a long discussion on 'dev', mentioning other remedies. >> >>> >> >> >> >> >> >> ___ >> >> talk mailing list >> >> talk@openstreetmap.org >> >> http://lists.openstreetmap.org/listinfo/talk >> >> >> > >> > >> > -- >> > John J. Mitchell >> > >> > ___ >> > talk mailing list >> > talk@openstreetmap.org >> > http://lists.openstreetmap.org/listinfo/talk >> > >> >> ___ >> talk mailing list >> talk@openstreetmap.org >> http://lists.openstreetmap.org/listinfo/talk > > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
That is very deep c++ code! care to comment on how it works? would be very interested to understand its performance ! looks very fast. mike On Sat, Mar 13, 2010 at 7:06 PM, Nic Roets wrote: > My understanding is that all Xml compliant* parsers will abort at the > file offsets that Frederik mentions. > My advice is to use the egrep filter when in doubt, because you will > loose no more than a dozen lines in a planet file of billions of > lines. > > *: (My split program is not compliant and will happily ignore these errors: > > http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp > ) > > On Sat, Mar 13, 2010 at 7:44 PM, John Mitchell > wrote: > > Will this also be a problem if you try to import via osm2pgsql into > postgres? > > > > Thanks, > > > > John > > > > On 3/13/10, hbogner wrote: > >> Thx for help, I'll try it. > >> > >> Now I have to follow 'dev' too :D > >> > >> Nic Roets wrote: > >>> There's a bug in the code that generated this week's planet. You > >>> should either wait until next week or filter the planet with the > >>> following command: > >>> bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... > >>> > >>> There has been a long discussion on 'dev', mentioning other remedies. > >>> > >> > >> > >> ___ > >> talk mailing list > >> talk@openstreetmap.org > >> http://lists.openstreetmap.org/listinfo/talk > >> > > > > > > -- > > John J. Mitchell > > > > ___ > > talk mailing list > > talk@openstreetmap.org > > http://lists.openstreetmap.org/listinfo/talk > > > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
My understanding is that all Xml compliant* parsers will abort at the file offsets that Frederik mentions. My advice is to use the egrep filter when in doubt, because you will loose no more than a dozen lines in a planet file of billions of lines. *: (My split program is not compliant and will happily ignore these errors: http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp) On Sat, Mar 13, 2010 at 7:44 PM, John Mitchell wrote: > Will this also be a problem if you try to import via osm2pgsql into postgres? > > Thanks, > > John > > On 3/13/10, hbogner wrote: >> Thx for help, I'll try it. >> >> Now I have to follow 'dev' too :D >> >> Nic Roets wrote: >>> There's a bug in the code that generated this week's planet. You >>> should either wait until next week or filter the planet with the >>> following command: >>> bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... >>> >>> There has been a long discussion on 'dev', mentioning other remedies. >>> >> >> >> ___ >> talk mailing list >> talk@openstreetmap.org >> http://lists.openstreetmap.org/listinfo/talk >> > > > -- > John J. Mitchell > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
Will this also be a problem if you try to import via osm2pgsql into postgres? Thanks, John On 3/13/10, hbogner wrote: > Thx for help, I'll try it. > > Now I have to follow 'dev' too :D > > Nic Roets wrote: >> There's a bug in the code that generated this week's planet. You >> should either wait until next week or filter the planet with the >> following command: >> bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... >> >> There has been a long discussion on 'dev', mentioning other remedies. >> > > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk > -- John J. Mitchell ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
Thx for help, I'll try it. Now I have to follow 'dev' too :D Nic Roets wrote: > There's a bug in the code that generated this week's planet. You > should either wait until next week or filter the planet with the > following command: > bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... > > There has been a long discussion on 'dev', mentioning other remedies. > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
There's a bug in the code that generated this week's planet. You should either wait until next week or filter the planet with the following command: bzcat /osm/planet-10*.osm.bz2 |egrep -v '&#[0-9]*;'|... There has been a long discussion on 'dev', mentioning other remedies. On Sat, Mar 13, 2010 at 6:29 PM, hbogner wrote: > Nic Roets wrote: >> (since we got rid of the segments) >> >> From 8.2 GB to 8.1 GB: >> http://planet.openstreetmap.org/ > Maybe something is wrong with it. > I don't know if anybody has the same problem but I can't manage to > complete an extract with osmosis. I'm doing the same thing as everytime > and it doesn't work. I tried downloading it again, nothing, another > version of osmosis, nothing, another version of .poly, nothing. > At the moment I'm waiting for it to extract from bz2 and try again, > maybe it's just bz2 ... > > error: > SEVERE: Thread for task 1-read-xml failed > org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse > xml file /dev/stdin. publicId=(null), systemId=(null), > lineNumber=529642199, columnNumber=27. > > osmosis: > bzip2 -d -c planet-100310.osm.bz2 | osmosis/bin/osmosis --read-xml > /dev/stdin --bounding-polygon clipIncompleteEntities="true" > file="croatia50km.poly" --write-xml file=- | bzip2 -c > > 20100310-croatia50km.osm.bz2 > > > ___ > talk mailing list > talk@openstreetmap.org > http://lists.openstreetmap.org/listinfo/talk > ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] First drop in planet size ?
Nic Roets wrote: > (since we got rid of the segments) > > From 8.2 GB to 8.1 GB: > http://planet.openstreetmap.org/ Maybe something is wrong with it. I don't know if anybody has the same problem but I can't manage to complete an extract with osmosis. I'm doing the same thing as everytime and it doesn't work. I tried downloading it again, nothing, another version of osmosis, nothing, another version of .poly, nothing. At the moment I'm waiting for it to extract from bz2 and try again, maybe it's just bz2 ... error: SEVERE: Thread for task 1-read-xml failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file /dev/stdin. publicId=(null), systemId=(null), lineNumber=529642199, columnNumber=27. osmosis: bzip2 -d -c planet-100310.osm.bz2 | osmosis/bin/osmosis --read-xml /dev/stdin --bounding-polygon clipIncompleteEntities="true" file="croatia50km.poly" --write-xml file=- | bzip2 -c > 20100310-croatia50km.osm.bz2 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM for walkers / hikers - getting it going!
> So mtb:scale=5 would be a vertical cliff? > Have you seen the video that plays in the Blacks outdoor shops?? "Yes" would be the answer! Incredible. -- ___ Surf the Web in a faster, safer and easier way: Download Opera 9 at http://www.opera.com Powered by Outblaze ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Freemap - OpenStreetMap for walkers (hikers) - feature ideas?
Hi, > Walking isn't just about long-distance stuff. > Being able to say: x is my starting point and I have [10|30|60mins|...] > and I am [slow as a snail|average|running from mad mappers], please take > me on a circular route that avoids busy roads, goes through nice parks, > maybe goes to places people marked as good viewpoints, and accounts for > me being slow uphill. If you have an algorithm which does that you should be able to add: x is my starting point and I have [10|30|60mins|...] and I am [slow as a snail|average|a mad mapper running], please take me on a circular route that covers some entries in OpenStreetBugs, preferably going along streets which have buildings tagged but no house numbers. Patrick "Petschge" Kilian ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
