Re: [OSM-talk] Statistics on road network length?
On 20/11/2011 20:48, Frederik Ramm wrote: http://www.remote.org/frederik/tmp/osmium_road_length.cpp Does all of Germany (from a PBF file) in less than 5 minutes and using less than 2 GB of RAM. I haven't got the time to do this properly... Thanks for this one. I just managed to do germany.osm.pbf in 3 minutes and 3 seconds. Memory usage only went up to 1.3G. Hermann ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
Hi, On 11/18/2011 10:58 PM, Frederik Ramm wrote: On Fri, 18 Nov 2011 11:56:53 -0800 (PST) ThomasBtoba0...@yahoo.de wrote: not sure we are using the same script. With osmosis --read-pbf file=/home/wicking/osm/germany.osm.pbf --write-xml file=- | perl osm-length-2.pl /dev/null I am 15 minutes after the start at 14GB RAM usage and still increasing. Yes, the osm-length perl script wastes a lot of memory by keeping all nodes in a Perl hash. It would be relatively easy to write an efficient road length calculator based on the Osmium framework; that one should then work with a far smaller memory footprint. http://www.remote.org/frederik/tmp/osmium_road_length.cpp Does all of Germany (from a PBF file) in less than 5 minutes and using less than 2 GB of RAM. I haven't got the time to do this properly right now - it just dumps a list of results for all highway types to stdout and sadly includes all the tag rubbish: *0.2km Access_Ramp 0.0km Altmarkt 0.2km FIXME0.8km Fußweg 0.0km Hinterer Spielbach 0.1km Hohndorfer Weg 1.2km Privatstraße0.1km Sandstraße 0.1km Weikersdorfer Weg0.1km abandoned 23.4km abandonned 0.3km abendoned0.5km access 0.8km access_ramp 0.1km agriculture 0.8km almost_path 0.4km animal_pass 0.1km animal_trail 0.2km area:unterschiedlich 0.1km bicycle 0.1km bollard 0.4km bridge 0.0km bridleway 2198.5km bridleway, track 0.5km bus_guideway 5.9km ... (full result file for Germany here: http://www.remote.org/frederik/tmp/osmium_road_length.result) But anyone with minimal C++ skills should be able to polish that. (If you do, be so kind and make your polished version available somewhere stable, e.g. send it to Jochen for inclusion in his examples directory or so.) A power user might even want to take Peter Koerner's history splitter which is OSM based and has the ability to cut out OSM data along polygon boundaries, and add the length formula from this script so you can easily make a separate counting for lots of places. Best wishes Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
On 18/11/2011 23:30, Jo wrote: Concerning the issue with dual carriageways, I am using a brute-force approach and divide the calculated distance by 2 if these conditions are true: highway ~ /^(motorway|trunk|primary|secondary|tertiary)/ oneway ~ /yes|true|1/ You probably want to add oneway=-1 to that list. There are some oddballs who used that indicate oneway is against the vector sense. Also JOSM proposes to use it when reversing a way. Jo Thanks for the hint. I already thought about it but forgot to mention that the above tests are using regular expressions. ^motorway also matches motorway_link, etc. and 1 also matches -1. Hermann ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
unfortunately osmium(osmjs) ends with /# Fatal error in CALL_AND_RETRY_2 # Allocation failed - process out of memory/ although there is still sufficient (physical) memory available. A change of osmjs -l array - disk did not help. Frederik Ramm wrote: Hi, On Fri, 18 Nov 2011 11:56:53 -0800 (PST) ThomasB lt;toba0211@gt; wrote: not sure we are using the same script. With osmosis --read-pbf file=/home/wicking/osm/germany.osm.pbf --write-xml file=- | perl osm-length-2.pl /dev/null I am 15 minutes after the start at 14GB RAM usage and still increasing. Yes, the osm-length perl script wastes a lot of memory by keeping all nodes in a Perl hash. It would be relatively easy to write an efficient road length calculator based on the Osmium framework; that one should then work with a far smaller memory footprint. Bye Frederik ___ talk mailing list talk@ http://lists.openstreetmap.org/listinfo/talk -- View this message in context: http://gis.638310.n2.nabble.com/Statistics-on-road-network-length-tp759p7011662.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
Hi! That error comes from the Javascript part, not from the node location store, so changes to the -l parameter will not help. It's probably the same problem Martijn reported on dev recently. There seems to be a memory leak somewhere in the Javascript code of Osmium. What OSM extract did you try to run it on? Jochen On Sat, Nov 19, 2011 at 08:52:39AM -0800, ThomasB wrote: Date: Sat, 19 Nov 2011 08:52:39 -0800 (PST) From: ThomasB toba0...@yahoo.de To: talk@openstreetmap.org Subject: Re: [OSM-talk] Statistics on road network length? unfortunately osmium(osmjs) ends with /# Fatal error in CALL_AND_RETRY_2 # Allocation failed - process out of memory/ although there is still sufficient (physical) memory available. A change of osmjs -l array - disk did not help. Frederik Ramm wrote: Hi, On Fri, 18 Nov 2011 11:56:53 -0800 (PST) ThomasB lt;toba0211@gt; wrote: not sure we are using the same script. With osmosis --read-pbf file=/home/wicking/osm/germany.osm.pbf --write-xml file=- | perl osm-length-2.pl /dev/null I am 15 minutes after the start at 14GB RAM usage and still increasing. Yes, the osm-length perl script wastes a lot of memory by keeping all nodes in a Perl hash. It would be relatively easy to write an efficient road length calculator based on the Osmium framework; that one should then work with a far smaller memory footprint. Bye Frederik ___ talk mailing list talk@ http://lists.openstreetmap.org/listinfo/talk -- View this message in context: http://gis.638310.n2.nabble.com/Statistics-on-road-network-length-tp759p7011662.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Jochen Topf joc...@remote.org http://www.remote.org/jochen/ +49-721-388298 ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
!i! wrote: Well even if it's bad style to ask if someone else can do the job, but couldn't we create an addon for existing statistic services, that can calculate this values? I contacted the people behind http://osmstats.altogetherlost.com but even if they hope to provide some country specific calculations, they say that exactly this value would make to heavy load for their servers. Too bad, cause I think a monthly run would be ok for everybody :/ Matthias I guess there are not so many people out there having a machine with more than 20GB RAM. I tried to calculate Germany with a 2009 planet file but failed, I guess because of RAM. So calculating large countries with 2011 data may require RAM well in excess of 20GB. Smaller countries work fine. -- View this message in context: http://gis.638310.n2.nabble.com/Statistics-on-road-network-length-tp759p7009261.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
On 18/11/2011 19:20, ThomasB wrote: I guess there are not so many people out there having a machine with more than 20GB RAM. I tried to calculate Germany with a 2009 planet file but failed, I guess because of RAM. So calculating large countries with 2011 data may require RAM well in excess of 20GB. Smaller countries work fine. One idea could be to use Geofabrik's OSM files by country. My rather crude script needs 2 seconds for processing Albania and 6 minutes each for France and Germany (on a machine with 8G RAM). See below. Hermann Fri Nov 18 19:36:28 CET 2011 +++ albania.osm.pbf Fri Nov 18 19:36:30 CET 2011 +++ andorra.osm.pbf Fri Nov 18 19:36:30 CET 2011 +++ austria.osm.pbf Fri Nov 18 19:37:40 CET 2011 +++ azores.osm.pbf Fri Nov 18 19:37:40 CET 2011 +++ belarus.osm.pbf Fri Nov 18 19:37:52 CET 2011 +++ belgium.osm.pbf Fri Nov 18 19:38:19 CET 2011 +++ bosnia-herzegovina.osm.pbf Fri Nov 18 19:38:23 CET 2011 +++ bulgaria.osm.pbf Fri Nov 18 19:38:29 CET 2011 +++ croatia.osm.pbf Fri Nov 18 19:38:42 CET 2011 +++ cyprus.osm.pbf Fri Nov 18 19:38:45 CET 2011 +++ czech_republic.osm.pbf Fri Nov 18 19:39:37 CET 2011 +++ denmark.osm.pbf Fri Nov 18 19:40:13 CET 2011 +++ estonia.osm.pbf Fri Nov 18 19:40:19 CET 2011 +++ faroe_islands.osm.pbf Fri Nov 18 19:40:19 CET 2011 +++ finland.osm.pbf Fri Nov 18 19:40:54 CET 2011 +++ france.osm.pbf Fri Nov 18 19:46:46 CET 2011 +++ germany.osm.pbf Fri Nov 18 19:53:23 CET 2011 +++ great_britain.osm.pbf Fri Nov 18 19:55:47 CET 2011 +++ greece.osm.pbf Fri Nov 18 19:56:10 CET 2011 +++ hungary.osm.pbf Fri Nov 18 19:56:20 CET 2011 +++ iceland.osm.pbf Fri Nov 18 19:56:24 CET 2011 +++ ireland.osm.pbf Fri Nov 18 19:56:41 CET 2011 +++ isle_of_man.osm.pbf Fri Nov 18 19:56:41 CET 2011 +++ italy.osm.pbf Fri Nov 18 19:59:07 CET 2011 +++ kosovo.osm.pbf Fri Nov 18 19:59:13 CET 2011 +++ latvia.osm.pbf Fri Nov 18 19:59:18 CET 2011 +++ liechtenstein.osm.pbf Fri Nov 18 19:59:18 CET 2011 +++ lithuania.osm.pbf Fri Nov 18 19:59:23 CET 2011 +++ luxembourg.osm.pbf Fri Nov 18 19:59:26 CET 2011 +++ macedonia.osm.pbf Fri Nov 18 19:59:27 CET 2011 +++ malta.osm.pbf Fri Nov 18 19:59:28 CET 2011 +++ moldova.osm.pbf Fri Nov 18 19:59:32 CET 2011 +++ monaco.osm.pbf Fri Nov 18 19:59:32 CET 2011 +++ montenegro.osm.pbf ... ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
Hermann Peifer wrote: One idea could be to use Geofabrik's OSM files by country. My rather crude script needs 2 seconds for processing Albania and 6 minutes each for France and Germany (on a machine with 8G RAM). See below. not sure we are using the same script. With osmosis --read-pbf file=/home/wicking/osm/germany.osm.pbf --write-xml file=- | perl osm-length-2.pl /dev/null I am 15 minutes after the start at 14GB RAM usage and still increasing. -- View this message in context: http://gis.638310.n2.nabble.com/Statistics-on-road-network-length-tp759p7009497.html Sent from the General Discussion mailing list archive at Nabble.com. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
Hi, On Fri, 18 Nov 2011 11:56:53 -0800 (PST) ThomasB toba0...@yahoo.de wrote: not sure we are using the same script. With osmosis --read-pbf file=/home/wicking/osm/germany.osm.pbf --write-xml file=- | perl osm-length-2.pl /dev/null I am 15 minutes after the start at 14GB RAM usage and still increasing. Yes, the osm-length perl script wastes a lot of memory by keeping all nodes in a Perl hash. It would be relatively easy to write an efficient road length calculator based on the Osmium framework; that one should then work with a far smaller memory footprint. Bye Frederik ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
Concerning the issue with dual carriageways, I am using a brute-force approach and divide the calculated distance by 2 if these conditions are true: highway ~ /^(motorway|trunk|primary|secondary|tertiary)/ oneway ~ /yes|true|1/ You probably want to add oneway=-1 to that list. There are some oddballs who used that indicate oneway is against the vector sense. Also JOSM proposes to use it when reversing a way. Jo ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
Am 16. November 2011 12:22 schrieb Matthias Meißer dig...@arcor.de: Hi, currently I looking for some analysis on the length of the OSM road/footway networks. Even this is just a very simple compare, it's a very common way to compare data completeness in professional GIS scene. For Italy there is detailed statistics available on gfoss.it : http://www.gfoss.it/osm/stat/ AFAIK these are done by Diego Guidotti. cheers, Martin ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] Statistics on road network length?
For Italy there is detailed statistics available on gfoss.it : http://www.gfoss.it/osm/stat/ AFAIK these are done by Diego Guidotti. Diego is available to the release of the code :) ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
