Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
> > There are different schools on the topic. I know Frank is not too keen on > mentionning versionning, although personnaly, I try to mention version > differences in the format page itself (http://www.gdal.org/ogr/drv_osm.html > mentions GDAL/OGR >= 1.10.0). But at the end, in drivers with a lot of > activity, > it becomes cluttered with mentions like "in version X, in version Y". Having > different version of the documentation, for each version, on the server could > solve that, although it is sometimes usefull to identify quickly that a > particular feature has been introduced in a version. > >> Yes, I know that situation all too well (same issues with mapserver.org). I agree, mentioning GDAL/OGR version in the format page itself is very good (I push for this in mapserver.org as well, as it is very helpful for users to know what version a feature was added). K thanks for chatting. -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Selon Jeff McKenna : > On 13-01-08 9:26 PM, Even Rouault wrote: > > Ah ok, so I must mention that the online documentation is always up-to-date > > with the latest trunk version (it is refreshed each night). So the fact > that > > something is documented is not a sign of stability by itself. It can be a > new > > development committed just a few hours ago. > > Hello Even, Stefan, > > I wonder if that ogr_formats.html table needs a new column for "GDAL > Version", that could let users know what GDAL version works with each > driver. There are different schools on the topic. I know Frank is not too keen on mentionning versionning, although personnaly, I try to mention version differences in the format page itself (http://www.gdal.org/ogr/drv_osm.html mentions GDAL/OGR >= 1.10.0). But at the end, in drivers with a lot of activity, it becomes cluttered with mentions like "in version X, in version Y". Having different version of the documentation, for each version, on the server could solve that, although it is sometimes usefull to identify quickly that a particular feature has been introduced in a version. > > -jeff > > > > -- > Jeff McKenna > MapServer Consulting and Training Services > http://www.gatewaygeomatics.com/ > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
On 13-01-08 9:26 PM, Even Rouault wrote: > Ah ok, so I must mention that the online documentation is always up-to-date > with the latest trunk version (it is refreshed each night). So the fact that > something is documented is not a sign of stability by itself. It can be a new > development committed just a few hours ago. Hello Even, Stefan, I wonder if that ogr_formats.html table needs a new column for "GDAL Version", that could let users know what GDAL version works with each driver. -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Le mercredi 09 janvier 2013 02:11:21, Stefan Keller a écrit : > Even, > > Thanks for the fast response! > > 2013/1/9 Even Rouault : > > (Just curious what in the doc makes you call it "stable", and what you > > consider as "stable") > > Just the fact that it's there and especially that is't also in the > overview table alongside with all other (stable) drivers > http://www.gdal.org/ogr/ogr_formats.html . Ah ok, so I must mention that the online documentation is always up-to-date with the latest trunk version (it is refreshed each night). So the fact that something is documented is not a sign of stability by itself. It can be a new development committed just a few hours ago. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even, Thanks for the fast response! 2013/1/9 Even Rouault : > (Just curious what in the doc makes you call it "stable", and what you > consider as "stable") Just the fact that it's there and especially that is't also in the overview table alongside with all other (stable) drivers http://www.gdal.org/ogr/ogr_formats.html . Yours, Stefan 2013/1/9 Even Rouault : > Le mercredi 09 janvier 2013 01:02:22, Stefan Keller a écrit : >> Hi Even >> >> Documentation of OpenStreetMap driver suggest its stable > > (Just curious what in the doc makes you call it "stable", and what you > consider as "stable") > >> - but when I >> recently installed the newest OGR version on a linux it was'nt there. >> Is it not yet stable enough? >> If not, what do you recommend? >> Should I include the tagged version (currently 1.9.2) and add the >> trunk version (subdirectory) of the OpenStreetMap driver? > > Stephan, > > The OSM driver is only available in the trunk version (definitely not in the > 1.9.X branch which is in maintenance mode only), the future 1.10 to be > released (hopefully in the following weeks). > > Copying the OSM directory from trunk into 1.9 will work. It won't compile > since there are a few services in port/ that have been added in trunk and are > needed by the OSM driver. > > So if you are interested in the OSM driver, I suggest that you just check out > and compile the trunk and use it. It passes currently the autotest regresssion > suite on both Linux and Windows, so I consider it stable enough for wide > testing. And we definitely need beta testers for the release anyway ! > > Best regards, > > Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Le mercredi 09 janvier 2013 01:02:22, Stefan Keller a écrit : > Hi Even > > Documentation of OpenStreetMap driver suggest its stable (Just curious what in the doc makes you call it "stable", and what you consider as "stable") > - but when I > recently installed the newest OGR version on a linux it was'nt there. > Is it not yet stable enough? > If not, what do you recommend? > Should I include the tagged version (currently 1.9.2) and add the > trunk version (subdirectory) of the OpenStreetMap driver? Stephan, The OSM driver is only available in the trunk version (definitely not in the 1.9.X branch which is in maintenance mode only), the future 1.10 to be released (hopefully in the following weeks). Copying the OSM directory from trunk into 1.9 will work. It won't compile since there are a few services in port/ that have been added in trunk and are needed by the OSM driver. So if you are interested in the OSM driver, I suggest that you just check out and compile the trunk and use it. It passes currently the autotest regresssion suite on both Linux and Windows, so I consider it stable enough for wide testing. And we definitely need beta testers for the release anyway ! Best regards, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Hi Even Documentation of OpenStreetMap driver suggest its stable - but when I recently installed the newest OGR version on a linux it was'nt there. Is it not yet stable enough? If not, what do you recommend? Should I include the tagged version (currently 1.9.2) and add the trunk version (subdirectory) of the OpenStreetMap driver? Yours, Stefan 2012/10/24 Even Rouault : > Le mercredi 24 octobre 2012 09:57:58, Frank Broniewski a écrit : >> Hi, >> >> I had this up my schedule for this week, but I see I need GDAL >= 2.0. >> Where might I get this? From svn (https://svn.osgeo.org/gdal/trunk/gdal)? > > Yes, svn trunk = GDAL 2.0dev. > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Le mercredi 24 octobre 2012 09:57:58, Frank Broniewski a écrit : > Hi, > > I had this up my schedule for this week, but I see I need GDAL >= 2.0. > Where might I get this? From svn (https://svn.osgeo.org/gdal/trunk/gdal)? Yes, svn trunk = GDAL 2.0dev. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Hi, I had this up my schedule for this week, but I see I need GDAL >= 2.0. Where might I get this? From svn (https://svn.osgeo.org/gdal/trunk/gdal)? Thanks, Frank Am 2012-10-17 11:54, schrieb Even Rouault: Selon Frank Broniewski : Hi Even, I need to reinstall my OSM database due to the license change to ODBL. Usually I use osm2pgsql for that, but I am willing to sacrifice a little downtime of my DB in order to test the GDAL implementation. Before storming ahead I wanted to know how far you are with the driver implementation - is there anything I need to be aware of? Apart reading carefully the documentation of the drivers at http://www.gdal.org/ogr/drv_osm.html and http://www.gdal.org/ogr/drv_pg.html to be aware of the various optimization hints, nothing comes to find. There have been several optimization passes and improvements done with Jukka's feedback in the last few months, so now I consider the gist of the developmement phase to be done. Are you interested at all in a benchmark comparison between GDAL and osm2pgsql? I'll be using the planet file, so thinks will take a while ... Yes, sure. Feedback welcome. Note that the driver hasn't specifically been developed to compete with osm2pgsql (for example, the OGR OSM driver, or more exactly the OGR model, cannot deal with OSM diff files), but more to address the capability of reading/converting "modest" sized .osm/.pbf files with little "infrastructure". Even -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Selon Frank Broniewski : > Hi Even, > > I need to reinstall my OSM database due to the license change to ODBL. > Usually I use osm2pgsql for that, but I am willing to sacrifice a little > downtime of my DB in order to test the GDAL implementation. Before > storming ahead I wanted to know how far you are with the driver > implementation - is there anything I need to be aware of? Apart reading carefully the documentation of the drivers at http://www.gdal.org/ogr/drv_osm.html and http://www.gdal.org/ogr/drv_pg.html to be aware of the various optimization hints, nothing comes to find. There have been several optimization passes and improvements done with Jukka's feedback in the last few months, so now I consider the gist of the developmement phase to be done. > Are you > interested at all in a benchmark comparison between GDAL and osm2pgsql? > I'll be using the planet file, so thinks will take a while ... Yes, sure. Feedback welcome. Note that the driver hasn't specifically been developed to compete with osm2pgsql (for example, the OGR OSM driver, or more exactly the OGR model, cannot deal with OSM diff files), but more to address the capability of reading/converting "modest" sized .osm/.pbf files with little "infrastructure". Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Hi Even, I need to reinstall my OSM database due to the license change to ODBL. Usually I use osm2pgsql for that, but I am willing to sacrifice a little downtime of my DB in order to test the GDAL implementation. Before storming ahead I wanted to know how far you are with the driver implementation - is there anything I need to be aware of? Are you interested at all in a benchmark comparison between GDAL and osm2pgsql? I'll be using the planet file, so thinks will take a while ... Frank Am 2012-07-10 19:23, schrieb Even Rouault: Hi, Following the recent brainstorming with Jukka, I've pushed into trunk a driver to read OpenStreetMap .osm / .pbf files . No particularly exotic dependencies : SQLite (and Expat for OSM XML files) See http://www.gdal.org/ogr/drv_osm.html for the details (will be available in a few hours). The performance to convert http://download.geofabrik.de/osm/europe/finland.osm.pbf into a Spatialite DB is the following one on my PC (Core i5 @ 2.67 GHz with 64bit GDAL) : $ time ogr2ogr finland.sqlite finland.osm.pbf -f SQLite -dsco SPATIALITE=YES -gt 1 -progress --config OGR_SQLITE_SYNCHRONOUS OFF real4m31.194s user3m33.020s sys 0m46.070s Testing with larger areas, like whole France or Europe, shows sluggish performance when ways are built from nodes, but that's perhaps expected. I didn't compare with other tools to know if the indexing or request strategy is particularly bad. The data/osmconf.ini configuration file is pretty basic and its settings could likely be improved with some tweaking. Contributions welcome. An improved version of the driver could allow specifying custom layers, instead of the 4 fixed ones. Happy testing, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Frank BRONIEWSKI METRICO s.à r.l. géomètres technologies d'information géographique rue des Romains 36 L-5433 NIEDERDONVEN tél.: +352 26 74 94 - 28 fax.: +352 26 74 94 99 http://www.metrico.lu ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault wrote: 23. heinäkuuta 2012 13:27 > Le lundi 23 juillet 2012 00:09:05, Jukka Rahkonen a écrit : >> Even Rouault mines-paris.org> writes: >> > > > However, select with SQL feels sub-optimal. >> > > >> > > Yes, when you use ogr2ogr with explicit layer names, there are >> > > optimizations. For example, when you only specify the layer 'points', >> > > the OSM driver will not even try to index the nodes into the temporary >> > > database because it is not needed. However, as you noticed, there is >> > > not yet any optimization when a SQL request is specified. >> > >> > Optimization for SQL request added in r24690 > >> I had a try with r24696 today, downloaded from >> http://gisinternals.com/sdk/Download.aspx?file=release-1500-gdal-mapserver. >> zip >> >> Filtered commands give me errors. An example: >> ogr2ogr -f "ESRI Shapefile" test germany.osm.pbf multipolygons -gt 2 >> -progress --config OGR_SQLITE_SYNCHRONOUS OFF -where "natural='forest'" >> >> ERROR 1: Failed inserting node 420797898: database schema has changed > I could also reproduce and I suppose there was at the begnning of the sequence > of errors : "ERROR 1: Failed inserting node : I/O error" Most probably yes but I saw only the last lines of the first thousand which were printed on the screen. > It turned out that the mechanism to transfer the temporary in-memory DB file > to > disk when it is too big had been broken by a previous commit, so it stayed on > RAM and at some point, it couldn't fit in RAM, hence the error. Should be > fixed > now with r24699. > I'm experimenting with removing the internal use of SQLite for the temporary > database and replacing it with something custom. Actually, it won't replace it > completely in all cases, but it could definitely be used in well-behaved cases > where the elements in the .osm/.pbf are listed in increasing id order, which > is the case of the data in geofabrik files for example. The first results seem > to show increased performance. > Note: in your above example, you don't need to specify -gt and --config > OGR_SQLITE_SYNCHRONOUS OFF when the output format is not sqlite/spatialite. > And the internal use of SQLite in the OSM driver already sets the > corresponding parameters to the values that give the best performance. Sure true with extra parametes, I just forgot to remove them when doing a quick verification for seeing if the error occurs similarly for spatialite and shapefile outputs. -Jukka- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Le lundi 23 juillet 2012 00:09:05, Jukka Rahkonen a écrit : > Even Rouault mines-paris.org> writes: > > > > However, select with SQL feels sub-optimal. > > > > > > Yes, when you use ogr2ogr with explicit layer names, there are > > > optimizations. For example, when you only specify the layer 'points', > > > the OSM driver will not even try to index the nodes into the temporary > > > database because it is not needed. However, as you noticed, there is > > > not yet any optimization when a SQL request is specified. > > > > Optimization for SQL request added in r24690 > > I had a try with r24696 today, downloaded from > http://gisinternals.com/sdk/Download.aspx?file=release-1500-gdal-mapserver. > zip > > Filtered commands give me errors. An example: > ogr2ogr -f "ESRI Shapefile" test germany.osm.pbf multipolygons -gt 2 > -progress --config OGR_SQLITE_SYNCHRONOUS OFF -where "natural='forest'" > > ERROR 1: Failed inserting node 420797898: database schema has changed I could also reproduce and I suppose there was at the begnning of the sequence of errors : "ERROR 1: Failed inserting node : I/O error" It turned out that the mechanism to transfer the temporary in-memory DB file to disk when it is too big had been broken by a previous commit, so it stayed on RAM and at some point, it couldn't fit in RAM, hence the error. Should be fixed now with r24699. I'm experimenting with removing the internal use of SQLite for the temporary database and replacing it with something custom. Actually, it won't replace it completely in all cases, but it could definitely be used in well-behaved cases where the elements in the .osm/.pbf are listed in increasing id order, which is the case of the data in geofabrik files for example. The first results seem to show increased performance. Note: in your above example, you don't need to specify -gt and --config OGR_SQLITE_SYNCHRONOUS OFF when the output format is not sqlite/spatialite. And the internal use of SQLite in the OSM driver already sets the corresponding parameters to the values that give the best performance. > > Same error with Spatialite output. > > -Jukka Rahkonen- > > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault mines-paris.org> writes: > > > > > > However, select with SQL feels sub-optimal. > > > > Yes, when you use ogr2ogr with explicit layer names, there are > > optimizations. For example, when you only specify the layer 'points', the > > OSM driver will not even try to index the nodes into the temporary > > database because it is not needed. However, as you noticed, there is not > > yet any optimization when a SQL request is specified. > > > > Optimization for SQL request added in r24690 I had a try with r24696 today, downloaded from http://gisinternals.com/sdk/Download.aspx?file=release-1500-gdal-mapserver.zip Filtered commands give me errors. An example: ogr2ogr -f "ESRI Shapefile" test germany.osm.pbf multipolygons -gt 2 -progress --config OGR_SQLITE_SYNCHRONOUS OFF -where "natural='forest'" ERROR 1: Failed inserting node 420797898: database schema has changed Same error with Spatialite output. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
> > However, select with SQL feels sub-optimal. > > Yes, when you use ogr2ogr with explicit layer names, there are > optimizations. For example, when you only specify the layer 'points', the > OSM driver will not even try to index the nodes into the temporary > database because it is not needed. However, as you noticed, there is not > yet any optimization when a SQL request is specified. > Optimization for SQL request added in r24690 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Rahkonen Jukka mmmtike.fi> writes: > > > Even Rouault wrote: > > > Do you have comparisons of the performance with osm2pgsql on the same PC and > > with the same data ? I'd be curious if that slow down effect is found with every > > tool, or if it is something specific to the way sqlite is used, or if other > > tools do more clever things when indexing or retrieving nodes. > This test may be interesting. I took timings from converting finland.osm.pbf and germany.osm.pbf with spatialite_osm_map. It is doing roughly same thing as ogr2ogr, that is, reads nodes from OSM file and resolves point, line and polygon features. Enviroment: Windows 7, 64-bit. 6 GB memory 64-bit spatialite_osm_map.exe, version numbers: SQLite version: 3.7.13 SpatiaLite version: 3.1.0-RC2 Command: spatialite_osm_map -d osm_map_testfi.sqlite -o finland.osm.pbf -jo Results: Finland 11 minutes Germany 10 hours Memory usage of spatialite_osm_map process was very low, under 50 MB. Similar behaviour of ogr2ogr and spatialite_osm_map makes me think that perhaps it is SQLite that does not scale up well. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault wrote: >> Windows 7, 64-bit, SATA disk and 2x3 GHz is converting Finland.osm.pbf in >> about >> 8 minutes for me. But execution time does not increase at all linearly. >> Germany.osm.pbf is about 10 times larger in filesize but ogr2ogr had to work >> about 8 hours with it, thus roughly 70 times longer. Obviously I would save 6 >> hours by splitting the German pbf file into 10 smaller ones, running ogr2ogr >> ten >> times and combining results with ogrtileindex for use with Mapserver. > I suppose there is a big discontinuity at some point. While the temporary > database can fit into RAM, and then in the I/O cache of the operating system, > the performance must be reasonably good. But when it grows above, you'll get > disk access for almost every way and this will be sluggish. Test with Germany.osm.pbf was more sluggish than I thought. Process was not complete and and finally it took 15 hours to run. Solving the relations must have been heavy for ogr2ogr. > Do you have comparisons of the performance with osm2pgsql on the same PC and > with the same data ? I'd be curious if that slow down effect is found with > every > tool, or if it is something specific to the way sqlite is used, or if other > tools do more clever things when indexing or retrieving nodes. I fear that my comparisons would not give very much information. Osm2pgsql is not at all optimised for this kind of task. I would need to run it in a slim mode and then osm2pgsql is doing whole lot of things for preparing database to accept updates with the diff files. I suppose that osm2pgsql could be much faster if it had some kind of "slim, with no diff-support" mode. A great part of the job is also performed by PostgreSQL and db parameters seem to have a big influence. >From my own experience I can say that osm2pgsql gets very slow and unrieliable on weak computers. Linux box with 700 MB of memory cannot import finland.osm extract at all, and with my Window laptop with 2 GB it takes two or three hours. Ogr2ogr on the same machine did the conversion in 40 minutes. Let's hope that some OSM developer gets interested in this task. > I suppose splitting could help, but I don't see an obvious reason why an > intelligent splitting would be faster. I mention "intelligent" because you can > do a rough splitting based on bounding box for example, but this will lead to > ways that have unresolved nodes. I forgot how much resolving there is in OSM format. Compared to that it is ridiculously simple to read already resolved OSM features from a WFS service http://188.64.1.61/cgi-bin/tinyows?service=WFS&version=1.0.0&request=GetFeature&typeName=lv:osm_polygon&maxfeatures=20 -Jukka- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault wrote: >> Amenity is included in my osmconf.ini and it can be used inside -sql. >> However, your example gives me this >> >> ERROR 1: 'amenity' not recognised as an available field. >> FAILURE: SetAttributeFilter(amenity='toilets') failed. > Hum, I think I know what's wrong. I see that, even if a subset of layers is > specified, the attribute filter is applied on all layers, so I suppose that > one > of the lines, polygons or multipolygons layers has no amenity field, right ? > You > could workaround the error by editing osmconf.ini to add that field in all > layer > definitions. But this should be fixed : would you mind opening a ticket about > that ? Thanks Adding "amenity" for all the layers really made filter to work and it is fast. Ticket #4751 created. -Jukka- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
> > Windows 7, 64-bit, SATA disk and 2x3 GHz is converting Finland.osm.pbf in > about > 8 minutes for me. But execution time does not increase at all linearly. > Germany.osm.pbf is about 10 times larger in filesize but ogr2ogr had to work > about 8 hours with it, thus roughly 70 times longer. Obviously I would save 6 > hours by splitting the German pbf file into 10 smaller ones, running ogr2ogr > ten > times and combining results with ogrtileindex for use with Mapserver. I suppose there is a big discontinuity at some point. While the temporary database can fit into RAM, and then in the I/O cache of the operating system, the performance must be reasonably good. But when it grows above, you'll get disk access for almost every way and this will be sluggish. Do you have comparisons of the performance with osm2pgsql on the same PC and with the same data ? I'd be curious if that slow down effect is found with every tool, or if it is something specific to the way sqlite is used, or if other tools do more clever things when indexing or retrieving nodes. I suppose splitting could help, but I don't see an obvious reason why an intelligent splitting would be faster. I mention "intelligent" because you can do a rough splitting based on bounding box for example, but this will lead to ways that have unresolved nodes. Actually, you could try that by adding a -spat argument to your ogr2ogr command line. Nodes that fall outside of the spatial filter are not added to the temporary database. But of course ways that cross the boundary of the bounding box will be truncated and some relations too. And no guarantee it will speed up the global process, because some time will still be lost trying to resolve the nodes of ways that are outside of the spatial filter. > > -Jukka Rahkonen- > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
> Amenity is included in my osmconf.ini and it can be used inside -sql. > However, your example gives me this > > ERROR 1: 'amenity' not recognised as an available field. > FAILURE: SetAttributeFilter(amenity='toilets') failed. Hum, I think I know what's wrong. I see that, even if a subset of layers is specified, the attribute filter is applied on all layers, so I suppose that one of the lines, polygons or multipolygons layers has no amenity field, right ? You could workaround the error by editing osmconf.ini to add that field in all layer definitions. But this should be fixed : would you mind opening a ticket about that ? Thanks > > -Jukka- > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault mines-paris.org> writes: > > Selon Jukka Rahkonen mmmtike.fi>: ... > > > It is > > much faster to convert all the points than select only a part of those. The > > error message suggest that ogr2ogr is inspecting the input file more closely > > than necessary for this use case: > > > > ogr2ogr -f "ESRI Shapefile" poitest.shp finland.osm.pbf > > -sql "select name, amenity from points where amenity='toilets'" > > > > ERROR 1: Too many features have accumulated in lines layer. Use > > OGR_INTERLEAVED_ > > READING=YES mode > > You can easily transform the above SQL into something equivalent that will > benefit from the optimization : > > ogr2ogr poitest.shp finland.osm.pbf points -select name,amenity -where > "amenity='toilets'"" Amenity is included in my osmconf.ini and it can be used inside -sql. However, your example gives me this ERROR 1: 'amenity' not recognised as an available field. FAILURE: SetAttributeFilter(amenity='toilets') failed. -Jukka- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault mines-paris.org> writes: > > Hi, > > Following the recent brainstorming with Jukka, I've pushed into trunk a > driver > to read OpenStreetMap .osm / .pbf files . . > > Testing with larger areas, like whole France or Europe, shows sluggish > performance when ways are built from nodes, but that's perhaps expected. I > didn't compare with other tools to know if the indexing or request strategy > is > particularly bad. Windows 7, 64-bit, SATA disk and 2x3 GHz is converting Finland.osm.pbf in about 8 minutes for me. But execution time does not increase at all linearly. Germany.osm.pbf is about 10 times larger in filesize but ogr2ogr had to work about 8 hours with it, thus roughly 70 times longer. Obviously I would save 6 hours by splitting the German pbf file into 10 smaller ones, running ogr2ogr ten times and combining results with ogrtileindex for use with Mapserver. -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
> Even Rouault mines-paris.org> writes: >> Following the recent brainstorming with Jukka, I've pushed into trunk a >> driver >> to read OpenStreetMap .osm / .pbf files . > Fascinating. Actually I think once someone imports the points, lines, polys into a DB, then (cool!) someone could then run the SQL script that Mike Smith and I made recently (https://github.com/mapserver/basemaps/blob/master/contrib/osm2pgsql-to-imposm-schema.sql) that created generalized and specific tables, and then, could even (if that someone wanted to publish through MapServer) run those tables through the MapServer/basemaps utility to generate the magical mapfile (https://github.com/mapserver/basemaps) -jeff -- Jeff McKenna MapServer Consulting and Training Services http://www.gatewaygeomatics.com/ ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Selon Jukka Rahkonen : > Even Rouault mines-paris.org> writes: > > > > > Hi, > > > > Following the recent brainstorming with Jukka, I've pushed into trunk a > driver > > to read OpenStreetMap .osm / .pbf files . > > Driver seems to do what it promises. It is super fast in converting POI data > because it is possible to skip the slower layers (lines, polygons and > especially > the tricky multipolygons). Speed with my slow Windows laptop is at least > 30 > converted POIs per minute. However, select with SQL feels sub-optimal. Yes, when you use ogr2ogr with explicit layer names, there are optimizations. For example, when you only specify the layer 'points', the OSM driver will not even try to index the nodes into the temporary database because it is not needed. However, as you noticed, there is not yet any optimization when a SQL request is specified. > It is > much faster to convert all the points than select only a part of those. The > error message suggest that ogr2ogr is inspecting the input file more closely > than necessary for this use case: > > ogr2ogr -f "ESRI Shapefile" poitest.shp finland.osm.pbf > -sql "select name, amenity from points where amenity='toilets'" > > ERROR 1: Too many features have accumulated in lines layer. Use > OGR_INTERLEAVED_ > READING=YES mode You can easily transform the above SQL into something equivalent that will benefit from the optimization : ogr2ogr poitest.shp finland.osm.pbf points -select name,amenity -where "amenity='toilets'"" > > -Jukka Rahkonen- > > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Even Rouault mines-paris.org> writes: > > Hi, > > Following the recent brainstorming with Jukka, I've pushed into trunk a > driver > to read OpenStreetMap .osm / .pbf files . Driver seems to do what it promises. It is super fast in converting POI data because it is possible to skip the slower layers (lines, polygons and especially the tricky multipolygons). Speed with my slow Windows laptop is at least 30 converted POIs per minute. However, select with SQL feels sub-optimal. It is much faster to convert all the points than select only a part of those. The error message suggest that ogr2ogr is inspecting the input file more closely than necessary for this use case: ogr2ogr -f "ESRI Shapefile" poitest.shp finland.osm.pbf -sql "select name, amenity from points where amenity='toilets'" ERROR 1: Too many features have accumulated in lines layer. Use OGR_INTERLEAVED_ READING=YES mode -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] New OGR driver to read OpenStreetMap .osm / .pbf files
Hi, Following the recent brainstorming with Jukka, I've pushed into trunk a driver to read OpenStreetMap .osm / .pbf files . No particularly exotic dependencies : SQLite (and Expat for OSM XML files) See http://www.gdal.org/ogr/drv_osm.html for the details (will be available in a few hours). The performance to convert http://download.geofabrik.de/osm/europe/finland.osm.pbf into a Spatialite DB is the following one on my PC (Core i5 @ 2.67 GHz with 64bit GDAL) : $ time ogr2ogr finland.sqlite finland.osm.pbf -f SQLite -dsco SPATIALITE=YES -gt 1 -progress --config OGR_SQLITE_SYNCHRONOUS OFF real4m31.194s user3m33.020s sys 0m46.070s Testing with larger areas, like whole France or Europe, shows sluggish performance when ways are built from nodes, but that's perhaps expected. I didn't compare with other tools to know if the indexing or request strategy is particularly bad. The data/osmconf.ini configuration file is pretty basic and its settings could likely be improved with some tweaking. Contributions welcome. An improved version of the driver could allow specifying custom layers, instead of the 4 fixed ones. Happy testing, Even ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
