Re: [OSM-dev] osm2pgsql slow on update import
Markus Wagner mar...@mwagner.info wrote: After vacuuming and fiddling with those values I am much closer to realtime. Currently at approx ~120% of realtime. So I think, there is hope, once I get faster disks. I'm reopening this thread because I have a very simular Problem and I suspect that this could be an issue of the particular Software versions and/or psql configuration. I'm using Debian squeeze (64 Bit) with postgresql 9.0.3/postgis 1.5.2 (homebrew backport) and I've got a Problem with very slow updates as well. Disk speed can not be the Problem as I'm using a 2-Disk lvm on SSD here. The initial import is very fast (with pbf) and still fast (with .osm.bz2). It is just updates which are very slow here. Two differences to the standard setup are a hstore column and that my database is using lat/long instead of sperical mercator. osm2pgsql is a very recent version from svn (25689) including a few non invasive minor changes to the hstore support which I added myself. Of course these can not be the cause of the problem ;) The Machine I'm using is quite fast (8 cores) and should have enough RAM (32GB). Here is what postgresql.conf looks like: --cut-- data_directory = '/var/lib/postgresql/9.0/main' hba_file = '/etc/postgresql/9.0/main/pg_hba.conf' ident_file = '/etc/postgresql/9.0/main/pg_ident.conf' external_pid_file = '/var/run/postgresql/9.0-main.pid' listen_addresses = '*' port = 5432 max_connections = 80 unix_socket_directory = '/var/run/postgresql' ssl = true shared_buffers = 7680MB work_mem = 192MB maintenance_work_mem = 1GB fsync = off synchronous_commit = off wal_buffers = 8MB checkpoint_segments = 16 checkpoint_completion_target = 0.9 effective_cache_size = 22GB default_statistics_target = 50 constraint_exclusion = on log_line_prefix = '%t ' autovacuum = off datestyle = 'iso, dmy' lc_messages = 'de_DE.UTF-8' lc_monetary = 'de_DE.UTF-8' lc_numeric = 'C' lc_time = 'de_DE.UTF-8' default_text_search_config = 'pg_catalog.german' --cut-- I still suspect that there is some setting here which I should change to speed this up. For some strange reason it would currently be fastest to use complete reimports of the planet rather that updates :( Regards Sven -- We don't know the OS that God uses, but the Vatican uses Linux (Sister Judith Zoebelein, Vatican Webmaster) /me is giggls@ircnet, http://sven.gegg.us/ on the Web ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and other solutions
Serge Wroclawski emac...@gmail.com wrote: Yes, but they are huge... My original place=* for france is 50 MB (not compressed). 50 meg doesn't seem so big. Yes 50MB is fine. 2 GB (compressed) is huge. France extract if 2 GB compressed, i don't think my python script was able to handle such a huge data. What kind of processing were you doing? I'm assuming some kind of XML (unless you were using the protobuffs). If thats the case, you can use a simple SAX parser. If you need an example, I'll send you one. Yes i preprocess XML to extract city data (place=town, place=village, etc...) to match them with an official source with population data. For that i uses elementTree lib (python) that's fine. I don't know about SAX, but i just see that contrary to DOM technique it do not need to load the whole XML into memory, so it can be helpful if i swtich to huge XML data file. I'll check. -- Pierre-Alain Dorange OSM experiences : http://www.leretourdelautruche.com/map/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and other solutions
On 28.04.2011, at 13:57, Pierre-Alain Dorange wrote: What kind of processing were you doing? I'm assuming some kind of XML (unless you were using the protobuffs). If thats the case, you can use a simple SAX parser. If you need an example, I'll send you one. Yes i preprocess XML to extract city data (place=town, place=village, etc...) to match them with an official source with population data. For that i uses elementTree lib (python) that's fine. I don't know about SAX, but i just see that contrary to DOM technique it do not need to load the whole XML into memory, so it can be helpful if i swtich to huge XML data file. I'll check. If you are using Python, you should use iterparse from the ElementTree API, it is much nicer and faster. Here is a complete OSM parser from an earlier version of Imposm: http://pastie.org/1843106 It just 40 lines of code and can parse arbitrary large OSM files. Regards, Oliver -- Oliver Tonnhofer| Omniscale GmbH Co KG| http://omniscale.de http://mapproxy.org | https://bitbucket.org/olt | @oltonn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and other solutions
On 28.04.2011, at 14:46, Serge Wroclawski wrote: On Thu, Apr 28, 2011 at 8:30 AM, Oliver Tonnhofer o...@omniscale.de wrote: If you are using Python, you should use iterparse from the ElementTree API, it is much nicer and faster. Faster than SAX? I'm not so sure. SAX calls the C directly. Here is a benchmark: http://effbot.org/zone/celementtree.htm Regards, Oliver -- Oliver Tonnhofer| Omniscale GmbH Co KG| http://omniscale.de http://mapproxy.org | https://bitbucket.org/olt | @oltonn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and other solutions
On 28.04.2011, at 16:00, Martijn van Exel wrote: Interesting, this totally uproots my firm conviction that SAX parsing is always more time- and memory-efficient than tree/DOM-based parsing. Sure. DOM based parser need to keep the whole tree in memory which doesn't work well with large XML files. SAX and other stream based parser (like iterparse for Python) do not keep any information in memory. Stream based parser require more work from the developer, you can't use XPath expressions for example. So DOM based parsing still makes sense for documents with a known size, but for OSM data SAX/iterparse is the way to go. The article is 6 years old, do the benchmarks still hold in your experience? Yes, for iterparse is still faster than xml.sax for Python. Regards, Oliver -- Oliver Tonnhofer| Omniscale GmbH Co KG| http://omniscale.de http://mapproxy.org | https://bitbucket.org/olt | @oltonn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] XAPI and other solutions
Am 28.04.2011 16:11, schrieb Oliver Tonnhofer: On 28.04.2011, at 16:00, Martijn van Exel wrote: Interesting, this totally uproots my firm conviction that SAX parsing is always more time- and memory-efficient than tree/DOM-based parsing. From my experience the expat-sax immplementation is faster then the libxml sax-parser. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] Which seperator is used for keyborad shortcuts in wiki/josm-jar
Am 26.04.2011 16:35, schrieb Dirk Stöcker: On Tue, 26 Apr 2011, colliar wrote: I have found different seperators for keyboard shortcuts in JOSM. * On the wiki pages mainly - is used. * In JOSM menus a wild mixture of + and - is used. In the menus is no wild mixture, but: * control-keys are added with + * keys are appended with - Means: Ctrl+Alt-U and Alt-U or Ctrl-U. I don't know whether this is Java standard or produced by Shortcut.java class. At least we have only two systems, but which one to use ? We should state our decision on the wiki. cu ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] How to present results of my conflation analysis
I've implemented the algorithmic part of a conflation tool in a JOSM plugin, pretty much following what I wrote down here: http://wiki.openstreetmap.org/wiki/Conflation/Nodes Right now I'm just using the Euclidean distance in the cost function, but we can of course add other costs like dissimilarity between key values. My big question now is how to present the results to the user, and to allow them to manually confirm/reject matches. All I've done so far is to create a new layer which draws arrows connecting the matched nodes/ways. However that's only a useful visualization tool but doesn't manage the actual conflation process. You can see the option dialog and arrow layer here: http://tinypic.com/r/23sw0u8/7 http://tinypic.com/r/jhd3j7/7 The first question is whether this tool should only merge data from one layer to another, or if it should allow syncing (i.e. bidirectional merging) between layers. The former is of course easier, and can possibly take advantage of existing functionality in JOSM, namely the conflict resolution tools. In fact I've added conflicts which appear in the conflict dialog when the Conflation layer is selected. Some things that are missing with that approach is to highlight the arrow connecting the matched nodes, and being able to select the arrow to conflate/resolve the pair of primitives. Also this conflation approach is slightly different from conflict resolution since I'm mostly concerned about resolving tags, and possibly center locations. Right now the conflict tools only support resolving conflicts between like primitives, i.e. Node-Node, however I need to allow any combination of Node, Way, or Relation. In other words I may want to match a GNIS feature node (from a fresh GNIS data set) to an existing feature in OSM that may be a node, area, or multipolygon. I'm not so much concerned about the geometry but rather the tags. Conflating geometry is much more complicated and is a future project. :) Any ideas would be much appreciated! -Josh ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] How to present results of my conflation analysis
On Thu, 28 Apr 2011, Josh Doe wrote: Any ideas would be much appreciated! A long term plan is to use the mapview also to view subsets of the map. So we could have e.g. a dialog with two small sections, each showing a partial map. One map shows the old state, one map the new state. This would improve conflict resolution a lot and probably also be the solution for your tasks. If you would like to implement this ... Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
