Re: [osmosis-dev] Osmosis replication fails
Brett, On Tue, Dec 13, 2011 at 3:20 AM, Brett Henderson br...@bretth.com wrote: On 13 December 2011 06:31, Martijn van Exel m...@rtijn.org wrote: Hi all, I have a replication task set up, initially with a longer interval because my planet file is a few weeks old. I have had it running in a cron job with a two hour interval but I get a lot of errors similar to this one: SEVERE: Thread for task 1-rri failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to parse xml file /tmp/change8301792328184763034.tmp. publicId=(null), systemId=(null), lineNumber=7384, columnNumber=3. Caused by: org.xml.sax.SAXParseException; lineNumber: 746; columnNumber: 3; The element type osmChange must be terminated by the matching end-tag /osmChange. It sounds like the change files are incomplete. Perhaps some of the downloads are failing. Is your network connection usually reliable? Well, it's comcast cable, so no. Out of 50 executions, this error appeared 23 times. I have my replication interval set to one day, could that be the problem? Processing (when it succeeds) takes about 90 minutes. I have the cron job set to execute every two hours. How is your replication configured? Specifically which replication files are you using (ie. minute, hour or day)? It may be worth switching to files with a longer interval if you are patching a file to reduce the number of downloads required. Minute replication files would typically be more suitable to patching a database where small files can be applied quickly. Hmm. I had this set to retrieve the minutely updates. I set it to hourly now and am expecting much better results. Thanks for the pointers, Brett, Martijn -- martijn van exel geospatial omnivore 1109 1st ave #2 salt lake city, ut 84103 801-550-5815 http://oegeo.wordpress.com ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
-Oprindelig meddelelse- Fra: Peter Körner [mailto:osm-li...@mazdermind.de] Sendt: 12. december 2011 21:28 Til: dev@openstreetmap.org Emne: Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT Am 12.12.2011 20:49, schrieb Serge Wroclawski: HOT gets around issues by downloading an area ahead of time and loading it into the editor. Well, you could download the area of interest via the homepage export function as .osm file and place it on a network share / usb stick. The Editors of the mapping party would open that file, do their changes and upload from there. Peter Hi, thanks for the answers, The solution with a local copy sounds interesting, but because of the time available and the average computer skills of the course participants, we teach Potlatch2. Is it possible to do this with Potlatch? /Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
Andreas Hammershøj wrote: Is it possible to do this with Potlatch? Afraid not - P2 doesn't yet have any ability to load local files from disk. I'm currently looking into a couple of possible solution, but because it's Christmas I don't have a whole bunch of time for P2 stuff for a couple of weeks. cheers Richard -- View this message in context: http://gis.638310.n2.nabble.com/IP-limit-issue-when-doing-mapping-parties-on-a-network-with-NAT-tp7086366p7100086.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSM-3D tagging info
Hi everyone: My name is Ander Pijoan and I'm a student at the University of Deusto. We are working on an application to convert data from a goverment database called Catastro (which has been openned to all users little time ago) to osm format in order to do massive imports. This database contains all the urban and rustical info about landuse, buildings... and lots of interesting things to add like sub_stations, amenities or even streetlamps and trees. Here is the wiki where you can follow the project: http://wiki.openstreetmap.org/wiki/Spanish_Catastro For big cities maybe there is stuff that is not necessary to import but for small villages its a great improvement. The conversion part is almost ready but we have one bit missing, the building tagging for 3d renderers to render propperly. We've seen you will have an OSM Dev weekend but we don't know if we will be able to attend. Example of current data stored in OSM: http://i.imgur.com/vComo.png Data we can import from Catastro: http://i.imgur.com/kqCex.jpg The tags on the wiki related to 3D rendering are quite reduced for all the information Catastro database gives. We have information about part of a building are balconies, terraces, porchs... http://i.imgur.com/i6LJ5.png So our question is if there are some more tags already standarized or recommended for this data, it is expected to have some or we should make some operations to calculate the shape it should draw on 3d and type it like min-height and height. Thanks a lot, Cheers. -- Ander Pijoan Lamas Ingeniero Técnico en Informática de Gestión Universidad de Deusto Contacto: Email: ander.pij...@deusto.es Móvil: +34 664471228 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM-3D tagging info
3d tagging is not widely supported. I think itd be of questionable value. What software are you using to convert, and whats the license of the data? Just a reminder, dont forget to follow the import guidelines, which includes discussing with the local community and impo...@openstreetmap.org. From: Ander Pijoan [mailto:ander.pij...@deusto.es] Sent: Friday, December 16, 2011 1:10 AM To: dev@openstreetmap.org Subject: [OSM-dev] OSM-3D tagging info Hi everyone: My name is Ander Pijoan and I'm a student at the University of Deusto. We are working on an application to convert data from a goverment database called Catastro (which has been openned to all users little time ago) to osm format in order to do massive imports. This database contains all the urban and rustical info about landuse, buildings... and lots of interesting things to add like sub_stations, amenities or even streetlamps and trees. Here is the wiki where you can follow the project: http://wiki.openstreetmap.org/wiki/Spanish_Catastro For big cities maybe there is stuff that is not necessary to import but for small villages its a great improvement. The conversion part is almost ready but we have one bit missing, the building tagging for 3d renderers to render propperly. We've seen you will have an OSM Dev weekend but we don't know if we will be able to attend. Example of current data stored in OSM: http://i.imgur.com/vComo.png Data we can import from Catastro: http://i.imgur.com/kqCex.jpg The tags on the wiki related to 3D rendering are quite reduced for all the information Catastro database gives. We have information about part of a building are balconies, terraces, porchs... http://i.imgur.com/i6LJ5.png So our question is if there are some more tags already standarized or recommended for this data, it is expected to have some or we should make some operations to calculate the shape it should draw on 3d and type it like min-height and height. Thanks a lot, Cheers. -- Ander Pijoan Lamas Ingeniero Técnico en Informática de Gestión Universidad de Deusto Contacto: Email: mailto:ander.pij...@deusto.es ander.pij...@deusto.es Móvil: +34 664471228 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM-3D tagging info
Hi Ander, the data you have looks great and it certainly would be a huge improvement, particularly in the 3D context. As you've already recognised, the 3D development in OSM is still in its early stages and tagging schemas are not yet established but still lively discussed. The most extensive proposal for tag-based building and roof modelling is made here: http://wiki.openstreetmap.org/wiki/Roof_table http://wiki.openstreetmap.org/wiki/DE:Roof_table (German) http://wiki.openstreetmap.org/wiki/3D_building You can also see what kind of different 3D OSM viewers exist so far and which tags are processed by them: http://wiki.openstreetmap.org/wiki/3D_Development/Tagging At the moment it is still very difficult to manually apply all these tags and it is also currently not recommended to do that since there is no consensus if this is the right way. However, it would already be great if you just included basic tags like the building height, the general roof type, number of floors etc. We will discuss and try to push forward this development during our workshop in Garching in March 2012. Best, Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
Am 16.12.2011 09:39, schrieb Andreas Hammershøj: Well, you could download the area of interest via the homepage export function as .osm file and place it on a network share / usb stick. The Editors of the mapping party would open that file, do their changes and upload from there. The solution with a local copy sounds interesting, but because of the time available and the average computer skills of the course participants, we teach Potlatch2. Is it possible to do this with Potlatch? I don't think so, but maybe then Potlatch is the wrong tool for your task. If you get to that conclusion, there are two possibilities: modify the tool or choose one that suits your needs better. Maybe you should give JOSM a try ;) Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
-Oprindelig meddelelse- Fra: Richard Fairhurst [mailto:rich...@systemed.net] Sendt: 16. december 2011 09:59 Til: dev@openstreetmap.org Emne: Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT Andreas Hammershøj wrote: Is it possible to do this with Potlatch? Afraid not - P2 doesn't yet have any ability to load local files from disk. I'm currently looking into a couple of possible solution, but because it's Christmas I don't have a whole bunch of time for P2 stuff for a couple of weeks. cheers Richard Ok, thanks a lot for looking into it:). I respect that this is all voluntary, so all help will be greatly appreciated. This is an ongoing challenge for us, and our next course on January 14th already has people on a waiting list because of a limited number of 3G modems. It'd be nice to one day avoid that sort of limits. /A View this message in context: http://gis.638310.n2.nabble.com/IP-limit-issue-when-doing-mapping-parties-on-a-network-with-NAT-tp7086366p7100086.html Sent from the Developer Discussion mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On 15.12.2011 12:10, Frederik Ramm wrote: But: Anyone who really wants to, and has the resources to, can set up a full database today, feed it with minutely diffs through Osmosis, and allow a merry band of replication clients down the line. problem is that Postgres repliation (the bundled one in 9.0+) replicates physical pages and page deltas and the exact page format is architecture dependant. So you can only replicate between systems that share the same byte order, integer size, maybe even same C compiler padding ... (unlike MySQL that logs logical changes into its replication log in a portable format and even has architecture agnostic physical storage formats for most storage engines so that moving logs or even complete data directory snapshots between different architectures has been a no-brainer ever since MySQL 3.23 ...) So even a public PG replication master would only make sense for those who run exactly the same architecture, or multiple masters for different architectures would be needed ... :/ -- hartmut ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM-3D tagging info
Ander, On 12/16/11 10:09, Ander Pijoan wrote: to osm format in order to do massive imports. Imports are not universally welcome in OSM. Please discuss, ideally on the imports list, exactly what you plan to import and how. The tags on the wiki related to 3D rendering are quite reduced for all the information Catastro database gives. We have information about part of a building are balconies, terraces, porchs... Such information is almost certainly not suitable for OSM. Remember that OSM is a user-editable database, so you should not import anything that cannot be meaningfully edited by users. So our question is if there are some more tags already standarized or recommended for this data Even if there were - OSM cannot be a repository of detailed 3D shapes for all the buildings in this world, this is simply outside our scope and outside of what our editors and our database can handle. Imports may make the map look nice in the short term, but they have the potential to make editing harder. Also, if you import data where there is no local community, you risk creating huge data dumps for which nobody really cares, and which nobody maintains. The value of OSM is not in the huge amount of data that has been collected, but in the community that keeps this data current. Under no circumstances should you import data for all of Spain while sitting at your desk - if at all, then you can help local users to import cadastre data where there's interest and where there *are* local users to begin with. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Am 16.12.2011 11:09, schrieb Hartmut Holzgraefe: So even a public PG replication master would only make sense for those who run exactly the same architecture, or multiple masters for different architectures would be needed ... :/ At one of the Hack-Weekends someone played around with distributing the SQL-Commands issued by osm2pgsql via XMPP. Such a server could distribute full-import-sql-dumps in a weekly shedule as files and the minutely updates as XMPP messages. Not sure if it would sove any real problems. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
-Oprindelig meddelelse- Fra: Peter Körner [mailto:osm-li...@mazdermind.de] Sendt: 16. december 2011 10:56 Til: Andreas Hammershøj Cc: dev@openstreetmap.org Emne: Re: SV: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT Am 16.12.2011 09:39, schrieb Andreas Hammershøj: Well, you could download the area of interest via the homepage export function as .osm file and place it on a network share / usb stick. The Editors of the mapping party would open that file, do their changes and upload from there. The solution with a local copy sounds interesting, but because of the time available and the average computer skills of the course participants, we teach Potlatch2. Is it possible to do this with Potlatch? I don't think so, but maybe then Potlatch is the wrong tool for your task. If you get to that conclusion, there are two possibilities: modify the tool or choose one that suits your needs better. Maybe you should give JOSM a try ;) Peter Hi Peter, we're aware that using JOSM would counter this problem, but we have deemed JOSM too complicated. Many of our participants are recently retired, with limited computer skills. They have enough of a challenge just learning to use P2. Also, we normally have only around 7 hours at our disposal to get people editing safely, without being menaces to the map;) Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
Am 16.12.2011 11:21, schrieb Andreas Hammershøj: Hi Peter, we're aware that using JOSM would counter this problem, but we have deemed JOSM too complicated. Many of our participants are recently retired, with limited computer skills. They have enough of a challenge just learning to use P2. Also, we normally have only around 7 hours at our disposal to get people editing safely, without being menaces to the map;) I think Mercator is also able to do offline editing: http://wiki.openstreetmap.org/wiki/Mercator I don't want to vote for the one or the other, just point out that if a tool doesn't solve your problem, your possibly better off with trying another tool. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
Am 16.12.2011 11:33, schrieb Peter Körner: Am 16.12.2011 11:21, schrieb Andreas Hammershøj: Hi Peter, we're aware that using JOSM would counter this problem, but we have deemed JOSM too complicated. Many of our participants are recently retired, with limited computer skills. They have enough of a challenge just learning to use P2. Also, we normally have only around 7 hours at our disposal to get people editing safely, without being menaces to the map;) Correct link: http://wiki.openstreetmap.org/wiki/Merkaartor I don't want to vote for the one or the other, just point out that if a tool doesn't solve your problem, your possibly better off with trying another tool. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] IP limit issue when doing mapping parties on a network with NAT
Merkaator is not much easier. I've got same issue a few times. Would it be possible to whitelist temporarily certain pre-registered IPs? Jaak Sent from my hiPhone On 16.12.2011, at 12:35, Peter Körner osm-li...@mazdermind.de wrote: Am 16.12.2011 11:33, schrieb Peter Körner: Am 16.12.2011 11:21, schrieb Andreas Hammershøj: Hi Peter, we're aware that using JOSM would counter this problem, but we have deemed JOSM too complicated. Many of our participants are recently retired, with limited computer skills. They have enough of a challenge just learning to use P2. Also, we normally have only around 7 hours at our disposal to get people editing safely, without being menaces to the map;) Correct link: http://wiki.openstreetmap.org/wiki/Merkaartor I don't want to vote for the one or the other, just point out that if a tool doesn't solve your problem, your possibly better off with trying another tool. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On 16.12.2011 11:21, Peter Körner wrote: At one of the Hack-Weekends someone played around with distributing the SQL-Commands issued by osm2pgsql via XMPP. with the SQL command execution, especially the index creation, being the most expensive part of an osm2pgsql run this would onyl save about 1/4 to 1/3 of the total planet import execution time i'm afraid ... -- hartmut ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Am 16.12.2011 12:47, schrieb Hartmut Holzgraefe: On 16.12.2011 11:21, Peter Körner wrote: At one of the Hack-Weekends someone played around with distributing the SQL-Commands issued by osm2pgsql via XMPP. with the SQL command execution, especially the index creation, being the most expensive part of an osm2pgsql run this would onyl save about 1/4 to 1/3 of the total planet import execution time i'm afraid ... It would help with keeping updated: you would not need the --slim tables anymore (on each server - only on the master) Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Hi, On 12/16/11 12:49, Peter Körner wrote: with the SQL command execution, especially the index creation, being the most expensive part of an osm2pgsql run this would onyl save about 1/4 to 1/3 of the total planet import execution time i'm afraid ... It would help with keeping updated: you would not need the --slim tables anymore (on each server - only on the master) At the cost of high network traffic - a single node moved in the outline of the Germany polygon would mean that a geometry with tens of thousands of points would have to be pushed from the master to the clients. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM-3D tagging info
El 16/12/2011 10:22, Paul Norman penor...@mac.com escribió: 3d tagging is not widely supported. I think it’d be of questionable value. I think that importing the number of floors and/or the height of the building is something really useful. Look at those pretty buildings in 3d renderings. Google folks love them. :-) Time to establish one? What software are you using to convert, and what’s the license of the data? I am in no way related to Ander, but I extensively know about his team's work because he has providing feedback for some months in the Spanish list. I told them, however, to follow the gidelines (which he is doing well, considering the import is not yet prepared) and inform at the international lists. Most Spanish mappers (including the most veterans) are aware and prividing feedback, corrections, opinions on data selection. The data he is using is the one I announced early this year in the talk list about Spanish Cadastre release (nobody seamed to care much back then). It is not only compatible with CTs, but they are giving full rights ownership to any Spanish citizen, to do anything they want with the data set. It is mostly landuse and building shapes. Just to be clear: must be Spanish to download, any derivative work is yours and can be exported. The software is a custom made still-in-progress converter to osm xml format divided in 3000 or so areas (municipalities). Just a reminder, don’t forget to follow the import guidelines, which includes discussing with the local community and impo...@openstreetmap.org. From: Ander Pijoan [mailto:ander.pij...@deusto.es] Sent: Friday, December 16, 2011 1:10 AM To: dev@openstreetmap.org Subject: [OSM-dev] OSM-3D tagging info Hi everyone: My name is Ander Pijoan and I'm a student at the University of Deusto. We are working on an application to convert data from a goverment database called Catastro (which has been openned to all users little time ago) to osm format in order to do massive imports. This database contains all the urban and rustical info about landuse, buildings... and lots of interesting things to add like sub_stations, amenities or even streetlamps and trees. Here is the wiki where you can follow the project: http://wiki.openstreetmap.org/wiki/Spanish_Catastro -- Jaime Crespo ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM-3D tagging info
El 16/12/2011 11:15, Frederik Ramm frede...@remote.org escribió: Ander, On 12/16/11 10:09, Ander Pijoan wrote: Such information is almost certainly not suitable for OSM. Under no circumstances should you import data for all of Spain while sitting at your desk - if at all, then you can help local users to import cadastre data where there's interest and where there *are* local users to begin with. Even if I agree that I like user contributed content above cold imports, when I learnt about the availability of this dataset, I knew it could be very useful. We are talking here about official administrative staff (similar to boundaries) which cannot be surveyed: landuses and building shapes. Buildings could be traced from ortophotography, but with the relation stuf it is both boring and very imprecise (compared to 1-metre-acurate always-updatable import). Sincerely, I prefer people working in surveying amenities, mountain tracks, names, routing, etc. This import have nothing of this. I, as one of the local users, am the one that is fearing any problem and checking progresses. But I want also thank Ander for his work, because he is doing all the hard work that nobody had time to do for this import (and even answering all suggestions). Anyway, I am the first person that could see problems for future contributors. But my beliefs are that they should be addressed within the editors' scope: things like better handling of layers and relations for buildings and landuses. 90% of the corrections I point to new users have to do with multipoligons. -- Jaime Crespo ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM-3D tagging info
Ander Pijoan wrote: We are working on an application to convert data from a goverment database called Catastro (which has been openned to all users little time ago) to osm format in order to do massive imports. [...] The conversion part is almost ready but we have one bit missing, the building tagging for 3d renderers to render propperly. We've seen you will have an OSM Dev weekend but we don't know if we will be able to attend. Imports are a very sensitive topic as they can do a lot of damage when done wrong. Unfortunately, you will simply not be able to do a good import of 3D information within in the next few months. As you have noticed, there is no established tagging scheme for 3d information, and I expect that this situation will persist at least until the Dev weekend in March. Even then, it would not advisable to immediately perform an import with a newly invented tagging scheme. Any tagging idea needs to prove its usability, i.e. be used by normal mappers manually. To sum this up, the proper order is like this: 1. We invent and agree on a tagging scheme. 2. A lot of mappers use it for manual mapping. 3. Imports help local mappers to improve coverage of a type of data that they could have mapped themselves anyway. (optional) As the maintainer of OSM2World, I'm obviously not opposed to adding 3D information to OSM. But it needs to be done in a manner that is compatible with the way our community works. You could consider importing some more established categories of data from your source earlier and leave out the 3D building stuff for the moment. Even then it is recommended that you don't push data into OSM, but instead give local contributors the ability to pull parts of the data and manually integrate it with existing information. Tobias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On 01/-10/-28163 12:59 PM, Peter Körner wrote: Am 16.12.2011 12:47, schrieb Hartmut Holzgraefe: On 16.12.2011 11:21, Peter Körner wrote: At one of the Hack-Weekends someone played around with distributing the SQL-Commands issued by osm2pgsql via XMPP. with the SQL command execution, especially the index creation, being the most expensive part of an osm2pgsql run this would onyl save about 1/4 to 1/3 of the total planet import execution time i'm afraid ... It would help with keeping updated: you would not need the --slim tables anymore (on each server - only on the master) The biggest --slim table is the planet_osm_nodes table which is roughly a 60 Gb table plus a 25Gb index. Having to access the nodes table is also where most of the slowness in a memory constrained import and during any diff application comes from. So the biggest benefit for speed would probably come from optimising the way osm2pgsql access the nodes. I have thought about moving the planet_osm_nodes table out of the postgis database and into a flat file. Really the only information needed about the nodes from the slim table is the lat / lon coordinates. I don't know how exactly postgres stores its data-structures, but it uses more than 8 bytes per row for sure. The flat file would simply be a huge array where you store the lats and lons as 8 byte tuples. The highest node id is currently about 1 600 000 000, which results in about a 13Gb file. As it is much smaller, a bigger proportion should be in disk cache and a node can be retrieved in O(1). So the idea would be that it is faster than using the database for this purpose. My initial tests weren't so promising, but I tried to use sparse files to save disk space, which was probably not a good idea, as that looses the O(1) access characteristics and also makes writing to them much more complicated. I don't yet know when I will get around trying these things again, but hopefully at some point I can figure out if this idea helps. Of cause, if anyone else wants to try it... Kai Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Patch for osm2pgsql duplicate keys
Nick Whitelegg nick.whitel...@solent.ac.uk writes: I know on my (rather old) version of osm2pgsql, append fails with duplicate ways. Will have to try the new version and see if I get the same problem. Sorry, this is with slim mode only. Dup ways give no error with regular mode. Does the patch relate to slim mode or regular mode? Yes, I use slim mode and this patch was written for slim mode. I didn't realise that regular mode was different in this respect but I don't personally have enough RAM to use that anyway. -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
Frederik Ramm wrote: You can try out my script here, by adding a way/node/relation id to the URL like so: http://wtfe.gryph.de/harmless/way/40103577 Here's an oddity... http://wtfe.gryph.de/harmless/way/9178258 suggests This object remains problematic even after looking at harmless edits. From looking at: http://osm.mapki.com/history/way.php?id=9178258 it seems that the CT-nonacceptor's change was to add the ref=A154 tag, which I removed in the next revision. I also added a history tag and removed a created_by tag, since I did a Potlatch 1 revert. I'd have expected (regardless of those two changes) that the CT-nonacceptor's change was harmless because it was merely the adding of one tag which was removed in the next version. Of course, it could always be me missing the blindingly obvious... Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osm2pgsql failure with low-end server
Martijn van Oosterhout klep...@gmail.com writes: On 8 December 2011 11:13, Nick Whitelegg nick.whitel...@solent.ac.uk wrote: 700 MB is a tiny machine, but then the Finland data set isn't that large either... it must be possible somehow ;) Incidentally why would it run out of memory in slim mode? I had the same problem too some time ago when trying to import the whole of England, back in the days before the county extracts. I did some research into this a while back and it has to do with the code that goes over all pending ways after the import, to deal with polygons. It looks like the number of pending ways is a lot more than it used to be. Osm2pgsql requests all the pending ways in a single query which fails on small machines. Can someone check if there is really the case. i.e. show the number of pending ways after a simple import. With this command line: osm2pgsql --create --database GIS --slim --cache 128 great_britain.osm I get this output: osm2pgsql great_britain.osm Reading in file: great_britain.osm Processing: Node(32899k) Way(4022k) Relation(80855) parse time: 3481s Node stats: total(32899023), max(1541436207) Way stats: total(4022307), max(140763013) Relation stats: total(80855), max(1905258) Going over pending ways processing way (1639k) Going over pending relations node cache: stored: 8047817(24.46%), storage efficiency: 47.97%, hit rate: 25.52% ... osm2pgsql SVN version 0.70.5 osm2pgsql great_britain.osm This runs on a VPS with ~512MB RAM (I think that this is the guaranteed level but I have certainly used ~50% more at times). -- Andrew. -- Andrew M. Bishop a...@gedanken.demon.co.uk http://www.gedanken.org.uk/mapping/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
Andy, On 12/16/2011 06:40 PM, SomeoneElse wrote: http://wtfe.gryph.de/harmless/way/9178258 suggests This object remains problematic even after looking at harmless edits. Yes. The script is not clever enough to find out what you did. It would have classed the non-agreer's change as harmless if it had been in between two *identical* versions of the object (i.e. if a full revert had taken place later). In your case, with the history and created_by coming into play, this was not the case and so the change was considered not harmless. I'll have to look into how I could improve this. The obvious choice would be: if someone adds something and whatever they added is not present in the current version any more, then that edit was harmless. However: What if the non-agreer adds the tag nmae=Aunt Gertrud's Home for Orphans and an agreer later fixes this to name=Aunt Gertrud's Home for Orphans ... the simple analysis sketched above would say clearly the non-agreer's change is harmless because the nmae tag is not present any more. But in this situation that would be wrong (I think). So while in your case the harmlessness is obvious to the human eye, I struggle to find a good algorithm that captures it. Any ideas? Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09 E008°23'33 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
My opinion is that the agree-er's change of the (apparently, but who knows for sure?) mis-spelling of the nmae= tag to name= brings the information into the realm of agreement by the adoption of the most recent edit of the tag. It is the responsibility, I would think, of the correcting user to ensure the validity of the change and is no different than an agree-er entering their own name= tag based on what some non-agree-ing person told them about the object. The disappearance of the nmae= tag makes the original addition a harmless edit. The edit is no longer present. The appearance of a name= tag must be, IMHO, considered acceptable if done by an agree-ing user. There is no way nor reason to infer that it was simply a correction of a spelling error without also assuming that the responsibility and agreement status for the information has transitioned to the most recent editor. Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32 (Watching the discussion from the side-lines) On 12/16/2011 12:58 PM, Frederik Ramm wrote: Andy, On 12/16/2011 06:40 PM, SomeoneElse wrote: http://wtfe.gryph.de/harmless/way/9178258 suggests This object remains problematic even after looking at harmless edits. Yes. The script is not clever enough to find out what you did. It would have classed the non-agreer's change as harmless if it had been in between two *identical* versions of the object (i.e. if a full revert had taken place later). In your case, with the history and created_by coming into play, this was not the case and so the change was considered not harmless. I'll have to look into how I could improve this. The obvious choice would be: if someone adds something and whatever they added is not present in the current version any more, then that edit was harmless. However: What if the non-agreer adds the tag nmae=Aunt Gertrud's Home for Orphans and an agreer later fixes this to name=Aunt Gertrud's Home for Orphans ... the simple analysis sketched above would say clearly the non-agreer's change is harmless because the nmae tag is not present any more. But in this situation that would be wrong (I think). So while in your case the harmlessness is obvious to the human eye, I struggle to find a good algorithm that captures it. Any ideas? Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
Hi, We're working hard on getting the relevant hardware in place to start trialling this out, but it's a big project. Many thanks for the insight The original topic was about replication for rendering, so a comment on that Whooops, I haven't read that thread far enough in the past to discover I'm off topic. I do agree that livre replication for rendering has few to no interests, and I was talking about live replication in order to load balance editing read calls. I'll start a new topic about that later but we are in the process of acquiring 3 huge servers for our activities in France, one of wich might be used for API ro calls and I'll conduct a few tests to check if a 3 minutes lag behing main db doesn't have significant edits drawbacks. Final goal is having faster and less limited ro api for editing -- sly qui suis-je : http://sly.letuffe.org email perso : sylvain chez letuffe un point org ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] OSM-3D tagging info
On Dec 16, 2011, at 4:48 AM, Jaime Crespo wrote: El 16/12/2011 10:22, Paul Norman penor...@mac.com escribió: 3d tagging is not widely supported. I think it’d be of questionable value. I think that importing the number of floors and/or the height of the building is something really useful. Look at those pretty buildings in 3d renderings. Google folks love them. :-) no importing has no value. Yes rendering has HUGE value!! Please merge this kind of data in a rendering DB not the main DB. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] OSM-3D tagging info
Hi! Yes rendering has HUGE value!! Please merge this kind of data in a rendering DB not the main DB. So, renderings like this must not be in OSM? Why? :3 http://latlon.org/buildings?zoom=16lat=53.90347lon=27.44665layers=BT -- Darafei Komяpa Praliaskouski OSM BY Team - http://openstreetmap.by/ xmpp:m...@komzpa.net mailto:m...@komzpa.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Imports] OSM-3D tagging info
Please read again what I wrote the answer is there!! Can I do more can shout in CAPS to make it clear? long answer in many posts from Frederick and others about imports On Dec 16, 2011, at 12:07 PM, Komяpa wrote: Hi! Yes rendering has HUGE value!! Please merge this kind of data in a rendering DB not the main DB. So, renderings like this must not be in OSM? Why? :3 http://latlon.org/buildings?zoom=16lat=53.90347lon=27.44665layers=BT -- Darafei Komяpa Praliaskouski OSM BY Team - http://openstreetmap.by/ xmpp:m...@komzpa.net mailto:m...@komzpa.net ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
From: Lynn W. Deffenbaugh (Mr) [mailto:ldeff...@homeside.to] Subject: Re: [OSM-dev] Harmless edits My opinion is that the agree-er's change of the (apparently, but who knows for sure?) mis-spelling of the nmae= tag to name= brings the information into the realm of agreement by the adoption of the most recent edit of the tag. It is the responsibility, I would think, of the correcting user to ensure the validity of the change and is no different than an agree-er entering their own name= tag based on what some non- agree-ing person told them about the object. The disappearance of the nmae= tag makes the original addition a harmless edit. The edit is no longer present. The appearance of a name= tag must be, IMHO, considered acceptable if done by an agree-ing user. There is no way nor reason to infer that it was simply a correction of a spelling error without also assuming that the responsibility and agreement status for the information has transitioned to the most recent editor. I cannot agree with your view that the responsibility for agreement transfers to the most recent editor. As an example, xybot fixes some common spelling errors in tags, as well as some other common mistakes. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Nominatim (npi / osm)?
Hello! I am doing some development and experimentation with nominatim and am confused by what is the best path forward with creating/searching nominatim databases One seems to be NPI (http://wiki.openstreetmap.org/wiki/Nominatim/Structured) and the other is via osm2pgsql/gazetteer ( http://wiki.openstreetmap.org/wiki/Nominatim) In some ways the two different software distributions referenced in those discussions do not seem to conflict, but in others they do -- e.g. conflicting documentation, database/table names, shared objects etc. I've looked for an active discussion to explain the situation, but can't seem to find anything. Thanks. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
On 12/16/2011 4:14 PM, Paul Norman wrote: From: Lynn W. Deffenbaugh (Mr) [mailto:ldeff...@homeside.to] Subject: Re: [OSM-dev] Harmless edits My opinion is that the agree-er's change of the (apparently, but who knows for sure?) mis-spelling of the nmae= tag to name= brings the information into the realm of agreement by the adoption of the most recent edit of the tag. It is the responsibility, I would think, of the correcting user to ensure the validity of the change and is no different than an agree-er entering their own name= tag based on what some non- agree-ing person told them about the object. The disappearance of the nmae= tag makes the original addition a harmless edit. The edit is no longer present. The appearance of a name= tag must be, IMHO, considered acceptable if done by an agree-ing user. There is no way nor reason to infer that it was simply a correction of a spelling error without also assuming that the responsibility and agreement status for the information has transitioned to the most recent editor. I cannot agree with your view that the responsibility for agreement transfers to the most recent editor. As an example, xybot fixes some common spelling errors in tags, as well as some other common mistakes. 'bots are one thing, manual edits are another. If a live editor fixes the nmae= to name= then would not responsibility transition to the entity that made the decision to implement said change? Lynn (D) - Just trying to understand ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Harmless edits
From: Lynn W. Deffenbaugh (Mr) [mailto:ldeff...@homeside.to] Subject: Re: [OSM-dev] Harmless edits On 12/16/2011 4:14 PM, Paul Norman wrote: From: Lynn W. Deffenbaugh (Mr) [mailto:ldeff...@homeside.to] Subject: Re: [OSM-dev] Harmless edits My opinion is that the agree-er's change of the (apparently, but who knows for sure?) mis-spelling of the nmae= tag to name= brings the information into the realm of agreement by the adoption of the most recent edit of the tag. It is the responsibility, I would think, of the correcting user to ensure the validity of the change and is no different than an agree-er entering their own name= tag based on what some non- agree-ing person told them about the object. The disappearance of the nmae= tag makes the original addition a harmless edit. The edit is no longer present. The appearance of a name= tag must be, IMHO, considered acceptable if done by an agree-ing user. There is no way nor reason to infer that it was simply a correction of a spelling error without also assuming that the responsibility and agreement status for the information has transitioned to the most recent editor. I cannot agree with your view that the responsibility for agreement transfers to the most recent editor. As an example, xybot fixes some common spelling errors in tags, as well as some other common mistakes. 'bots are one thing, manual edits are another. If a live editor fixes the nmae= to name= then would not responsibility transition to the entity that made the decision to implement said change? Users may fix tagging mistakes without verifying the data. The alternative is leaving an obvious mistake in place when you know how to fix it. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] speeding up loading an OSM dump into PostGIS?
On 01/-10/-28163 12:59 PM, sly (sylvain letuffe) wrote: Hi, We're working hard on getting the relevant hardware in place to start trialling this out, but it's a big project. Many thanks for the insight The original topic was about replication for rendering, so a comment on that Whooops, I haven't read that thread far enough in the past to discover I'm off topic. I do agree that livre replication for rendering has few to no interests, and I was talking about live replication in order to load balance editing read calls. I'll start a new topic about that later but we are in the process of acquiring 3 huge servers for our activities in France, one of wich might be used for API ro calls and I'll conduct a few tests to check if a 3 minutes lag behing main db doesn't have significant edits drawbacks. Final goal is having faster and less limited ro api for editing There was some talk a while back about having sub-minutely diffs. If I remember correctly, there is no technical reason to not offer replication-diffs at a granularity of less than a minute. E.g. every 30 seconds or perhaps even every 10 seconds. However, so far there hasn't really been any compelling use cases for sub minutely diffs. If you really are going to set up a ro mirror for editing, perhaps this idea can be revisited. However, I would also suspect that a lag of 2 - 3 minutes would be mostly fine anyway. The current work flow is to download some data, then edit for a while after which you upload changes. The edit for a while stage is probably typically longer than 2 - 3 minutes anyway so by the time you upload you are working with old data. One would presumably have to adapt JOSM and Potlatch slightly anyway to be able to handle a secondary read only mirror. By default they would download data from the ro-mirror. In the case of conflict, uploads, or an explicit update request by the user, it would still use the main api for reads to make sure it has the most up-to-date data. Kai ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
