[OSM-dev] New feature: OAuthed rate limits!
Hi!
A version of CGImap [1] has been deployed which supports "OAuthed rate
limits". What this means is that requests with a valid OAuth signature
will be rate-limited by user account rather than source IP address
[2].
This probably makes no difference for most of us, most of the time.
But it potentially could make a huge difference to people at mapping
parties or any other situation where a large number of people are
trying to map from behind a NAT router or firewall.
At the moment, I don't think any editors send OAuth-signed requests
unless authorisation is required for that particular API call, but I
hope that it will be easy to integrate this feature (perhaps
optionally after receiving "509 Bandwidth Limit Exceeded"). Please let
me know or file an issue [3] if you are getting unexpected errors with
OAuth-signed requests, or if there's anything I can help with adding
this feature to your editor.
Another feature released along with this is CGImap support for the
/changesets/#{id} call, including discussions. Again - please let me
know or file an issue [3] if you notice anything amiss with this call.
There is on-going work to support historical versions of elements [4],
which I hope will lead to support for history calls as well as
changeset/#{id}/download (osmChange) API calls. And eventually, the
whole API [5]. If you'd like to help, please take a look at the source
code and get in touch!
Cheers,
Matt
[1] https://github.com/zerebubuth/openstreetmap-cgimap/releases/tag/v0.5.5
[2] https://github.com/openstreetmap/operations/issues/36
[3] https://github.com/zerebubuth/openstreetmap-cgimap/issues
[4] https://github.com/zerebubuth/openstreetmap-cgimap/tree/history
[5] http://www.asklater.com/matt/blog/2015/11/18/the-road-to-api-07/
___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] London development event, Aug 8-9
The OpenStreetMap London group would like to invite you to our next development event. If you'd be interested in a weekend of collaborative development of OpenStreetMap and open geodata technologies, please sign up at http://www.eventbrite.com/e/osm-london-hack-weekend-tickets-17511175397 Sadly, space is limited and signing up for a ticket is necessary. The theme for this event is mobile development and we would especially like to welcome anyone with an interest or experience in that area. If mobile isn't your thing, that's cool too; everyone's welcome to come and hack on OSM and open geodata projects! This isn't just an event for developers - we need designers, UX experts, testers, users, mappers and all kinds of people to help us make the best possible result! Over the course of the weekend, we'll figure out a plan and then try to implement it. Note that this isn't one of those competitive hacking competitions; there might be different teams working on different things, but we're all working together. The planning will take place on Saturday morning, so please take that into account in your plans. Our thanks to our hosts for the weekend, Geovation Hub, for letting us use their office, which is fully accessible. For more info and tickets, please see http://www.eventbrite.com/e/osm-london-hack-weekend-tickets-17511175397 ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Changeset discussions: dumps and replication
at the recent Karlsruhe hack weekend [1] (thanks to Geofabrik for hosting), Paul Norman and i made some changes to changeset replication [2] and planet dumps [3] to support discussions. the first of the dumps is now available [4] as, for the moment, a separate file and the plan is to merge that back into the "main" changesets dump in about 6 months time - if you consume changeset dumps, please test your code against the discussions dump! the discussions dump is largely the same as the changesets dump, but includes the discussions like this: From humble beginnings… which is the same format as the API (with comments included) [5]. these are also present in the same format in the changeset replication feed [6], which allows you to keep a dump or replicated database up to date with the latest comments & changesets. i hope this will give everyone the data they need to build interesting new ways of visualising and interacting with the changeset comment data, for example; better UIs to display discussions near me or on the map, analysis tools to check for spam or other bad comments, and entirely new and unexpected cool things. cheers & happy hacking, matt [1] http://wiki.openstreetmap.org/wiki/Karlsruhe_Hack_Weekend_February_2015 [2] https://github.com/openstreetmap/chef/pull/18 [3] https://github.com/zerebubuth/planet-dump-ng/pull/10 [4] http://planet.openstreetmap.org/planet/2015/discussions-150223.osm.bz2 [5] http://api.openstreetmap.org/api/0.6/changeset/1?include_discussion=true [6] http://planet.openstreetmap.org/replication/changesets/ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM software repositories -- git and svn
On Fri, May 23, 2014 at 8:44 PM, Jochen Topf wrote: > On Fri, May 23, 2014 at 03:19:07PM +0200, Frederik Ramm wrote: >> 2. discoverability > > Lots of little repositories and a 101st with repo with list of repo URLs > sound good to me. This would also allow different ownership/rights for > all of the little repos. Why a one-size-fits-all solution when some repos > could be free-for-all and some more managed. Just allow everybody to add > any repository to your 101st "list repo". > > If you want to, you could ask people to add some standardized meta.json > file or so that you can then crawl to build some kind of index to make > it even easier to find repos by keyword or whatever. i started working on a site for discovery, learning and code documentation, at http://code4osm.org/ but haven't got very far with it yet. the site is just some scripts for processing other repositories [1]. it would be awesome to get some PRs to add other projects, improve documentation or improve the site. cheers, matt [1] https://github.com/zerebubuth/code4osm.org ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] London Hack Weekend - Mar 8 & 9
we're having a hack weekend on the 8th and 9th March, graciously hosted by AOL / MapQuest at their offices in central London. it would be great to see you there! full details are available on the wiki page [1], and there's also an event on Lanyrd [2]. please sign up to at least one of these if you are planning to attend, both so that we're able to judge the amount of power sockets we'll need, and for fire safety / security. in addition, we'll be socialising in the pub on the Friday and Saturday evenings - if hacking isn't your thing, we'd be very happy to see you there. venues TBA at the moment, and suggestions welcome. cheers, matt [1] http://wiki.openstreetmap.org/wiki/London/London_Hack_Weekend_Mar_2014 [2] http://lanyrd.com/2014/osmhack/ ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] London hack weekend
after the last hack weekend, i wanted to have another around the beginning of March, and that's rapidly approaching. here's a doodle poll for the weekends and if you're interested in coming, please indicate which weekends you'd be available. http://www.doodle.com/hh4vnx2p8kzrnypv#table thanks, matt ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Experimental history files too
On Tue, Jan 28, 2014 at 11:11 AM, Peter Körner wrote: > Am 28.01.2014 03:18, schrieb Matt Amos: >> On Fri, Jan 24, 2014 at 2:56 PM, Peter Körner >> wrote: >>> What do you think about adopting the osmium-naming-scheme for history files? >> >> personally, i think it's misleading >> [...] >> in the case of .osm files, they're all potentially history files, and >> the file format does not change depending on whether multiple versions >> are present for a single ID or not. > > History-Files carry an extra attribute (bool visible) that distinguishes > them from regular osm files. all .osm files can carry the visible attribute, and they're present in all responses from the API. they are only elided from the "current" planet dump because they all have the value "true", so would simply be a waste of space. to be clear: the presence of the visible attribute does not mean the file contains multiple versions with the same ID. > Also it's not only about the parser. [...] > So from a data-user point of view it does not matter if the format is > actually-quite-similar, as long as there are separate programs and tools > required to handle the two types of file, they are actually different. the "current" planet is a strict subset of the "history" planet - anything which can load and process a "history" planet file should be able to process the "current" planet. also, it is trivial to turn a "history" planet into a "current" planet by discarding old versions and deleted elements. > I'd compare it more to tiff/aiff. While they are actually quite similar > from a file-format point of view, no one would argue that audio-files > and image-files should be handled as if there were no difference. i would agree that audio files and image files are completely different :-) but that is not the case with "history" and "current" planet files - as i've said above, the former is a strict superset of the latter, so the comparison to tiff/aiff is misleading. i can't think of a good comparison - the best one i've thought of so far is one version of HTML compared with a previous version, and programs which understand only previous versions will ignore some elements of a later version file. >> whether something is a "history" >> osm file or a "current" osm file is a matter of the content - so >> wanting a different extension is a bit like wanting .png for >> truecolour images and .pgr for greyscale images (in the same PNG >> format). > The main difference here is that all applications capable of reading png > MUST be capable of reading pgr as well. That's not true for osm/osh > applications. > > Also the tasks you can perform on both file formats are the same > (display, crop, combine, ...). This is also not true for osm/osh files. > One can import both into a database but the database-format required and > the actions possible on those databases are quite different. indeed, the analogy with image files was not good. however, there are tasks that one can perform on osm files regardless of whether they contain multiple versions with the same ID - one can pull out elements matching certain tags & attributes, perform transformations on the basis of tags & attributes, reproject nodes or extract those within certain areas, etc... >> having said that, it would seem reasonable to add a flag to >> the document element to indicate whether the .osm file is a special >> case, having a single version for each ID, as many programs seem to >> rely on this assumption and it would be better to be able to check it. > Isn't that already implemented via the required_features header of > pbf-files? > > See > <https://github.com/joto/osmium/blob/master/include/osmium/output/pbf.hpp#L880> > for a reference. no, actually, that flag has a different meaning. setting "HistoricalInformation" in the PBF header simply means that some elements may be deleted (have visible="false"). one can create a conforming file, with all elements having visible="true", and yet still have multiple versions for the same ID. to correct this, PBF should probably deprecate "HistoricalInformation"[1] and have a header field "CurrentInformationOnly", which indicates that *both* there are no visible="false" elements and no two elements of the same type share an ID. and our XML should probably have the same attribute on the header. cheers, matt [1]: one could simply change what "HistoricalInformation" meant, but that runs the risk of already-written (and previously conforming) files becoming non-conforming, which will be confusing. ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Experimental history files too
On Fri, Jan 24, 2014 at 2:56 PM, Peter Körner wrote: > Am 23.01.2014 18:28, schrieb Matt Amos: >> i encourage everyone to take a look and report back any problems you >> find. my thanks to Peter Körner, who seems to already be doing this - >> with no problems? > > No problems yet. Have run two splits of those files already > (http://osm.personalwerk.de/full-history-extracts/) > > What do you think about adopting the osmium-naming-scheme for history files? > .osm.[bz2|gz|pbf] -> regular osm files > .osh.[bz2|gz|pbf] -> history files > .osc.[bz2|gz]-> changeset-files > > That would make detecting the kind of file at first glance more easy and > it also fits nicely int othe .osc-file nameing convention. personally, i think it's misleading. osmchange is a related, but different, format from osm xml and a parser which works for one will not necessarily work for the other. therefore, having a different extension seems reasonable. in the case of .osm files, they're all potentially history files, and the file format does not change depending on whether multiple versions are present for a single ID or not. whether something is a "history" osm file or a "current" osm file is a matter of the content - so wanting a different extension is a bit like wanting .png for truecolour images and .pgr for greyscale images (in the same PNG format). having said that, it would seem reasonable to add a flag to the document element to indicate whether the .osm file is a special case, having a single version for each ID, as many programs seem to rely on this assumption and it would be better to be able to check it. > I'm going to implement a regular run that generates fresh extracts every > week from the available file. Is there any note on which weekday the > full-history-dumps are generated, so I can loosely sync my split-script > to that rhythm? great! :-) the generation is synced to the backup database dumps, so the clock starts running early Tuesday, when Monday's backup is complete. they seem to be fairly reliably finished by Wednesday morning, so it's probably safe to start looking for them then - although they'll be named for Monday's date. > I xml-writing takes only half as long as xml-reading I'd double-think > about supplying xml-based files. nobody really has fun reading such huge > files with expat. And if it's really neccessary, there's always > osmium_convert which will generate xmls from pbf-dumps or -extracts locally. this is a discussion which could probably continue forever. my opinion is that it's worthwhile distributing files which are sort-of human-readable, in a well-known format/markup for which many libraries exist in many languages, compressed with standard tools, and in the same format as the API. this way, it's possible for people to develop tools which work against small map call downloads, then scale them to extracts and even the whole planet. of course, it's a widely-held belief that xml sucks irretrievably and, while it's certainly true that pbf is smaller and parses faster, distributing only pbf would mean someone would have to learn those extra tools/commands to start using the data. xml, despite its many flaws, at least has myriad libraries, bindings and tools which make it easier to experiment with processing and transforming osm data. these experimental planet/history files are also line-oriented, which means one can even do quick-and-dirty grep/sed/awk work for ad-hoc analysis. cheers, matt ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] London Hack Weekend - Nov 30 / Dec 1
On Sat, Oct 26, 2013 at 3:57 PM, Matt Amos wrote: > full details are available on the wiki page [1], and there's also an > event on Lanyrd [2]. of course, it would have been more helpful if i'd actually put the links in... [1] http://wiki.openstreetmap.org/wiki/London/London_Hack_Weekend_Nov_2013 [2] http://lanyrd.com/crxqm ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] London Hack Weekend - Nov 30 / Dec 1
we're having a hack weekend on the 30th November and 1st December, graciously hosted by AOL / MapQuest at their offices in central London. it would be great to see you there! full details are available on the wiki page [1], and there's also an event on Lanyrd [2]. please sign up to at least one of these if you are planning to attend, both so that we're able to judge the amount of power sockets we'll need, and for fire safety / security. in addition, we'll be socialising in the pub on the Friday and Saturday evenings - if hacking isn't your thing, we'd be very happy to see you there. venues TBA at the moment, and suggestions welcome. cheers, matt ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Fun documentation questionnaire!
we at EWG are interested in which OSM-related projects would benefit from some help improving their documentation for developers. that is; the documentation which you'd look at if you wanted to start contributing to that project, not any documentation for using it. to help collect some data to inform any discussion, we would be very grateful if you could spend a few minutes filling out this short (and fun) questionnaire: http://bit.ly/17OxvKa if you see a project which you've not heard of before, or haven't seen the developer documentation, it would be extremely helpful if you could spend a short time looking for it. it may not even exist - but it's good data! the list of projects is not supposed to be exhaustive, and there are many worthy projects missing from the list. if there is something which we've missed, please use this thread to discuss it and its developer documentation situation. cheers, matt ___ dev mailing list dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Changeset metadata replication stream not working
On Fri, Apr 12, 2013 at 7:18 PM, Paweł Paprota wrote: > Please either fix it or take it down so I know what to do in my code. apologies, should be working again now. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Latest full-history planet
now available from: http://planet.openstreetmap.org/planet/full-history/ get it while it's hot :-) and please let me know if you find any errors with it. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Status of the Mapnik stylesheets
On Wed, Nov 14, 2012 at 12:23 PM, Richard Fairhurst wrote: > Matt Amos wrote: > > i'd sound a note of caution about having separate "clean" and > > "detailed" styles. we sort-of did that before with mapnik and > > osmarender respectively and... well, we don't have > > osmarender any more. > > That was a technology failure, though, rather than anything wrong with the > concept itself. > the greater resources required to render osmarender worldwide may have played a part, but that part of the problem was the one getting the most attention with tiles@home and so forth. my take on it was that many, perhaps most, people felt the extremely "detailed" style that osmarender/tiles@home was producing wasn't as pleasant as the mapnik style, so it got second billing, less re-use, and that's what led to its decline. In principle, there are clear, identifiable, distinct needs for a "showcase > style" and a "debugging view". We want the great unwashed to look at OSM > and > say "wow, that's a complete, accurate map"; and we want our mappers to > enjoy > the gratification of seeing their changes rendered, because that's a > powerful incentive to keep contributing. > > (I use the word "view" rather than "style" because it's conceivable that > the > latter could be provided some other way than a traditional Mapnik > stylesheet, perhaps something along the lines of Kothic-JS.) > i agree, but would go further and suggest that the debugging view might be better constructed on top of the "showcase style", rather than being able to (d)evolve independently of it. i would further suggest that the gratification of seeing changes rendered is strongly reliant on those changes being visible directly on osm.org, which (i think) leads to the conclusion that these two "views" cannot be separate, and must be integrated somehow. > (For the showcase style, personally I think XML vs CSS is more of a problem > than svn vs git, but that's an implementation detail.) > i'd like to applaud Andy Allan's recent work in this area [1]. this is excellent stuff, and many thanks to Andy for starting it :-) cheers, matt [1] https://github.com/gravitystorm/openstreetmap-carto ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Status of the Mapnik stylesheets
On Tue, Nov 13, 2012 at 10:36 PM, Tom MacWright wrote: > There is no way to 'just do it' until the style ... has active > maintainers. Until then we're just talking. seems like this might be the major hurdle - there do seem to be people willing to contribute on this thread, but uncertainty about who is / will be the maintainer and take on the long-term responsibility of choosing which contributions to take. i'd sound a note of caution about having separate "clean" and "detailed" styles. we sort-of did that before with mapnik and osmarender respectively and... well, we don't have osmarender any more. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Minutely changeset feed
On Wed, Nov 7, 2012 at 5:22 PM, Toby Murray wrote: > On Wed, Nov 7, 2012 at 10:15 AM, Matt Amos wrote: > > On Fri, Nov 2, 2012 at 6:34 AM, Toby Murray > wrote: > >> > >> It looks like changesets don't show up until they are closed. This > >> makes sense since then you don't have to worry about information > >> changing. > > > > > > hmm... this shouldn't be the case - changesets should show up when > they're > > opened and again when they're closed. if they're opened and closed in the > > same period, it will only show the closed state. this seems to be what > > happens with most changesets - they open and close within a few seconds. > > I looked at 5 consecutive diffs just now and there was not a single > open changeset. Other random ones I've looked at in the past also > don't have any open ones. I find this highly unlikely since it is > actually kind of hard to manually close a changeset in P2. So I > suspect you have a bug. Possibly the same bug that affected the weekly > planet dump until a few weeks ago when I noticed it and Tom fixed it: > ah... just noticed a massive bug - i was assigning both created_at and closed_at from the database's created_at! no wonder it thought everything was already closed. right, i'll go fix that. > > http://git.openstreetmap.org/planetdump.git/commitdiff/e406711139b9eb8db9d32fa4094325ec3841082c?hp=beb331bcdf6e5565b02fe61be397c150bc03 > > The problem was that all changesets have the closed_at field set to > one hour in the future whenever the changeset is touched. This is > stored in UTC. If one hour has passed since the last change, meaning > that closed_at is in the past, the changeset is considered closed. But > the SQL that determines the changeset's openness uses the now() > function which by default returns server local time. During daylight > savings time in London this means that, the "closed_at > now()" check > always returned false so all changesets were marked as being closed in > the weekly dump. > > But then again, DST is no longer in effect... So it may be something else. yeah, i thought i handled that - all calls in the code are UTC, although i see i've used now() in one SQL statement, but it's just to get a starting list and is filtered against the correct time later in the code. i will have to do some investigation... > >> > >> However, what will it do in the edge case where a changeset is closed > >> but then reopened? Not possible you say? I once had an upload take > >> over an hour to process. It did eventually succeed but the changeset > >> was marked as closed in my changeset list for several minutes before > >> the upload finished and then it was suddenly open again. I am assuming > >> in this case the changeset would show up twice in the minutely diffs > >> and cause the INSERT queries of all consumers to explode violently :) > > > > > > yeah... i don't think the code is able to handle this at the moment... > maybe > > it would see the re-opening, but probably not. any specific examples of > this > > would be great, btw, to help me start building up some test cases. > > Ok well if the changeset normally shows up twice in the feed anyway > then tools will need to handle this so it shouldn't be a problem on > the consumer side. But yes, the code generating the diffs does need to > take this into account. I know the osmosis diffs are based on database > transactions so they pick up any row that has changed since the last > diff. Applied to changesets, this would actually produce an entry in > the stream every time a chunk was uploaded (in JOSM chunked upload > mode) > yeah, i wanted to avoid the changeset being in the feed too many times (e.g: chunked upload). ideally only when it is opened and closed, although there are situations where it probably prints out more often. > By examples, do you mean a changeset ID? Since this is all time based > that won't do much good since now the changeset will look like any > other changeset that I held open for 2 hours. i'll have to find more inventive ways to test the code, then ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Minutely changeset feed
On Fri, Nov 2, 2012 at 6:34 AM, Toby Murray wrote: > Since I have been playing with changeset metadata[1] a bit, Paul > Norman recently pointed out to me that there are now minutely diffs of > changesets available from planet.osm [2]. Would anyone care to share > some juicy details like where it came from and how it works? > Apparently it was mentioned in an EWG meeting. I should really check > one of those out... > > Anyway, some observations/questions: > > I see the state file is in a different format than all the other > replication state files. Probably not a big deal but is there value in > consistency? > perhaps. ultimately, i'd like this to be rolled into the osmosis diff production processes, but doing that now might cause some breakage. in the interim, i thought a YAML file is probably standard enough that not re-using the non-standard osmosis parser would be easy enough. > Are there any existing tools to use this stream? > i don't know of any, but i'd be very happy to learn of new tools / support in existing tools :-) > It looks like changesets don't show up until they are closed. This > makes sense since then you don't have to worry about information > changing. > hmm... this shouldn't be the case - changesets should show up when they're opened and again when they're closed. if they're opened and closed in the same period, it will only show the closed state. this seems to be what happens with most changesets - they open and close within a few seconds. > However, what will it do in the edge case where a changeset is closed > but then reopened? Not possible you say? I once had an upload take > over an hour to process. It did eventually succeed but the changeset > was marked as closed in my changeset list for several minutes before > the upload finished and then it was suddenly open again. I am assuming > in this case the changeset would show up twice in the minutely diffs > and cause the INSERT queries of all consumers to explode violently :) > yeah... i don't think the code is able to handle this at the moment... maybe it would see the re-opening, but probably not. any specific examples of this would be great, btw, to help me start building up some test cases. cheers, matt > [1] https://github.com/ToeBee/ChangesetMD > [2] http://planet.openstreetmap.org/replication/changesets/ > > Toby > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] ODbL full history planet
now available from: http://planet.openstreetmap.org/planet/full-history/ only limited testing has been done on this file (checking for XML well-formed-ness and various spot-checks on elements), so please let me know if you find any errors with it. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Why are so many changeset so large?
On Wed, 2012-10-17 at 00:28 +0100, Tom Hughes wrote: > On 17/10/12 00:04, Alex Barth wrote: > > - Are there technical reasons why changesets should tend to be > > large? Are they expensive on some level? > > I believe it's entirely because we've got so many people doing > mechanical or semi-mechanical edits. > > That includes bots but also things like people using xapi or overpass to > download all objects matching some set of tags, then change those tags > and reupload. the historical answer to this is that when changesets were added to the OSM API there were two different intentions for their use which got conflated: first, that changesets were structures for grouping edits sharing common attributes. and second, that changesets were VCS-style 'commits' which would be uploaded in a single request and applied atomically. effectively, the first use-case was for users, and tried to make changesets as open-ended as possible. from this, we get tags on changesets for comments, editor, bot-ness, etc... and the ability to keep uploading into an open changeset. the second use-case was a technical thing - the sheer number of API calls to individual elements, even from normal-sized editing sessions, could cause problems. and, for small calls, HTTP headers and round-trip latencies would dominate the cost of an upload. further, editors had to cope with the situation where an upload failed half-way through and to re-try the failed calls. from this, we get a single changeset/#id/upload call which applies atomically. at the time, this seemed like a good way to satisfy both use-cases. and, while it does what it set out to, i think we should consider splitting these in the next API version; explicitly reifying uploads at which bboxes / coverage sets and change counts can be stored. changesets can then simply be collections of uploads. getting to the point: this might to some extent mitigate the "large changesets" issue, as it would allow bboxes to be collected at a smaller granularity. however, it wouldn't be a full solution and we'd probably still need something like OWL to break down the geographic footprint of changesets further. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Older version of OWL alive?
On Tue, 2012-10-16 at 09:32 -0400, Alex Barth wrote: > Matt (or others who might know) - > > Is there a staging (or some other kind of) version of OWL alive that I could > look at? sadly not. the version on dev (which was crushing it) died under its own weight a while ago. the situation back in May [1] is pretty much the same situation we're in now, despite great work by Ian Dees on trying to get the front-end working. cheers, matt [1] http://lists.openstreetmap.org/pipermail/talk/2012-May/062959.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
On Mon, 2012-10-15 at 18:13 +0200, Paweł Paprota wrote: > > "top ten tasks" is "These are the Top Ten Tasks that the OSM System > > Administrators" > > What about the community ? This only is a todo list by the admins, for > > the > > admins coded by the admins. So far so good, but that's not a wishlist, > > or, at > > least, not a wishlist of the community. the page says: "These are the Top Ten Tasks that the OSM System Administrators would really like your development help on." so i think it's unfair to say it's a list by the admins, for the admins. the important part of the sentence is "... would really like your development help on." we (EWG) formulated this list as a curated selection of tasks that we thought were generally important so that people who were looking for something to do would have some idea of what "the most important"[1] items were, and where their efforts would be most appreciated. > Disclaimer: I have just started working on OSM a few weeks ago so I may > be wrong with my impression about how things work. > > Generally I would say that OSM works like a typical open source project > - people who do the actual programming work choose what they want to > work on. That's OK since this characteristic is the main attraction to > open source for programmers around the world - they can work on what > they like, instead of working on what their boss or a customer order > them to work on. exactly - there are no restrictions on what you should work on. the data is open, the software is open, and you can work on whatever is interesting to you. and if, surrounded by this huge number of different things to work on, you want to work on something that some people who are knowledgeable about OSM think is important enough to have short-listed; that's the Top Ten Tasks[1]. > I think every open source project, including the big ones has some > challenges with "user voice being heard" or at least that's the > impression. If you propose to change it by creating a community-driven > (instead of "admin"-driven as you put it) wishlist, by any means - do > it. The operative word being "do". one problem (which probably started this thread) is that a wishlist is potentially infinite. and, even treating it as an ideas-gathering forum, the value becomes diluted with the quantity of ideas presented. the hardest part of making the process useful is curating the list of wishes and doing the work of turning it into a list of achievable and practical items. and, as you rightly point out, the operative word is "do". cheers, matt [1] this is not to say that other things aren't important. when the Top Ten Tasks were written, our crystal ball was out of order so we sadly couldn't predict the future. things change, and ten is much too small a number to get everything that's important onto the list, so the TTTs are a just a suggestion - they're not gospel. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fwd: Streaming Replication
On Sun, 2012-10-14 at 22:39 +1100, Brett Henderson wrote: > The timestamp columns in the database are set to "timestamp without > time zone" which presumably means the timezones of dates aren't > automatically converted to the correct timezone upon querying. I'm a > bit confused though because I believe PostgreSQL itself is running in > the BST timezone. I'd like to investigate further but I don't have > time at the moment. bare timestamps are stored in the database in UTC always [1]. as far as i know, there's no way to decorate the column with this information, so postgres doesn't have it to do any conversions. it seems likely that somewhere along the stack of software doing the query something is adding timezone information by defaulting to the current timezone, and it shouldn't be doing that. cheers, matt [1] https://github.com/openstreetmap/openstreetmap-website/blob/master/app/models/node.rb#L290 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM Wishlist
On Fri, Oct 12, 2012 at 11:50 PM, Iván Sánchez Ortega wrote: > A diff/patch tool for imported data. opengeo presented at SOTM about their plans for "geogit" [1]. from what i understood one of their aims is interoperability between separately-maintained OGC model datasets and OSM. i think i'm right in saying that there's a fair way to go before it's a finished product, but i'm sure they'd welcome any support. cheers, matt [1] http://blog.opengeo.org/tag/geogit/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Hello World
On Fri, Oct 12, 2012 at 6:14 PM, Michal Migurski wrote: > * Improved relation + data model support in a editors, either a new one or a > retrofit onto Potlatch. Things like route relations or addressing schemas are > important, for example, and should see the same kind of first-class support > in an editor that the primary/secondary/etc. road schema does. It's time for > these higher-order features to come out of the lab. another item, if i may add to your wishlist, is turn restrictions; there's already some special-purpose support for this in JOSM and Potlatch 2, and i think it's something where a little more work could really help reduce the complexity of adding and maintaining these features. they tend to be found around complex junctions and there have been many ideas in the past for what a UI specifically for junction editing might look like (for example [1]). > * Better time-awareness in planet dumps. I want to see a "days since last > edit" or similar attribute on all entities in the planet file, which will > make it possible to create downstream displays and styles that address hot, > contentious information. We're starting to see two tiers of OSM data > consumers, those who want the latest-newness and those who want to treat OSM > like a slower-moving data source. please could you explain a little more? i think perhaps i've misunderstood what you're asking for. all nodes, ways and relations in the planet file have a timestamp, and the planet has a date header that would allow the calculation of "days since last edited". would you like this information on the sub-elements (tags, nds, members) as well? > Stepping back, the story of OSM *is* that things happen in other tools, so > I'm in agreement with Paweł Paprota that Mapbox should "build something that > needs JSON support, filtering and tag API" before those features get fed into > the core infrastructure. The minute diffs support the ability to develop > real, working systems in an offboard manner, and I think we should take > advantage of that. and systems developed in this way are loosely coupled to the rails code and main database, making it easier to admin and scale these services later without adding complexity to the rails code. in my opinion, it's a huge advantage. cheers, matt [1] http://www.slideshare.net/yutik/mapzen-junction-editor ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Will the real OpenStreetBugs stand up?
On Tue, Oct 9, 2012 at 9:48 PM, Mikel Maron wrote: >> What part of "I will take an action to get something pushed out before the >> next meeting" did you fail to understand yesterday? > > This sounds promising, but I fail to understand it fully. Is there someone > else from the EWG meeting (not TomH, who I don't want to bother) who would > like to fill us in on these developments with integrating bug tracking into > OSM.org? i've just finished putting the minutes up [1] and the relevant extract is: * ACTION: TomH to push current OSB/notes branch public (which he has been improving on a local branch) * (OSB/notes) there are concerns about the UI implementation of the current branch, but it may be possible to merge the API separately so that UI development can proceed independently cheers, matt [1] http://www.osmfoundation.org/wiki/Working_Group_Minutes/EWG_2012-10-08 > -Mikel > > * Mikel Maron * +14152835207 @mikel s:mikelmaron > > > From: Tom Hughes > To: Tom MacWright > Cc: dev@openstreetmap.org > Sent: Tuesday, October 9, 2012 4:31 PM > Subject: Re: [OSM-dev] Will the real OpenStreetBugs stand up? > > On 09/10/12 21:24, Tom MacWright wrote: > >>All those are independent third party sites created by individuals >>and are not directly related to core site. >> >> Aren't they using the same database somehow? > > No idea. > >>What we were talking about in the EWG meeting was adding a "bug" >>reporting system to the main site that records things in the main >>database and is integrated with the API etc. >> >>It is not directly related to any of the sites you mention. >> >> Okay, then what is it? :) Is it not open-source at all? I thought that >> you were working on a branch of the 'official' OSB project and just >> needed to merge/publish that? > > What part of "I will take an action to get something pushed out before the > next meeting" did you fail to understand yesterday? > > The story is that Kai created something that was literally based on taking > one of the existing OSB systems and bolting that javascript onto the rails > code but it didn't produce a something that was very coherent with the rest > of the site and API so I have been reworking it. > > There is a branch out there that you may stumble across but it bears no > resemblence to the current code. > > Now if you want me to get what I have cleaned up and published I should > probably stop writing emails about it and actually work on it instead... > > Tom > > -- Tom Hughes (t...@compton.nu) > http://compton.nu/ > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > > > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OWL + OSM Activity Server
On Mon, Oct 8, 2012 at 9:33 PM, Paweł Paprota wrote: > Hi Ian, > > Thanks for the response. What you wrote about partitioning the data etc. > is exactly what concerns me with the Changeset Activity Publisher > implementation - it will get too big with time. what Ian says is exactly right - the current status is that work needs to be done to either denormalise the data or figure out a clever way to handle non-spatial queries over this spatial data structure. the least horrific option seems to be the denormalisation, but i've also been playing around with trying to write an sp-gist index and toying with the question of whether some other form of database might be a better fit. > On the other hand, for "social" purposes I guess there is no need to > hold on to every single change since forever... Does anyone read > Facebook posts from few months ago? Guess not... depending on how fast the generation process is, could it potentially be done on-the-fly and cached? > Slightly different use cases I think but still it would be good to learn > from OWL devs knowledge about scaling something like this. i can definitely tell you what doesn't work: sticking it all in one table and hoping the database will magically sort it out ;-) as yet i haven't seen a solution which doesn't incur significant drawbacks. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Re-booting EWG
in the shadow of the big, bad license change, i'm afraid EWG rather fell by the wayside. apologies about that, and for taking so long to re-start it. in keeping with the spirit of continuity, the next meeting will be on #osm-ewg in a week's time: monday 8th october at 1800 GMT (see [1] for times in various time zones). on the agenda for this meeting: review of the top ten tasks [2] and planning upcoming events. cheers, matt [1] http://www.timeanddate.com/worldclock/fixedtime.html?iso=20121008T19&p1=136 [2] http://wiki.openstreetmap.org/wiki/Top_Ten_Tasks ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] ［Building Lane Model for OSM］How can I get multiple gpx from osm for repeated roads?
On Thu, Nov 17, 2011 at 4:47 PM, donglai wrote: > Hi all: > > I'm thinking to build the lane model for OSM using machine learning > techniques. How can I get multiple gpx files from osm database for certain > roads? the GPX tracks in the OSM database are not explicitly related to the roads - it is a completely different dataset in that regard. to download tracks for an area of interest, there is the "trackpoints" API call [1]. however, for bulk downloading large areas there is no good method (yet). perhaps using the trackpoints API call near some major intersections would yield the data you need? cheers, matt [1] http://wiki.openstreetmap.org/wiki/API_v0.6#Retrieving_GPS_points > Thanks in advance for any advice or feedback. > Best > D. > > Below are my half-baked ideas: > > 1) Motivation: > > a) According to Sebastian Thrun, in order to build the autonomous driving > car, they push google map to 15cm accuracy with lane models !!! (I guess > they do it in an expensive way, maybe only for part of CA to test their car) > http://www.youtube.com/watch?v=bp9KBrH8H04 > > b) Also, several OSM members have already proposed the lane tag for the map. > However, it'll be a pain for our dear brave volunteers to track and mark the > lane they were in. > > > 2) One possible way out: > a) the bottleneck is that gpx files uploaded by users are often too noisy to > figure out the lane assignment. But we have the "turn constraint" (change to > inner/outer lane before turning left/right) and relative geometry shape of > the route. > b) Given multiple gpx files for one road segment, we can have a robust > estimation of the lane model using machine learning techniques. > c) Also, such technique may be useful for map refinement instead of asking > users to correct by their own where the accuracy is in meters. > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Where is the controler for requests?
the OSM rails server follows a pretty standard layout for the controllers, models, views and routes and so forth as used by any Ruby on Rails project. the Rails tutorial has an overview of the directory structure of a rails app [1] and there is more detail in the rails official guides [2]. cheers, matt [1] http://ruby.railstutorial.org/ruby-on-rails-tutorial-book#sec:the_first_application [2] http://guides.rubyonrails.org/ On Wed, Nov 16, 2011 at 9:21 AM, Alexandru wrote: > I want to add some static html pages to my server, can someone pls tell me > where the controller is so that i can add my paths? > Thanks in advance. > PS: a reference for osm's general file structure would be very appreciated, > too! > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Segmentation fault in osm
On Fri, Nov 11, 2011 at 1:29 PM, Alexandru wrote: > /usr/lib/ruby/gems/1.8/gems/i18n-0.6.0/lib/i18n/core_ext/hash.rb:13: [BUG] > Segmentation fault > ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] > > Aborted this would seem to indicate a pretty serious problem with your ruby installation. what distro / OS are you running? and is it possible there are some rogue out-of-date or ABI-incompatible libraries somewhere on your disk? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] EWG & London Hack Weekend
based on the doodle poll dates, it looks like the weekend of the 26th & 27th november is most convenient. i look forward to a great weekend of hacking, discussion, education and maybe a tiny little bit of drinking, and i hope to see you there. if you're interested in coming, please sign up on the wiki: http://wiki.openstreetmap.org/wiki/London/London_Hack_Weekend_Nov_2011 and a reminder that the EWG meeting is tomorrow at 5pm GMT. cheers, matt On Sun, Oct 2, 2011 at 5:22 PM, Matt Amos wrote: > first, a quick reminder that the Engineering Working Group (EWG) [1] > is having meetings on #osm-ewg Mondays at 5pm GMT. if you'd like to > get involved in OSM-related development, or have ideas about how to > support it, then please come along. this week, we're going to be > mostly looking at "what's the biggest single thing stopping you > developing with OSM data?" and probably a bunch more stuff :-) > > for a more hands-on event, i'm looking to start planning the next > London hack weekend (previous [2]) some time in late November / early > December. judging by previous events, it'll be a smorgasbord of > hard-core development, tutorials and hanging around in pubs! if you're > interested in coming along please enter your availability at this > doodle poll [3] so that we can find out when has maximal utility [4]. > > cheers, > > matt > > [1] http://www.osmfoundation.org/wiki/Engineering_Working_Group > [2] http://wiki.openstreetmap.org/wiki/London_Hack_Weekend_May_2011 > [3] http://www.doodle.com/z5brf6c5wa9biwix > [4] http://en.wikipedia.org/wiki/Utility > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] EWG & London Hack Weekend
first, a quick reminder that the Engineering Working Group (EWG) [1] is having meetings on #osm-ewg Mondays at 5pm GMT. if you'd like to get involved in OSM-related development, or have ideas about how to support it, then please come along. this week, we're going to be mostly looking at "what's the biggest single thing stopping you developing with OSM data?" and probably a bunch more stuff :-) for a more hands-on event, i'm looking to start planning the next London hack weekend (previous [2]) some time in late November / early December. judging by previous events, it'll be a smorgasbord of hard-core development, tutorials and hanging around in pubs! if you're interested in coming along please enter your availability at this doodle poll [3] so that we can find out when has maximal utility [4]. cheers, matt [1] http://www.osmfoundation.org/wiki/Engineering_Working_Group [2] http://wiki.openstreetmap.org/wiki/London_Hack_Weekend_May_2011 [3] http://www.doodle.com/z5brf6c5wa9biwix [4] http://en.wikipedia.org/wiki/Utility ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] 'tile' - what it is used for in some tables in the schema?
On Fri, Sep 30, 2011 at 10:00 AM, Zsombor Szabó wrote: > Hi, > > studying the OSM database schema over at > http://wiki.openstreetmap.org/wiki/Database_schema I was wondering for > what the 'tile' attribute is used for in the current_nodes, nodes, > gps_points tables. > > If you know then please share. it's a geometric index using a B-tree of the (truncated) morton code of the lat/lon of those elements. for the gory details, see the sql_for_area function in the source code [1]. there's also some general discussion on the wiki [2], info about the general method on wikipedia [3] and in the mailing list archives [4]. cheers, matt [1] http://git.openstreetmap.org/rails.git/blob/HEAD:/lib/quad_tile.rb#l63 [2] http://wiki.openstreetmap.org/wiki/Quadtile#Indexing_single-point_geo-locations [3] http://en.wikipedia.org/wiki/Z-order_curve [4] http://lists.openstreetmap.org/pipermail/dev/2008-September/011694.html ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSMF Engineering Working Group inauguration
ern with uservoice and similar systems was that it's not a good forum for bi-directional discussion about things like feasibility and compromise. but the difficulty of the task should not dissuade us ;-) > But overall, it seems like it was a reasonable productive first meet for > the EWG. thanks to everyone who came. hopefully we'll have another productive meeting next week. cheers, matt > Kai > > > On 01/-10/-28163 12:59 PM, Kate Chapman wrote: >> >> Hi Matt, >> >> Just wanted to mention I'm interested in helping. Sadly I am >> unavailable today, though normally I would be at the time selected. >> >> Couple thoughts I have about the rough agenda: >> >> * Barriers for entry for getting started: >> >> People seem to have issues getting the rails port working. Perhaps >> creation of VM with development environment? That would allow those >> who maybe can just do JavaScript and CSS still to contribute >> something. >> >> Documentation (this of course depends on the specific project). Are >> there holes in the documentation that could be improved? >> >> * Small Projects that are mentored to get developers started: >> >> I don't have any great ideas about this. I don't actively contribute >> to any of the main OSM projects by actually coding, though I'm willing >> to help mentor if there is a way that makes sense. Are there >> individuals who come in and want to contribute code and don't map that >> need basic OSM mentoring? >> >> >> Thanks! >> >> -Kate >> >> >> >> >> On Mon, Aug 15, 2011 at 8:09 PM, Matt Amos wrote: >>> >>> the "Engineering Working Group"[1] is a new group set up with the >>> purpose of helping stimulate the development community surrounding >>> OSM. you don't need to be a developer to join - we're interested in >>> hearing from everyone and ideas are welcome. software enables us to do >>> the mapping we love, and to make use of the data we collect, so making >>> our software better is part of making OSM better. >>> >>> although IRC isn't a perfect communication medium it's easy to set up >>> and it's interactive, so i've set one up over at #osm-ewg on OFTC. >>> just to kick off, i suggest a meeting next week on: >>> >>> Monday 22nd at 5pm GMT >>> >>> (5pm CEST, 6pm BST, 1pm east-US, 10am pacific-US)[2] to start the >>> conversation. >>> >>> a very rough agenda for the meeting, including but certainly not limited >>> to: >>> * what are the barriers-to-entry for new developers? >>> * are there small projects (perhaps mentored?) which new developers >>> can take on? >>> * what can be done to improve the usefulness of communication between >>> developers and users? >>> >>> hope to see you there, >>> >>> matt >>> >>> [1] it's not necessarily the best or most descriptive name, but it'll >>> do for now. other suggestions include "Development Support Group" and >>> "Software Working Group". >>> [2] yeah, i know, but no matter what time is chosen it'll be bad for >>> someone. i suggest we start with this and i'm open to suggestions >>> about rotating the time or setting up other communication channels >>> which are less time-zone sensitive. >>> >>> ___ >>> dev mailing list >>> dev@openstreetmap.org >>> http://lists.openstreetmap.org/listinfo/dev >>> >> >> > > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] OSMF Engineering Working Group inauguration
the "Engineering Working Group"[1] is a new group set up with the purpose of helping stimulate the development community surrounding OSM. you don't need to be a developer to join - we're interested in hearing from everyone and ideas are welcome. software enables us to do the mapping we love, and to make use of the data we collect, so making our software better is part of making OSM better. although IRC isn't a perfect communication medium it's easy to set up and it's interactive, so i've set one up over at #osm-ewg on OFTC. just to kick off, i suggest a meeting next week on: Monday 22nd at 5pm GMT (5pm CEST, 6pm BST, 1pm east-US, 10am pacific-US)[2] to start the conversation. a very rough agenda for the meeting, including but certainly not limited to: * what are the barriers-to-entry for new developers? * are there small projects (perhaps mentored?) which new developers can take on? * what can be done to improve the usefulness of communication between developers and users? hope to see you there, matt [1] it's not necessarily the best or most descriptive name, but it'll do for now. other suggestions include "Development Support Group" and "Software Working Group". [2] yeah, i know, but no matter what time is chosen it'll be bad for someone. i suggest we start with this and i'm open to suggestions about rotating the time or setting up other communication channels which are less time-zone sensitive. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] JSON-output for xapi
On Sun, Jun 26, 2011 at 6:17 PM, Ian Dees wrote:
> I'm not sure it's specified anywhere, but it is customary (and many tools
> expect) to receive a sorted OSM file: nodes first, followed by ways,
> followed by relations. This makes constructing an in-memory model of the
> data easier on the client side.
indeed. the rails port generates XML with the nodes, ways and
relations in order and cgimap is designed to be compatible with that,
in both XML and JSON outputs. i figure it also makes parsing the data
easier, which is probably more important for JSON as the format is
often imported directly into working data structures in the browser.
have you considered using temporary tables in the database to order
the data so that it can be streamed? this is how cgimap (and, iirc,
jxapi?) solve the issue of building up search results in a performant
manner that can be streamed while also being correctly ordered.
cheers,
matt
> On Sun, Jun 26, 2011 at 12:05 PM, Dario Brandes wrote:
>>
>> hi matt,
>>
>> the main difference between our solutions is that you sort the data in
>> category's (nodes, ways, relations). We have only one list, in
>> which this elements are unsorted. This asynchronism is necessary to
>> keep our good server performance, because the output-module does not
>> have to await for all the data. It can start the output stream before.
>> What's your point of view? Do you really need this type of sorting?
>>
>> I hope we can find a solution, to make all happy :)
>>
>> Regards Dario
>>
>>
>> On Thu, 16 Jun 2011 19:22:54 +0100
>> Matt Amos wrote:
>>
>> > hi dario,
>> >
>> > there's experimental support in cgimap for JSON format, which will
>> > hopefully get enabled at 0.7 time (whenever that is). i've attached an
>> > example of the output in both JSON and XML for comparison. it looks
>> > extremely similar to the examples you've included below, so hopefully
>> > we can work together on this. :-)
>> >
>> > cheers,
>> >
>> > matt
>> >
>> > On Thu, Jun 16, 2011 at 5:04 PM, Dario Brandes
>> > wrote:
>> > > Hello everyone,
>> > >
>> > > we from the xappy.js Team want talk with you about a JSON Standard
>> > > for osm. The GeoJSON Format is inapt to safe relations and so we
>> > > created something new.
>> > > Here the link to our wikipage:
>> > > https://github.com/slomo/osm-spline-xapi/wiki/JSON-Output
>> > >
>> > > We would like to hear your comments about our proposal.
>> > >
>> > > Regards Dario
>> > >
>> > >
>> > >
>> > > # JSON output format definition by example.
>> > >
>> > > **Note:** for simple reading we included whitespaces which are of
>> > > course omitted in the actual implementation.
>> > >
>> > > ## skeleton
>> > > {
>> > > "version": 0.6,
>> > > "generator": "xappy.js",
>> > > "xapi": {
>> > > "uri": "XXX",
>> > > "planetDate": 201106161601,
>> > > "copyright": "XXX",
>> > > "instance": "XXX"
>> > > },
>> > > "elements": [
>> > > ...
>> > > ...
>> > > ...
>> > > ]
>> > > }
>> > >
>> > > where the `elements` array obviously contains all the elements.
>> > >
>> > > ## node
>> > > {
>> > > "type": "node",
>> > > "id": 3596186,
>> > > "lat": 53.4633699598014,
>> > > "lon": -2.22667910006381,
>> > > "timestamp": "2007-06-21T17:10:58+01:00",
>> > > "version": 2,
>> > > "changeset": 2213,
>> > > "tags": [
>> > > "amenity": "hospital",
>> > > "name": "Manchester Royal Infirmary"
>> > > ]
>> > > }
>> > >
>> > > ## way
>> > > {
>> > > "type": "way",
>> > > "id": 4958218,
>> > > "version": 3,
&g
Re: [OSM-dev] openid
On Sat, Jun 18, 2011 at 3:57 PM, Manuel Reimer wrote: > Another question about openid: Is it possible to use it to, for example, to > let an editor get access to OSM or is there still a "regular" OSM profile > needed? no, an OSM account is needed, although of course the user can choose to sign up with OpenID if they want. the editor can then use OAuth to access the OSM API. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] JSON-output for xapi
hi dario,
there's experimental support in cgimap for JSON format, which will
hopefully get enabled at 0.7 time (whenever that is). i've attached an
example of the output in both JSON and XML for comparison. it looks
extremely similar to the examples you've included below, so hopefully
we can work together on this. :-)
cheers,
matt
On Thu, Jun 16, 2011 at 5:04 PM, Dario Brandes wrote:
> Hello everyone,
>
> we from the xappy.js Team want talk with you about a JSON Standard for
> osm. The GeoJSON Format is inapt to safe relations and so we created
> something new.
> Here the link to our wikipage:
> https://github.com/slomo/osm-spline-xapi/wiki/JSON-Output
>
> We would like to hear your comments about our proposal.
>
> Regards Dario
>
>
>
> # JSON output format definition by example.
>
> **Note:** for simple reading we included whitespaces which are of
> course omitted in the actual implementation.
>
> ## skeleton
> {
> "version": 0.6,
> "generator": "xappy.js",
> "xapi": {
> "uri": "XXX",
> "planetDate": 201106161601,
> "copyright": "XXX",
> "instance": "XXX"
> },
> "elements": [
> ...
> ...
> ...
> ]
> }
>
> where the `elements` array obviously contains all the elements.
>
> ## node
> {
> "type": "node",
> "id": 3596186,
> "lat": 53.4633699598014,
> "lon": -2.22667910006381,
> "timestamp": "2007-06-21T17:10:58+01:00",
> "version": 2,
> "changeset": 2213,
> "tags": [
> "amenity": "hospital",
> "name": "Manchester Royal Infirmary"
> ]
> }
>
> ## way
> {
> "type": "way",
> "id": 4958218,
> "version": 3,
> "timestamp": "2007-07-25T01:55:35+01:00",
> "changeset": 2211,
> "nodes": [
> 218963,
> 331193
> ],
> "tags":[
> "landuse": "residential",
> "source": "landsat"
> ]
> }
>
> ## relation
> {
> "type": "relation",
> "id": 2670,
> "timestamp": "2007-10-25T03:05:34Z",
> "version": 32,
> "changeset": 2211,
> "members": [
> {
> "type":"way",
> "ref":3992472,
> "role": ""
> },
> {
> "type":"way",
> "ref":3992524,
> "role": ""
> }
> ...
> ],
> "tags":[
> "name": "Fonnereau Way",
> "network": "Ipswich foothpaths",
> "type": "route"
> ]
> }
>
> ___
> dev mailing list
> dev@openstreetmap.org
> http://lists.openstreetmap.org/listinfo/dev
>
>
foo.osm.json
Description: application/json
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] rails_port seg. fault
On Tue, May 17, 2011 at 9:05 PM, Jaroslaw Wozny
wrote:
> Hi,
>
>> interesting. i've just done a test with the latest git code (e120e59)
>> and got no errors. unfortunately, my setup is different from yours:
>
> I have also the latest (head version e120e59).
>
>>> 187 tests, 2413 assertions, 3 failures, 0 errors
>
>> what's your log/test.log? are you using the postgres extensions in
>> db/functions/?
>
> Ok. I have installed extra functions for PG and no fails on tests. But I
> have still problem with seg fault. Problem is repeated by two independent
> people on Ubuntu 11.04 (two different machines). Import is performed by
> bulk_upload.py. :(
>
> Maybe something is specific in bulk_upload.py that shows problem during
> import? I don't know.
yes, it appears to be. i was able to reproduce both the segfault you
reported and the long-looked-for infinite loop bug, both with
bulk_upload.py! unfortunately, the segfault seems to be happening
because the libxml data structure is being reclaimed before the ruby
object wrapping it is dead, and it's very difficult to debug stuff
across that ruby/libxml interface.
however, there is good news: i've updated rails_port to use
libxml-ruby 2.0.5 and it seems much more robust - i haven't been able
to get it to segfault. it'll be in git master soon, but if you'd like
to try it out then a patch is attached. i'd like to know how you get
on with it and whether it has any problems.
cheers,
matt
From 5472095287a651d9915805ab7bd3deadfe8d6c3d Mon Sep 17 00:00:00 2001
From: Matt Amos
Date: Sat, 21 May 2011 15:15:49 +0100
Subject: [PATCH] Updated to libxml-ruby 2.0.5 and fixed code accordingly.
I've tested this through the unit tests and by hitting it with
bulk_upload.py and neither fail or cause the server to crash or
go into an infinite loop. Both of these things happened randomly
with 1.1.3/4 due to an apparent early deregistration of the
expanded nodes.
---
config/environment.rb |2 +-
lib/diff_reader.rb| 19 ++-
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/config/environment.rb b/config/environment.rb
index 58d9432..d5cee81 100644
--- a/config/environment.rb
+++ b/config/environment.rb
@@ -18,7 +18,7 @@ Rails::Initializer.run do |config|
unless STATUS == :database_offline
config.gem 'composite_primary_keys', :version => '2.2.2'
end
- config.gem 'libxml-ruby', :version => '~> 1.1.1', :lib => 'libxml'
+ config.gem 'libxml-ruby', :version => '>= 2.0.5', :lib => 'libxml'
config.gem 'rmagick', :lib => 'RMagick'
config.gem 'oauth', :version => '>= 0.4.3'
config.gem 'oauth-plugin', :version => '0.3.14'
diff --git a/lib/diff_reader.rb b/lib/diff_reader.rb
index 0f0492f..de2da3c 100644
--- a/lib/diff_reader.rb
+++ b/lib/diff_reader.rb
@@ -20,6 +20,10 @@ class DiffReader
def initialize(data, changeset)
@reader = XML::Reader.string(data)
@changeset = changeset
+# document that's (re-)used to handle elements expanded out of the
+# diff processing stream.
+@doc = XML::Document.new
+@doc.root = XML::Node.new("osm")
end
##
@@ -85,8 +89,21 @@ class DiffReader
model = MODELS[model_name]
raise OSM::APIBadUserInput.new("Unexpected element type #{model_name}, " +
"expected node, way or relation.") if model.nil?
- yield model, @reader.expand
+ # new in libxml-ruby >= 2, expand returns an element not associated
+ # with a document. this means that there's no encoding parameter,
+ # which means basically nothing works.
+ expanded = @reader.expand
+
+ # create a new, empty document to hold this expanded node
+ new_node = @doc.import(expanded)
+ @doc.root << new_node
+
+ yield model, new_node
@reader.next
+
+ # remove element from doc - it will be garbage collected and the
+ # rest of the document is re-used in the next iteration.
+ @doc.root.child.remove!
end
end
--
1.7.4.1
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] rails_port seg. fault
On Mon, May 16, 2011 at 9:49 PM, Jaroslaw Wozny wrote: > >> libxml-ruby (1.1.4) should be 1.1.3 > > Hi, > > Reinstalled and I have got similar problem. > > Error: > 2011-05-16 22:39:13.659425 #13825] > ERROROAuth::Signature::UnknownSignatureMethod > [2011-05-16 22:39:13.661038 #13825] Filter chain halted as [:authorize] > rendered_or_redirected. > [2011-05-16 22:39:13.661322 #13825] Completed in 39ms (View: 1, DB: 1) | 401 > Unauthorized [http://127.0.0.1/api/0.6/changeset/1/upload] > /home/jarekw/rubygems-1.3.7/rails/app/models/node.rb:82: [BUG] Segmentation > fault > ruby 1.8.7 (2010-08-16 patchlevel 302) [x86_64-linux] > > My list of gems: > *** LOCAL GEMS *** > > actionmailer (2.3.8) > actionpack (2.3.8) > activerecord (2.3.8) > activeresource (2.3.8) > activesupport (2.3.8) > httpclient (2.2.0.2) > i18n (0.5.0) > libxml-ruby (1.1.3) > nokogiri (1.4.4) > oauth (0.4.4) > oauth-plugin (0.3.14) > pg (0.11.0) > rack (1.1.2) > rails (2.3.8) > rake (0.8.7) > rmagick (2.13.1) > sanitize (2.0.1) > SystemTimer (1.2.3) > timecop (0.3.5) interesting. i've just done a test with the latest git code (e120e59) and got no errors. unfortunately, my setup is different from yours: $ ruby -v ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux] $ gem list *** LOCAL GEMS *** actionmailer (2.3.8) actionpack (2.3.8) activerecord (2.3.8) activeresource (2.3.8) activesupport (2.3.8) gash (0.1.3) gemcutter (0.7.0) grancher (0.1.5) haml (2.2.24) hanna (0.1.12) httpclient (2.2.0.2) i18n (0.5.0) json (1.5.1) libxml-ruby (2.0.3, 1.1.3) nokogiri (1.4.4) oauth (0.4.4) oauth-plugin (0.3.14) open4 (0.9.6) pg (0.11.0) rack (1.1.2) rails (2.3.8) rake (0.8.7) rake-compiler (0.7.8) rake-rubygems (0.2.0) rake-tasks (0.2) rdoc (2.3.0) rmagick (2.13.1) sanitize (2.0.1) SystemTimer (1.2.3) test-unit (2.3.0) test-unit-ext (0.5.0) timecop (0.3.5) wirble (0.1.3) and i'm (still) running ubuntu 10.10. > Note: > I executed tests and it shows 3 problems: > 1) Failure: > test_changes_simple(ApiControllerTest) > [/test/functional/api_controller_test.rb:207]: > Expected response to be a <:success>, but was <500> > <""> > > 2) Failure: > test_changes_zoom_valid(ApiControllerTest) > [/test/functional/api_controller_test.rb:231:in `test_changes_zoom_valid' > /test/functional/api_controller_test.rb:229:in `upto' > /test/functional/api_controller_test.rb:229:in > `test_changes_zoom_valid']: > Expected response to be a <:success>, but was <500> > <""> > > 3) Failure: > test_hours_valid(ApiControllerTest) > [/test/functional/api_controller_test.rb:260:in `test_hours_valid' > /test/functional/api_controller_test.rb:258:in `upto' > /test/functional/api_controller_test.rb:258:in `test_hours_valid']: > Expected response to be a <:success>, but was <500> > <""> > > 187 tests, 2413 assertions, 3 failures, 0 errors what's your log/test.log? are you using the postgres extensions in db/functions/? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Filename extensions for OSM files
On Wed, May 11, 2011 at 12:23 PM, Peter Körner wrote: > Am 11.05.2011 13:17, schrieb Matt Amos: >>> >>> We should probably introduce an attribute on the tag that says that >>> this is a history file. Just as we are currently discussing for the PBF >>> version. >> >> while we're at it, let's have a flag to indicate if the elements are >> sorted by ID, how many prime-numbered elements there are, whether >> there was a full moon when it was generated and what the colour of the >> file is. i'm up for all of that as long as it's blue. > > history files have similar syntax as normal osm files, but very different > semantics. A program processing a history file needs to be aware or it > - having multiple objects with the same id > - having objects that stop being existent in some point in time > - that have an extra information attached > > that way it is useful for a program to know, if a file fed into it via stdin > will be processed fine right from the start. nobody likes processes that > fail after 10 hours because of a missing --this-is-an-history-file flag. it'll be far less than 10 hours before the first element with visible="false" or multiple versions comes through. in fact, i bet it'd happen within the first 10 lines. all OSM files are potentially history files (e.g: http://osm.org/api/0.6/node/1/history) - they all have the visible and version attributes on all elements, despite that not being needed for dealing with a snapshot. some programs may make assumptions about the contents of OSM files, but that's for the program in question to assert, and for the user to be aware of. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Filename extensions for OSM files
On Wed, May 11, 2011 at 11:26 AM, Jochen Topf wrote: > On Wed, May 11, 2011 at 11:13:37AM +0200, Peter Körner wrote: >> Am 10.05.2011 22:31, schrieb David Paleino: >> >On Tue, 10 May 2011 20:58:40 +0200, Peter Körner wrote: >> > >> >>Frederik mentioned mime-types, I'd suggest (not knowing if I'd break any >> >>convention using multiple + symbols): >> >> >> >>application/x-osm+xml for .osm >> >>[..] >> > >> >I'd suggest we should also implement some magic(5) lines for file(1). >> >> Well, that would only work if you could differentiate between osm, >> osh and osc files by looking at the first lines. >> >> osm-files have an -tag, osc-files start with > ../> but osh-files are, currently, only osm-files with a >> visible-attribute on each entity - so no way to distinguish them >> from osm-files. > > We should probably introduce an attribute on the tag that says that > this is a history file. Just as we are currently discussing for the PBF > version. while we're at it, let's have a flag to indicate if the elements are sorted by ID, how many prime-numbered elements there are, whether there was a full moon when it was generated and what the colour of the file is. i'm up for all of that as long as it's blue. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] how long does it take to generate the full-history dump?
the last one took about 60 hours - 2011-04-18 00:00 is the time according to the file name, 2011-04-20 12:11 is the file timestamp. that's not completely accurate, but is probably to within an hour or two. note that the full planet dump is not optimised for speed - nor would we want it to be, as the more resources are devoted to the dump, the fewer are available to editors - the availability of minutely / hourly replication dumps makes it kinda pointless anyway. cheers, matt On Tue, May 3, 2011 at 10:06 PM, Manchun Yao wrote: > Hi Peter, > > I am not using the tool. I just want to get a general idea about how long it > take to produce the full-history dump. > > -Manchun > -Original Message- > From: Peter Körner [mailto:osm-li...@mazdermind.de] > Sent: Tuesday, May 03, 2011 1:27 PM > To: Manchun Yao > Cc: dev@openstreetmap.org > Subject: Re: [OSM-dev] how long does it take to generate the full-history > dump? > > Am Dienstag, den 03.05.2011, 20:14 + schrieb Manchun Yao: >> How many hours does it take to generate the full-history dump using >> the historydump tool? > > Can't you look into the result-file using the tools tail or less? So you can > check how much of the dump has already been written. > > Peter > > > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Using osmosis to generate a full-planet.osm file
On Thu, Apr 7, 2011 at 8:30 AM, Frederik Ramm wrote: > Scott, > > On Wed, 6 Apr 2011 22:06:57 -0500 >> I believe the osmosis database representation for a planet is not the >> same as what is used by osm servers. OSM servers use a special dump >> program for generating planets. > > I think Eric was talking about the history planets, > not the normal planet files. Neither of these is created with Osmosis > as far as I know but I'm not sure what software is used for the full > history dumps. the full history planets are created using lars francke's historydump [1] tool. cheers, matt [1] https://bitbucket.org/lfrancke/historydump/overview ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] next London hack day
thanks to everyone who came to the hack day last weekend in London [1] - i had a lot of fun and i hope everyone else did too. :-) i'd like to keep the ball rolling by asking if anyone's interested in having another of these events in the summer, possibly around the end of May or the start of June. i've set up a wiki page to collect everyone's thoughts and availability [2] and hopefully we'll have the same great mix of talks, development and social events. happy hacking, matt [1] http://wiki.openstreetmap.org/wiki/London_Hack_weekend_April_2011 [2] http://wiki.openstreetmap.org/wiki/London_Hack_Weekend_Summer_2011 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenID for OpenStreetMap?
On Thu, Feb 10, 2011 at 8:09 PM, Serge Wroclawski wrote: > On Thu, Feb 10, 2011 at 2:24 PM, Matt Amos wrote: > >> i might be missing the point here, but are you suggesting that OSM >> would be an openID provider, consumer or both? > > I'm suggesting that instead of taking on a technology, we take on a > set of requirements, or needs, and address them. And from there I'm > saying that OpenID itself is a side issue. ok - so what's the problem, or "requirement" in corporate-speak, that we're trying to address here? >>> The second issue is OpenID support in general. >>> >>> MHO on the issue is that while it might seem that OpenID support is >>> great and widely useful, I think there are a number of implementation >>> issues and concerns which are really hard to overcome. >>> >>> For example, how does one handle OpenID with an API? This has been >>> discussed in other contexts, and unfortunately there's no good single >>> answer. >> >> my understanding is that openID would be usable on the website to >> generate an OAuth token, which would then be used to access the API. >> this seems like a good enough solution to me, but i could be missing >> something. > > That sounds like it makes non-web based editors hard to develop. nope. the user has to log into the website to authorize an OAuth token - what does it matter whether they log in via the usual username and password or OpenID? certainly it doesn't matter to the editor, even if it's doing some sort of ugly hack like JOSM's "fully automatic" mode. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OpenID for OpenStreetMap?
On Thu, Feb 10, 2011 at 7:11 PM, Serge Wroclawski wrote: > I think the conversation is really two different questions. > > First is the original question about OpenID in the context of OSM and > the other is about OpenID support generally. > > I think part of the feeling for a need for OpenID in OSM is about the > fact that for several of our services, authentication isn't unified. > That is the wiki credentials don't map to the OSM API credentials. > help.osm does though. It's confusing. i might be missing the point here, but are you suggesting that OSM would be an openID provider, consumer or both? i had thought that the discussion was limited to being an openID consumer - i'm not sure we'd want to be a provider, with all that it entails. > The second issue is OpenID support in general. > > MHO on the issue is that while it might seem that OpenID support is > great and widely useful, I think there are a number of implementation > issues and concerns which are really hard to overcome. > > For example, how does one handle OpenID with an API? This has been > discussed in other contexts, and unfortunately there's no good single > answer. my understanding is that openID would be usable on the website to generate an OAuth token, which would then be used to access the API. this seems like a good enough solution to me, but i could be missing something. > I personally believe OpenID is a great idea, but as Tom and others > point out, the devil's in the details. > > OTOH the former issue, of a unified authentication mechanism- that's > not technically hard (for the most part)- that's just a lot of work > and a lot of disruption to the community while the transition takes > place. the devil's in the details and the SMOP. for instance, case sensitivity of usernames, allowable characters, length and other conditions. iirc, this wasn't a big deal when the wiki was set up, and many people signed up with the same username. but since then things have drifted a little bit (including the slightly inadvertent switch to case-sensitive usernames for the main site) and it seems like way more trouble than it's worth to transition now. cheers, matt > That is, I'm sure the technical team knows how to solve it, but as > someone whose completed a user authentication merge, I can tell you > that it's painful. It's especially painful on something like a wiki, > where not only must a user remapping take place, but that user mapping > must be applied to each and every page. > > I think for new services (like help), we've solved the issue, but we > may have to either accept the pain of the current situation for the > wiki, or get buy in from the community for the disruption any change > would cause. > > - Serge > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] scaling
On Tue, Jan 11, 2011 at 1:28 PM, Kai Krueger wrote: > What this shift of pattern means for scaling is imho not entirely clear. On > the one hand more and large imports would likely put more burden on the > write paths of the API, as imports don't need to download data first to edit > it. On the other hand with the growing density of map data, each map call is > likely becoming much more complex than it used to be, shifting the burden > back towards read loads. however, those imports probably *should* be doing some sort of reading in order to ensure they're not duplicating or stomping on existing data. the fact that they don't generally at the moment is (probably) not a reflection on the speed of the API ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] scaling
On Mon, Jan 10, 2011 at 11:09 PM, Frederik Ramm wrote: > Brett Henderson wrote: >> >> That's not ideal. What's the implication of this? What OSM code uses >> temp tables? >> >> Osmosis uses them for extraction of minutely diffs. Are there others? > > Matt's cgimap code does. it's possible to rewrite it so that it doesn't use temp tables, but that would be at the cost of more complex queries or higher memory usage. i presume that postgres doesn't currently allow it because it's technically non-trivial, but beyond that i have no idea what the level of difficulty is... i had a quick look on the postgres mailing lists and didn't spot anything obvious about it - is it worth bringing it up there and seeing what their take on it is? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] scaling
On Mon, Jan 10, 2011 at 1:31 PM, Andy Allan wrote: > On Mon, Jan 10, 2011 at 12:44 PM, Robert Scott wrote: >> On Monday 10 January 2011, Kai Krueger wrote: >>> Depending on how far you really want to scale, I think a lot of the >>> necessary components are already in place. If none of the "out of the box" >>> solutions such as the new Postgresql 9.0 replication mechanism work, we >>> would possibly get a fair distance by splitting out reads and writes onto >>> separate db servers. >> >> This is how the postgres 9 replication works. The replicating servers become >> "hot standbys" which you can use for read requests. So in theory the read >> requests could be scaled quite easily once set up. Atomicity of the API >> would potentially suffer though. > > We need to be careful about what purposes we scale read requests. A > lot of the read-load on the API is using the data for non-editing > purposes (such as rendering or other 'cool' things that need /map > calls) and this should be avoided[1]. The advantage of making sure > that read-scaling for non-editing purposes is *not* via db-replication > is that anyone (and everyone) can add their own servers to scale > things out. For example, everything that currently fetches live data > via the diffs at planet.openstreetmap.org (or a service that depends > on them, like xapi, geofabrik extracts etc) is a Good Thing, and > everything that currently fetches live read-only data via > osm.org/api/0.6/map is a Bad Thing. > > It's a bad thing since there's only so much hardware the OSMF can > own/host/run, but there's nothing stopping 50 other organisations > running cool stuff based off of planet.openstreetmap.org it seems postgres 9 isn't able to do temporary table creation on hot-standby servers [1]. maybe this is something that'll be fixed in future versions (but it doesn't seem to be on the TODO [2]) or maybe it's an opportunity to put a bounty to good use. cheers, matt [1] http://www.postgresql.org/docs/9.0/interactive/hot-standby.html#HOT-STANDBY-USERS [2] http://wiki.postgresql.org/wiki/Todo ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Node stuck in the database?
On Thu, Dec 23, 2010 at 10:15 PM, Phil! Gold wrote: > I'm not sure whether this is the right venue, but I was directed here from > IRC. > > I can't upload a changeset (from JOSM); the upload seems to hang on node > 37521665. I've had a few problems with this set of changes and I finally > dialed JOSM down to sending one object at a time. Every other object > seems to be uploaded perfectly well, but when it hits that node (which I > moved), it prints > > PUT http://api.openstreetmap.org/api/0.6/node/37521665... > > on the console, displays "Uploading node: '37,521,665' (id: 37,521,665)" > in the upload progress dialog box, and just sits there like that until I > cancel the upload. I'm using JOSM 3739, though the past couple testing > builds have all had the same issue, as does the current tested release, > 3701. > > Occasionally, I've had things stall on nodes 37521667 and 37719724. > > This seems like a database issue, but could it be something in my upload > that's causing problems? there were a few dead processes holding locks in the database, mainly changeset closes for some reason. i've killed these and the locks have been released. could you please try again and let me know if you're still able to reproduce the stall on any of those nodes? cheers, matt > -- > ...computer contrarian of the first order... / http://aperiodic.net/phil/ > PGP: 026A27F2 print: D200 5BDB FC4B B24A 9248 9F7A 4322 2D22 026A 27F2 > --- -- > 'Twas brillig, and the slithy toves > Did gyre and gimble in the wabe. > All mimsy were the borogroves > And the mome raths outgrabe. > -- Lewis Carroll, "Jabberwocky" > --- -- > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Compression types in PBF Format
On Tue, Nov 30, 2010 at 8:41 PM, Stefan de Konink wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hi, > > > Op 30-11-10 20:49, Frederik Ramm schreef: >> Stefan de Konink wrote: >>> This is the place for the 'too little, too late'. We are beyond the >>> point of 'what' the bitstream should look like: you ought to handle >>> what is defined now. >> >> This is not how we work in OSM. We don't have standards. > > For some reason we do, this is not a free form fight. And if we can > change the API every week, I wonder why we are still at XML then. because XML is a nearly human-readable, easy to explain and inspect format. the same cannot be said of the PBF format, but then the declared design goals of it were reduction in parsing time and file size, not readability - and it achieves those goals superbly. however, i think using it in the API wouldn't provide enough of a speedup, limited by Amdahl's law, to offset the loss of those other benefits. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] full history dump 2010-10-22
On Mon, Oct 25, 2010 at 8:20 PM, Marco Lechner - FOSSGIS e.V. wrote: > Hi Matt, > > thank you. Is it just a more recent one or are there any changes in the > schema? it's just a more recent one for convenience. it should be identical to the intervening replication diffs applied to the previous full history planet. cheers, matt > Marco > > Am 25.10.2010 21:03, schrieb Matt Amos: >> there's a new full history file available: >> >> http://planet.openstreetmap.org/full-experimental/full-planet-101022.osm.bz2 >> >> cheers, >> >> matt >> >> ___ >> dev mailing list >> dev@openstreetmap.org >> http://lists.openstreetmap.org/listinfo/dev >> > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] full history dump 2010-10-22
there's a new full history file available: http://planet.openstreetmap.org/full-experimental/full-planet-101022.osm.bz2 cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Data source for robot
On Tue, Oct 12, 2010 at 6:20 PM, Jan Sandbrink wrote: > sorry, but what i see happening on the list a lot of times is, that people > who just want to contribute somehow are smashed into the ground. i welcome mr budny's contribution - as a tool for users to identify and fix problems with the map. since the beginning of this thread the advice has simply been "don't run a bot". as you rightly point out, we've had bad experiences with people running bots and we'd rather it didn't happen again. i encourage mr budny to do his project and release the results (and hopefully even the source code, but some universities have issues with that), but *not* to automatically run them against the live database. putting them up on a site, or making them visible on a map, would allow them to be integrated by individual mappers with local knowledge and without damaging any existing data. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Data source for robot
On Tue, Oct 12, 2010 at 5:50 PM, Andrew wrote: > Matt Amos gmail.com> writes: > >> of course, the best thing is that these automated edits never happen >> at all, instead that tools are provided (like the geofabrik inspector, >> keep-right or the duplicate nodes map) to help the community fix these >> errors themselves. we need to start cracking down on these disruptive >> and counter-productive bots. > > The duplicate nodes map is not the best example of this. People have sometimes > jumped in and done heavy edits not based on local knowledge. i totally agree, which is why the about page of the duplicate nodes map says: "This means that each point marked on the map isn't definitely an error, but it could be one" and the wiki page says "It is important to note that, despite the design of the map, these are not all necessarily errors, and should not all be 'fixed' by merging nodes". no-one should be doing edits, especially on a large scale, without a healthy dose of common sense and, preferably, local knowledge. > A bot that joined > roads to other roads (and only to other roads) at county boundaries would > actually be more constructive as we could then add a notice that this issue > with TIGER roads has been fixed completely. i don't think so - it might be more constructive to have a map which only displayed duplicates between roads and other roads at county boundaries so that US mappers could concentrate their efforts - in fact, it's something i once ran and (time permitting) hope to develop again. a bot, however, would always be less constructive than real people looking at the data. see http://matt.dev.openstreetmap.org/dupe_nodes/about.html#just_use_a_bot cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Data source for robot
On Tue, Oct 12, 2010 at 3:48 PM, Peter Budny wrote: > Matt Amos writes: > >> On Tue, Oct 12, 2010 at 11:56 AM, Serge Wroclawski wrote: >>> 3) The road to hell in OSM is paved with bot intentions. >>> >>> OSM has a long, negative history with bots. We have a very small >>> number of good imports, and dozens (if not more) bad imports. Bad >>> imports are so commonplace in OSM that within the OSM community, bots >>> of any sort are discouraged, but especially any imports, and >>> especially (as you appear to be proposing), merging existing data with >>> imported data. >> >> +1 >> >> bots and imports are also fantastic ways of washing away community >> consensus for the ideas of a single person. > > http://wiki.openstreetmap.org/wiki/Interstate_Highway_relations > http://wiki.openstreetmap.org/wiki/United_States_Numbered_Highway_Relations > http://wiki.openstreetmap.org/wiki/Michigan/Highway_Relations > > Maybe I'm unclear what you mean by "consensus"? Those look pretty > agree-upon to me. what i mean by "consensus" is "general agreement by a group of people". but this doesn't mean everyone agrees and then bots go and stomp over all the data. i mean the kind of consensus where each individual mapper is editing in a way that makes sense to them and a common system emerges. the very concept of a mass edit or bot is, by definition, unilateral and not consensual. secondly, just because something says "agreed" on the wiki doesn't make it so. the wiki is for documentation and discussion of existing tags. the real test of whether a scheme is "agreed" is whether it is popularly-used - by many users, not by a small number of proponents and bot-writers. have a look on tag[watch|info|stat] to see whether these schemes are in use and encourage people to use them - provide them tools and let them vote with their feet, rather than imposing a unilateral "agreement" by bot. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Data source for robot
On Tue, Oct 12, 2010 at 11:56 AM, Serge Wroclawski wrote: > 3) The road to hell in OSM is paved with bot intentions. > > OSM has a long, negative history with bots. We have a very small > number of good imports, and dozens (if not more) bad imports. Bad > imports are so commonplace in OSM that within the OSM community, bots > of any sort are discouraged, but especially any imports, and > especially (as you appear to be proposing), merging existing data with > imported data. +1 bots and imports are also fantastic ways of washing away community consensus for the ideas of a single person. there has been, for a very long time, a code of conduct for such automated or mechanical edits (not just bots and imports, but also mass edits with the usual editors) [1]. it's interesting that almost no-one has followed the code of conduct, including logging their activity on the log [2]. of course, the best thing is that these automated edits never happen at all, instead that tools are provided (like the geofabrik inspector, keep-right or the duplicate nodes map) to help the community fix these errors themselves. we need to start cracking down on these disruptive and counter-productive bots. cheers, matt [1] http://wiki.openstreetmap.org/wiki/Automated_Edits/Code_of_Conduct [2] http://wiki.openstreetmap.org/wiki/Automated_Edits/Log ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Fail to get OsmChange for some changesets
On Sat, Oct 9, 2010 at 1:04 AM, Manchun Yao wrote: >> Yes, it's perfectly possible. Individual changes to the data can occur >> without the changeset being closed. This is completely by design, although >> many people are confused into thinking that changesets are atomic or delayed >> until they are closed or similar. They aren't, and that's deliberate. As for >> how likely it is to happen without someone trying to do it deliberately? I >> can't answer that. > > I can imagine this now.: > Suppose we start with: node a, version 7; node b version 7 > Two persons operate on the these two nodes in change set A and change > B respectively with the following sequence: > - A changes node a first: node a version 7 -> 8 > - B changes node b: node b version 7->8 > - A changes node b: node b version 8->9 > - B changes node a: node a version 8->9 > - A change set closes > - B change set closes > So we have: > Change A: node a, version 7->8, node b version 8->9 > Change B: node a, version 8->9, node b version 7->8 > Although change A closes before change B, if we applied change A first > without considering change B at all, there will be problems. > > If this is the case, will the accumulative replication process (applying osc > changes daily/minutely) ran into problems? not in my experience. i've been running a tool consuming minutely replication osc files since feb and i haven't seen these problems. this is because the replication osc files do not wait to contain whole changesets - since they're not atomically committed it wouldn't make sense. however, the osc files do contain the changes which are applied to the history tables on the level of database transactions rather than changesets. cheers, matt > -Manchun > -Original Message- > From: Andy Allan [mailto:gravityst...@gmail.com] > Sent: Thursday, October 07, 2010 3:14 AM > To: Manchun Yao > Cc: dev@openstreetmap.org > Subject: Re: [OSM-dev] Fail to get OsmChange for some changesets > > On Wed, Oct 6, 2010 at 5:59 PM, Manchun Yao wrote: >> Hi Andy, >> >> If all change sets were created and operated strictly based on API 0.6, >> would the scenario you described (two change sets affecting different >> objects in non-sequential orders) ever happen? Is the optimistic locking >> mechanism meant to avoid such things? The problem might exist in the >> database because of historical reasons though. > > Yes, it's perfectly possible. Individual changes to the data can occur > without the changeset being closed. This is completely by design, although > many people are confused into thinking that changesets are atomic or delayed > until they are closed or similar. They aren't, and that's deliberate. As for > how likely it is to happen without someone trying to do it deliberately? I > can't answer that. > >> I indeed want to build the history of all the changes so that I can find the >> map at some time (for example, Jan. 1, 2008) before. The full-planet dump >> files at (http://planet.openstreetmap.org/full-experimental) contain >> historical info. However, the file is a bit too big for my computer. I was >> trying to test whether I can build the history from ground up for a small >> area by getting the individual change sets one by one. Looks like there are >> some issues because some "changes" were made before API 0.6 and I have >> problem to access them using 0.6 API calls. > > You could do it by collecting all the data that those changesets refer to, > and then "replay" all the data changes in the order the individual changes > occurred (not in the order that the changesets were either opened or closed, > which I've explained isn't necessarily the correct order). > > The wider point is that the history dumps are, almost entirely, currently > unusable for most people. If anyone has any suggestions for solving this > problem, or wants to try making a toolchain to process the history dumps into > history extracts, that would be great. There was discussion previously on > adapting osmosis to handle history dumps, but I don't know if that resulted > in any code. > >> Do you have any suggestions about how to get these problematic change sets? >> Maybe someone with a database populated with the full-planet dump can get >> these change sets for me (145015, 4318130, 4318132, 4318136). > > Perhaps this is supported by one of the read-only api services? I'm not sure > if any of them (XAPI, ROMA, OWL etc) have access to enough history though. > > Cheers, > Andy > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OAuth - 401 Authorization Required (OAuth::Unauthorized)
On Mon, Sep 27, 2010 at 4:24 PM, Christoph Bünte wrote: > Ok, it seems to me, that I'am the only one having this problem. That makes me > think, that either I'm the only one who recently tried to connect an oauth > consumer with a provider. Or it is just a ruby issue, like some weeks ago, > when a new commit broke this functionality. Or, just to be complete, it is > just me having this problem at all, cause my code is somewhat faulty. > > Could anyone try the example code with your app? Cause this issue keeps us > from starting our beta phase and publish wheelmap to a broader audience. > > As always: Any help is highly appreciated. it looks like oauth now needs the verification token from the callback / printed on the website when a token is authorised for OAuth 1.0a support [1]. i've updated the example at http://wiki.openstreetmap.org/wiki/OAuth/Examples to reflect this and it now works for me. hopefully it'll work for you too. cheers, matt [1] http://git.openstreetmap.org/?p=rails.git;a=commit;h=1c3a9ee62b7d1a0dc97d52b1a498be1339d49ebf ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
Kai just showed me your demo running on dev, which is great - sorry I missed that. The data looks pretty up-to-date, is it being updated weekly or more often? Is it world-wide? Cheers, Matt On Sep 4, 2010 6:55 PM, "Matt Amos" wrote: On Fri, Sep 3, 2010 at 10:14 AM, Nic Roets wrote: > On Thu, Sep 2, 2010 at 5:20 P... are you saying that kernel.org can be unfriendly because the amount of effort put in by each contributor is higher? i don't think that's the right way to look at it - kernel.org doesn't need to showcase the capabilities of linux because there are many other providers and distributions doing this. the people who are using linux, for example on an android phone, don't need to know about kernel.org, and don't need to care. the people who are using ubuntu don't need to know about kernel.org, since ubuntu provides almost all of their needs. in my opinion it can be the same with osm.org - people can be using the data (for example, on bing, mapquest or cloudmade) without ever needing to see the osm.org site. those who are interested enough to want to contribute can do it directly through the provider, or be directed to osm.org. > So we need to make the environment as > friendly as possible. i wouldn't disagree with you there, and providing routing on the main page might make the experience slightly nicer, but it comes with a cost to OSM in terms of developing, integrating and running the service which is non-zero. anyone who's interested in getting a service on the front page has to make it as friendly as possible, not just to the users, but also to the admins. my suggestion for getting anything integrated with the main site is to set up a patched rails port* and your running service on dev.osm.org, announce it and keep it up to date for a while. when people have been using it, and it's clear that it's stable and what computational and admin requirements it has, it has a much higher chance of being integrated into the main page. cheers, matt * it has to be a rails port and not a static page. either integrating it into rails port is easy (in which case you can do it - it's easy) or it's hard (in which case there's a big barrier to getting it adopted, so you should do it to make it easier to integrate). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Fri, Sep 3, 2010 at 10:14 AM, Nic Roets wrote: > On Thu, Sep 2, 2010 at 5:20 PM, Ævar Arnfjörð Bjarmason > wrote: >> >> > You'd want me to spend $12 a day to provide geolocalisation for an >> > OSM editor (if you didn't read the thread, I remind you I'm speaking >> > of Merkaartor)!!?? >> >> What sort of crazy hosting costs $360 per month (asking Nic Roets)? I > > I said $12 a day for a server that can provide a decent routing service. > (EC2 High Memory Instance). > > The kernel.org analogy is not really valid. They don't call themselves the > "wiki kernel". When I hack a kernel for an embedded project running on > specialized hardware, they certainly are interested in my changes. > > By contrast, we will accept users who come in and make a few changes, > spending as little as 10 minutes. are you saying that kernel.org can be unfriendly because the amount of effort put in by each contributor is higher? i don't think that's the right way to look at it - kernel.org doesn't need to showcase the capabilities of linux because there are many other providers and distributions doing this. the people who are using linux, for example on an android phone, don't need to know about kernel.org, and don't need to care. the people who are using ubuntu don't need to know about kernel.org, since ubuntu provides almost all of their needs. in my opinion it can be the same with osm.org - people can be using the data (for example, on bing, mapquest or cloudmade) without ever needing to see the osm.org site. those who are interested enough to want to contribute can do it directly through the provider, or be directed to osm.org. > So we need to make the environment as > friendly as possible. i wouldn't disagree with you there, and providing routing on the main page might make the experience slightly nicer, but it comes with a cost to OSM in terms of developing, integrating and running the service which is non-zero. anyone who's interested in getting a service on the front page has to make it as friendly as possible, not just to the users, but also to the admins. my suggestion for getting anything integrated with the main site is to set up a patched rails port* and your running service on dev.osm.org, announce it and keep it up to date for a while. when people have been using it, and it's clear that it's stable and what computational and admin requirements it has, it has a much higher chance of being integrated into the main page. cheers, matt * it has to be a rails port and not a static page. either integrating it into rails port is easy (in which case you can do it - it's easy) or it's hard (in which case there's a big barrier to getting it adopted, so you should do it to make it easier to integrate). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: gazetteer.osm.org being retired
On Mon, Aug 30, 2010 at 10:05 PM, Nic Roets wrote: > On Mon, Aug 30, 2010 at 10:05 PM, Matt Amos wrote: >> >> > Is there a list available of APIs and services that OSM provides which >> > are >> > considered "enterprise ready"? In terms of "enterprise ready" I'd >> > expect a >> > few commitments to be made, most importantly: >> >> i don't think any of the services OSM provides are "enterprise ready". >> OSM runs these services as aids to mapping and to show what can be >> done with the data. all of the software used to run OSM services is >> free, so any enterprise needing those services can run them itself. > > Matt, my calculations show that running such services is an extremely cheap > way of improving the OSM database. Serving a query can cost as little as > 0.1 cent. So if there is just a one percent chance that the data > underlying the query is wrong and if only 1 in 100 users actually decides to > fix the data, then the cost to fix one mistake is still less than one cent. > Compare that to the fuel cost of someone driving just one block to capture > an unnamed street. absolutely. but you're missing the opportunity cost of using that hardware to run other services which might work better, or give new abilities. note that no-one is saying that OSM should do without a geocoding server - nominatim is that service and (to my knowledge) there are no plans to retire it. it's simply that namefinder, as a secondary, older, geocoder takes some hardware to run that could be put to better purposes. >> alternatively, a commercial service from one of the companies selling >> OSM-based services might be willing to give an "enterprise" guarantee. > > Let me just mention to Torsten that you have conflict of interest here: Your > employer is such a company. not since november - the london development team was let go due to lack of funding. for the last 9 months my employer is someone entirely unrelated to OSM. > Torsten, I think the option you would be more interested in would be to run > the software on your own servers, or to collaborate with the providers of > such services that use open source software. which is pretty much what i was saying, thanks ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: gazetteer.osm.org being retired
On Mon, Aug 30, 2010 at 4:33 PM, Torsten Rahn wrote: > Would there be a way of leaving nominatim support intact via the given API > just with more limited capabilities (so that it still works with reduced > quality and taking less hardware resources)? nominatim will continue to work and if someone were to write a forwarding service which talks namefinder protocol, it could provide a stop-gap. > Is there a low traffic OSM announcement list that provides announcements like > these for people who rely on OpenStreetMap services? there is an announce list, but for development announcements like this one, this dev list is the best place. > Is there a list available of APIs and services that OSM provides which are > considered "enterprise ready"? In terms of "enterprise ready" I'd expect a > few commitments to be made, most importantly: i don't think any of the services OSM provides are "enterprise ready". OSM runs these services as aids to mapping and to show what can be done with the data. all of the software used to run OSM services is free, so any enterprise needing those services can run them itself. alternatively, a commercial service from one of the companies selling OSM-based services might be willing to give an "enterprise" guarantee. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] preconditions_ok? in app/models/relation.rb and way.rb
On Sun, Aug 22, 2010 at 5:24 PM, Mitja Kleider wrote:
> Richard Fairhurst wrote:
>> I think you're misreading how the relation integrity checks work. The
> API
>> only checks whether an entity is still visible when the entity is
> _added_
>> to a relation.
>> For performance reasons, it does not check the entities
>> formerly in the relation.
>
> Thanks for clarifying. That is probably the cause.
>
>
>> It assumes that they're ok, which in the case of historic
>> (pre-integrity check) data may not be the case. See preconditions_ok? in
>> app/models/relation.rb in the Rails port for the code.
>
> When were the integrity checks introduced?
many of them were in-place before API 0.6 went live on 2009-04-21, but
0.6 (i think) included far more of them and in a more consistent
manner.
> Are you sure that checking only _added_ entities is enough? I read it like
> that: for performance reasons the checks are done when the entity is
> deleted (way node in the next case).
when a referenced entity is deleted it is checked for membership in
any way or relation and, if it's a member of another entity, the
deletion is disallowed. when a new entity is created, or elements
added to an existing entity, then those members are checked to ensure
they're not deleted. in a fully serialised model this should be enough
to prevent any referential integrity problems. but...
> Looking at the other example
> ("http://www.openstreetmap.org/api/0.6/way/60263972/1";), the deleted nodes
> contained in that way (e.g.
> "http://www.openstreetmap.org/api/0.6/node/338392109/5";) were deleted May
> 2010 (2010-05-26).
>
> In my understanding it still should not have happened, or am I missing
> something?
unfortunately, we don't have a fully serialised model and many
uploads, particularly from potlatch, execute in parallel. this leads
to the case above where the node is being deleted and the way created
at very similar times, but in different transactions. the timeline
probably looks like:
1. (transaction A) request is received for node deletion. checks in
software indicate that it isn't part of any ways or relations. the
node is deleted.
2. (transaction B) request is received for way creation. checks in
software indicate that the nodes are all not-deleted (since
transaction A hasn't committed yet). the way is created.
3. (transaction B) commits without problems, as the FK constraint is satisfied.
4. (transaction A) commits without problems.
since nodes are stored in the database with a "visible" flag to
indicate whether they're deleted or not, the FK constraint can't catch
this error. it's a similar story with relation members, as the current
schema can't be FK checked.
API 0.6 drastically improved things over 0.5 in its referential
integrity, but we didn't manage to catch all the problems. hopefully
we'll get them in 0.7. until then, when you find one of these it's
best just to fix it.
cheers,
matt
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to clean up Negative IDs
On Tue, Aug 3, 2010 at 7:28 PM, Eric Wolf wrote: > I assume there is something in the API that says "negative IDs == BAD". I've > been trying to test that theory but keep hitting stumbling blocks. Postgres > doesn't seem to want to let me defer integrity constraints, so my efforts to > change a few IDs to positive values keeps failing. Maybe I've lost my SQL > chops (or maybe I just can't do that as the "openstreetmap" database user). > Am I barking up the right tree? Should I just go ahead and destroy the > database and repopulate it using bulk_upload.py instead of osmosis? all IDs in the database should be positive - negative IDs are only used on the client side to indicate elements which need to be created by the server (so that they can be referenced by other elements in the file). if you have negative IDs then you should use an uploader against an API, rather than directly inserting the data into the database with osmosis. a further complication is that postgres allocates new IDs from a "sequence", which is separate from the table and needs to be kept up-to-date after any direct-to-database import (not sure if osmosis does this?), otherwise postgres may attempt to allocate duplicate primary key IDs... uploading using bulk_upload.py is probably safer, but much slower, unfortunately... cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] relations refering to deleted objects
On Sun, May 23, 2010 at 6:28 PM, Frederik Ramm wrote: > Mitja Kleider wrote: >> Shouldn't the API reject a changeset creating such inconsistencies? > > It should. I haven't looked at your list but be aware that (contrary to > what some people think) the planet dump does NOT possess referential > integrity. this should no longer be the case. recent planets (i can't remember exactly when it changed) should be dumped with full integrity, so any examples of inconsistencies that you find in the planet should also be in the database. if they're not that's a bug in the dump process i'd like to know about, please. :-) as for the inconsistencies in the database, some pre-date API 0.6 and, although efforts were made to clean it up when we migrated we obviously missed some. there were also some created since then, presumably due to bugs in the API. i'd be very interested if anyone can find a reproducible way of creating these errors (on api06.dev, please). my hunch is that it's something to do with concurrent editing and insufficient locking, but i've never been able to pin it down. the ultimate solution is to have the database enforce this referential integrity, but that's a big enough change it'll have to wait until 0.7. in any case, these problems are (should be?) much, much rarer than they were under 0.5, and if you encounter one please fix it by removing deleted nodes from the ways that reference them or deleted members from relations that reference them. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Planet OSM 'bad result during COPY'
On Thu, Apr 15, 2010 at 8:01 PM, Seth Voltz wrote: > My installation is CentOS 5.4 with the latest SVN (20912) osm2pgsql. > PostgreSQL 5.4.3 and postgis 1.3.6-1 were installed via Yum from the PGDG84 > repository. PostgreSQL 5.4.3? that seems awfully old. are you sure? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Hourly/Daily Updates
On Sat, Mar 27, 2010 at 9:31 PM, Simone Cortesi wrote: > 2010/3/27 Matt Amos : >> i've cleaned up the wiki page describing the use of the diffs >> http://wiki.openstreetmap.org/wiki/Planet.osm/diffs and please let me >> know if you have any more questions so it can be further improved. > > What would be the best option for people wishing to keep minutely > updated mapnik installation on their own server? definitely the minute replicate diffs using the instructions on http://wiki.openstreetmap.org/wiki/Minutely_Mapnik modified to use the rrii and rri tasks rather than rcii and rci as described in Ldp's post at the bottom of that page. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Talk-GB] OSM shortlinks problem
On Mon, Mar 29, 2010 at 11:34 PM, Gregory wrote: > On 29 March 2010 15:15, Matt Amos wrote: >> >> http://trac.openstreetmap.org/changeset/16271 >> >> yeah, it was originally =, but changed to - to work with twitter. >> given that the characters A-Z, a-z, 0-9, _, @, - and = have already >> been used, what's the next best character? ~? +? > > Well I suggested that you remove problematic symbols, and just use A-Z, a-z, > 0-9 if need be. we're already using those symbols in a way that makes it difficult to reuse them for trailing characters without breaking existing links. if anyone can find a character which isn't in the list above, and doesn't break at least one URL-detector out there, then that's the best way to fix this. the alternative is to use some other scheme, for example appending .1/.2 instead of -/-- which makes some zoom levels a char longer. or possibly pre-pending the dashes, since they're usually picked up in the middle of an URL - but this makes char deletion from the end work slightly differently. it's also worth noting that, in the current scheme, shortlinks can end in @ or _ - do these also break the URL detectors? > Hmm, sorry Matt I see you only sent to Dev so might have missed the > discussion on Talk-GB. gmail seems to be clever enough to bring it into the thread. but i figured this was a development discussion, so better to keep it here. > It was said (if we have powers over Google, Microsoft, and other big e-mail > applications) we tell them to change... > It's also allowed behaviour. '-' is a valid URI character, so any user > agent that chops them off the end of a URI is broken. I realise that > doesn't help the people using them, but it would be better to fix the > problem in the correct place. sure, and if someone can find a char which works in more clients than -, it's an easy fix. > Robert Scott referenced RFC4648 base32 and said we could change it to = > perhaps (changing it back and breaking twitter again!). using = still works, so if that's working better then i guess a quick fix is to edit it manually... > Steve Doerr wondered if replacing it with %2D would solve it. I'm not sure > at what point it would be best to do this (in the link or manually). it might do, but the idea of shortlinks is that they're short and, if possible, easy to type on an ascii keyboard. but it seems like a difficult brief to fill. ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Talk-GB] OSM shortlinks problem
http://trac.openstreetmap.org/changeset/16271 yeah, it was originally =, but changed to - to work with twitter. given that the characters A-Z, a-z, 0-9, _, @, - and = have already been used, what's the next best character? ~? +? cheers, matt On Mon, Mar 29, 2010 at 10:52 AM, Gregory wrote: > Forwarding to dev as requested. > I think it has been noticed already, it can also break in applications such > as twitter. But is anything going to be changed? > I understand avoiding the ending character from being non-alphanumeric(not > letters or numbers) would mean a serious rewrite of how the shortlinks are > made/decoded. How much longer would short links be if all digits were > alphanumeric only? > Also the code is public so people can create these themselves or decode > them, has anyone actually done that? > On 29 March 2010 02:34, Tom Chance wrote: >> >> One for devs really but I'm not on the list. Perhaps someone could pass >> this on? >> >> I've noticed that the osm.org shortlinks ending with a dash "-" often >> don't work in emails. The dash gets missed off from the link. In my area, at >> least, this seems to move the centre slightly and switch from zoom level 17 >> to 18. >> >> For example this: >> http://osm.org/go/euut_VcMR- >> >> Gets interpreted as this: >> http://osm.org/go/euut_VcMR >> >> Is there any chance the shortlinks could avoid ending in a punctuation >> mark? >> >> Regards, >> Tom >> >> -- >> http://tom.acrewoods.net http://twitter.com/tom_chance >> >> ___ >> Talk-GB mailing list >> talk...@openstreetmap.org >> http://lists.openstreetmap.org/listinfo/talk-gb >> > > > > -- > Gregory > o...@livingwithdragons.com > http://www.livingwithdragons.com > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Hourly/Daily Updates
2010/3/26 Andreas Höschler : > Hi all, > >>> I'll leave the daily changesets running for now because they are >>> running with a much longer delay and shouldn't miss data. > > I need to incorporate changes into our map server and am currentlöy > building on > > http://planet.openstreetmap.org/daily > > and plan down make a wget on > > wget http://planet.openstreetmap.org/daily/20100325-20100326.osc.gz > > once a day and then import that changes into our database. Is this > save? yes. obviously assuming you change the date each day ;-) > Will http://planet.openstreetmap.org/daily be maintained in the > future. i don't think there are any plans to disable it, but keep an eye on the lists - it'll be announced here if the plans change. > Hourly files would be even better for us. Is this maintained, > if so where. It seems to be disabled right now!? there are hourly replication diffs at http://planet.openstreetmap.org/hour-replicate which are recommended for hourly use. i've cleaned up the wiki page describing the use of the diffs http://wiki.openstreetmap.org/wiki/Planet.osm/diffs and please let me know if you have any more questions so it can be further improved. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inconsistency in database?
Has anyone found any more of these? I had a look through the changesets, but the only commonality I could find is they're all tagged as PL in live mode. However 3 is a small sample size - it would be better to have more examples to track down the bug. Or, even better, if anyone can find a repeatable method for reproducing it. Cheers, Matt On Mar 1, 2010 8:49 PM, "Frank Bielig" wrote: Hallo, I found two more inconsistent ways: 45298547: http://api.openstreetmap.org/api/0.6/way/45298547 http://api.openstreetmap.org/api/0.6/way/45298547/1 45770380: http://api.openstreetmap.org/api/0.6/way/45770380 http://api.openstreetmap.org/api/0.6/way/45770380/4 44141527: (from previous mail) http://api.openstreetmap.org/api/0.6/way/44141527 http://api.openstreetmap.org/api/0.6/way/44141527/2 The problem of inconsistency seems to affect more than one way. It might be good idea to fix these special ways in database directly. BR, Frank > Hallo, > > I just checked the way 44141527. Calling > http://api.openstreetmap.org/api/0.6/way/4... -- Frank Bielig Tel: +49 33398 687848 OneStepAhead AGFax: +49 33398 687849 Research & Development Mobil: +49 177 3339868 Leuschnerstr. 45 eMail: frank.bie...@onestepahead.de D-70176 Stuttgart Web: http://www.OneStepAhead.de Firma: OneStepAhead AG Firmensitz/Amtsgericht:Stuttgart, HRB 22686 Vorstand/Vorsitzender: Nihat Kücük Aufsichtsratvorsitzender: Prof. Dr. Reinhold von Schwerin ___ dev mailing list dev@openstreetmap.org http://list... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Mon, Jan 18, 2010 at 12:05 PM, Steve Hill wrote: > On Mon, 18 Jan 2010, John Smith wrote: > >> You can stuff the new diff files straight into osm2pgsql, it's all I'm >> doing, I coded my own shell script to keep track of where my system is >> up to and to pull new ones when they are available. > > I think thats what I'm going to have to do. I can't figure out how to get > Osmosis to work - it just sits there doing nothing when I fire it up with > "--rri --simc --wxc blah.osc". Even with "-v 9" on the commandline all I > get is it sitting there saying "FINE: Reading current server state." and > doing nothing more. :( this is very odd... for me, it completes in about 6s after downloading one hour's worth of changes and merging them. (i didn't change maxInterval in configuration.txt, and that's the default). if you're running under linux it would be very helpful if could you run osmosis via strace and let us know what syscall it's in when it's hanging. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql deltas
On Sat, Jan 16, 2010 at 11:27 PM, Steve Hill wrote: > I've just realised that the hourly and minutely planet deltas have been > replaced by the new transaction based formats, but I can't seem to find > any documentation regarding their use with osm2pgsql. > > Can anyone provide any info on this? for the things i'm using them for they seem to work fine with osm2pgsql without any changes. if you're using osmosis to grab the changes then you'll need to change the rcii/rci tasks to rrii/rri tasks [1], though. cheers, matt [1] http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#Replication_Tasks ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
more here http://wiki.openstreetmap.org/wiki/Editor_usage_stats cheers, matt On Sun, Jan 10, 2010 at 12:05 PM, Matt Amos wrote: > On Wed, Dec 16, 2009 at 4:20 PM, Matt Amos wrote: >> On Wed, Dec 16, 2009 at 11:25 AM, Nick Black wrote: >>> Wondered if there had been any update on these numbers from the latest >>> planet? > > after a christmas hiatus, here's the numbers from the 2010/01/06 planet: > > editor | num | num_data | num_users > -+-+--+--- > Potlatch | 2123705 | 1607511 | 60405 > JOSM | 2369729 | 2268500 | 13978 > Merkaartor | 269419 | 254868 | 2305 > Mapzen POI Collector | 4975 | 4901 | 505 > Mapzen Beta | 4537 | 3092 | 477 > BigTinCan Upload Script | 642 | 537 | 181 > iLOE | 2551 | 2355 | 134 > osm2go | 1806 | 1740 | 99 > Osmose Raw Editor | 1879 | 1013 | 96 > bulk_upload | 124159 | 114749 | 66 > osmtools | 20108 | 18405 | 64 > Vespucci | 1273 | 892 | 62 > Mapzen Alpha | 666 | 370 | 37 > andnav | 432 | 394 | 35 > QGIS OSM v | 270 | 234 | 32 > OpenSeaMap-Editor- | 168 | 155 | 22 > PythonOsmApi | 1892 | 1522 | 18 > upload | 71282 | 68044 | 14 > KMLManager | 34886 | 34626 | 9 > GpsMid_ | 246 | 200 | 7 > > cheers, > > matt > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
On Wed, Dec 16, 2009 at 4:20 PM, Matt Amos wrote: > On Wed, Dec 16, 2009 at 11:25 AM, Nick Black wrote: >> Wondered if there had been any update on these numbers from the latest >> planet? after a christmas hiatus, here's the numbers from the 2010/01/06 planet: editor | num | num_data | num_users -+-+--+--- Potlatch| 2123705 | 1607511 | 60405 JOSM| 2369729 | 2268500 | 13978 Merkaartor | 269419 | 254868 | 2305 Mapzen POI Collector|4975 | 4901 | 505 Mapzen Beta |4537 | 3092 | 477 BigTinCan Upload Script | 642 | 537 | 181 iLOE|2551 | 2355 | 134 osm2go |1806 | 1740 |99 Osmose Raw Editor |1879 | 1013 |96 bulk_upload | 124159 | 114749 |66 osmtools| 20108 |18405 |64 Vespucci|1273 | 892 |62 Mapzen Alpha| 666 | 370 |37 andnav | 432 | 394 |35 QGIS OSM v | 270 | 234 |32 OpenSeaMap-Editor- | 168 | 155 |22 PythonOsmApi|1892 | 1522 |18 upload | 71282 |68044 |14 KMLManager | 34886 |34626 | 9 GpsMid_ | 246 | 200 | 7 cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Querying changesets
On Sat, Dec 26, 2009 at 8:48 PM, Karl Guggisberg wrote: > Hmm, strange, now > http://api.openstreetmap.org/api/0.6/changesets?closed=true > seems to hang (no response after 60s yet). > http://api.openstreetmap.org/api/0.6/changesets?open=true > still works fine, though. > > Any hint what is going on? Is there maintenance work underway? probably not, but there was a bug in the query for closed changesets (which i've now fixed) which was causing postgres to scan the whole changesets table, which would take a while. should be fixed after r19215 is deployed, though. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] X_HTTP_METHOD_OVERRIDE
On Fri, Dec 18, 2009 at 9:16 PM, hy-soft wrote: > Matt Amos wrote: >> On Fri, Dec 18, 2009 at 2:27 PM, hy-soft wrote: > >>> I can not get it to work, neither with curl >>> which hasn't implemented a delete Method, so I tried >>> X_HTTP_METHOD_OVERRIDE: DELETE >>> >>> curl -v -G -H "X_HTTP_METHOD_OVERRIDE: DELETE" -T DOSM0134.XML >>> "http://api.openstreetmap.org/api/0.6/node/595647268"; >> >> that's not the command listed on the wiki page. the command listed >> there (updated for delete) is: >> curl -v -v -u username:password -d @node.osm -H >> "X_HTTP_METHOD_OVERRIDE: DELETE" >> "http://api06.dev.openstreetmap.org/api/0.6/node/1168959"; > > Well I omitted user&password in the example above > The command I sent did actually trigger an update of the node. > > Now when I use a post [option -d] as given in your example it works - > with curl. > > But using POST with indy-sockets returns an Error 405: method not allowed. > > So thanks for clearing up the matter regarding curl, but actually I > still can not use the api via indy-sockets. > > The header is being sent - but obviously not recognized then. if the header is being sent then i don't see why that would be any different from what curl is doing. do you have a trace of the conversation between your app and the server to help debug what's going wrong? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] X_HTTP_METHOD_OVERRIDE
On Fri, Dec 18, 2009 at 2:27 PM, hy-soft wrote: > Has anybody tried this workaround as descibed in the api page of the > wiki: http://wiki.openstreetmap.org/index.php/OSM_Protocol_Version_0.6 > ? it works fine for me. see http://api06.dev.openstreetmap.org/browse/node/1168959/history which is a node i created with JOSM and deleted using curl. :-) > I can not get it to work, neither with curl > which hasn't implemented a delete Method, so I tried > X_HTTP_METHOD_OVERRIDE: DELETE > > curl -v -G -H "X_HTTP_METHOD_OVERRIDE: DELETE" -T DOSM0134.XML > "http://api.openstreetmap.org/api/0.6/node/595647268"; that's not the command listed on the wiki page. the command listed there (updated for delete) is: curl -v -v -u username:password -d @node.osm -H "X_HTTP_METHOD_OVERRIDE: DELETE" "http://api06.dev.openstreetmap.org/api/0.6/node/1168959"; where node.osm contains the node xml downloaded from http://api06.dev.openstreetmap.org/api/0.6/node/1168959 modified to reference the correct changeset, which i also opened with curl. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
for JOSM: lang | num | num_users ---++--- de| 491078 | 6457 en| 240905 | 3086 | 76081 | 2704 fr| 91720 | 919 en_GB | 55408 | 852 ru| 45467 | 527 it| 34936 | 357 es| 21888 | 248 nl| 10850 | 177 fi| 18956 | 159 sv| 8482 | 149 cs| 12716 | 144 pl| 7006 | 126 ja| 9313 |66 da| 3892 |63 sk| 3120 |42 nb| 1047 |35 pt|583 |23 ro|994 |23 bg| 1762 |13 for potlatch: lang | num | num_users ---++--- | 800309 | 46679 en| 109151 | 8023 de| 63330 | 4901 fr| 11479 | 981 ru| 10075 | 732 es| 5274 | 549 it| 7374 | 523 nl| 6520 | 347 pl| 1733 | 226 sv| 3447 | 155 pt-BR |989 | 152 fi| 1410 | 105 no| 1763 |84 cs|505 |82 ja| 1242 |80 da| 2261 |77 hu|382 |67 ro| 1041 |59 pt|163 |29 tr| 88 |26 not sure what the null language results are - presumably at some point the editors weren't putting a language in their changeset comments or something? cheers, matt On Fri, Dec 4, 2009 at 11:45 PM, Ævar Arnfjörð Bjarmason wrote: > On Fri, Dec 4, 2009 at 20:09, Matt Amos wrote: >> as a massive self-plug, here's a more in-depth look at some of the >> editor data. unfortunately, there isn't enough data to do this for any >> editor other than the "big three", but hopefully in six months time... > > This looks nice. It's certainly a lot better than my ad-hoc statistics > I posted in september: > > http://lists.openstreetmap.org/pipermail/talk/2009-September/042902.html > > One thing that I missed about your stats though is language statistics > like the ones I posted (but done better with fancy graphs, obviousl > :). > > Since I posted my stats I've added information about the user language > to Potlatch (previously only JOSM had them), but unfortunately it > looks like the Merkaartor people have ignored my request of adding it > to their created_by string. > > It would be really cool to see statistics per-editor and per-language > presented in such a way that one could gauge whether an editor being > localized had an effect on its update. > > I recently found out for example that the Potlatch translation into > Italian was really incomplete while JOSM was almost 100% translated > into Italian (and JOSM has like 4000 strings while Potlatch has around > 300). I contacted some Italian translators about this and Potlatch now > has a much better Italian translation. > > Perhaps some Italians where shunning Potlatch because of this, it > would be interesting to see stats to confirm or disprove that. > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
On Wed, Dec 16, 2009 at 11:25 AM, Nick Black wrote: > Wondered if there had been any update on these numbers from the latest planet? these are the numbers from today's planet: editor | num | num_data | num_users -+-+--+--- Potlatch| 1029797 | 775495 | 57637 JOSM| 1139962 | 1090173 | 13282 Merkaartor | 129510 | 122438 | 2160 Mapzen POI Collector|2124 | 2088 | 389 Mapzen Beta |1261 | 821 | 271 BigTinCan Upload Script | 293 | 242 | 143 iLOE|1202 | 1106 | 117 osm2go | 903 | 870 |99 Osmose Raw Editor | 813 | 399 |85 bulk_upload | 61908 |57209 |64 osmtools|8951 | 8614 |59 Vespucci| 614 | 426 |52 Mapzen Alpha| 333 | 185 |37 andnav | 214 | 195 |33 QGIS OSM v | 128 | 110 |28 PythonOsmApi| 946 | 761 |18 upload | 35585 |33968 |14 OpenSeaMap-Editor- | 66 | 60 |11 KMLManager | 17443 |17313 | 9 GpsMid_ | 123 | 100 | 7 impressive gains for mapzen POI collector, up 150 users over two weeks. potlatch is up 1,696, but that's only +2.9% whereas mapzen's gain is +38.6%. the most impressive fractional gain is mapzen beta, by +76%. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 6:24 PM, Ian Dees wrote: > Actually, I was just now creating a stub page for API 0.7 brainstorming: > > http://wiki.openstreetmap.org/wiki/API_v0.7 > > Remember, it's a brainstorm: all ideas are good ideas at this point... ;-). cool! although i'd consider "verified users / locked tags" to be an anti-feature ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Structured error messages from API
On Fri, Dec 11, 2009 at 5:05 PM, Matthias Julius wrote: > Richard Fairhurst writes: > >> Tom Hughes wrote: >>> Basically, even if sending a streaming response from rails was >>> possible (which I'm not sure it is - it's certainly hard) >> >> You can do it using render :text=>proc, as amf_controller does. But I >> suspect this would be non-trivial to work into the existing XML API. > > The API could also return an URL which the client can poll to get the > status for an upload. > > This would also avoid issues with client timeouts. > > Currently, when the client hits a timeout and aborts the connection it > has no way of knowing whether the upload succeeded or not. For new > data this creates duplicates when the user tries to upload again. i have been thinking about something similar, which would work along the following lines. the changeset is a container and POSTing a diff to it returns a 202 Accepted result if diff processing takes more than a few seconds. the result is a Location redirect to the individual resource for the diff upload, which could be polled for status. unfortunately, that's where it gets complicated. because the diff upload occurs in a transaction, none of its outputs are visible until it commits. therefore any status would need to be posted on a different connection, in a different transaction. this makes things annoying a messy. if anyone's got any ideas then i'd be very interested to talk. this is something that we can put into API 0.7 - which it might be a good idea to start thinking about now, since the last API took about a year to develop ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] full history planet (experimental)
hi devs, i've just finished running Lars' full history planet dumper and the results are here: http://planet.openstreetmap.org/full-experimental/full-planet-091207.osm.bz2 (13Gb) although a skim through the file shows nothing obviously wrong, this should be considered experimental. please report any problems you find to us and we'll try and figure it out. also, the dump should have referential integrity (a way might reference a deleted node, but shouldn't reference a non-existent node) so please report if that isn't the case. it seems to take about 2-3 days to dump, although much of that is dependent on pbzip2 performance, and causes enough of a load on the DB to delay the daily dumps by an hour or two. so i don't think this is something that's worth running every week, but maybe every month or two. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
On Fri, Dec 4, 2009 at 6:49 PM, Shaun McDonald wrote: > > On 4 Dec 2009, at 17:50, John Smith wrote: > >> 2009/12/5 Nick Black : >>> Thanks for the update Matt. As this is changeset based, this means >>> that Mapzen POI Collector is the fourth most used OSM editor ever, >>> since changesets were introduced? >> >> The order is by numbers of users, not number of uses. > > +1 yeah, i think it's fair to say that Mapzen POI collector is the fourth most popular OSM editor over the last 6 months. as a massive self-plug, here's a more in-depth look at some of the editor data. unfortunately, there isn't enough data to do this for any editor other than the "big three", but hopefully in six months time... http://www.asklater.com/matt/wordpress/2009/12/editor-retention/ cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
here's the results for wednesday's (2nd) planet dump, again over the history since each editor started adding changeset tags. Mapzen POI collector has gained over 200 users in the past week, adding 636 changesets. well done to the Mapzen team! editor | num | num_data | num_users -+-+--+--- Potlatch| 988916 | 739623 | 55941 JOSM| 1092702 | 1043776 | 12945 Merkaartor | 124756 | 117812 | 2100 Mapzen POI Collector|1374 | 1341 | 239 BigTinCan Upload Script | 266 | 219 | 124 iLOE|1113 | 1019 | 105 osm2go | 901 | 868 |99 Osmose Raw Editor | 756 | 358 |82 Mapzen Beta | 218 | 155 |65 bulk_upload | 60110 |55417 |61 osmtools|8326 | 8021 |57 Vespucci| 558 | 374 |49 Mapzen Alpha| 333 | 185 |37 andnav | 207 | 191 |31 QGIS OSM v | 117 | 100 |24 PythonOsmApi| 937 | 753 |18 upload | 35560 |33959 |14 KMLManager | 17443 |17313 | 9 GpsMid_ | 121 | 98 | 7 OpenSeaMap-Editor- | 41 | 35 | 6 cheers, matt On Sun, Nov 29, 2009 at 4:21 PM, Matt Amos wrote: > On Sunday, November 29, 2009, Simone Cortesi wrote: >> On Sun, Nov 29, 2009 at 15:46, Nick Black wrote: >>> Interesting data. If there are 190,000 OSM contributors and around >>> 10% of them are active, how come there are 50,000 users using Potlatch >>> on one day? >> >> probably that is the total changesets "produced" with the given >> editor. not on a single day. 25th is the date of the snapshot. > > That's right, the numbers are from last week's changeset dump. So it > only counts changesets created between then and whenever the editor > started writing created_by tags on changesets. About six months or so. > > Since mapzen wasn't out on the 25th those numbers are low, but I > expect they'll pick up on the 2nd. > > Cheers, > > Matt > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Distance Grid
On Tue, Dec 1, 2009 at 7:03 PM, Ævar Arnfjörð Bjarmason wrote: > On Tue, Dec 1, 2009 at 17:06, Matt Amos wrote: >> On Tue, Dec 1, 2009 at 4:04 PM, Vitor George wrote: >>> I want to start a project in Brazil that is similar to the Tiger Fixup 250 >>> Cities [1], and I looking forward to build a script that generates a >>> distance grid, like this: >>> >>> http://matt.sandbox.cloudmade.com/usa-routes.html >>> >>> I've tried to contact the authors via wiki, but I was unsuccessful, so I >>> want to develop it by myself. I'm not a professional programmer, but I have >>> some experience in java. Is the LibOSM the easiest way tool to implement it? >> >> sorry you missed us on the wiki. i have to say, i don't really check >> the discussion pages like a good wikian should :-( >> >> the script is very simple, and uses cloudmade's ruby API [1] to get >> the routes. you'll need to sign up for a cloudmade API key, if you >> haven't already. the script i've attached will output CSV on stdout, >> but i usually put them in a database for easy access. >> >> run it like "ruby routes.rb ". i've attached an example >> configuration file, which is just a yaml map of string to lat/lon >> array. >> >> i look forward to seeing brazil's 250 cities :-) > > I made a hacky script to generate .yml from XAPI output (attached): > > wget > 'http://www.informationfreeway.org/api/0.6/node[place=city|town|village|hamlet][bbox=-25.74085,62.84553,-12.41708,67.50085]' > -O iceland-places.osm > perl -CI place2yaml.pl < place.osm > place.yml > > But I just get internal server errors from the CM API: > > $ ruby routes.rb place.yml > [Tue Dec 01 18:59:57 + 2009] HTTP error: Couldn't read data. HTTP > status: #, retrying... interesting. if you could send me a trace of what's going on that would be very helpful. > Anyway do you have a script to generate that cute HTML matrix? And how > can I link to a route on CM's website. I have to supply lat/lon/zoom > it seems and not just starting/ending lat/lon (at least the URLs I've > seen are all like that). i've just thrown something together here - might be buggy, etc... you'll need to play with the factor variable to see what looks right for your area. cheers, matt to_html.rb Description: application/ruby ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Distance Grid
On Tue, Dec 1, 2009 at 4:04 PM, Vitor George wrote: > I want to start a project in Brazil that is similar to the Tiger Fixup 250 > Cities [1], and I looking forward to build a script that generates a > distance grid, like this: > > http://matt.sandbox.cloudmade.com/usa-routes.html > > I've tried to contact the authors via wiki, but I was unsuccessful, so I > want to develop it by myself. I'm not a professional programmer, but I have > some experience in java. Is the LibOSM the easiest way tool to implement it? sorry you missed us on the wiki. i have to say, i don't really check the discussion pages like a good wikian should :-( the script is very simple, and uses cloudmade's ruby API [1] to get the routes. you'll need to sign up for a cloudmade API key, if you haven't already. the script i've attached will output CSV on stdout, but i usually put them in a database for easy access. run it like "ruby routes.rb ". i've attached an example configuration file, which is just a yaml map of string to lat/lon array. i look forward to seeing brazil's 250 cities :-) cheers, matt [1] http://developers.cloudmade.com/projects/show/ruby-lib routes.rb Description: application/ruby usa_extract.yml Description: Binary data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
On Sunday, November 29, 2009, Simone Cortesi wrote: > On Sun, Nov 29, 2009 at 15:46, Nick Black wrote: >> Interesting data. If there are 190,000 OSM contributors and around >> 10% of them are active, how come there are 50,000 users using Potlatch >> on one day? > > probably that is the total changesets "produced" with the given > editor. not on a single day. 25th is the date of the snapshot. That's right, the numbers are from last week's changeset dump. So it only counts changesets created between then and whenever the editor started writing created_by tags on changesets. About six months or so. Since mapzen wasn't out on the 25th those numbers are low, but I expect they'll pick up on the 2nd. Cheers, Matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Tagwatch Editor Counts
in case anyone is interested, here's the data from the 25th. num is the number of changesets, num_data is those with bboxes, num_users is the number of distinct (public) user ids. of course, this was before mapzen poi collector was released in the app store, so it's not very helpful for that ;-) creator | num | num_data | num_users --+-+--+--- Potlatch | 964387 | 718768 | 54823 JOSM | 1064183 | 1015659 | 12741 Merkaartor | 121672 | 114819 | 2057 BigTinCan Upload Script | 245 | 201 | 109 osm2go | 898 | 865 | 98 iLOE | 940 | 850 | 84 Osmose Raw Editor | 714 | 330 | 78 bulk_upload | 59740 | 55051 | 58 osmtools | 7914 | 7622 | 56 Vespucci | 542 | 359 | 48 Mapzen Alpha | 310 | 164 | 37 andnav | 207 | 191 | 31 Mapzen POI Collector | 738 | 709 | 27 QGIS OSM v | 114 | 98 | 23 PythonOsmApi | 935 | 753 | 18 upload | 35026 | 33431 | 14 KMLManager | 17443 | 17313 | 9 cheers, matt On Sat, Nov 28, 2009 at 6:37 PM, Shaun McDonald wrote: > > On 28 Nov 2009, at 13:00, Nick Black wrote: > >> Hi Guys, >> >> A few people have been asked if I have any stats on Mapzen POI >> Collector usage. I took a look at the tagwatch editor page [1] but >> neither Mapzen Alpha or Mapzen POI Collector in any of its versions >> appears in the list. Both apps tag the change set [2] rather than the >> node or way that is created. I'm wondering if this is why there are >> no stats on the editor's usage in Tagwatch? If this is the case, how >> can I help update Tagwatch to look at change sets? >> >> [1] http://tagwatch.stoecker.eu/Europe/En/top_used_editors.html >> [2] http://www.openstreetmap.org/browse/changeset/2996362 >> >> > > It looks like that up until the 26 November there have been less than 783 > changesets with the exact editor name that mapzen is using. > > You can use the changeset info dump to do this analysis. It is available from > http://planet.openstreetmap.org/ on a weekly basis. > > Shaun > > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complete history of OSM data - questions and discussion
On Thu, Nov 12, 2009 at 4:28 PM, Lars Francke wrote: > - As of now the XML is not indented. I use Woodstox[1] for XML output > and that doesn't have an option to "pretty print" the output. It is > not a problem for me but if it is requested I can use StaxMate or > something else to properly indent the XML i'm pretty sure no-one minds. as you say, anyone who wants it indented can easily do it with xmllint and friends. > - Changesets: num_changes from the database isn't dumped in planet.c. > It is queried from the database but not used anywhere. The data _can_ > be calculated but it isn't that easy if not using the standard db > schema and not easily done by reading the XML stream. I could just > dump it too. I haven't had a look at the API if this field is set > correctly at all?! it should be set correctly. you're welcome to dump it out on the changesets if you think it's useful. > - I'm using the same technique as planet.c in regards to the output of > the data (just streaming it to standard output), I just assume that > this is okay? Are there any other things I'll have to change in > comparison to the way planet.c works? yeah. the output will be piped directly into pbzip2, most probably. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complete history of OSM data - questions and discussion
On Wed, Nov 11, 2009 at 1:29 PM, Lars Francke wrote: > I had > brief discussions with Brett about Osmosis and incorporating certain > changes into it so I've spent quite some time in its source code. > Having said that: I probably won't program this as a new task for > Osmosis but as a standalone program as this probably won't be used > widely and doesn't justify the extra work required to incorporate this > into Osmosis. just remember that new code = new bugs ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Complete history of OSM data - questions and discussion
On Wed, Nov 11, 2009 at 6:41 AM, Lars Francke wrote: > There are a few questions that probably need answering first and I > hope we can start a discussion about this: > - Am I correct in assuming that there are no general objections from > the OSM server folks against such a dump? (Which would render the rest > of this E-Mail useless ;-) the response has always been "if someone writes it, and it's good, we'll run it" :-) > - Is anyone else currently working on this? for some values of "working", yes. it's on my list of things to do for the license change plan - clearly we'll need a full data dump before we can re-license. > - Which format should the data be dumped in (3) is the easiest to get done and most easily supported, in my opinion. > - Distribution of the data and storage space requirements i have a feeling that the data, while big, won't be so big that the usual method of planet.osm.org + heanet mirror won't work. > - Interval of dumps based on back-of-the envelope calculations, a full dump in planet format would take something like 7-10 days to do in parallel with normal server activity. so it couldn't be run every week and would probably be cumbersome to do every month. in my opinion, we should be looking at every 3-6 months. > 3) A dump of all OSM elements in OSM format > (http://www.openstreetmap.org/api/0.6/node/60078445/history) this is my favourite method as well. the easiest approach would be to modify planet.c to dump the full history, instead of just the current_* tables. note that brett has been working on option (2) by using osmosis to dump very historical diffs going back to the inception of the database. you can see the experimental results in http://planet.openstreetmap.org/history/ for my money, if we do both (2) and (3), then we cater for all consumers, and in a standard format. the output of the COPY command, while good for backups, isn't really suited to dumping the information that we have in the planet (given there will be edits by users who are still not public, etc...) if you want to get started hacking on planet.c then i'm happy to help. otherwise i'm hoping to get around to it by the end of the month, but there are never any guarantees ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Missing changes in minute-replicate
On 11/9/09, Jon Burgess wrote: > On Mon, 2009-11-09 at 21:17 +0100, Peter Körner wrote: >> > >> > The changes in changeset 3073502 are completely missing from the >> > minute-replicate diffs. They appear in both the minute and hourly >> diffs. >> > >> >> I checked node 559977476 [1] that has been created in changeset >> 3073502 >> [2]. I can't find them in the minute-replicate diffs. They should be >> in >> the index between 000/084/536 - 000/084/537 but grepping for this id >> returns nothing. > > I looked on the mapnik database being fed by these diffs and it does not > show this node either. There is not a single reference to changeset > 3073502 in any of the 100 - 999 diffs in the 000/084 directory. > > References to the changesets on either side (3073501 & 3073503) appear > in the 539.osc.gz file. I have no idea why this one has gone missing. possibly it's a coincidence, but changeset 3073502 was committed in transaction 174978321, which is the txnMax of replication diff 84538. i don't see anything special about that number, though. is it possible that the connection pooling in the apache DBCP BasicDataSource means that the transaction state query is occurring in a connection which has txn 174978321 committed, but the replication query happens in one where it hasn't been committed yet? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] usernames, keys, and values
On Thu, Oct 29, 2009 at 6:16 PM, Anthony wrote: > On Thu, Oct 29, 2009 at 1:31 PM, Matt Amos wrote: >> On Wed, Oct 28, 2009 at 3:38 PM, Andy Allan wrote: >>> I would say that if the dump code and >>> http://www.w3.org/TR/REC-xml/#NT-Char >>> are in conflict, there's a bug in the dump code. But since I'm not >>> going to fix it, maybe I'll keep my opinions quiet :-) >>> >>> As for the rails code, there is (AFAIK) no explicit character >>> checking. The server implicitly relies on libxml to ensure the >>> characters in the XML requests and responses are only those allowed by >>> the XML spec above. >> >> there is explicit checking in the potlatch API, as that doesn't go >> through libxml: >> >> http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb#L909 > > There doesn't seem to be a spec, so everyone's just making it up as > they go along. sort of. the "spec" is just that the API talks XML, and we aim not to have any restrictions beyond that. so anything that's a valid XML character (http://www.w3.org/TR/REC-xml/#charsets) should be allowed in OSM. > But I'm going to attempt to clarify, with a quote from W3: "In > attribute values, the character information items TAB (#x9), newline > (#xA), and carriage-return (#xD) are represented by " ", " ", > and " " respectively." > (http://www.w3.org/TR/2000/WD-xml-c14n-2119.html) > > Including a tab, newline, or carriage return unescaped in an xml > attribute would clearly be incorrect. But as long as it's escaped, > it's valid xml. is valid xml. > It may or may not represent valid OSM data. This is why I'm saying my > question has nothing at all to do with XML. but there are other escaped values which are invalid XML (e.g: �). > Apparently under that potlatch code, tabs, carriage returns, and > newlines are not allowed in keys or values (I don't actually know > ruby/rails enough to say for sure, but that seems to be what Matt just > pointed out). the code, which is a bit opaque: delete "\000-\037", "^\011\012\015" says that it'll delete any char in the range 0-37 (octal), but not 11, 12 or 15. in ascii this corresponds to anything "less than" a space, but not tab, newline or carriage return. it should be (modulo bugs) the same as the XML valid character productions. > On the other hand, usernames apparently *can*, at this > point, contain these characters. Actually changing one's username to > include them would require using an input method other than the web > page, but I don't see any code to forbid this. yes, we should probably add a rails validation to stop people using those chars in their username (who needs a tab in their username anyway?) > On the other hand, the planet dump code is silently changing control > characters to "?". This could cause problems (for instance, two > usernames might wind up being silently changed to identical values), > though it would probably require a deliberate attack. the planet dump format includes the uids, so this case would be detectable. however, this behaviour in planet dump is a bug and i'll have a go at fixing it. > I wonder, what happens if someone enters tabs into keys or values > through the API (where there apparently are no checks for this), and > then someone tries to edit it in potlatch? Looks like a denial of > service attack to me. there don't need to be any checks - the xml is parsed using an xml parser, which would barf if those chars were in the API calls. potlatch also can't enter those chars via its API. i don't think there's any attacks possible here, but i'd like to know if there are! > It would be a good idea to release an official spec on exactly what > characters are allowed in keys, values, and usernames. Just > disallowing control characters (decimal value less than 32) altogether > would probably be the best. But if the decision is made to allow > them, fine, they need to be handled properly. that's a good idea. i'll stick something up on the wiki - for reference i think the current "spec" is the XML valid character productions, although i can't think of any particular reason to keep \t, \n or \r: Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x1-#x10] http://www.w3.org/TR/REC-xml/#NT-Char cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] usernames, keys, and values
On Wed, Oct 28, 2009 at 3:38 PM, Andy Allan wrote:
> On Wed, Oct 28, 2009 at 3:22 PM, Anthony wrote:
>> On Wed, Oct 28, 2009 at 10:21 AM, Frederik Ramm wrote:
>>> Hi,
>>>
>>> Anthony wrote:
Just took a look at the planet dump code at
http://svn.openstreetmap.org/applications/utils/planet.osm/C/output_osm.c
In xmlescape(): "} else if ((*in >= 0) && (*in < 32)) {
escape_tmp[len] = '?'; len++; } else {"
>>>
>>> The planet dump code decides what gets *out* (to the planet file), not what
>>> gets in (to the database) and neither what gets out over the API. If you
>>> want to look at code to answer your questions then you should look at the
>>> rails port (/sites/rails_port) which guards the API.
>>
>> My real question was "how can I parse the planet dump file". :)
>>
>> I briefly skimmed through the rails_port, but not knowing ruby/rails I
>> didn't get very far. I suppose if I care I'll make a few tests
>> through the dev server api. But for now I'm content with adding "if
>> (buf[ptr]<32) { fail(buf, "unexpected character data"); }" to my dump
>> parser.
>>
>> If someone decides that's a bug in the dump code, and not a feature,
>> please let us know here.
>
> I would say that if the dump code and
> http://www.w3.org/TR/REC-xml/#NT-Char
> are in conflict, there's a bug in the dump code. But since I'm not
> going to fix it, maybe I'll keep my opinions quiet :-)
>
> As for the rails code, there is (AFAIK) no explicit character
> checking. The server implicitly relies on libxml to ensure the
> characters in the XML requests and responses are only those allowed by
> the XML spec above.
there is explicit checking in the potlatch API, as that doesn't go
through libxml:
http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb#L909
cheers,
matt
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] usernames, keys, and values
On 10/28/09, Anthony wrote: > Regarding usernames, keys, and values in the OSM database, are there > are restrictions on what characters are allowed? Tabs, carriage > returns, line feeds, nulls? All allowed? i think the current set of allowed characters are those which are in the XML character range: http://www.w3.org/TR/REC-xml/#NT-Char this includes \t, \n and \r, but not nulls. to be honest, i can't see a good use for these characters, but someone out there is bound to be using them for something ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] get user id on api
2009/10/12 Frederik Ramm : > Hi, > >> Etienne Chové a écrit : >>> How to get user id when we have the username ? (GET changesets needs >>> userid and do not accept username). > > Then fix the API ;-) done. http://trac.openstreetmap.org/changeset/18103 http://wiki.openstreetmap.org/wiki/API_v0.6#Query:_GET_.2Fapi.2F0.6.2Fchangesets note that it might not be deployed yet by the time you read this. ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Hourly diffs are missing edits (too)
On Wed, Oct 7, 2009 at 11:22 AM, Frederik Ramm wrote: > Hi, > > Tom Hughes wrote: >>> Could Postgres be persuaded to abort any transaction that runs longer >>> than "n" minutes (e.g. 30), and the we run the hourlies at hh:31 or so? >>> That would probably be a slight inconvenience to those who happen to >>> start 35-minute transactions but they should just learn to do their bulk >>> imports properly ;-) >> >> Um... Are you being serious? > > Yes. In most cases, whatever HTTP client they were using to upload will > already have terminated the TCP connection and complained about a > timeout anyway, leading them to upload the same thing again... unless they're on an extremely slow connection and the diff is trickling into the server at a few bytes per second. >> Why? We've already solved the problem with the transactional diffs... > > For many uses, I view the hourly diffs as superior as they will contain > less "noise" (edits that cancel each other out will not be present in > the hourlies). So I would really like to have reliable hourly diffs - > even if they come with a considerable delay. it's possible to do this by aggregating the replication diffs and dropping intermediate versions. of course, this won't exactly match an hour boundary, but should usually be pretty close. > My reasoning is that I feel uneasy when these transactions are > completely unlimited. For all I know, there might be a freak case where > a transaction runs for 8 days and I suddenly have an object in this > week's planet which is 8 days old but wasn't in last week's planet (etc.). but since the transaction commits atomically, the only evidence of this is would be the timestamp on the element. the timestamp on the element may as well be replaced with the timestamp of the replication diff it came in. > So I would like to have *some* certainty about the run time of > transactions. Even if it is 24 hours or so - just some value where I > know that no transaction can possibly exceed this duration. And if we > should find that only one promille of all transactions takes longer than > thrity minutes then I'd be prepared to sacrifice that one transaction if > that buys me accurate hourly diffs at 31 minutes past the hour. aggregating replication diffs gives you accurate nearly-hourly diffs at 1 minute past the hour - wouldn't you prefer those? all it takes is a little mental re-adjustment away from a time-based stream towards a transaction-based stream. come on in, the mental water is fine ;-) cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
