Re: [OSM-dev] How reconstrucing a way from the history?
2009/1/13 Nevel Gandish : > 2009/1/13 Robert Vollmert : >> On Jan 12, 2009, at 13:45, Nevel Gandish wrote: >>> >>> And can I be sure that when moving a node caused a new history entry >>> for the node and the way that both have the exact timestamp or might >>> they differ by a few seconds? >> >> Just moving a node doesn't create a new history entry for the way, I think. > > Um, that seems true indeed. I just assumed it would. > > Nevel > However, I think potlatch does work in this way.. -- Regards, Thomas Wood (Edgemaster) ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Limit on the number of tags on a node
Hmm, trying my post again with a message created from scratch! I didn't realise I couldn't just reply all to another message, change the subject and delete the old text! Its a bad habit anyway so time I stopped it. I get a 400 error when I try and upload nodes with more than 50 tags and about 4,300 bytes. This is to a test database, not www.openstreetmap.org/api. Is anyone aware of any limitations? Regards Neil Penman Stay connected to the people that matter most with a smarter inbox. Take a look http://au.docs.yahoo.com/mail/smarterinbox___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Dropping Osmosis Named Pipes (was: Osmosis --ac changed?)
Frederik Ramm wrote: > Hi, > > Karl Newman wrote: >> Why drop them? > > Because it makes documenting what Osmosis does easier, and it makes > understanding what Osmosis does easier for those who start to work > with it now. Keeping everything that made sense at some time in the > past unnecessarily increases complexity. My first reaction was the same as Karl. The named pipe stuff has never been the source of bugs and does give a way for those struggling to get tasks connected properly to "force" them to connect the way they wish. However on further thought I've never used named pipes since the stack connection mechanism was introduced and they're not terribly easy to understand. Internally pipes don't exist anyway, they were always just a logical command line thing to help people visualise the pipeline. So far I've kept the command line very consistent between releases to try to avoid breaking scripts constantly but this might be a good case for simplifying. The command line parsing and pipeline building code has also been very reliable but the code isn't simple. Anyway, if nobody uses named pipes then I'm not opposed to dropping them. One problem is that I've never had a great feel for who uses osmosis so I don't know the best way of finding out usage patterns. Perhaps I need to introduce some evil phone home functionality providing user statistics ;-) If we do drop them we might have to improve the verbose logging that helps diagnose task connectivity problems. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How reconstrucing a way from the history?
2009/1/13 Robert Vollmert : > On Jan 12, 2009, at 13:45, Nevel Gandish wrote: >> >> And can I be sure that when moving a node caused a new history entry >> for the node and the way that both have the exact timestamp or might >> they differ by a few seconds? > > Just moving a node doesn't create a new history entry for the way, I think. Um, that seems true indeed. I just assumed it would. Nevel ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis --ac changed?
Hi, Karl Newman wrote: > Why drop them? Because it makes documenting what Osmosis does easier, and it makes understanding what Osmosis does easier for those who start to work with it now. Keeping everything that made sense at some time in the past unnecessarily increases complexity. Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis --ac changed?
On Tue, Jan 13, 2009 at 8:18 AM, Frederik Ramm wrote: > Hi, > > There was a reason behind it though. With the old queue based approach >> sometimes there was no way to get the tasks connected properly without using >> named pipes. With the stack based approach I think it's always possible to >> connect tasks without using named pipes. >> > > It seems you're right. With the stack approach, anything is possible ;-) > any plans to drop named pipes then? > > Bye > Frederik > > Why drop them? They're optional, they work, they make connecting some tasks unambiguous and some people undoubtedly use them in scripts. I don't think they're a maintenance headache (they're contained to a small portion of the code, and their presence is transparent to the code for individual pipeline tasks), so no reason not to keep them, right? Karl ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Martijn van Exel wrote: > Is there another source explaining the architecture of the Rails port? > Preferably with a > diagram or such. > Hardware wise... 1x Database Server (MySQL) 3x Dedicated Rails Application Servers (Ruby 1.8.6, Rails 2.0.x) 1x Frontend Web Server (lighttpd + Ruby 1.8.6, Rails 2.0.x) Dynamic requests are broken down into 4 types: Web: Website, Login, Signup, Messages etc (Handled by frontend web) TAH: All ti...@home requests BulkAPI: API map, trackpoint, amf/reads and swf/trackpoints API: New nodes, rest of API stuff See: http://svn.openstreetmap.org/sites/rails_port/config/lighttpd.conf There have been problems in the past with Rail's libxml binding leaking memory. Rails seams to leak memory on some of the BulkAPI and occasionally API requests. Particularly /api/0.5/map API code has a forced Rails daemons death after ~1 requests to free leaked memory. / Grant ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis --ac changed?
Hi, > There was a reason behind it though. With the old queue based approach > sometimes there was no way to get the tasks connected properly without > using named pipes. With the stack based approach I think it's always > possible to connect tasks without using named pipes. It seems you're right. With the stack approach, anything is possible ;-) any plans to drop named pipes then? Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Data Consistency Checks now available
Hi Alessandro! You're right, this shouldn't be an error. But in my copy of the database (2009-01-02) SS12 and Corso Verona are not yet connected. So the error is not related to roundabouts, Corso Verona just looked like an open end. Best regards, HArald > Hi Harald, > the one you pointed out seem pretty much to be errors. > > Please see this: > http://www.openstreetmap.org/?lat=45.872858&lon=11.031874&zoom=18&layers=B000FTF > > > http://keepright.ipax.at/report_map.php?ch30=30&ch40=40&ch50=50&ch60=60&ch70=70&ch80=80&ch90=90&ch100=100&ch110=110&ch120=120&ch130=130&ch150=150&ch160=160&ch170=170&ch180=180&ch190=190&ch200=200&lat=45.87352910639793&lon=11.032413884766799&zoom=17&requery=requery > > > > As on hi-zoom the error icons disappear for some reason, I confused the > error from the adiacent one-way road with the roundabout. > > So sorry for the wrong bug report. :) > > Alessandro > > Harald Kleiner ha scritto: >> Hi, Alessandro! >> [blablabla] ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
2009/1/13 "Marc Schütz" : >> Diff uploads will not help online editor such as Potlatch. The >> addition of transactions and exposure of the version numbers will help >> more. > > Why not? It may well utilize it for changes to multiple objects that should > be atomic, e.g. the example mentioned here. > Because potlatch uses a different API (found in amf_controller). However, in 0.6 the upload way call can be contained within a database transaction and so make the edit atomic. Dave ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Matt Amos wrote: > in the case of quadtile, i think the extra complexity was > worthwhile. in the case of SQL in the amf_controller, i'm not > so sure. this is just my opinion. Sure. We can even do more nuanced than that, if you like. :) In my opinion - and again, just that - the clarity of Rails is worthwhile and important in the write operations. putway is a lot more readable in Rails than it ever was in raw SQL, and it's more reassuring in terms of data integrity, too. Speed isn't really an issue here as these are called comparatively rarely. Where the raw SQL approach scores is in the read operations, which are incredibly simple, easy-to-understand queries. In Potlatch these are whichways and getway, in the XML API it'd be the /map call. The disadvantage of using Rails in this context is obviously that you have this huge overhead to create objects for every way or node, simply to serialise them and dispose of the objects immediately. Because they're reads, there are (generalising a bit) no integrity issues: and because they're very common calls, you get a significant speedup. cheers Richard -- View this message in context: http://www.nabble.com/Ruby-developers-in-Amsterdam-tp21433520p21437005.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 1:53 PM, Richard Fairhurst wrote: > Matt Amos wrote: >> in which case we're trading development effort for performance. >> i think we're operating in areas constrained by the lack of >> available developers, so making the barrier to entry higher is a >> tough decision. > > (Speaking seriously for once...) > > Kind of, but it's not a simple trade-off. Certainly as far as Potlatch is > concerned, server performance issues require development effort on the > client. The problem that's been brought up both here and on talk, where the > server dies halfway through a write operation and leaves the database in an > inconsistent state, is an example of this - there are others. these problems are not due to rails - most are due to the lack of transactions, which is an implementation problem, and the many others are version race conditions, which is an API design problem. lets not confuse a performance issue with a bug or correctness issue. > When TomH did the quadtile stuff, that added a degree of complexity for a > boost in performance. I doubt anyone now would say it was the wrong > decision. absolutely, but i'd say that was a small increase in complexity in a small corner of the code (which is well hidden) for a large boost in performance. i'm sorry if i gave the impression i was totally against any form of increased complexity, because i'm not. i just think these things need to be balanced on a case by case basis. in the case of quadtile, i think the extra complexity was worthwhile. in the case of SQL in the amf_controller, i'm not so sure. this is just my opinion. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Matt Amos wrote: > in which case we're trading development effort for performance. > i think we're operating in areas constrained by the lack of > available developers, so making the barrier to entry higher is a > tough decision. (Speaking seriously for once...) Kind of, but it's not a simple trade-off. Certainly as far as Potlatch is concerned, server performance issues require development effort on the client. The problem that's been brought up both here and on talk, where the server dies halfway through a write operation and leaves the database in an inconsistent state, is an example of this - there are others. When TomH did the quadtile stuff, that added a degree of complexity for a boost in performance. I doubt anyone now would say it was the wrong decision. cheers Richard -- View this message in context: http://www.nabble.com/Ruby-developers-in-Amsterdam-tp21433520p21436160.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 1:07 PM, Frederik Ramm wrote: > Matt Amos wrote: >> >> what do you see as the benefits of an old-fashioned compiled language >> (presumably you mean C/C++/Java) over just plain ruby? just because >> we're using ruby doesn't mean we have to use rails+activerecord :-) > > I think nothing beats C/C++ (not so sure about Java) when it comes to > pumping bytes from one source to one destination with as little memory usage > and as little copying of data as possible. any java fan would tell you that java is much better nowadays at that sort of thing. but i'm not a java fan, so i'm going to agree with you - when memory management really matters, nothing beats doing it manually. > I also think that we're at least > partially operating in areas where it really makes a difference what kind of > read or write operation you use, how many bytes you send in one buffer, how > many you request from the database driver, and so on. (And the optimum > performance settings may well be different for a MySQL and a PostGIS > backend, so I tend to say "no thank you" if anyone wants to sell me a super > duper database wrapper that makes everything database "agnostic".) in which case we're trading development effort for performance. i think we're operating in areas constrained by the lack of available developers, so making the barrier to entry higher is a tough decision. in the short term, i think its the wrong one. > In 99.9% of applications nobody gives a damn whether some dynamic > programming language allocates the same memory three times in a row and > copies the same data from one location to the next, to the next - it's over > in microseconds and nobody notices, and you're better off writing > worse-performing better-looking code than squeezing the last juice out of > your CPU. and ruby will be getting faster and better with the new VM and GC in 1.9. rails is (allegedly) getting faster with each release. if we're prepared to stick it out for a while, we may see improved performance for little effort on our part. also, the front end is one place where we can use multiple machines to spread the cost of dynamic languages and gain some redundancy as well. but we've had continuing problems with high memory usage which may not be fixed, so there are arguments both ways. i think tomh and firefishy definitely care about ruby/rails' memory usage :-) > OSM might be one of the 0.1%. But I might also be wrong. you identified a real problem - part of OSM is ideally suited to rails and part of it isn't. assuming that we're talking about after 0.6 - what should we be doing? is the inclusion of OAuth the right point to start separating out the different parts of the OSM codebase? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Op 13 jan 2009, om 13:17 heeft Frederik Ramm het volgende geschreven: > Martijn, > >> The goal of the talk will be to interest the (Amsterdam) Ruby >> community in OSM in general, but also in becoming involved in OSM >> Rails development. Is that helpful? > > My personal take on this is that we actually have two, very > different, kinds of "web interface", or better "http interface". > [...] The distinction between interfacing with end users (the front end) and interfacing between the database through the API with non-human clients is a useful one. I don't want to get into a discussion as to whether Ruby / Rails is the best choice for this - but at least I'm prepared for a discussion among the Ruby devs :) > > In my eyes, OSM is not a "typical Rails project", nor even a project > where Rails can demonstrate superiority to anything else. In parts, > OSM is even a "we're Rails here but it's really not optimal" project. I don't think it needs to be, at least not for knowledgable Ruby professionals. It might be interesting for them to see Ruby on rails used in a non-typical use case. Best practices are a dime a dozen I guess. Let's see. Thanks for your take on this. Martijn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 1:02 PM, Richard Fairhurst wrote: > Matt Amos wrote: >> just because we're using ruby doesn't mean we have to use >> rails+activerecord :-) > > You mean I can put all the SQL back in amf_controller and get it running at > a reasonable speed again? ;) not yet :-) if, in the future, we reckon that moving away from rails is worthwhile then the whole API will be changed. at present, having some bits rails-y and other bits not just makes it more difficult to develop, test and debug, in my opinion. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Coastline data for mapnik renders "hole" near rio de janeiro
Can someone help? http://www.openstreetmap.org/?lat=-21.776&lon=-41.863&zoom=9&layers=B000FTF I oppened Ticket #1490 for this. thanks ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] JOSM Mappaint - major improvements
Hi, Dirk Stöcker wrote: > That would be easy for encapsulated data access. For current JOSM it will > be lots of work. Explain? Can't you just "when in doubt" always put the object on the visible list? It doesn't hurt if a few are not actually visible. Bye Frederik ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Ruby developers in Amsterdam
Hi, Matt Amos wrote: > what do you see as the benefits of an old-fashioned compiled language > (presumably you mean C/C++/Java) over just plain ruby? just because > we're using ruby doesn't mean we have to use rails+activerecord :-) I think nothing beats C/C++ (not so sure about Java) when it comes to pumping bytes from one source to one destination with as little memory usage and as little copying of data as possible. I also think that we're at least partially operating in areas where it really makes a difference what kind of read or write operation you use, how many bytes you send in one buffer, how many you request from the database driver, and so on. (And the optimum performance settings may well be different for a MySQL and a PostGIS backend, so I tend to say "no thank you" if anyone wants to sell me a super duper database wrapper that makes everything database "agnostic".) In 99.9% of applications nobody gives a damn whether some dynamic programming language allocates the same memory three times in a row and copies the same data from one location to the next, to the next - it's over in microseconds and nobody notices, and you're better off writing worse-performing better-looking code than squeezing the last juice out of your CPU. OSM might be one of the 0.1%. But I might also be wrong. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Matt Amos wrote: > just because we're using ruby doesn't mean we have to use > rails+activerecord :-) You mean I can put all the SQL back in amf_controller and get it running at a reasonable speed again? ;) cheers Richard -- View this message in context: http://www.nabble.com/Ruby-developers-in-Amsterdam-tp21433520p21435286.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 12:17 PM, Frederik Ramm wrote: > One is talking to users - letting them register, write blogs, write > messages (and if the trend continues, one day they'll be able to upload > images, have multi-page user profiles and forums and a full-blown E-Mail > system all within OSM). maybe we should just bow to the inevitable and re-write OSM as a facebook app :-P > This is > the part where a Rails-like system excels, but it is not in any way > mission critical for us and could also be done in a scripting language > or web templating system of your choice. arguably user logins are pretty critical, but i get your point. > here, the API is actually just a rather thin > REST layer on top of the database. a large amount of the work done in the rails layer when writing is validation and the rails validations are a nice way of doing that. > And this is where, in my eyes, the > whole Rails migrations and ActiveRecord automatism is of no use at all > and is in fact circumvented more than it is used. This part should > probably be separated from the rest and implemented in an old-fashioned > compiled language. certainly for things like the map call the rails machinery is not very useful. i would agree that, at some point in the future, we'd like to move as much of the logic and constraints of the API into the database and use things like triggers to handle updating the history tables. there are downsides to this approach as well. for example, in an old-fashioned compiled language it would have taken much longer to generalise the database backend to both mysql and postgres (and the barrier to entry would have been higher). yes, yes, i know if it were designed "properly" it would have been database agnostic from the beginning. what do you see as the benefits of an old-fashioned compiled language (presumably you mean C/C++/Java) over just plain ruby? just because we're using ruby doesn't mean we have to use rails+activerecord :-) > In my eyes, OSM is not a "typical Rails project", nor even a project > where Rails can demonstrate superiority to anything else. In parts, OSM > is even a "we're Rails here but it's really not optimal" project. there are many kinds of optimality. in this case i think the brevity, readability and testability of the API code is improved by adopting rails-isms, even though they're not a perfect fit to the kinds of things we're trying to do. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Data Consistency Checks now available
Hi Harald, the one you pointed out seem pretty much to be errors. Please see this: http://www.openstreetmap.org/?lat=45.872858&lon=11.031874&zoom=18&layers=B000FTF http://keepright.ipax.at/report_map.php?ch30=30&ch40=40&ch50=50&ch60=60&ch70=70&ch80=80&ch90=90&ch100=100&ch110=110&ch120=120&ch130=130&ch150=150&ch160=160&ch170=170&ch180=180&ch190=190&ch200=200&lat=45.87352910639793&lon=11.032413884766799&zoom=17&requery=requery As on hi-zoom the error icons disappear for some reason, I confused the error from the adiacent one-way road with the roundabout. So sorry for the wrong bug report. :) Alessandro Harald Kleiner ha scritto: > Hi, Alessandro! > > I'm not sure if I fully understand your post. > I do think that a roundabout should be a circular way where the first > node is connected to the last one. There need not necessarily be another > road connected to the first node of such a loop. > > Please give me an example so I can reproduce that! > Don't you think that this > http://www.openstreetmap.org/?lat=43.2850843&lon=5.5819796&zoom=18 > http://www.openstreetmap.org/api/0.5/way/3617368 > is an error? > > Regards, > > Harald > > Alessandro Briosi schrieb: >> Hi, >> this tool is very helpful. >> >> I found a potential false positive which is related to roundabouts. >> >> A roundabout which has node 1 not connected to any way. It's considered >> as a oneway (which is correct) but it does not matter if it's start and >> end nodes are not connected, imho. >> >> Alessandro > > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
> Diff uploads will not help online editor such as Potlatch. The > addition of transactions and exposure of the version numbers will help > more. Why not? It may well utilize it for changes to multiple objects that should be atomic, e.g. the example mentioned here. Regards, Marc -- Pt! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Limits on numbers of tags
I get a 400 error when I try and upload nodes with more than 50 tags and about 4,300 bytes. This is to a test database, not www.openstreetmap.org/api. Is anyone aware of any limitations? Regards Neil Penman Stay connected to the people that matter most with a smarter inbox. Take a look http://au.docs.yahoo.com/mail/smarterinbox___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
On 13 Jan 2009, at 12:14, Marc Schütz wrote: >>> It has been explained many times in the >>> past, not just >>> by me, that database freakiness _will_ happen because we don't >>> have >>> transactions, our server is often under high load, and there >>> may be memory >>> leaks or blocking processes in some of the software we use. >> >> Do I understand correctly that the introduction of changesets in >> API 0.6 >> should go some way towards addressing this problem? As I understand >> it the >> whole changeset needs to make it to the server for it to be >> applied, and >> therefore I've been assuming that when updating a way the addition, >> updating >> and removal of nodes to a way would all be part of a single >> changeset. Or >> have I misunderstood? > > Changesets merely group changes together, they are not atomic. > However, API 0.6 will also implement diff uploads [1], which are > atomic and can therefore be used for this. Diff uploads will not help online editor such as Potlatch. The addition of transactions and exposure of the version numbers will help more. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
On 13 Jan 2009, at 09:22, Ed Loach wrote: > Richard wrote: > >> It has been explained many times in the >> past, not just >> by me, that database freakiness _will_ happen because we don't >> have >> transactions, our server is often under high load, and there >> may be memory >> leaks or blocking processes in some of the software we use. > > Do I understand correctly that the introduction of changesets in API > 0.6 should go some way towards addressing this problem? As I > understand it the whole changeset needs to make it to the server for > it to be applied, and therefore I've been assuming that when > updating a way the addition, updating and removal of nodes to a way > would all be part of a single changeset. Or have I misunderstood? Changesets are groups of edits. A whole changeset is not atomic (i.e. does not need to be sent to the server in one go). Each edit will be atomic (i.e. the whole request will all complete, or will all fail if there is a problem with one part of the data, or someone has modified the data since the start of the request or the user last downloaded.) The exposing of version numbers will help too. Shaun ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Martijn, > The goal of the talk will be to interest the (Amsterdam) Ruby > community in OSM in general, but also in becoming involved in OSM > Rails development. Is that helpful? My personal take on this is that we actually have two, very different, kinds of "web interface", or better "http interface". One is talking to users - letting them register, write blogs, write messages (and if the trend continues, one day they'll be able to upload images, have multi-page user profiles and forums and a full-blown E-Mail system all within OSM). This part includes minor database activity, like storing user preferences in a table when the user requests it. This is the part where a Rails-like system excels, but it is not in any way mission critical for us and could also be done in a scripting language or web templating system of your choice. The other is the hard-core database part where we're not talking to humans at their web browsers but to editors or other clients which want to download or upload data. We talk XML, not HTML, here, and we have massive database activity; here, the API is actually just a rather thin REST layer on top of the database. And this is where, in my eyes, the whole Rails migrations and ActiveRecord automatism is of no use at all and is in fact circumvented more than it is used. This part should probably be separated from the rest and implemented in an old-fashioned compiled language. In my eyes, OSM is not a "typical Rails project", nor even a project where Rails can demonstrate superiority to anything else. In parts, OSM is even a "we're Rails here but it's really not optimal" project. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
> > It has been explained many times in the > > past, not just > > by me, that database freakiness _will_ happen because we don't > > have > > transactions, our server is often under high load, and there > > may be memory > > leaks or blocking processes in some of the software we use. > > Do I understand correctly that the introduction of changesets in API 0.6 > should go some way towards addressing this problem? As I understand it the > whole changeset needs to make it to the server for it to be applied, and > therefore I've been assuming that when updating a way the addition, updating > and removal of nodes to a way would all be part of a single changeset. Or > have I misunderstood? Changesets merely group changes together, they are not atomic. However, API 0.6 will also implement diff uploads [1], which are atomic and can therefore be used for this. [1] http://wiki.openstreetmap.org/wiki/0.6#Diff_uploads Regards, Marc -- Sensationsangebot verlängert: GMX FreeDSL - Telefonanschluss + DSL für nur 16,37 Euro/mtl.!* http://dsl.gmx.de/?ac=OM.AD.PD003K1308T4569a ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Ruby developers in Amsterdam
Hi dev, I've been given the opportunity to give a talk during a gathering of Ruby developers in Amsterdam[1] next month. I want to present them with a general high level introduction of OSM and then make the link to Ruby by explaining a bit about the Rails part of the OSM infrastructure. Problem is - I'm not a Ruby developer myself and I don't know much about this part of the OSM infrastructure. I have looked at http://wiki.openstreetmap.org/wiki/Rails briefly, but it's mostly installation instructions. Is there another source explaining the architecture of the Rails port? Preferably with a diagram or such. Any pointers welcome :) The goal of the talk will be to interest the (Amsterdam) Ruby community in OSM in general, but also in becoming involved in OSM Rails development. Is that helpful? Martijn [1] http://rubyandrails.org/usergroups/amsterdam/ -- martijn van exel -+- mve...@gmail.com -+- http://www.schaaltreinen.nl/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
Richard Fairhurst a scris:
> D Tucny wrote:
>> Eddy Petrișor wrote:
>>> What's even more weirder is the fact that if I try to edit with
>>> Potlatch, the way is displayed correctly, as if stale data was read of
>>> as if Mapnik's data was read.
>>> [...]
>> There was a post 12 hours ago to the talk list about the exact same
>> thing...
>>
>> Richard the author of potlatch explained the issue in his reply to that
>> post 11 hours ago as being a partial written save due to server load
>> or network problems or some other weird issue, the result being that
>> ways can contain nodes that are deleted...
I am not subscribed to the talk list. Is not exactly clear which lists
should be of interest, depending on the usage of the user goals and
which are a *must* if one intents to do some type of work, like development.
> Exactly (and thanks for posting before I got a chance to :) ).
Hello,
Thanks for the explanations.
> When Potlatch writes a way it first of all updates the nodes:
>
> http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb?rev=11420#L356
>
> then it deletes any nodes that were used by the old version, but not the new
> one:
>
> http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb?rev=11420#L400
>
> and finally, it writes the new version of the way:
>
> http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb?rev=11420#L411
>
> What appears to be happening in these cases is that the server is timing out
> while deleting the surplus ("unique") nodes. Therefore there is a
> discrepancy between the state of the nodes, and of the way.
>
> I have just committed a change to swap the last two stages around. Therefore
> if the server breaks when deleting the unused nodes, the way will already
> have been written. This of course is not really fixing the problem, it's
> moving it elsewhere, but hey.
As I understand it, in the worse case scenario, this could lead to
possible dead nodes appearing, right?
(Of course, this wouldn't be as bad as having corrupt ways, so it's an
improvement.)
> On a side issue it is offensive and demoralising when people jump to
> conclusions in this way, especially when they clearly haven't bothered to
> search the mailing list archives first: this has been discussed several
Sorry for not searching the archives, it is indeed my fault, and I am
sorry if you found my initial mail offensive.
I spite of that, I always advice people not to take criticism of the
projects the work on as a personal offensive, but only pick the
objective information since, in most cases, people act on either panic
(OMG, this is breaking data, I'd better tell people since it might break
anything is touched), frustration (damn, this made me waste 3 hours of
work) or some other badly charged emotional state and seldom out of pure
pleasure. I acted on panic, so please accept my apologies if I managed
to negatively charge the discussion from the get go.
Looking on the bright side, I still prefer Potlatch to JOSM just because
is accessible anywhere and it doesn't make my system crawl. I can start
reasonably fast working on stuff I want to correct (without a GPX trace)
without trying to spot the area close and small enough for me to get the
exact rectangle that interests me (like JOSM does without a planet
file). I like the "parallel" way feature and I like that the yahoo!
aerials work (as opposed to JOSM with firefox3).
OTOH, back to the discussion, I am not subscribed to talk and I didn't
knew it was a precondition to joining dev. If this is true, maybe the
description of dev should say "OpenStreetMap developer discusssion. Is
recommended you also join "talk", if you're subscribing to this list".
Please note that I am not trying to act smart, but I am really trying to
help and suggest improvements.
> times in the past, and as D said, most recently 12 hours ago! It is not a
> "really BAD bug" in Potlatch, it is poor server performance allied to the
> lack of transactions. It has been explained many times in the past, not just
Which leads to what looks like a path for data breakage, as some people
on the RO-OSM list feared. Taking into account that the history of the
way was truncated and that it was missing nodes (for any pov, except
Potlatch itself) this was a valid concern. I understand you're caught
between a rock and a hard place since transactions do not exist yet in
the API, so there's isn't much you can do now to fix this.
> by me, that database freakiness _will_ happen because we don't have
> transactions, our server is often under high load, and there may be memory
> leaks or blocking processes in some of the software we use. I don't mind you
> getting annoyed because data isn't how you expect but I do object to you
> blaming one particular component when it actually ain't that simple.
So, AIUI, even JOSM or any other sw that works with relatively high
loads of data at a t
Re: [OSM-dev] How reconstrucing a way from the history?
On Jan 12, 2009, at 13:45, Nevel Gandish wrote: > And can I be sure that when moving a node caused a new history entry > for the node and the way that both have the exact timestamp or might > they differ by a few seconds? Just moving a node doesn't create a new history entry for the way, I think. Cheers Robert ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
Richard wrote: > It has been explained many times in the > past, not just > by me, that database freakiness _will_ happen because we don't > have > transactions, our server is often under high load, and there > may be memory > leaks or blocking processes in some of the software we use. Do I understand correctly that the introduction of changesets in API 0.6 should go some way towards addressing this problem? As I understand it the whole changeset needs to make it to the server for it to be applied, and therefore I've been assuming that when updating a way the addition, updating and removal of nodes to a way would all be part of a single changeset. Or have I misunderstood? Ed ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Potlatch - really BAD bug
D Tucny wrote:
>Eddy Petrișor wrote:
>> What's even more weirder is the fact that if I try to edit with
>> Potlatch, the way is displayed correctly, as if stale data was read of
>> as if Mapnik's data was read.
>> [...]
>
> There was a post 12 hours ago to the talk list about the exact same
> thing...
>
> Richard the author of potlatch explained the issue in his reply to that
> post 11 hours ago as being a partial written save due to server load
> or network problems or some other weird issue, the result being that
> ways can contain nodes that are deleted...
Exactly (and thanks for posting before I got a chance to :) ).
When Potlatch writes a way it first of all updates the nodes:
http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb?rev=11420#L356
then it deletes any nodes that were used by the old version, but not the new
one:
http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb?rev=11420#L400
and finally, it writes the new version of the way:
http://trac.openstreetmap.org/browser/sites/rails_port/app/controllers/amf_controller.rb?rev=11420#L411
What appears to be happening in these cases is that the server is timing out
while deleting the surplus ("unique") nodes. Therefore there is a
discrepancy between the state of the nodes, and of the way.
I have just committed a change to swap the last two stages around. Therefore
if the server breaks when deleting the unused nodes, the way will already
have been written. This of course is not really fixing the problem, it's
moving it elsewhere, but hey.
On a side issue it is offensive and demoralising when people jump to
conclusions in this way, especially when they clearly haven't bothered to
search the mailing list archives first: this has been discussed several
times in the past, and as D said, most recently 12 hours ago! It is not a
"really BAD bug" in Potlatch, it is poor server performance allied to the
lack of transactions. It has been explained many times in the past, not just
by me, that database freakiness _will_ happen because we don't have
transactions, our server is often under high load, and there may be memory
leaks or blocking processes in some of the software we use. I don't mind you
getting annoyed because data isn't how you expect but I do object to you
blaming one particular component when it actually ain't that simple.
cheers
Richard
--
View this message in context:
http://www.nabble.com/Potlatch---really-BAD-bug-tp21427998p21431363.html
Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com.
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
