Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 2:39 PM, Richard Fairhurst rich...@systemed.net wrote: Where the raw SQL approach scores is in the read operations, which are incredibly simple, easy-to-understand queries. In Potlatch these are whichways and getway, in the XML API it'd be the /map call. The disadvantage of using Rails in this context is obviously that you have this huge overhead to create objects for every way or node, simply to serialise them and dispose of the objects immediately. Because they're reads, there are (generalising a bit) no integrity issues: and because they're very common calls, you get a significant speedup. As far as I'm concerned this is the most interesting bit for a rails discussion. Given that, especially in the map call, we take many rows from a db table and serialise them to xml, it's a complete waste of time to try to read all the rows first, then serialise them all after that. They really don't depend on one another in any way. The (apparent) lack of a railsy way to read-serialise-stream row-by-row is a PITA. Cheers, Andy ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Martijn, The goal of the talk will be to interest the (Amsterdam) Ruby community in OSM in general, but also in becoming involved in OSM Rails development. Is that helpful? My personal take on this is that we actually have two, very different, kinds of web interface, or better http interface. One is talking to users - letting them register, write blogs, write messages (and if the trend continues, one day they'll be able to upload images, have multi-page user profiles and forums and a full-blown E-Mail system all within OSM). This part includes minor database activity, like storing user preferences in a table when the user requests it. This is the part where a Rails-like system excels, but it is not in any way mission critical for us and could also be done in a scripting language or web templating system of your choice. The other is the hard-core database part where we're not talking to humans at their web browsers but to editors or other clients which want to download or upload data. We talk XML, not HTML, here, and we have massive database activity; here, the API is actually just a rather thin REST layer on top of the database. And this is where, in my eyes, the whole Rails migrations and ActiveRecord automatism is of no use at all and is in fact circumvented more than it is used. This part should probably be separated from the rest and implemented in an old-fashioned compiled language. In my eyes, OSM is not a typical Rails project, nor even a project where Rails can demonstrate superiority to anything else. In parts, OSM is even a we're Rails here but it's really not optimal project. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 12:17 PM, Frederik Ramm frede...@remote.org wrote: One is talking to users - letting them register, write blogs, write messages (and if the trend continues, one day they'll be able to upload images, have multi-page user profiles and forums and a full-blown E-Mail system all within OSM). maybe we should just bow to the inevitable and re-write OSM as a facebook app :-P This is the part where a Rails-like system excels, but it is not in any way mission critical for us and could also be done in a scripting language or web templating system of your choice. arguably user logins are pretty critical, but i get your point. here, the API is actually just a rather thin REST layer on top of the database. a large amount of the work done in the rails layer when writing is validation and the rails validations are a nice way of doing that. And this is where, in my eyes, the whole Rails migrations and ActiveRecord automatism is of no use at all and is in fact circumvented more than it is used. This part should probably be separated from the rest and implemented in an old-fashioned compiled language. certainly for things like the map call the rails machinery is not very useful. i would agree that, at some point in the future, we'd like to move as much of the logic and constraints of the API into the database and use things like triggers to handle updating the history tables. there are downsides to this approach as well. for example, in an old-fashioned compiled language it would have taken much longer to generalise the database backend to both mysql and postgres (and the barrier to entry would have been higher). yes, yes, i know if it were designed properly it would have been database agnostic from the beginning. what do you see as the benefits of an old-fashioned compiled language (presumably you mean C/C++/Java) over just plain ruby? just because we're using ruby doesn't mean we have to use rails+activerecord :-) In my eyes, OSM is not a typical Rails project, nor even a project where Rails can demonstrate superiority to anything else. In parts, OSM is even a we're Rails here but it's really not optimal project. there are many kinds of optimality. in this case i think the brevity, readability and testability of the API code is improved by adopting rails-isms, even though they're not a perfect fit to the kinds of things we're trying to do. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Matt Amos wrote: just because we're using ruby doesn't mean we have to use rails+activerecord :-) You mean I can put all the SQL back in amf_controller and get it running at a reasonable speed again? ;) cheers Richard -- View this message in context: http://www.nabble.com/Ruby-developers-in-Amsterdam-tp21433520p21435286.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Hi, Matt Amos wrote: what do you see as the benefits of an old-fashioned compiled language (presumably you mean C/C++/Java) over just plain ruby? just because we're using ruby doesn't mean we have to use rails+activerecord :-) I think nothing beats C/C++ (not so sure about Java) when it comes to pumping bytes from one source to one destination with as little memory usage and as little copying of data as possible. I also think that we're at least partially operating in areas where it really makes a difference what kind of read or write operation you use, how many bytes you send in one buffer, how many you request from the database driver, and so on. (And the optimum performance settings may well be different for a MySQL and a PostGIS backend, so I tend to say no thank you if anyone wants to sell me a super duper database wrapper that makes everything database agnostic.) In 99.9% of applications nobody gives a damn whether some dynamic programming language allocates the same memory three times in a row and copies the same data from one location to the next, to the next - it's over in microseconds and nobody notices, and you're better off writing worse-performing better-looking code than squeezing the last juice out of your CPU. OSM might be one of the 0.1%. But I might also be wrong. Bye Frederik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 1:02 PM, Richard Fairhurst rich...@systemed.net wrote: Matt Amos wrote: just because we're using ruby doesn't mean we have to use rails+activerecord :-) You mean I can put all the SQL back in amf_controller and get it running at a reasonable speed again? ;) not yet :-) if, in the future, we reckon that moving away from rails is worthwhile then the whole API will be changed. at present, having some bits rails-y and other bits not just makes it more difficult to develop, test and debug, in my opinion. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Op 13 jan 2009, om 13:17 heeft Frederik Ramm het volgende geschreven: Martijn, The goal of the talk will be to interest the (Amsterdam) Ruby community in OSM in general, but also in becoming involved in OSM Rails development. Is that helpful? My personal take on this is that we actually have two, very different, kinds of web interface, or better http interface. [...] The distinction between interfacing with end users (the front end) and interfacing between the database through the API with non-human clients is a useful one. I don't want to get into a discussion as to whether Ruby / Rails is the best choice for this - but at least I'm prepared for a discussion among the Ruby devs :) In my eyes, OSM is not a typical Rails project, nor even a project where Rails can demonstrate superiority to anything else. In parts, OSM is even a we're Rails here but it's really not optimal project. I don't think it needs to be, at least not for knowledgable Ruby professionals. It might be interesting for them to see Ruby on rails used in a non-typical use case. Best practices are a dime a dozen I guess. Let's see. Thanks for your take on this. Martijn ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 1:07 PM, Frederik Ramm frede...@remote.org wrote: Matt Amos wrote: what do you see as the benefits of an old-fashioned compiled language (presumably you mean C/C++/Java) over just plain ruby? just because we're using ruby doesn't mean we have to use rails+activerecord :-) I think nothing beats C/C++ (not so sure about Java) when it comes to pumping bytes from one source to one destination with as little memory usage and as little copying of data as possible. any java fan would tell you that java is much better nowadays at that sort of thing. but i'm not a java fan, so i'm going to agree with you - when memory management really matters, nothing beats doing it manually. I also think that we're at least partially operating in areas where it really makes a difference what kind of read or write operation you use, how many bytes you send in one buffer, how many you request from the database driver, and so on. (And the optimum performance settings may well be different for a MySQL and a PostGIS backend, so I tend to say no thank you if anyone wants to sell me a super duper database wrapper that makes everything database agnostic.) in which case we're trading development effort for performance. i think we're operating in areas constrained by the lack of available developers, so making the barrier to entry higher is a tough decision. in the short term, i think its the wrong one. In 99.9% of applications nobody gives a damn whether some dynamic programming language allocates the same memory three times in a row and copies the same data from one location to the next, to the next - it's over in microseconds and nobody notices, and you're better off writing worse-performing better-looking code than squeezing the last juice out of your CPU. and ruby will be getting faster and better with the new VM and GC in 1.9. rails is (allegedly) getting faster with each release. if we're prepared to stick it out for a while, we may see improved performance for little effort on our part. also, the front end is one place where we can use multiple machines to spread the cost of dynamic languages and gain some redundancy as well. but we've had continuing problems with high memory usage which may not be fixed, so there are arguments both ways. i think tomh and firefishy definitely care about ruby/rails' memory usage :-) OSM might be one of the 0.1%. But I might also be wrong. you identified a real problem - part of OSM is ideally suited to rails and part of it isn't. assuming that we're talking about after 0.6 - what should we be doing? is the inclusion of OAuth the right point to start separating out the different parts of the OSM codebase? cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Matt Amos wrote: in which case we're trading development effort for performance. i think we're operating in areas constrained by the lack of available developers, so making the barrier to entry higher is a tough decision. (Speaking seriously for once...) Kind of, but it's not a simple trade-off. Certainly as far as Potlatch is concerned, server performance issues require development effort on the client. The problem that's been brought up both here and on talk, where the server dies halfway through a write operation and leaves the database in an inconsistent state, is an example of this - there are others. When TomH did the quadtile stuff, that added a degree of complexity for a boost in performance. I doubt anyone now would say it was the wrong decision. cheers Richard -- View this message in context: http://www.nabble.com/Ruby-developers-in-Amsterdam-tp21433520p21436160.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
On Tue, Jan 13, 2009 at 1:53 PM, Richard Fairhurst rich...@systemed.net wrote: Matt Amos wrote: in which case we're trading development effort for performance. i think we're operating in areas constrained by the lack of available developers, so making the barrier to entry higher is a tough decision. (Speaking seriously for once...) Kind of, but it's not a simple trade-off. Certainly as far as Potlatch is concerned, server performance issues require development effort on the client. The problem that's been brought up both here and on talk, where the server dies halfway through a write operation and leaves the database in an inconsistent state, is an example of this - there are others. these problems are not due to rails - most are due to the lack of transactions, which is an implementation problem, and the many others are version race conditions, which is an API design problem. lets not confuse a performance issue with a bug or correctness issue. When TomH did the quadtile stuff, that added a degree of complexity for a boost in performance. I doubt anyone now would say it was the wrong decision. absolutely, but i'd say that was a small increase in complexity in a small corner of the code (which is well hidden) for a large boost in performance. i'm sorry if i gave the impression i was totally against any form of increased complexity, because i'm not. i just think these things need to be balanced on a case by case basis. in the case of quadtile, i think the extra complexity was worthwhile. in the case of SQL in the amf_controller, i'm not so sure. this is just my opinion. cheers, matt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Matt Amos wrote: in the case of quadtile, i think the extra complexity was worthwhile. in the case of SQL in the amf_controller, i'm not so sure. this is just my opinion. Sure. We can even do more nuanced than that, if you like. :) In my opinion - and again, just that - the clarity of Rails is worthwhile and important in the write operations. putway is a lot more readable in Rails than it ever was in raw SQL, and it's more reassuring in terms of data integrity, too. Speed isn't really an issue here as these are called comparatively rarely. Where the raw SQL approach scores is in the read operations, which are incredibly simple, easy-to-understand queries. In Potlatch these are whichways and getway, in the XML API it'd be the /map call. The disadvantage of using Rails in this context is obviously that you have this huge overhead to create objects for every way or node, simply to serialise them and dispose of the objects immediately. Because they're reads, there are (generalising a bit) no integrity issues: and because they're very common calls, you get a significant speedup. cheers Richard -- View this message in context: http://www.nabble.com/Ruby-developers-in-Amsterdam-tp21433520p21437005.html Sent from the OpenStreetMap - Dev mailing list archive at Nabble.com. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Ruby developers in Amsterdam
Martijn van Exel wrote: Is there another source explaining the architecture of the Rails port? Preferably with a diagram or such. Hardware wise... 1x Database Server (MySQL) 3x Dedicated Rails Application Servers (Ruby 1.8.6, Rails 2.0.x) 1x Frontend Web Server (lighttpd + Ruby 1.8.6, Rails 2.0.x) Dynamic requests are broken down into 4 types: Web: Website, Login, Signup, Messages etc (Handled by frontend web) TAH: All ti...@home requests BulkAPI: API map, trackpoint, amf/reads and swf/trackpoints API: New nodes, rest of API stuff See: http://svn.openstreetmap.org/sites/rails_port/config/lighttpd.conf There have been problems in the past with Rail's libxml binding leaking memory. Rails seams to leak memory on some of the BulkAPI and occasionally API requests. Particularly /api/0.5/map API code has a forced Rails daemons death after ~1 requests to free leaked memory. / Grant ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
