Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Hello, 1) I had a long chat with James Mike about his project. we basically have a same idea of a osm-inputoutput, tiles-from-stored-osm-output machine image as to the outer interfaces. a) James Mike wants osm inputoutput in git (sftp access) or later some very light webservice (not the rails port, a custom c++ thing) accepting osm read/write (eg. from JOSM) b) he wants a hot rendering (in c++ with a modified mapnik build he's started work on) of tiles outdated from new osm data. Those tiles would be served over git or some http daemon Below, I'm presenting my project's aims which are less demanding in terms of required speed CPU power (James Mike told me that his solution could possibly run in embedded devices like a mobile phone !!). What I'll write replies namely to Emilie's comment on my 2nd e-mail in this mail discussion. Well, one quick point: database performance are going to be bad. Someone is currently trying to do something similar in one of the mailing list. Cloud vendor are very good for serving content like a web page. They are extremely poor for database performance. The IO is usually catastrophic. I wouldn't expect any real good performance out of such a system especially for rendering. I think that using cloud technology is for the moment at least a bit pointless. Emilie Laffray For my project (indooroutdoor mapping of small scale objects (you're able to see the coffee machine and tables rendered as rectangles or point POIs) rather small concentrations (eg. spots the size of a building, european-size (wink to north america :-) ) university campus, or a small city), I just want a working machine image that's ready to redeploy and takes : a) josm data as input over http (the rails port fits me for this) b) a sync between the rails port's db and mapnik's db. So the standard implementation we don't mind using for now is : mapnik(+postgis) rendering tiles on demand through mod_tiles,renderd,minutely or hourly change interval osmosis technique We don't mind: 1) having perfectly optimized instance images (eg. we'll have 5 buildings per instance) 2) working on the cloud despite low database IO, as the cloud is advantageous for us in that we can deploy new instances remotely with simple scripting (eg. 1 company dedicated building indoor mapping space = 1 new instance for them, or if we have 5 buildings on each of our instances already and are requested to host 2 new building's maps, we run another instance filling it with those 2...). If my project (Geopard) goes into a start up something, having a dedicated machine instead of an EC2 instance for indooroutdoor high precision maps hosting could be part of some business model. I would appreciate having either a ready to use : - normal machine image (eg. vmware, virtualbox) or - Amazon Machine Images (AMI) ... with easy osm inputoutput + hot rendering into tiles Please give me contacts of people doing any of the two. Anyway, we will release by the end of January 2010 an AMI with a vanilla install of rails+timely change interval sync+mapnik, shipping with nice scripts to install (maybe reconfigure) the server with your settings (eg. for the rails port, but imagine it for mapnik and the sync part too : just edit in a single config file your wished rails db names, login+password, posgres memory limits and rails website login+password accounts, and some script you run by hand will install the rails port with those settings for you). We will try to release in this AMI, some default settings that you can reuse for having indoor mapping working (ie. tiles rendering with increased max zoom + 1 floor=1 TMS layer) possibility to disable sea limits land zones so that your map is full white/empty at the beginning, possibility to sync timely only a bbox (not the planet file) from osm.org into your db). Our scripts will be in Ruby (regular shell ruby, not rails) I think. Who wants to collaborate on those magic deployment scripts ? In the end, maybe we could have a vanilla OSM instance be configured/installed from a php page filled by some enduser, a bit like tiledrawer does (well.. the config is in JSON data passed to the AMI at startup, but then you see installationrendering progress in status.php). Take care, Jonathan On Fri, Jan 8, 2010 at 6:13 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: Ok, this looks great. So, now, that would be my suggestion to use on this amazon webservice. A lot of small osm files, the size of the smallest tile. Now, on thing did occur to me while walking the dog, we could render multiple zoom levels at once. Every time you render the smallest unit (tile) you would render the sections of all the zoom levels and that should prevent you from touching the data at the same time. mike On Fri, Jan 8, 2010 at 6:08 PM, Dane Springmeyer bl...@hailmail.net wrote: On Jan 8, 2010, at 7:19 AM, jamesmikedup...@googlemail.com wrote:
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Hello, Please tell me if I'm replicating any initiative or doing something the wrong or a not-so-good way. I've started this week from an EBS backed (or boot) Ubuntu 8.04 image ami-f6ad409f. I agree to give the .pem file for ssh connection to an instance to anyone willing to collaborate. My idea is to provide an EC2 EBS image with rails port(done) + tile rendering setup(started) + rails port-tile rendering sync scripts(todo) + easy config scripts (why not triggerable on startup with JSON config ala tilecache.org) (todo, not sure I'll have time to it). The list of commands to install is very long. I note down all the commands I type so as to be able to one or more bash scripts to be able to reinstall a similar solution on an Ubuntu N.NN later (by assuming that commands/package names for 8.04 are similar to those for 9.04 etc..). Maybe I'll have 3 bash scripts such as : install_railsport_u804.sh, install_mapnik_u804.sh, install_railsport_mapnik_sync.sh. And I'd need other scripts to go along : setup_rails_port_u804.sh, setup_mapnik_u804.sh and setup_rails_mapnik_sync_u804.sh... What do you think of this ? A recall on the requirements (reply to change them if you'd like to collaborate or request something): - postgres+osmosis+rails port (with 3 dbs : development, testing and production) : filled with nothing for now. setup_rails_port_u804.sh would have a simple calling syntax like : sudo setup_rails_port_u804.sh [--bz2url= http://domain/someRegion.osm.bz2|--bz2file=/path/to/file.osm.bz2http://domain/someRegion.osm.bz2%7C--bz2file=/path/to/file.osm.bz2] [--forceResetDB (false if omitted)] [--dbName=development(default),other,dbs,to,make,identical,too] [--guessDBLoginPassFromRailsConfig (default is true)] - mapnik tile rendering setup. I want to have a simple script to insert a bz2 + tell to use some style sheets + to tell to render at z=N (max zoom) - cron'able synced file making the Mapnik setup be filled with the Rails ports's data. I plan to get inspiration from the myanmar scripts for this. I'm sorry to repeat this, though, please reply if you know of some linux image (on EC2 or not) hanging somewhere which almost does what I'm trying to achieve and that image is easy to redeploy (ex : .iso or vmware/virtual bos EC2 ami). Jonathan On Sun, Dec 20, 2009 at 6:20 PM, Jonathan-David SCHRODER jonathan.schro...@gmail.com wrote: Hello, I'd like to create a platform like opencyclemap.org for indoor stuff easily, and more on more people on the Internet seem to want to setup their own openstreetmap server. Some tutorials exist describing the steps to obtain an OSM DB with data in it + a first Mapnik rendering : http://weait.com/content/build-your-own-openstreetmap-server Now what I would sincerely like is a quick and easy solution providing me an openstreetmap server distribution/image that : - ships on liveCD / CD / DVD or not but preferably for me as an AMI (Amazon Machine Image) - has : - an OSM psgql db setup with [SomeGeoData] - a mapnik or other renderer - an HTTP Rails port so as to have something like osm.org's slippymap+potlatch+HTTP API (at least TMS tiling API for tiles streaming and v0.6 features filtering API for POIs search and display) for mobiles applicationsOSM editors (JOSM, Merkaartor etc..) ...where [SomeGeoData] is a flexible set (imported/synchronized frequently from other sources, or stored on machine and editable by remote over HTTP with a JOSM-like editor) of : - layers (containing features that will be drawn but not-clickable), - POIs (clickable features), - interchangeable GeoStyleSheets or how map should be drawn-rules ... and that [SomeGeoData]'s flexible set of layers, POIs, stylesheets is remotely editable through a web interface, an HTTP interface or some shell scripts to run on the host. As far as LiveCDs / distributions are concerned, though I've seen only a few (one is Fedora based : https://fedoraproject.org/wiki/User:Ynemoy/Fedora_Geo_Spin ). I've experimented with tiledrawer.com for nearly a week on Amazon EC2. That server contains : - an openstreetmap database (postgresql + postGIS + the primary interfacing tools osm2pgsql / osmosis) - the TileCache python server which serves png tiles over http the TMS way* * this is openstreetmap.org's way of providing tiles for the Slippy Map one just needs to download each tile http://publicDNS/tilecache/x/y/z.png; see also http://en.wikipedia.org/wiki/Tile_Map_Service I need the Rails port on top of that. I can give free access to an amazon EC2 instance based on an Alestic Ubuntu EC2 image... based on TileDrawer.com's image or not. Who can comment on my need/request for help/proposal for collaboration ? Thanks in advance ! Take care ! Jonathan-David SCHRODER ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
2010/1/8 Jonathan-David SCHRODER jonathan.schro...@gmail.com Hello, Please tell me if I'm replicating any initiative or doing something the wrong or a not-so-good way. I've started this week from an EBS backed (or boot) Ubuntu 8.04 image ami-f6ad409f. I agree to give the .pem file for ssh connection to an instance to anyone willing to collaborate. My idea is to provide an EC2 EBS image with rails port(done) + tile rendering setup(started) + rails port-tile rendering sync scripts(todo) + easy config scripts (why not triggerable on startup with JSON config ala tilecache.org) (todo, not sure I'll have time to it). The list of commands to install is very long. I note down all the commands I type so as to be able to one or more bash scripts to be able to reinstall a similar solution on an Ubuntu N.NN later (by assuming that commands/package names for 8.04 are similar to those for 9.04 etc..). Maybe I'll have 3 bash scripts such as : install_railsport_u804.sh, install_mapnik_u804.sh, install_railsport_mapnik_sync.sh. And I'd need other scripts to go along : setup_rails_port_u804.sh, setup_mapnik_u804.sh and setup_rails_mapnik_sync_u804.sh... What do you think of this ? A recall on the requirements (reply to change them if you'd like to collaborate or request something): - postgres+osmosis+rails port (with 3 dbs : development, testing and production) : filled with nothing for now. setup_rails_port_u804.sh would have a simple calling syntax like : sudo setup_rails_port_u804.sh [--bz2url= http://domain/someRegion.osm.bz2|--bz2file=/path/to/file.osm.bz2http://domain/someRegion.osm.bz2%7C--bz2file=/path/to/file.osm.bz2] [--forceResetDB (false if omitted)] [--dbName=development(default),other,dbs,to,make,identical,too] [--guessDBLoginPassFromRailsConfig (default is true)] - mapnik tile rendering setup. I want to have a simple script to insert a bz2 + tell to use some style sheets + to tell to render at z=N (max zoom) - cron'able synced file making the Mapnik setup be filled with the Rails ports's data. I plan to get inspiration from the myanmar scripts for this. I'm sorry to repeat this, though, please reply if you know of some linux image (on EC2 or not) hanging somewhere which almost does what I'm trying to achieve and that image is easy to redeploy (ex : .iso or vmware/virtual bos EC2 ami). Well, one quick point: database performance are going to be bad. Someone is currently trying to do something similar in one of the mailing list. Cloud vendor are very good for serving content like a web page. They are extremely poor for database performance. The IO is usually catastrophic. I wouldn't expect any real good performance out of such a system especially for rendering. I think that using cloud technology is for the moment at least a bit pointless. Emilie Laffray ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
I would like to have an account. We could try out some ideas for replacing the database with a lot of small files. mike On Fri, Jan 8, 2010 at 1:42 PM, Emilie Laffray emilie.laff...@gmail.com wrote: 2010/1/8 Jonathan-David SCHRODER jonathan.schro...@gmail.com Hello, Please tell me if I'm replicating any initiative or doing something the wrong or a not-so-good way. I've started this week from an EBS backed (or boot) Ubuntu 8.04 image ami-f6ad409f. I agree to give the .pem file for ssh connection to an instance to anyone willing to collaborate. My idea is to provide an EC2 EBS image with rails port(done) + tile rendering setup(started) + rails port-tile rendering sync scripts(todo) + easy config scripts (why not triggerable on startup with JSON config ala tilecache.org) (todo, not sure I'll have time to it). The list of commands to install is very long. I note down all the commands I type so as to be able to one or more bash scripts to be able to reinstall a similar solution on an Ubuntu N.NN later (by assuming that commands/package names for 8.04 are similar to those for 9.04 etc..). Maybe I'll have 3 bash scripts such as : install_railsport_u804.sh, install_mapnik_u804.sh, install_railsport_mapnik_sync.sh. And I'd need other scripts to go along : setup_rails_port_u804.sh, setup_mapnik_u804.sh and setup_rails_mapnik_sync_u804.sh... What do you think of this ? A recall on the requirements (reply to change them if you'd like to collaborate or request something): - postgres+osmosis+rails port (with 3 dbs : development, testing and production) : filled with nothing for now. setup_rails_port_u804.sh would have a simple calling syntax like : sudo setup_rails_port_u804.sh [--bz2url=http://domain/someRegion.osm.bz2|--bz2file=/path/to/file.osm.bz2] [--forceResetDB (false if omitted)] [--dbName=development(default),other,dbs,to,make,identical,too] [--guessDBLoginPassFromRailsConfig (default is true)] - mapnik tile rendering setup. I want to have a simple script to insert a bz2 + tell to use some style sheets + to tell to render at z=N (max zoom) - cron'able synced file making the Mapnik setup be filled with the Rails ports's data. I plan to get inspiration from the myanmar scripts for this. I'm sorry to repeat this, though, please reply if you know of some linux image (on EC2 or not) hanging somewhere which almost does what I'm trying to achieve and that image is easy to redeploy (ex : .iso or vmware/virtual bos EC2 ami). Well, one quick point: database performance are going to be bad. Someone is currently trying to do something similar in one of the mailing list. Cloud vendor are very good for serving content like a web page. They are extremely poor for database performance. The IO is usually catastrophic. I wouldn't expect any real good performance out of such a system especially for rendering. I think that using cloud technology is for the moment at least a bit pointless. Emilie Laffray ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
I have been working on c++ code with rtrees and other nice things. depending on what you want to do with it, I think it is fair to make updates very slow if the rendering is very fast... mike On Fri, Jan 8, 2010 at 2:39 PM, Stefan de Konink ste...@konink.de wrote: Op 08-01-10 13:49, jamesmikedup...@googlemail.com schreef: I would like to have an account. We could try out some ideas for replacing the database with a lot of small files. We could just switch to MonetDB get deduplication, and don't require a lot of space ;) ...in other news did anyone try SphinxSearch on OSM data? Since I dislike PostgreSQL's geo-non-indiced functionality. Hence: an index on IsValid(...) is not materialised, I wonder if anyone succesfully applied stuff like: select planet_osm_point.osm_id, planet_osm_polygon.name, planet_osm_point.name from planet_osm_point, planet_osm_polygon where ST_Intersects(planet_osm_point.way, planet_osm_polygon.way) and planet_osm_polygon.name '' and planet_osm_point.name '' and planet_osm_polygon.name planet_osm_point.name; sql_query = select planet_osm_roads.osm_id, planet_osm_polygon.name, planet_osm_roads.name from planet_osm_roads, planet_osm_polygon where st_isvalid(planet_osm_roads.way) AND st_isvalid(planet_osm_polygon.way) AND ST_Intersects(planet_osm_roads.way, planet_osm_polygon.way) and planet_osm_polygon.name '' and planet_osm_roads.name '' and planet_osm_polygon.name planet_osm_roads.name; sql_query = select p2.osm_id, p1.name, p2.name from planet_osm_polygon as p2, planet_osm_polygon as p1 where ST_Intersects(p1.way, p2.way) and p1.name '' and p2.name '' and p1.name p2.name; Stefan ps. TomH thanks for the comment on IRC :D ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Now both to list. Op 08-01-10 14:53, jamesmikedup...@googlemail.com schreef: On Fri, Jan 8, 2010 at 2:50 PM, Stefan de Koninkste...@konink.de wrote: Op 08-01-10 14:46, jamesmikedup...@googlemail.com schreef: I have been working on c++ code with rtrees and other nice things. depending on what you want to do with it, I think it is fair to make updates very slow if the rendering is very fast... Do you mean small files as render tiles? something like that. I wonder if we were to store all the osm data for one tile and its index on a file system and just keep them all in synch. Anyone who edits the data would update all of them as needed. You would have to take in to account what data is required for rendering. This is basically what the column wise PostgreSQL Schema tries to achieve. I would find it difficult to believe that with for example a memorymapped file, that allocates for example 4k for a node would outperform a database method. I would see a better future in automatic analysing a stylesheet, determing enums, and using databases. This is already what we do on tile.openstreetmap.nl; I have modified renderd to render 'metatiles', but not to save them as metatiles. So on disk I have all seperate tiles. My webserver, based if the file exists, serves the file, or otherwise shoots in a render request like mod_tile would do. define metatile? Metatile is basically a tile that spawns a (much) greater space than the invidivual tile, for example 8x8. This reduces mapnik-overhead, since you always have to render a bit bigger than the tile anyway. An additional assumption has been made that it would be faster to store and server metatiles by extracting data, than invidivual tiles. Currently there is no benchmark that can confirm or deny that. So the main difference is that I never extract tiles, and never compress tiles. We still should plan some sort of benchmark to see if this method is better or worse then meta tiles. I think its better because the webserver is on less load, and the disk speed or filesystem should be tuned anyway. It will depend on your architecture, but I think on a distributed system where you have many processors, that lots of files , if connected to the processors will work well. Ideally you would have a set of processors to deal with a set of data and any requests would come to them and the whole thing would be in cache all the time... That would assume a big disk cache. For NL, it would be theoretically possible to use multiple renderd instances out of the box. But that would require to run renderd on an TCP/IP socket, instead of a Unixsocket. In my perspective the query time can be reduced anyway significantly if we want it. Mapnik is much harder to optimise, unless you want to go GPU accelerated rendering. Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Op 08-01-10 15:09, jamesmikedup...@googlemail.com schreef: I dont know all the details of mapnik, but from what I have seen, using the postgres database is not needed in all cases. I am thinking about It is not required to use the database, but the overhead is in the rendering. I do agree that PostgreSQL will not always give you the fastest results ;) Well you mean a 4k page in memory. But you dont need to have all that data memory mapped. It could be just 4k of data on disk in an array. It could also just be a tiny osm file that is parsed when needed. Tiny would have to be an osmtile in binary format, agreed? Well again, I have not really gotten into mapnik. But I can. Please do so :) But lets try and define the problem as rendering an osm file to a tile. That osm file is updated , and rerendered. All the data needed to render is just stored in osmxml in a nice sorted way. You never need all the data at once because you only render a tile at a time. You don't want to store it in XML, it will get huge... and requires again the parsing overhead. And as I pointed out before, we prefer to render 64 tiles at a time. but you dont need a full postgres database functionality, you just have very basic update of pages of data. In your idea, how would you *update* the storage? Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
On Fri, Jan 8, 2010 at 3:23 PM, Stefan de Konink ste...@konink.de wrote: Op 08-01-10 15:09, jamesmikedup...@googlemail.com schreef: I dont know all the details of mapnik, but from what I have seen, using the postgres database is not needed in all cases. I am thinking about It is not required to use the database, but the overhead is in the rendering. I do agree that PostgreSQL will not always give you the fastest results ;) Well you mean a 4k page in memory. But you dont need to have all that data memory mapped. It could be just 4k of data on disk in an array. It could also just be a tiny osm file that is parsed when needed. Tiny would have to be an osmtile in binary format, agreed? Yes, well if you sort the data properly then mapnik would just render the data in a sax callback, right? It would be a single pass over the data to render it, maybe with a second pass to do some touch ups. But given a street you could just draw the street. It has a vector of nodes. imagine the following sequence of events in a sax xml callback : begin tile : process nodes for next street (marking intersections) Process all the intersecting streets (there we need to preprocess the data) process street (process intersections as they go) next street. So you need to look for street crossings, and need to mark the points where they cross as such. I can imagine that might need some work. I am just working this out right now, but I can imagine that if you sorted the streets and knew all the intersections and marked each intersection, storing a ref to it like in nd ref and also had those segments available, then you could just draw the street. Well again, I have not really gotten into mapnik. But I can. Please do so :) Ok, well I will have to.! But lets try and define the problem as rendering an osm file to a tile. That osm file is updated , and rerendered. All the data needed to render is just stored in osmxml in a nice sorted way. You never need all the data at once because you only render a tile at a time. You don't want to store it in XML, it will get huge... and requires again the parsing overhead. And as I pointed out before, we prefer to render 64 tiles at a time. So, you would have 64 tiles of data in one page How big is that? the size of a city. Well I have gotten the parsing down to very small using sax2, it is just a bit slower than reading the file. If you sort the xml data topologically and pack it all in, it could be very easy to process. but you dont need a full postgres database functionality, you just have very basic update of pages of data. In your idea, how would you *update* the storage? My vision of a distributed GIT repository would have the user update the storage when they save it... Now if you have a web user that commits the data or uploads it, then the page of data would be updated by that process the rendering would be triggered and the whole thing checked in. Each tile would be checked in and older versions discarded as needed. Now, if you just have a nice xml file that is compact or an ASN.1 file for higher speed binary or something else, you need to update that. but how often are we talking about update here? Most people are using daily diff files from the main database. those would be split up into the tiles, and the xml would be replaced by a new page, sorting would have to take place. I need to work this all out, but it is possible. I will have to work through the entire pipeline of processing to see if it fits my model, I risk having to put my foot in my mouth here mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Op 08-01-10 15:39, jamesmikedup...@googlemail.com schreef: Tiny would have to be an osmtile in binary format, agreed? Yes, well if you sort the data properly then mapnik would just render the data in a sax callback, right? Jup, but you will never get better results of that storing in parts than having it stored in one file and having an index to it. So in that respect you have to mmap the file, for example a post/plane parser, extract the childs, send the childs to Mapnik. I have tested this using MonetDB/XML, that wasn't really a succes ;) Either way you need something that stores that creates an index by a bounding box. You can use rtree's for that sure. But we already have shown that sideway cracking gives far better results. [google: sideway cracking databases] You don't want to store it in XML, it will get huge... and requires again the parsing overhead. And as I pointed out before, we prefer to render 64 tiles at a time. So, you would have 64 tiles of data in one page How big is that? the size of a city. All depending on the zoom level :) Well I have gotten the parsing down to very small using sax2, it is just a bit slower than reading the file. If you sort the xml data topologically and pack it all in, it could be very easy to process. But how are you going to store this, for example if we take a *perfect* filesystem. Will you store everything by nodeid, wayid, relationid? OR will you store everything that is in a tile, in a file marking the tile, having the same structure as the tile server uses? I will have to work through the entire pipeline of processing to see if it fits my model, I risk having to put my foot in my mouth here Sure :) Just start with some smaller country and try to prevent bottlenecks. The worse thing that can happen is TomH saying you are a 'Bonker' (whatever that may mean). Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Ok, well I am going to first work on making the mapnik be able to render an osm file standalone without postgis, will report back when that works. mike On Fri, Jan 8, 2010 at 3:50 PM, Stefan de Konink ste...@konink.de wrote: Op 08-01-10 15:39, jamesmikedup...@googlemail.com schreef: Tiny would have to be an osmtile in binary format, agreed? Yes, well if you sort the data properly then mapnik would just render the data in a sax callback, right? Jup, but you will never get better results of that storing in parts than having it stored in one file and having an index to it. So in that respect you have to mmap the file, for example a post/plane parser, extract the childs, send the childs to Mapnik. I have tested this using MonetDB/XML, that wasn't really a succes ;) Either way you need something that stores that creates an index by a bounding box. You can use rtree's for that sure. But we already have shown that sideway cracking gives far better results. [google: sideway cracking databases] You don't want to store it in XML, it will get huge... and requires again the parsing overhead. And as I pointed out before, we prefer to render 64 tiles at a time. So, you would have 64 tiles of data in one page How big is that? the size of a city. All depending on the zoom level :) Well I have gotten the parsing down to very small using sax2, it is just a bit slower than reading the file. If you sort the xml data topologically and pack it all in, it could be very easy to process. But how are you going to store this, for example if we take a *perfect* filesystem. Will you store everything by nodeid, wayid, relationid? OR will you store everything that is in a tile, in a file marking the tile, having the same structure as the tile server uses? I will have to work through the entire pipeline of processing to see if it fits my model, I risk having to put my foot in my mouth here Sure :) Just start with some smaller country and try to prevent bottlenecks. The worse thing that can happen is TomH saying you are a 'Bonker' (whatever that may mean). Stefan ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
jamesmikedup...@googlemail.com wrote: Ok, well I am going to first work on making the mapnik be able to render an osm file standalone without postgis, will report back when that works. *cough* http://trac.mapnik.org/browser/trunk/plugins/input/osm Will not work with the postgis-based stylesheet. -- Lennard ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
So this renderes directly from an osm file? On Fri, Jan 8, 2010 at 4:02 PM, Lennard l...@xs4all.nl wrote: jamesmikedup...@googlemail.com wrote: Ok, well I am going to first work on making the mapnik be able to render an osm file standalone without postgis, will report back when that works. *cough* http://trac.mapnik.org/browser/trunk/plugins/input/osm Will not work with the postgis-based stylesheet. I have not looked into all of this, as I said. From what I know about mapnick, it renders from the database. Compiling now. mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
http://wiki.openstreetmap.org/wiki/Mapnik#Data_Sources This webpage only talkes about posgis, is there any documentation that covers the osm direct rendering? mike On Fri, Jan 8, 2010 at 4:08 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: So this renderes directly from an osm file? On Fri, Jan 8, 2010 at 4:02 PM, Lennard l...@xs4all.nl wrote: jamesmikedup...@googlemail.com wrote: Ok, well I am going to first work on making the mapnik be able to render an osm file standalone without postgis, will report back when that works. *cough* http://trac.mapnik.org/browser/trunk/plugins/input/osm Will not work with the postgis-based stylesheet. I have not looked into all of this, as I said. From what I know about mapnick, it renders from the database. Compiling now. mike ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
On Jan 8, 2010, at 7:19 AM, jamesmikedup...@googlemail.com wrote: http://wiki.openstreetmap.org/wiki/Mapnik#Data_Sources This webpage only talkes about posgis, is there any documentation that covers the osm direct rendering? http://trac.mapnik.org/wiki/OsmPlugin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] amazon AMIs for a full blown openstreetmap.org-like server
Ok, this looks great. So, now, that would be my suggestion to use on this amazon webservice. A lot of small osm files, the size of the smallest tile. Now, on thing did occur to me while walking the dog, we could render multiple zoom levels at once. Every time you render the smallest unit (tile) you would render the sections of all the zoom levels and that should prevent you from touching the data at the same time. mike On Fri, Jan 8, 2010 at 6:08 PM, Dane Springmeyer bl...@hailmail.net wrote: On Jan 8, 2010, at 7:19 AM, jamesmikedup...@googlemail.com wrote: http://wiki.openstreetmap.org/wiki/Mapnik#Data_Sources This webpage only talkes about posgis, is there any documentation that covers the osm direct rendering? http://trac.mapnik.org/wiki/OsmPlugin ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
