Mapumental is a website which shows contour maps of public transport travel times, house prices and other data. It's in closed beta.
http://mapumental.channel4.com/ It uses lots of CPU running the transport route finding for each postcode, and rendering the tiles as they are served. Before we can openly release it, we need to make it scale easily (say, on Amazon Web Services). Currently it is using * A PostgreSQL database to store the points behind the static datasets such as scenicness and house prices. * Binary files on NFS to store the generated datasets of travel times. PostgreSQL was too slow, and used too much memory, to load in the large number of rows that would be required (300,000 for each user entered postcode). * A rendered tile cache, containing PNG files on the NFS filesystem. * PostgreSQL for queueing the jobs for the transport route finder. We now want to: * make the site scale easily (on Amazon Web Service), * make it easy to add more data sets. We had problems with NFS, so I need something to replace the binary files in NFS and the tile cache. It might also be prudent to use something easier to scale than a PostgreSQL database, although I suspect the load on it would be low so perhaps it isn't a problem. So the new version of Mapumental that I'm currently plannning has to store: a) cache of tiles rendered (some fairly generated rarely and frequently accessed e.g. house prices, some not accessed often compared to generation times, e.g. public transport route) b) coordinates and values of arbitary point datasets (e.g. school quality, asthma air quality, wind speed, route by car to a particular postcode etc. etc.) I'm looking for good, open source, alternatives to NFS and PostgreSQL to do this. Distributed data stores and queueing systems. What should I look at? What can I trust? I've already surveyed the field, and have my own ideas about what to do, but would be interested if anyone here has some experience or views on any of the obvious technologies. I'd like it to be stable and mature, and realistically it would already be in a Debian package. Francis _______________________________________________ Mailing list [email protected] Archive, settings, or unsubscribe: https://secure.mysociety.org/admin/lists/mailman/listinfo/developers-public
