[OSM-talk] OSM server problems?
I'm getting a lot of internal server errors reported by JOSM when I download (small) areas from the server. Is there a problem with the server? Regards, Maarten ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
I think we may be talking about different things. I thought the question was about using a virtual server rather than a dedicated one. It sounds like it is about using a virtual machine on a desktop. In that case you are right that running the database on the host will work better than on the guest operating system. Some people are very attached to Windows though! Yes, that's what I thought the OP meant by VM, a virtual server such as those offered by Bytemark. If it's a desktop machine I'd recommend installing Ubuntu natively as a dual boot (backing up your data on Windows of course in case of problems!) and doing all your OSM stuff on that. Nick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
I think we may be talking about different things. I thought the question was about using a virtual server rather than a dedicated one. It sounds like it is about using a virtual machine on a desktop. In that case you are right that running the database on the host will work better than on the guest operating system. Some people are very attached to Windows though! Graham from my phone On 21 Dec 2010 23:49, David Murn da...@incanberra.com.au wrote: On Tue, 2010-12-21 at 14:31 +, Nick Whitelegg wrote: Bear in mind though that many of us have... Not entirely sure what you mean by 'for financial reasons', but I agree in part. I do wonder though, if youre using a windows host simply for financial book-keeping and a ubuntu database server as a guest, youve got things the wrong way around. Make your host ubuntu and windows your VM. This gets around the performance problems of running a database in a VM, and you will infact notice an inprovement with windows running in guest mode instead of host mode, as it will benefit from the caching available in ubuntu. Especially if the only reason you need non-ubuntu software is for financials, then youre better off having that running part-time in a VM and having your main system operation running as host. Unless you want to simply render once, dont care how long it takes to setup or complete, and then delete the whole renderer. If you expect to be using the setup more than once, a virtual machine is not the best way to go.. its not even 2nd best. David While trying to load the whole planet or even the whole of a country like the UK might be di... ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
http://wiki.openstreetmap.org/wiki/Rails_port will help with installing the main application. There are, of course, more pages on the wiki which can help, some of which are linked to from that page. Tim On 21 December 2010 00:12, Arlindo Pereira openstreet...@arlindopereira.com wrote: Hi there, I'm planning to build with OpenStreetMap some historic, out-of-copyright maps (such as [1] and [2]). They would feature many things (such as streets, mountains and beaches) that do not exist anymore. AFAIK, this kind of data is not supposed to live on the main OSM server. So, I'd like to ask: what is the easiest way to build a OSM server? I'm thinking about running the database and mapnik on a virtual machine (preferably Ubuntu Server, since I already use Ubuntu on desktop), and upload only the tiles and OpenLayers to a regular Apache server. A copy-and-paste sequence of apt-gets would be perfect - I couldn't find it on the wiki. My idea is to create a page similar to [3], being the left side the historical map and on the right the current OSM map. Thanks a lot, Arlindo Nighto Pereira 1: http://www.bondesrio.com/fotos/mapas/mapa_linhas_1878.jpg 2: http://www.bondesrio.com/fotos/mapas/mapa_linhas_1907.jpg 3: http://tools.geofabrik.de/mc/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
In general databases and Virtual machines do not work well together. Databases tend to want fast disk accesses and the virtual machine bit slows these down. Also typically virtual machines restrict the memory and databases use the memory to reduce disk accesses so you get a second hit there. Virtual machines work better for lightly loaded cpu intensive work. Cheerio John On 21 December 2010 06:50, Tim Waters chippy2...@gmail.com wrote: http://wiki.openstreetmap.org/wiki/Rails_port will help with installing the main application. There are, of course, more pages on the wiki which can help, some of which are linked to from that page. Tim On 21 December 2010 00:12, Arlindo Pereira openstreet...@arlindopereira.com wrote: Hi there, I'm planning to build with OpenStreetMap some historic, out-of-copyright maps (such as [1] and [2]). They would feature many things (such as streets, mountains and beaches) that do not exist anymore. AFAIK, this kind of data is not supposed to live on the main OSM server. So, I'd like to ask: what is the easiest way to build a OSM server? I'm thinking about running the database and mapnik on a virtual machine (preferably Ubuntu Server, since I already use Ubuntu on desktop), and upload only the tiles and OpenLayers to a regular Apache server. A copy-and-paste sequence of apt-gets would be perfect - I couldn't find it on the wiki. My idea is to create a page similar to [3], being the left side the historical map and on the right the current OSM map. Thanks a lot, Arlindo Nighto Pereira 1: http://www.bondesrio.com/fotos/mapas/mapa_linhas_1878.jpg 2: http://www.bondesrio.com/fotos/mapas/mapa_linhas_1907.jpg 3: http://tools.geofabrik.de/mc/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
In general databases and Virtual machines do not work well together. Databases tend to want fast disk accesses and the virtual machine bit slows these down. Also typically virtual machines restrict the memory and databases use the memory to reduce disk accesses so you get a second hit there. Virtual machines work better for lightly loaded cpu intensive work. Bear in mind though that many of us have to make do with VMs, for financial reasons. While trying to load the whole planet or even the whole of a country like the UK might be difficult, if you try and cut down the data, then it should (IMX) work reasonably well. For example, don't try to do the whole world or even the whole of your country, use a subset of data such as your region, or, if possible try and design your app to minimise the amount of data it needs (e.g. use osmosis to cut out irrelevant data). Nick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
Yes, the idea is to create a VM with the database and the render, then upload the files to regular Apache server I already got on a shared host. Therefore, it'd be no problem if it becomes slow, after the rendering this wouldn't make difference, since the VM wouldn't be exposed as a server to the internet. The wiki page mentions the import of planet.osm. Since I'm going to work with other data, can I just create a sketch on JOSM them import it or should I use anything else like the planet boundary? I really need only Rio de Janeiro's downtown bbox, so anything else would be a waste of database space/rendering time. Thanks a lot! Arlindo Nighto Pereira 2010/12/21 Graham Jones grahamjones...@gmail.com I agree with Nick - for small map rendering applications a virtual machine has worked ok for me. I got the impression from the original message that this was a relatively small application so should be ok? The main limitation with a virtual machine has been available storage space - a few extra GB of virtual disk makes the thing get a lot more expensive. That is why at the moment I am using my old laptop to do the database processing, but because it is connected to the internet via my domestic broadband connection it seems quite sluggish from the outside world. My intention is to separate out the web front end from the rendering and use a virtual machine for the web interface, and do rendering on the more powerful machine at home, but this does not work yet! Graham. On 21 December 2010 14:31, Nick Whitelegg nick.whitel...@solent.ac.ukwrote: In general databases and Virtual machines do not work well together. Databases tend to want fast disk accesses and the virtual machine bit slows these down. Also typically virtual machines restrict the memory and databases use the memory to reduce disk accesses so you get a second hit there. Virtual machines work better for lightly loaded cpu intensive work. Bear in mind though that many of us have to make do with VMs, for financial reasons. While trying to load the whole planet or even the whole of a country like the UK might be difficult, if you try and cut down the data, then it should (IMX) work reasonably well. For example, don't try to do the whole world or even the whole of your country, use a subset of data such as your region, or, if possible try and design your app to minimise the amount of data it needs (e.g. use osmosis to cut out irrelevant data). Nick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk -- Graham Jones Hartlepool, UK. ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
The wiki page mentions the import of planet.osm. Since I'm going to work with other data, can I just create a sketch on JOSM them import it or should I use anything else like the planet boundary? I really need only Rio de Janeiro's downtown bbox, so anything else would be a waste of database space/rendering time. In that case, if I've understood you right, you probably want to do something like this: - download a planet extract from your part of Brazil, see http://wiki.openstreetmap.org/index.php/Planet; - extract Rio using Osmosis (http://wiki.openstreetmap.org/index.php/Osmosis); - set up the PostGIS database and Mapnik (http://wiki.openstreetmap.org/index.php/Mapnik); - import OSM into the PostGIS database with osm2pgsql (http://wiki.openstreetmap.org/index.php/Osm2pgsql); - add your OOC data to the PostGIS database separately; - tweak the mapnik XML file to render your OOC data. Nick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
On Tue, 2010-12-21 at 14:31 +, Nick Whitelegg wrote: Bear in mind though that many of us have to make do with VMs, for financial reasons. Not entirely sure what you mean by 'for financial reasons', but I agree in part. I do wonder though, if youre using a windows host simply for financial book-keeping and a ubuntu database server as a guest, youve got things the wrong way around. Make your host ubuntu and windows your VM. This gets around the performance problems of running a database in a VM, and you will infact notice an inprovement with windows running in guest mode instead of host mode, as it will benefit from the caching available in ubuntu. Especially if the only reason you need non-ubuntu software is for financials, then youre better off having that running part-time in a VM and having your main system operation running as host. Unless you want to simply render once, dont care how long it takes to setup or complete, and then delete the whole renderer. If you expect to be using the setup more than once, a virtual machine is not the best way to go.. its not even 2nd best. David While trying to load the whole planet or even the whole of a country like the UK might be difficult, if you try and cut down the data, then it should (IMX) work reasonably well. For example, don't try to do the whole world or even the whole of your country, use a subset of data such as your region, or, if possible try and design your app to minimise the amount of data it needs (e.g. use osmosis to cut out irrelevant data). Nick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server on a (Ubuntu) VM?
On small databases where performance isn't that critical they will work under virtual machines but larger databases are different. I was responsible for running large central database servers at Statistics Canada and we had a server support area who thought Virtual Machines were the only way to go. Memory is measured in nanoseconds, disk speeds in milliseconds. 10,000 nano seconds equals one millisecond. Basically we got our performance by switching to 64 bit operating systems and dumping in lots of memory to primarily avoid dis accesses. I think we consolidated some 20 database servers on the one machine with a single cpu with multiple cores. By contrast those databases that were set up by the server group were often limited to 2 gigs of memory by the Virtual Machine environment and the virtual cpus were in fact a maximum of a single core. This has major licensing implications if you are running commercial database software. If you are running Oracle for example, unless things have changed, you need to license every cpu on the physical server not just the virtual machine. They also had performance issues in part because the virtual machine emulated a particular older server. Fine except our servers had later hardware on the disk side which the database and operating system could take advantage of. Under emulation they couldn't take the same advantage of the hardware. Cheerio John On 21 December 2010 18:47, David Murn da...@incanberra.com.au wrote: On Tue, 2010-12-21 at 14:31 +, Nick Whitelegg wrote: Bear in mind though that many of us have to make do with VMs, for financial reasons. Not entirely sure what you mean by 'for financial reasons', but I agree in part. I do wonder though, if youre using a windows host simply for financial book-keeping and a ubuntu database server as a guest, youve got things the wrong way around. Make your host ubuntu and windows your VM. This gets around the performance problems of running a database in a VM, and you will infact notice an inprovement with windows running in guest mode instead of host mode, as it will benefit from the caching available in ubuntu. Especially if the only reason you need non-ubuntu software is for financials, then youre better off having that running part-time in a VM and having your main system operation running as host. Unless you want to simply render once, dont care how long it takes to setup or complete, and then delete the whole renderer. If you expect to be using the setup more than once, a virtual machine is not the best way to go.. its not even 2nd best. David While trying to load the whole planet or even the whole of a country like the UK might be difficult, if you try and cut down the data, then it should (IMX) work reasonably well. For example, don't try to do the whole world or even the whole of your country, use a subset of data such as your region, or, if possible try and design your app to minimise the amount of data it needs (e.g. use osmosis to cut out irrelevant data). Nick ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] OSM server on a (Ubuntu) VM?
Hi there, I'm planning to build with OpenStreetMap some historic, out-of-copyright maps (such as [1] and [2]). They would feature many things (such as streets, mountains and beaches) that do not exist anymore. AFAIK, this kind of data is not supposed to live on the main OSM server. So, I'd like to ask: what is the easiest way to build a OSM server? I'm thinking about running the database and mapnik on a virtual machine (preferably Ubuntu Server, since I already use Ubuntu on desktop), and upload only the tiles and OpenLayers to a regular Apache server. A copy-and-paste sequence of apt-gets would be perfect - I couldn't find it on the wiki. My idea is to create a page similar to [3], being the left side the historical map and on the right the current OSM map. Thanks a lot, Arlindo Nighto Pereira 1: http://www.bondesrio.com/fotos/mapas/mapa_linhas_1878.jpg 2: http://www.bondesrio.com/fotos/mapas/mapa_linhas_1907.jpg 3: http://tools.geofabrik.de/mc/ ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [Talk-de] Fwd: [OSM-talk] OSM Server-Wartung Anfang Juli
Die Wartungsarbeiten scheinen nun erfolgreich beendet zu sein und OpenStreetMap sollte wieder voll funktionsfaehig sein Happy mapping, Kai -- View this message in context: http://gis.638310.n2.nabble.com/Fwd-OSM-talk-OSM-Server-Wartung-Anfang-Juli-tp5224085p5250761.html Sent from the Germany mailing list archive at Nabble.com. ___ Talk-de mailing list Talk-de@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-de
[Talk-de] Fwd: [OSM-talk] OSM Server-Wartung Anfang Juli
OpenStreetMap, Ab dem Abend des 1.7.2010, hauptsächlich aber am 2. und 3. Juli kommt es zu Wartungsarbeiten an dem Storagesytem des Datenbankservers. Das Editieren der Karte wird dann nicht möglich sein. Der Vorläufige Plan sieht folgenden Ablauf vor: 1.7.2010, 23:00 Uhr CEST: Die API und die Webseite sind nur noch lesend erreichbar (Kartendaten können zwar heruntergeladen, aber Edits nicht wieder hochgeladen werden) 2.7.2010, 14:00 Uhr CEST: API Abgeschaltet, Nur das Anschauen der Map und Exportfunktionen verfügbar 3.7.2010, (später) Nachmittag: API und Webseite stehen wieder voll zur Verfügung. Das Wiki ist von den Wartungsarbeiten nicht betroffen, und wird voll zur Verfügung stehen. Weitere Informationen und Fortschrittsmeldungen werden (auf Englisch) im Wiki veröffentlicht, unter folgender Adresse: http://wiki.openstreetmap.org/wiki/July_2010_DB_Upgrade Original-Nachricht Betreff: [OSM-talk] Upcoming OSM Server Maintenance Datum: Sat, 26 Jun 2010 00:16:03 +0100 Von: Grant Slater openstreet...@firefishy.com An: Talk Openstreetmap t...@openstreetmap.org, annou...@openstreetmap.org OpenStreetMap, Please translate and copy to local lists. On the 2nd (Friday) and 3rd (Saturday) of July 2010 the sysadmin team will be upgrading the database server's storage. Editing will not be available during this period. Initial plan, subject to revision: 1st July 21:00 (GMT) - API and website read-only. (no edits) 2nd July 12:00 (GMT) - API unavailable. Only Map browsing and map export available on www.openstreetmap.org 3rd July Afternoon (GMT) - API and website return. (edit resumes) The wiki will remain available for the duration. Futher information and progress status will be posted here: http://wiki.openstreetmap.org/wiki/July_2010_DB_Upgrade Regards Grant OpenStreetMap Sysadmin Team ___ talk mailing list t...@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk signature.asc Description: OpenPGP digital signature ___ Talk-de mailing list Talk-de@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk-de
[OSM-talk] OSM server...
Dear, I am looking at the best implementation of OSM local server for a public GIS organisation anywhere in the world dealing with street data update everyday? I am from Canada and I have heard about the Canvec project. However, I am looking at success stories which OSM server ( http://weait.com/content/build-your-own-openstreetmap-server) and code ( http://wiki.openstreetmap.org/wiki/The_Rails_Port) have been locally implemented and used for internal web applications. In my own public organisation, we do have some legal issues with different GIS datasets (e.g. street, aerial, thematic maps) from third parties, but we would like to build an internal web mapping application using the OSM philosophy and base maps, combining OSM data and our third-party datasets (that is not GPL licence) without breaking the legal agreement we have. After building a viable OSM internal server and once it for all dealt with our legal issues, we could be able to transfer some street datasets into OSM external site (www.openstreetmap.ca). We do have very good datasets in PostGIS, we are using UMN MapServer, OpenLayers (to combine OSM and other datasets), tilecache.org, web service and MapFish already for our internal apps, but we would like to avoid developing a complex WFS-T server and applications to monitor, validate and update our street datasets, that is why OSM local server option might be a good idea (example: using Merkatoor or other web OSM editing project for low-level editing users in our organisation). Thanks for letting me know good stories of this kind, options available or critical information I might need before building a OSM local server project involving other none-OSM datasets. Cheers, Nicolas ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
[OSM-talk] OSM Server Side Script
Hello, the project I've been working on the last few month now into some kind of beta status. So I you would like a reverse gazetter or a download an area of the size of a city, have a look at http://78.46.81.38 In particular, this might be relevant to the topic Advanced multipolygons - do we need area types? How well are they supported? of the London Hack Weekend http://wiki.openstreetmap.org/wiki/London_Hack_Weekend The idea behind the story is to have a server where one can obtain derived data as a web service. Areas are a standard example of that kind of things: the borders are as ways of use on their own, but they also define the area. In the OSM database, you find only the borders represented as ways and a relation declaring which borders constitute a certain area. So every application must figure out the areas on its own and has to rewrite the code and spent possibly substantial computation time (think of calculating a nation's borders on a mobile phone) on that. That's where the OSM Server Side Script server comes into the game: the derived data gets accessible to any application just with a single query, and the mappers still only need to edit and declare the independent data. And even the rules can be edited by the user as explained in documentation: http://78.46.81.38/#section.rule_example So in the long term, we may also do things like preparing the data for routing, deriving residental areas as desired here http://lists.openstreetmap.org/pipermail/dev/2009-February/014175.html or apply the machine readable version of the wiki as proposed here http://wiki.openstreetmap.org/wiki/Machine-readable_Map_Feature_list to detect conflicting objects in the database. There's a lot of work to do left. So I would like to get some feedback what to do first. And maybe there's even somebody who would like to join the project :) Some issues I see so far features: * A spatially intrinsic query for ways: At the moment, you only can query for nodes and then get the back references to get the data for an area. However, this would not include ways that cross an area without having a node inside of it. So this enhancement of the area-query would make it possible to include also those ways. * Mixed queries with spatial and tag-based criteria: an example would be to find all motorways in Germany. * Restriction of the output: If the size of the data is relevant (think of a mobile phone as a client), the server could omit certain useless tags (like the frequent created_by tag to reduce file size or processing complexity. * Or other things that come into your mind ... basics: * Proceed with the documentation: at the moment, the documentation is reduced to the essential things. And I don't even know whether the documentation is helpful or not. * Make the source code of the server accessible: The code is a bunch of C++ source files along with some bash scripts. It is quite a mess at the moment. And I'm even not sure whether I should place it in the OSM SVN or not. I would be grateful for every kind of feedback. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM Server Side Script
good work, roland. seems like a lot of useful work to me. thanks gerhard gary68 On Thu, 2009-05-28 at 17:47 +0200, Roland Olbricht wrote: Hello, the project I've been working on the last few month now into some kind of beta status. So I you would like a reverse gazetter or a download an area of the size of a city, have a look at http://78.46.81.38 In particular, this might be relevant to the topic Advanced multipolygons - do we need area types? How well are they supported? of the London Hack Weekend http://wiki.openstreetmap.org/wiki/London_Hack_Weekend The idea behind the story is to have a server where one can obtain derived data as a web service. Areas are a standard example of that kind of things: the borders are as ways of use on their own, but they also define the area. In the OSM database, you find only the borders represented as ways and a relation declaring which borders constitute a certain area. So every application must figure out the areas on its own and has to rewrite the code and spent possibly substantial computation time (think of calculating a nation's borders on a mobile phone) on that. That's where the OSM Server Side Script server comes into the game: the derived data gets accessible to any application just with a single query, and the mappers still only need to edit and declare the independent data. And even the rules can be edited by the user as explained in documentation: http://78.46.81.38/#section.rule_example So in the long term, we may also do things like preparing the data for routing, deriving residental areas as desired here http://lists.openstreetmap.org/pipermail/dev/2009-February/014175.html or apply the machine readable version of the wiki as proposed here http://wiki.openstreetmap.org/wiki/Machine-readable_Map_Feature_list to detect conflicting objects in the database. There's a lot of work to do left. So I would like to get some feedback what to do first. And maybe there's even somebody who would like to join the project :) Some issues I see so far features: * A spatially intrinsic query for ways: At the moment, you only can query for nodes and then get the back references to get the data for an area. However, this would not include ways that cross an area without having a node inside of it. So this enhancement of the area-query would make it possible to include also those ways. * Mixed queries with spatial and tag-based criteria: an example would be to find all motorways in Germany. * Restriction of the output: If the size of the data is relevant (think of a mobile phone as a client), the server could omit certain useless tags (like the frequent created_by tag to reduce file size or processing complexity. * Or other things that come into your mind ... basics: * Proceed with the documentation: at the moment, the documentation is reduced to the essential things. And I don't even know whether the documentation is helpful or not. * Make the source code of the server accessible: The code is a bunch of C++ source files along with some bash scripts. It is quite a mess at the moment. And I'm even not sure whether I should place it in the OSM SVN or not. I would be grateful for every kind of feedback. Cheers, Roland ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
Re: [OSM-talk] OSM server down
db went down for some reason, restarted On 21 Mar 2008, at 07:57, Mike Collinson wrote: Both http://www.openstreetmap.org/ and attempted JOSM download give 500 Internal Server Error ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk ___ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk
