Re: [OSM-dev] map disappear on YourS local installation
On Thu, Dec 22, 2011 at 5:36 PM, Anwar Azulfa an...@troyans.net wrote:
i have checkout from
http://svn.openstreetmap.org/applications/routing/yours/branches/version-1.0-via
now, map is appear. but when i try to make routing, i get error again
this is step which i did :
1. i following reference from http://wiki.openstreetmap.org/wiki/YOURS and
checkout from http://svn.openstreetmap.org/applicatio …
on-1.0-viahttp://svn.openstreetmap.org/applications/routing/yours/branches/version-1.0-via
2. i checkout gosmore source from http://svn.openstreetmap.org/applicatio
… ng/gosmore http://svn.openstreetmap.org/applications/rendering/gosmore
3.download file .pak and rename to gosmore.pak.i put this file inside
gosmore folder
4. i put YOURS/* to /var/www so i can acces it via browser with
http://localhost
now i can access map, but when i make a route from one location to another
location and click find route, error message :Status: An unexpected error
occured in Gosmore: appear
i check on /var/log/apache2/error.log and i get this error message :
*[Thu Dec 22 22:29:48 2011] [error] [client 127.0.0.1] PHP Notice:
Undefined index: format in /home/maple/yours/api/1.0/gosmore.php on line
105, referer: http://localhost/*
*nice: ./gosmore: No such file or directory*
*
*
What should i do to solve this ?
Inside the gosmore folder, you should run
./configure
make
Currently, gosmore will not compile with HEADLESS. I suggest you compile
with gtk+2 includes and libraries installed. Or you can check out a version
what is more than 3 months old.
It is also a good idea to test routing from the command line before you
test YOURS e.g.
QUERY_STRING=flat=52.396545flon=4.915714tlat=52.373075tlon=4.901638fast=0v=motorcar
./gosmore
Regards,
Nic
Help me, please
2011/12/21 Anwar Azulfa an...@troyans.net
I try install yours on my computer, i have followed reference from
http://wiki.openstreetmap.org/wiki/YOURS
But when i access it via browser, i can't found any map.
then i get idea to pointing js to my map on local computer because i have
own local tiles .
I try to change init function on yournavigation.js with :
function init() {
var options = {
projection: new OpenLayers.Projection(EPSG:900913),
displayProjection: new OpenLayers.Projection(EPSG:4326),
units: m,
maxResolution: 156543.0339,
maxExtent: new OpenLayers.Bounds(-20037508.34,
-20037508.34,
20037508.34,
20037508.34),
numZoomLevels: 16,
controls: [
new OpenLayers.Control.Navigation(),
new OpenLayers.Control.PanZoomBar(),
new OpenLayers.Control.Permalink(),
new OpenLayers.Control.ScaleLine(),
new OpenLayers.Control.MousePosition(),
new OpenLayers.Control.KeyboardDefaults()
]
};
myFirstMap = new OpenLayers.Map(map,options);
var newL = new OpenLayers.Layer.OSM(Default,
http://localhost/osm_tiles3/${z}/${x}/${y}.png;, {numZoomLevels: 19});
map.addLayer(newL);
if (!map.getCenter()) {
var center = new OpenLayers.LonLat(116.97,-2.56);
center.transform(options.displayProjection,
options.projection);
map.setCenter(center, 5);
}
}
But i stil l can't find map on it ?
What should i do to solve this ?
--
Regards,
M.Iftakhul Anwar
--
Regards,
M.Iftakhul Anwar
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] (gosmore:25678): Gtk-WARNING **: cannot open display:
Hello Anwar, Gosmore will not connect to the display if the QUERY_STRING environment variable was set. For example: unset DISPLAY QUERY_STRING=flat=52.396545flon=4.915714tlat=52.373075tlon=4.901638fast=0v=motorcar ./gosmore Yield on my computer: Content-Type: text/plain No route found On Thu, Dec 22, 2011 at 9:35 AM, Anwar Azulfa an...@troyans.net wrote: I have try using gosmore, but gosmore like need GTK.I rebuild on Ubuntu server which have only console screen. How to solve this ? Thanks -- Regards, M.Iftakhul Anwar ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Something wrong with the renderer?
On Thu, May 5, 2011 at 7:05 PM, Igor Brejc igor.br...@gmail.com wrote: On Thu, May 5, 2011 at 6:33 PM, M∡rtin Koppenhoefer dieterdre...@gmail.com wrote: 2011/5/5 Grant Slater openstreet...@firefishy.com: What about changing the tile urls frequently? This could be automated and synchronized with the start page and would stop those apps quite fast (at least until they start making the urls in their app dynamic and synchronize as well). That would punish everyone, even those apps that adhere to the policy. I see no reason to prevent access to an app which just mimics the browser and shows a limited map area, uses throttling and caches those tiles for more than a week. Besides, this would not really help OSM spread its word. I'm sure there are better ways of dealing with offenders. We could encourage all 3rd party websites and apps to get their tiles from commercial providers. As I explained in a previous post, placing the occasional advertisement on a map can have significant economic benefits. I see a lot of similarities between serving tiles and hosting Internet sites: Some of them are add supported (Geocities). Some of them are little sole proprietor operations being run a garage. Some focus on reliability (pair) and others on scalability / variety / flexibility (Amazon). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Some advice for a new OSM client
On Thu, Apr 7, 2011 at 4:00 PM, Richard Fairhurst rich...@systemed.net wrote: It still amazes me how many app developers have made the functioning of their paid-for app entirely dependent on what someone (OSM) decides to do with a third-party server. Clearly some developers think about it at all. But some developers may reason that OSMF is getting something from it: The most obvious choice would be exposure that OSMF then tries to translate into productive edits and perhaps even donations. After all, Google served tiles for free when serving map tiles were much more expensive (NT / TA license and hosting costs when they launched 6 years ago). Now OSMF can use this type of analysis to redesign the Tile Usage Policy. All you are really aiming for is (a) preventing increases in tile requests faster than procurement of servers and (b) maintaining a good tile request to edit ratio. (a) is not a problem because of the negative feedback loop. And (b) can be achieved with by simply placing a limit on the number of tiles served to a single IP address in a single day. If you can convince a user to start editing after serving him 20MB of tiles, you're most like not going to convince him with 100 MB of tiles. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Develop a Simple Mapping Tool for Mobile Phones
On Thu, Mar 24, 2011 at 12:08 PM, Yarik iaroslav.shepty...@hs-bremen.de wrote: You have offered help in triangulation in one of the prev emails. A couple links would be awesome for a quick initialization. Thanks in advance! Firstly, sensors give you magnetism and gravity as 3D vectors. For every picture, obtain magnetic East (in the horizontal plane) you take the cross product of the two vectors. Then you get magnetic North by taking the cross product of magnetic East and gravity. Then you project the camera vector onto the horizontal plane to get the azimuth angle. Then get elevation angle with the cosine rule. When the user selects multiple photos aimed at the same point, you can estimate that point using some least squares method. Preferably the photos should have a large angular separation (as seen from the point being observed), otherwise the results will be inaccurate (in GPS, it is called Dilution Of Precision). All these terms are described on Wikipedia, but they focus on linear variables. Angles are non-linear. So either one should look at books or papers describing theodolite computations, or just make your own approximations and simulations. For example it may be sufficient to assume the Earth is flat after applying the equirectangular projection. Greets, Yarik On Wed, 2011-03-23 at 20:22 +0200, Nic Roets wrote: On Wed, Mar 23, 2011 at 5:30 PM, Yarik iaroslav.shepty...@hs-bremen.de wrote: Hi Nic! Great to hear! I has been thinking of building a tool like that for quite a long time already with slightly different functionality than you describe but still very similar. I wrote already that I worked on location based mobile games. In many games players are encouraged to take pictures of different objects. The challenge I always faced in data analyzes was to extract any useful information out of those pics about features (landmarks in particular). You offer an interesting approach. Do you know anything about the last year SoC project? Would it make sense to extend its achievements? No, I haven't looked at it. Since you say such apps are useful for community, I will develop the idea further. I will also try to apply to SoC this year. But even in case I fail I would be interested in the development. Are you going to apply for SoC too / mentoring? I'm not going to apply for SoC. I'd be happy to informally mentor anyone developing software to process sensor and GPS data. I built up a little bit of experience with my KeypadMapper project, as well as porting Gosmore to Android. In the long term (i.e. not this year) it would be nice to detect logos (trademarks), perhaps do OCR or even photosynth type survey. But in the short term there is no image processing necessary. Simply show the user a crosshair. He can then take multiple shots of an object to collect it's height and location. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Develop a Simple Mapping Tool for Mobile Phones
Hello Yarik, I think an application that allows the user to take photos and records the position, azimuth and elevation angle will definitely be useful. It will allow people to anonymously contribute bug reports without typing anything. You can always start with the app and hope that other people contribute other parts. Like a central server of photo bugs, integration with editor(s), grouping of photos for triangulation purposes, off-line modes and mobile tagging. If you don't know how to compute azimuth and elevation angle from sensor inputs, I can help you with it. Regards, Nic On Tue, Mar 22, 2011 at 7:38 PM, Yarik iaroslav.shepty...@hs-bremen.de wrote: Hi Zhijie! Thanks for clarification! Still though it is interesting what the community thinks about such sort of apps. Does it worth putting an effort into it? Again referring to my previous experience would be interesting / cool to have an app implementing simple game mechanics in order to encourage more people to contribute data. Maybe evolving the app developed last year with these mechanics and publishing it at App markets. Cheers, Yarik On Wed, 2011-03-23 at 00:20 +0800, Zhijie Shen wrote: Hi everyone, I think I need to make some clarification here. Actually, in the past two days, some students who want to participant in GSoC sent email to me to show their interest in the idea Develop a Simple Mapping Tool for Mobile Phones. I'm sorry to not have reply immediately. Here I must clarify something: 1. I'm very glad to meet the students that are also interested in location based service on mobile phone here. It will be nice if we can discuss the mobile application development here. 2. Yes, I posted the idea, but I'm a student as well, and cannot be the mentor. In fact, I wrote the idea because I want to apply for GSoC with it. I also wish there will be some mentors here that are willing to guide the mobile application development. 3. I used the title of the idea proposed in the last year. However, this idea hasn't much relationship with what I proposed. Mine is quite related to my current research work. Hence who are interested in building mobile application with OSM doesn't need to follow my idea. Sorry again for the misunderstanding I've brought about. Wish these words explain myself well. Regards, Zhijie On Tue, Mar 22, 2011 at 11:42 PM, Yarik iaroslav.shepty...@hs-bremen.de wrote: Dear all! I am new to the developers list so lemme first introduce myself. My name is Yarik. I am 24. Study Digital Media at HS Bremen / Bremerhaven. The main topic of my studies are location based mobile games (like Foursquare). Currently I am writing my thesis about landmark identification using these sort of games. I am considering myself more of a programmer, therefore development of osm interests me for quite a long time. But SoC actually did the magic kick and led me to the page with implementation ideas. Checking those ideas out I have found an idea to Develop a Simple Mapping Tool for Mobile Phones. Mostly because of rich previous experience, I find it very interesting and would love to work on a tool like that within our outside of the SoC. Unfortunately I'm a newbie to osm and therefore I cannot judge myself the importance of this sort of tool for the osm community. Do we really need it? I have seen similar efforts from the last SoC. http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2010/AcceptedProjects/AndroidPOICollector Did they fail? Does it need to be further developed? I have also seen several other tools with similar functionality. How do you think does it make sense to keep up with the investigation of those and submitting the solution idea? Thanks Yarik ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev -- Zhijie Shen School of Computing National University of Singapore ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Develop a Simple Mapping Tool for Mobile Phones
On Wed, Mar 23, 2011 at 5:30 PM, Yarik iaroslav.shepty...@hs-bremen.de wrote: Hi Nic! Great to hear! I has been thinking of building a tool like that for quite a long time already with slightly different functionality than you describe but still very similar. I wrote already that I worked on location based mobile games. In many games players are encouraged to take pictures of different objects. The challenge I always faced in data analyzes was to extract any useful information out of those pics about features (landmarks in particular). You offer an interesting approach. Do you know anything about the last year SoC project? Would it make sense to extend its achievements? No, I haven't looked at it. Since you say such apps are useful for community, I will develop the idea further. I will also try to apply to SoC this year. But even in case I fail I would be interested in the development. Are you going to apply for SoC too / mentoring? I'm not going to apply for SoC. I'd be happy to informally mentor anyone developing software to process sensor and GPS data. I built up a little bit of experience with my KeypadMapper project, as well as porting Gosmore to Android. In the long term (i.e. not this year) it would be nice to detect logos (trademarks), perhaps do OCR or even photosynth type survey. But in the short term there is no image processing necessary. Simply show the user a crosshair. He can then take multiple shots of an object to collect it's height and location. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [GSoC] Video based speed limit detector
Personally I think that any tool that can process a video recording in a reasonable amount of time and detect most street signs will be a significant accomplishment. Even if it does not integrate with an editor. Let's suppose your video stream is of high quality and the lighting conditions are good. Now there is some stationary object like a pole that obscures the street sign during some of the frames. For the tool to be useful, it needs to handle it correctly. 2011/3/21 Matthias Meißer dig...@arcor.de: Hi Keshan, I'm currently working on the videomapping plugin as my bachelor thesis. I guess this would be a good starting point. http://wiki.openstreetmap.org/wiki/JOSM/Plugins/VideoMapping Personaly I'm not that sure, how a detection of signs can be added in the workflow but let's see ;) Matthias ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Community standards for writing editors
Hello Andrew, So far we found that crappy editors loose popularity. For example, Potlatch would not warn users that the two ways they are merging have two different names. Then some would call for it to be banned. Then they would incorporate the good features of Potlatch (e.g. Y! imagery) into other editors. There is an interesting trend though: All or nearly all editors have been open source. Regards, Nic On Thu, Mar 10, 2011 at 3:07 PM, Andrew andrewhain...@hotmail.co.uk wrote: Because anyone who imports data and runs a bot can make large changes to the database with little effort the OSM community expects them to behave themselves while doing so; this is, for instance, expressed in the wiki pages http://wiki.openstreetmap.org/wiki/Import/Guidelines and http://wiki.openstreetmap.org/wiki/Automated_Edits/Code_of_Conduct . People who write editing tools can also exert a lot of leverage, for instance by having their programs used by mappers who are less skilled and experienced than they are themselves. What do you see as being the standards the community expects of editor writers? -- Andrew ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Newbie questions, introductions and questions about Mac/Objective-C based mapping/Map View using openstreetmap...
On Thu, Mar 10, 2011 at 10:11 PM, Richard Fairhurst rich...@systemed.net wrote: It can do routing and is generally wondrous. Much much more potential than pratting around with X11 and Gosmore. I seriously doubt that that was what Frederik had in mind. A lot of developers use Gosmore only as a routing engine and something based on Mapnik for rendering, be it realtime rendering or prerendered tiles. The strong points of routing with Gosmore are: Supports turn restrictions and many advanced tag out of the box, fast, the ability to calculate very long routes and of course open source. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] OSM formats optimised for client-side vector rendering?
On Thu, Feb 17, 2011 at 7:02 PM, Tim Teulings t...@framstag.com wrote: There is always a trade of :-/ Absolutely. I think Nick refers to mobile users, in which case the tradeoffs are (a) the time of the user (b) mobile bandwidth (c) processing power, i.e. must the user buy a faster phone to run the app (d) RAM (e) flash (f) Wlan bandwidth that is often available at home / work. I think I have these resources in order from most significant to least significant for the average app store app. So when he says client-side, I'm starting to wonder because client-server apps are generally less responsive and require more mobile bandwidth, consuming 2 precious resources. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [OSM-talk] Nominatim updates
For me, reverse lookups are completely wrong right now. (I haven't tried forward lookups). And the status indication on nominatim.openstreetmap.org is blank. So I figure he's doing a DB rebuild or something. http://osm.org/?lat=-25.797306lon=28.289618zoom=18 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] House number interpolation tool / plugin ?
On Thu, Jan 27, 2011 at 7:32 PM, Lambertus o...@na1400.info wrote: Hi Nic, A while ago I've done a bit of work on the terracer plugin (lets you terrace buildings and easilly add housenumbers to buildings). I think I should be able to make what you want. How about: one adds the addr:housenumber tags to the two outer nodes, select both nodes including a street then hit the key combination that opens a screen which lets you add the rest of the addr:* values if you want and/or hit enter to have the plugin create the intermediate nodes, save the values to the nodes and also add the address relation. Hello Lambertus, I'm don't want to break the interpolated way down into nodes at this stage. If an inaccuracy creeps in, then any subsequent fix may require many nodes to be moved. I also don't want dialogs. Otherwise I will never finish. In Josm, is there a way to bind a key combination to a certain tag ? I want one key combination for addr:interpolation = even and another one for addr:interpolation = odd. This is very similar behavior like the terracer plugin. What do you think of that? Fidling with the math to get the interpolated node positions right with a curvy road will be the biggest challenge I reckon. I got a lot of experience with 2D vectors over the last few years. Equirectangular projection. Perpendicular vectors, projecting a point into a line without trig, testing that angles are within certain ranges with trig, etc. On 2011-01-22 16:46, Nic Roets wrote: Hello, I live in an area where very few house numbers have been collected. So I made a couple of experiments. One of them is an Android App called KeypadMapper that allows me to type in housenumbers while I cycle, at a rate of 300 per hour. The only disadvantage is that it does not do interpolation. So if the houses are on both sides of the street and are 30 meters apart, I cannot cycle faster than 300 / 2 * 30 = 4,5 km/h or I will miss some. http://help.openstreetmap.org/questions/1385/what-is-the-best-mobile-application-for-large-scale-house-number-collection The other experiment was to collect only the numbers at the end points and then draw interpolation ways. The good thing is that we already have a great open source geocoder that supports it, namely Nominatim. http://www.openstreetmap.org/?lat=-25.77512lon=28.2892zoom=17layers=M My only complaint is that it takes a lot of work to add the interpolation ways. So I want to ask anyone on this list with to write a plugin for Josm to speed up the process. It would be ideal if I can just: 1. Select 2 nodes that are tagged with addr:housenumber set and 2. Hit a button or key combination for Interpolate The plugin will then 3. For each node, calculate the nearest way tagged as highway. If it was 2 different ways, the plugin can abort or try to handle it more intelligently. 4. Generate a way between the two nodes that runs parallel runs to that way. 5. Automatically set the odd or even tag to match the contents of the addr:housenumber tags on the nodes. I know I sound lazy. But a tool like this will save me hundreds of man hours over the next couple of years. The area I want to cover includes over 20,000 streets and over 100,000 houses. Regards, Nic ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] House number interpolation tool / plugin ?
Hello, I live in an area where very few house numbers have been collected. So I made a couple of experiments. One of them is an Android App called KeypadMapper that allows me to type in housenumbers while I cycle, at a rate of 300 per hour. The only disadvantage is that it does not do interpolation. So if the houses are on both sides of the street and are 30 meters apart, I cannot cycle faster than 300 / 2 * 30 = 4,5 km/h or I will miss some. http://help.openstreetmap.org/questions/1385/what-is-the-best-mobile-application-for-large-scale-house-number-collection The other experiment was to collect only the numbers at the end points and then draw interpolation ways. The good thing is that we already have a great open source geocoder that supports it, namely Nominatim. http://www.openstreetmap.org/?lat=-25.77512lon=28.2892zoom=17layers=M My only complaint is that it takes a lot of work to add the interpolation ways. So I want to ask anyone on this list with to write a plugin for Josm to speed up the process. It would be ideal if I can just: 1. Select 2 nodes that are tagged with addr:housenumber set and 2. Hit a button or key combination for Interpolate The plugin will then 3. For each node, calculate the nearest way tagged as highway. If it was 2 different ways, the plugin can abort or try to handle it more intelligently. 4. Generate a way between the two nodes that runs parallel runs to that way. 5. Automatically set the odd or even tag to match the contents of the addr:housenumber tags on the nodes. I know I sound lazy. But a tool like this will save me hundreds of man hours over the next couple of years. The area I want to cover includes over 20,000 streets and over 100,000 houses. Regards, Nic ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Can't login to https://trac.openstreetmap.org/login
Hello MartinZ I had the same problem with trac.openstreetmap.org a couple of years ago. I have no idea who changed what, but eventually it just started working. If you have a secondary email address, register it with osm.org and try again ? Regards, Nic On Tue, Jan 18, 2011 at 7:17 PM, Martin Ždila m.zd...@gmail.com wrote: Hello On Tue, Jan 18, 2011 at 5:38 PM, Jonathan Bennett openstreet...@jonno.cix.co.uk wrote: On 17/01/2011 20:30, Martin Ždila wrote: Hello I can't login to https://trac.openstreetmap.org/login with my username mar...@zdila.sk mailto:mar...@zdila.sk and correct password. These credentials works on the http://www.openstreetmap.org site. Have you tried logging in with your display name rather than your email address? Trac is working fine here, and I can log in. Also, there's no HTTPS support on trac, but you should get redirected if you try. http://trac.openstreetmap.org/login redirects me to https protocol I also tried my screen name *Martin* - http://www.openstreetmap.org/user/*Martin*. This doesn't work either :-S. -- Ing. Martin Ždila tel:+421-908-363-848 mailto:mar...@zdila.sk http://www.zdila.sk/ ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] scaling
I'm not the one funding the hardware nor handing out the bounties, BUT I would like to see a little bit more accounting / budgeting, as in : 1. The current hardware costs $X and can serve Y million map queries per year, 2. A XAPI server costs ... and can serve ... million map queries per year, 3. We estimate that for every NNN map queries we serve to example.com, we get a user to make one correction to the DB. This compares to MMM map queries per edit for visitors to osm.org Part of the reason for 1 and 2 is that there is very little competition in the large server market and that translate into poor value for money. So economies of scale may not kick in. Then we can estimate how much amount of money will be saved by a rewrite in C. Then we can say to corporates and webmasters that they will be a drain on the project unless they donate $D to us. No need to be accurate to the second decimal. Or the first decimal. Regards, Nic On Mon, Jan 10, 2011 at 3:31 PM, Andy Allan gravityst...@gmail.com wrote: On Mon, Jan 10, 2011 at 12:44 PM, Robert Scott li...@humanleg.org.uk wrote: On Monday 10 January 2011, Kai Krueger wrote: Depending on how far you really want to scale, I think a lot of the necessary components are already in place. If none of the out of the box solutions such as the new Postgresql 9.0 replication mechanism work, we would possibly get a fair distance by splitting out reads and writes onto separate db servers. This is how the postgres 9 replication works. The replicating servers become hot standbys which you can use for read requests. So in theory the read requests could be scaled quite easily once set up. Atomicity of the API would potentially suffer though. We need to be careful about what purposes we scale read requests. A lot of the read-load on the API is using the data for non-editing purposes (such as rendering or other 'cool' things that need /map calls) and this should be avoided[1]. The advantage of making sure that read-scaling for non-editing purposes is *not* via db-replication is that anyone (and everyone) can add their own servers to scale things out. For example, everything that currently fetches live data via the diffs at planet.openstreetmap.org (or a service that depends on them, like xapi, geofabrik extracts etc) is a Good Thing, and everything that currently fetches live read-only data via osm.org/api/0.6/map is a Bad Thing. It's a bad thing since there's only so much hardware the OSMF can own/host/run, but there's nothing stopping 50 other organisations running cool stuff based off of planet.openstreetmap.org Cheers, Andy [1] It's against the policy at http://wiki.openstreetmap.org/wiki/API_usage_policy , for a start. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] scaling
Hello Steve, I have ideas for a super fast API mirror with the capability of answering historic queries i.e. contents of bbox l,t,r,b at time h. But it can't be the main server because it ignores changesets and other info. If we moved all the load that is not directly related to editing to such a server, would it make a massive difference ? Get rid of all the scraping and spidering ? http://www.google.co.za/search?q=edna+site%3Awww.openstreetmap.orgie=utf-8oe=utf-8aq=trls=org.mozilla:en-US:officialclient=firefox-a On Mon, Jan 10, 2011 at 12:27 AM, Igor Brejc igor.br...@gmail.com wrote: I'll play a heretic here, but my feeling is that openness in OSM will more and more come under question, and the reason is scaling. Yes, OSM can proclaim the access to its data is open, but in reality only someone (or better some organization/company) with enough HW resources to be able to process planetary OSM data can actually make use of it. In reality most potential users of OSM data don't really need global data, they want easy access to OSM data that's local to them. And that's what OSM infrastructure does not provide. XAPI and CloudMade and Geofabrik extracts are just poor workarounds (that's not to say that they aren't valuable). A simple example: if I need OSM data for European highways, why do I need to process the whole planet file? Europe is physically separated from America and I see very few reasons for having to share OSM data across continents in a single planetary file. Separating data in vertical layers could help too: country borders certainly belong to a completely different level than, say, park benches. And they change a bit less often. Why keep them in the same store? Igor On 9.1.2011 22:38, SteveC wrote: The amusing recent FakeSteveC ... I guess I will call it a LOLSCALE got me thinking about what people actually think of the boards comment on scaling; http://fakestevec.blogspot.com/2011/01/know-your-osm-memes-2.html As much as I want a dialogue with my fake self, a discourse on the thrust of the argument is I think merited. I think scaling is the number one issue OSM should tackle technically. The days of just 'buy a bigger database server' are I think over. It's not very elegant and it's just too damn expensive. Perhaps we could do another iteration, but if OSM bandwidth continues to outpace moore's law and donations then it just doesn't work. So that means scaling horizontally to more than one machine. And if you're doing that, you may as well do more than 2 machines, or more than 20, or whatever figure you have in your head. I think this is number one because I think the amount of data OSM is going to have to deal with is going to explode in a fairly short time scale. I don't mean just another big import. Sadly I can't be public but I had a conversation with a large company over a year ago (no, it's not MS or CM) who speculated about putting OSM on the front page of their maps product, which would approximately turn all of our yearly statistics to daily or weekly numbers. We went through a decision tree about how that could happen. Every leaf node on that tree came back as basically we couldn't do it. Could we accept the edit traffic? No, far too much. Could we provide a good user experience, clearly no. Could they help us scale? No they would be viewed as taking over on any kind of timescale they needed. Could they host us? Again no, it would be too slow of a process and it'd be a takeover and the community would probably reject it. I could continue, but the basic direction you can imagine. Imagine you had millions of daily users and you wanted to use OSM in a respectable community-driven community way. And let's say you get over the 4chan rhetoric over on t...@. If you think through it, within any reasonable time frame (like 6-12 months) it's very hard to make that happen, and so you may as well go build your own things. Which I think sucks and is a loss for OSM. Now this conversation has come up a few more times recently with other large mapping companies. And I feel like I'm rehashing those conversations above. I'd love to be public about it, but those companies aren't ready to talk yet. Even if people weren't privately proposing notching up our traffic a few orders of magnitude, it would still make a lot of sense to figure out how to scale. Back to FakeSteveC and the negative eye-rolling comment on thinking about this for a few seconds. Well it turns out we have. The board specifically didn't list any technical measure on purpose, that's not it's job. But the direction of supporting and encouraging basic things like scaling is I think well within the bounds. I haven't a clue what we should use to scale horizontally. There are a few major architectural choices and then within those there are lots of implementations. Some are too new and buggy, some are in the wrong language ... it's clearly a bit of a
Re: [OSM-dev] scaling
On Mon, Jan 10, 2011 at 3:43 AM, SteveC st...@asklater.com wrote: What you're effectively saying is move certain things to something that looks like xapi? Yes. Note that the rate of edits to the road network is limited to the amount of real world changes to the road network plus a once off catch up amount. Even if OSM becomes popular in the global South (India, China, etc) (something that not even Wikipedia achieved), then the increase will be closer to 4 fold, rather than 100 fold. If everyone in the world add their home address as a node, it will barely double the planet. Edits to buildings can cause a problem. But I don't see buildings as critical to the success of OSM, so we can take steps to deemphasize them if we have to. And the evidence is that the number of changesets have been growing linearly at +- 300,000 per month since late 2009. Read requests however will explode and keep growing exponentially for regions where OSM is becoming the authoritative source for geographical data. -- I think Frederik is right. If we grow 100 fold or even 10 fold, trolls, vandals and newbies will be our biggest problem. Fred May I paraphrase? You push here, and say in other replies, that you're concerned about the social impact (of a much larger community). As the guy who implemented the first friend features and was called an idiot for it, and doing a lot of community building, I think I know a thing or two about that stuff. I read the other day that proof-by-contradiction is too easy and you should try for others, but it's too easy to pass up. Let's say you are so concerned about the social aspects of OSM. Isn't it a contradiction given that there is no community on talk@ any more, and that the community is basically (give or take) the same for the in-person meetings it's been for what, 3 years? And we're clearly moving to tools that can handle radically bigger communities more efficiently like the osm help site? I don't think you (or anyone else) is much up for fixing the community on talk@ despite the obvious things to do, I don't think growing radically the community will impact the other things outlined above, therefore I don't think the community argument flies. Or am I wrong? On Jan 9, 2011, at 2:35 PM, Frederik Ramm wrote: Steve, SteveC wrote: I don't mean just another big import. Sadly I can't be public but I had a conversation with a large company over a year ago (no, it's not MS or CM) who speculated about putting OSM on the front page of their maps product, which would approximately turn all of our yearly statistics to daily or weekly numbers. We went through a decision tree about how that could happen. Every leaf node on that tree came back as basically we couldn't do it. This was not purely a techical issue. If we were set up, technically, to handle something like what you're describing here, the eternal september effect would kill off the community for good. So in a way, we must be glad that we have this technical limitation, because it protects us against someone dumping 1000s of new users every week onto our mailing lists, IRC channels, forums, onto help.osm.org, the Wiki, and int our regular pub meets. If you think through it, within any reasonable time frame (like 6-12 months) it's very hard to make that happen, and so you may as well go build your own things. Which I think sucks and is a loss for OSM. I honestly think that the project would degrade tremendously were something like this to happen. We can handle growth, and I also believe we can handle super-linear growth, but something that turns all our yearly statistics into daily or weekly numbers would be the end of everything that is sane about OSM. Now this conversation has come up a few more times recently with other large mapping companies. And I feel like I'm rehashing those conversations above. I'd love to be public about it, but those companies aren't ready to talk yet. Good for them because they would have to take the heat for killing OSM - or nonchalantly thinking about doing that. Even if people weren't privately proposing notching up our traffic a few orders of magnitude Please stop thinking of this as a technical issue. It isn't! But the direction of supporting and encouraging basic things like scaling is I think well within the bounds. I haven't a clue what we should use to scale horizontally. I think encouraging scaling is within the bounds of what OSMF could think about. Whether or not such scaling needs to be horizontal, vertical, deep-bore or whatever other method is currently en vogue, that's perhaps something best left to the TWG. If OSMF board wants to realistically construct a scenario that has the number of new users, and the amount of edits and traffic, multiplied by 100 in the frame of a few months, then I'm sure that TWG can provide ideas how to get there. But you would *have* to find someone
Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.
Hello Scott, How do you keep track of what bboxs each entity belongs to ? I'm not really asking a question, I'm just saying that I found a way to reduce the memory requirement for that considerably. Instead of a bit per bbox per entry, I store only 16 bits or 32 bits per entry. Here is the source. http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp?rev=24484 Ask if you have any questions. On Wed, Dec 1, 2010 at 2:59 PM, Scott Crosby scro...@cs.rice.edu wrote: I've been working on making various improvements to the mkgmap splitter. It was originally intended to split a map into smaller pieces which could then be converted into Garmin format and artifacts of that original purpose continue to exist. (Eg, its coordinate system for representing the areas.) However, I've committed several improvements to crosby_integration branch in mkgmap SVN that massively increase its scalability and may make it more useful to a general audience: On a 8gb core i7 machine, the new version can: Split the whole planet into 1200 tiles in about an hour (from pbf to osm.gz format) in a single pass. Split the whole planet into 2 tiles in about 4 hours over 3 passes. There are some limitations, both in the implementation, and design. Due to inherent serialization of the internal tracking tables, it only uses a few CPU cores. The code uses bounding boxes, not bounding polygons. In the original purpose of generating mkgmap tiles, tiles are square there are no polygon tests. Polygon support is straightforward to add, but a fileformat must be defined to specify bounding polygons instead of bounding boxes. At thousands of tiles being output concurrently in a pass, memory consumption in output buffering can be significant. The process assumes that the file is sorted with nodes first, then ways, then relations. Efficiency degrades if they are not further sorted based on ID. Ways are not 'completed', A way is included if any node of it is within the bounding box. Other nodes in the way are not included unless they too lie inside the bounding box. (So, typically, the bounding box is expanded to include an overlap region.) All metadata is dropped. The code hasn't been generalized to overcome these limitations, but might be useful to those creating lots of excerpts or breaking maps up by continent, nation, state, or county. As I don't expect to have much time to work on the code, I mention it now. Scott ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Splitting the planet into thousands of pieces in one pass.
On Wed, Dec 1, 2010 at 8:18 PM, Scott Crosby scro...@cs.rice.edu wrote:
On Wed, Dec 1, 2010 at 7:27 AM, Nic Roets nro...@gmail.com wrote:
Hello Scott,
How do you keep track of what bboxs each entity belongs to ?
An Int2ShortMultiMap implemented by composing two underlying
Int2ShortMap implementations with different space efficiency
tradeoffs, a custom sparsearray
implementation based on
http://google-sparsehash.googlecode.com/svn/trunk/doc/implementation.html,
and one imported from a library of java collections specialized to
primitive types ('fastutils') that uses a standard hashtable. The
hybrid has a memory overhead of about 4 bytes per node output, storing
750m keys with 800m vals in 3.2gb of heap.
The approach for generating an Int2ShortMultiMap from several
Int2ShortMap's is by layering them. When put()ing keys, I store in
Int2ShortMap[0], but if the key is already there, I try
Int2ShortMap[1], . until I find one that is free. I create
additional maps as needed. For dense maps, sparsehash is the more
efficient implementation, and for non-dense maps at the higher layers,
a hashtable is the more efficient implementation.
To avoid checking each bbox for each point, used a simpler design than
a quadtree or r-tree as andrzej suggests, some precomputing and binary
searches at a cost of (sqrt(n)) instead of O(log n) for mostly
non-overlapping-regions.
I'm not really asking a question, I'm just saying that I found a way to
reduce the memory requirement for that considerably. Instead of a bit per
bbox per entry, I store only 16 bits or 32 bits per entry. Here is the
source.
http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp?rev=24484
Definitely. Can you explain how you track the information?
You have a much more concise implememtation than mine, but I can't
easily figure it out what it is doing from the source code.
Let's say there are 80 bboxes, then you can use a 80 bit number for each
entity to record which bboxes they fell into and which they didn't. That
2^80 space is extremely sparse. For my regular splits of 80 bboxes there are
typically only 1000 combinations that actually occur. So I store 1000
younions and then a 16-bit index per entity. And my bbox are as chaotic as
they come:
http://dev.openstreetmap.de/gosmore/
Scott
___
dev mailing list
dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New public nominatim service (mapquest)
The search box on the left does not work because the demo is not a full implementation of the main page. (On my to do list). After enabling the routing layer, you can search for London. http://nroets.dev.openstreetmap.org/demo/index.html?layers=B000FTFTT Which unfortunately returns London in Ontario, Canada, which is the first entry of the list returned by Nominatim. Ideally cities should be listed in decreasing population size. Not a big job considering the data is in the geonames database. On Thu, Nov 18, 2010 at 9:14 PM, Wyo otto.w...@orpatec.ch wrote: Nic Roets wrote: I think nominatim is well documented on the wiki. Nomination well yes, but not how to access Nomination with Ajax. I used it to create the Osm.org routing demo. It queries nominatim using jquery. You can find the source here: http://nroets.dev.openstreetmap.org/ Thanks. I tried the demos, searching for london, none produced any results. Wyo ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] New public nominatim service (mapquest)
I think nominatim is well documented on the wiki. I used it to create the Osm.org routing demo. It queries nominatim using jquery. You can find the source here: http://nroets.dev.openstreetmap.org/ On Wed, Nov 17, 2010 at 10:48 PM, Wyo otto.w...@orpatec.ch wrote: Brian Quinion wrote: Mapquest have now created an instance of nominatim running on their network infrastructure Does Mapquest also have a nice sample page which shows how to access Nominatim? I'd like to access Nominatim via OpenLayers.Request if possible, yet I'm not very familiar with OpenLayers and such samples seems to be rather rare. So far I've just succeeded with setting up a demo map page with some small improvements ( http://www.orpatec.ch/osm/tools/main.php). Accessing Nominatim seems to be in yet another ligue. Wyo ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Q regarding export by RoadMap
Hello Danny, The Api/0.6 is used by editor software like JOSM because it always has the most up-to-date data. That quality does come at a price though: A big Xeon server with 32GB of RAM and many discs. I'm not sysadmin, but I suspect they are starting to clamp down on non-editors using this valuable resource. There are however a large number of alternatives to the api: XAPI and TRAPI servers will give near real-time data. Cloudmade Geofabrik offer. You can even make your own extracts using tools like osmosis and bboxSplit on a weekly or even daily basis. On Sun, Sep 5, 2010 at 10:50 PM, Danny Backx danny.ba...@scarlet.be wrote: Hi, The buildmap tool from the roadmap project (http://sf.net/projects/roadmap) accesses OSM to download map info, via queries like www.openstreetmap.org/api/0.6/map?bbox=5.053710,52.163085,5.097655,52.207030 It often gets HTTP/1.1 509 Bandwidth Limit Exceeded replies. My basic question is how to respond to that. Are there alternate servers ? Is there suggested behaviour for applications that try to download map info this way ? Thanks, Danny -- Danny Backx ; danny.backx - at - scarlet.be ; http://danny.backx.info ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Sat, Sep 4, 2010 at 7:55 PM, Matt Amos zerebub...@gmail.com wrote: are you saying that kernel.org can be unfriendly because the amount of effort put in by each contributor is higher? Yes. Actually kernel.org is very friendly to there audience (Geeks). I'm thinking of someone like Richard Stallman who does not even surf the web. lwn.net/Articles/262570/ people can be using the data (for example, on bing, mapquest or cloudmade) without ever needing to see the osm.org site. those who are interested enough to want to contribute can do it directly through the provider, I'm willing to consider that model. From Wikipedia's experience we know that people are hesitant to contribute to anything with a corporate affiliation to it. (Why should I do anything more than report a bug ? They have so much money that their people can fix the data ) The other choice would be a second non-profit site. Kind of like Debian is to kernel.org. The site would try to capture it's own donations. So it would limit the number of links to osm.org e.g. by embedding Potlatch. And it would not want to spend money on development, so it would copy large parts of osm.org. So it will be just a few steps away from becoming an outright fork... (Banish the thought). Kai just showed me your demo running on dev, which is great - sorry I missed that. The data looks pretty up-to-date, is it being updated weekly or more often? Is it world-wide? Daily worldwide updates since July 19. The 3 major things that surfaced / resurfaced were : 1. Gosmore did not handle access=yes correctly, but it's patched now. 2. Motorcars on tracks. 3. Connecting high speed bidirectional roads to the network. If anyone don't know what 2 3 are about, please ask me, or ask the routing list. * it has to be a rails port and not a static page. ... Thanks for the guidelines. Serge wrote Imagine if every Merkaartor user who did a search received This service is depricated. Please use Nomimatim. Good advice. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Sat, Sep 4, 2010 at 10:53 PM, Frederik Ramm frede...@remote.org wrote: Hi, Nic Roets wrote: From Wikipedia's experience we know that people are hesitant to contribute to anything with a corporate affiliation to it. (Why should I do anything more than report a bug ? They have so much money that their people can fix the data ) I would love that to be true but the popularity of Google Map Maker seems to prove otherwise. On the whole, I don't think people even care. I think many people are not yet seeing Google as a large, greedy corporate. Party because the only people who are paying Google any money are the people who advertise with them. And if you take the number of MM users as a percentage of the number of Google Maps users, I think it will be tiny. Only people who are genuinely annoyed with problems in Google Maps are contributing. But after doing a few searches like [1], I think we are still ahead of them. And as we gain market share in the online maps arena, we will steal MM contributors. I myself was contributing to Google Earth and Track4Africa* before I started contributing to OSM. [1] http://www.google.com/search?q=google+map+maker+OR+openstreetmap * They are currently one of the Google Maps sources for African countries. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Thu, Sep 2, 2010 at 5:20 PM, Ævar Arnfjörð Bjarmason ava...@gmail.comwrote: You'd want me to spend $12 a day to provide geolocalisation for an OSM editor (if you didn't read the thread, I remind you I'm speaking of Merkaartor)!!?? What sort of crazy hosting costs $360 per month (asking Nic Roets)? I I said $12 a day for a server that can provide a decent routing service. (EC2 High Memory Instance). The kernel.org analogy is not really valid. They don't call themselves the wiki kernel. When I hack a kernel for an embedded project running on specialized hardware, they certainly are interested in my changes. By contrast, we will accept users who come in and make a few changes, spending as little as 10 minutes. So we need to make the environment as friendly as possible. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] [Marble-devel] NOTICE: gazetteer.osm.org being retired
On Wed, Sep 1, 2010 at 4:14 PM, Chris Browet c...@semperpax.com wrote: While I'd completely agree in a business world, don't forget we are FLOSS developers working on software in our free time. We don't (at least I don't, and I doubt Marble has) have a server. OSMF do. Just put a big red sticker on the wiki saying ABSOLUTELY DO NOT USE OUR WEB SERVICES, SETUP YOUR OWN!, then... Just because OSMF, KDE and all the little projects are a non-profit, does not mean they cannot raise money. Last time OSMF raised money for hardware, the target was reached in 2 days. And it makes sense to have a few central servers serving many projects. Not only does it simplify things like applying patches and performance tuning, but the planet diffs also has a certain amount of fixed processing to them. All you do is form an association and pool the donations you receive. And we know that the service will not be swamped overnight by Google users or some other large site. Those large commercial companies have high standards, even for their free products. Our data is not good enough and they will loose too many users if the site gets overloaded. And while this discussion has been about searching (geocoding) it also applies to routing. For $12 a day you can rent a pretty nice server. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] NOTICE: gazetteer.osm.org being retired
On Mon, Aug 30, 2010 at 11:24 PM, Matt Amos zerebub...@gmail.com wrote: Let me just mention to Torsten that you have conflict of interest here: Your employer is such a company. not since november - the london development team was let go due to lack of funding. for the last 9 months my employer is someone entirely unrelated to OSM. Sorry Matt. Honest mistake. I'm really out of touch, e.g. not having watched a single SoTM video. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Gosmore routing engine
Hello Leonardo, For the routing demo I fixed a lot of bugs in the yours code. You can find the latest code as a tgz file below. The only 'hidden' file is gosmore.php. http://nroets.dev.openstreetmap.org/ There are still a few things I want to fix before checking it into SVN. The gosmore CGI interface does not run under Windows. On Mon, Aug 16, 2010 at 8:03 PM, Frederik Ramm frede...@remote.org wrote: There's also a specialist routing mailing list, see lists.openstreetmap.org for signup instructions! Frederik, my understanding is that he does not want to dig into routing behaviour (which is the subject of the routing list), but rather do 'normal' webmaster stuff, like customizing javascript. Are you saying we should create a new list for webmasters ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] robots.txt
Hello Anthony, AFAIK, robots.txt only applies to recursive downloads. Given that file names follow simple patterns and timestamp files exist, it is really not necessary to run recursive spiders. That said, wget and curl can be told to ignore robots.txt. Regards, Nic On Sun, Aug 15, 2010 at 6:39 PM, Anthony o...@inbox.org wrote: I see http://planet.openstreetmap.org/robots.txt now has User-agent: * and Disallow: / Are we allowed to download the minute-replicate files as they become available? If not, what's the point of having them? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] North America gone in geofabrik and tagwatch
On Tue, Jul 13, 2010 at 12:35 PM, Mike N. nice...@att.net wrote: Routing: http://www.openrouteservice.org/ - includes a workable pedestrian routing. But eventually routing may arrive on the main OpenStreetMap server. Note that for routing services to be added to the main osm site, we will in all likelyhood need a new server. If the intention is that is should only be used to debug the data, then a machine the size of Errol should be enough (24GB RAM, 8 cores). But my understanding is that the community wants a public service. The good thing about that is that we can then go to government and any other organization concerned with congestion and pollution (CO2 etc) and show them how people are using our server to plan their journeys. Then we can make a good case that our service is having small but measurable effect in cutting driving time and consequently congestion and pollution. When you compare that to the cost: A 40km journey can be calculated in 50ms for an approximate cost of 0.1 cent. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On Tue, Jun 22, 2010 at 9:40 AM, Tom Hughes t...@compton.nu wrote: On 22/06/10 08:27, Lambertus wrote: So what about my proposal to use the OSM dev server for the purpose of determining the server specs? http://wiki.openstreetmap.org/wiki/Servers/errol Sure, if somebody wants to run some tests there then that's fine. Can I have an account on it please ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] How to earn undying fame
The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. And most of the work has already been done. An opensource routing engine that can do the job already exists [2]. The routing database can be updated weekly. The hardware requirements are quite reasonable (64-bit processor with tens of GB of RAM). If a new server is needed, I'm sure we'll get the money in fairly quickly. All that remains is for someone with a little bit of programming experience to volunteer. Make a copy of the osm.org website* and add routing buttons. Then write some code that converts the routing engine output to a few formats. Open Layers, GPX, human readable text should be enough. [1] http://osm.uservoice.com/ [2] http://wiki.openstreetmap.org/index.php/Gosmore *: Copy only the html and the javascript and exclude /edit, /history, /api, /user etc. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On Mon, Jun 21, 2010 at 5:17 PM, Tom Hughes t...@compton.nu wrote: On 21/06/10 16:00, Nic Roets wrote: The result of the uservoice survey[1] is in: The community wants routing on our website more than anything else. Like we didn't know that already. Tom, you're in London and is an OSM insider. Even Steve Coast indicated that he finds uservoice a useful tool. I feel very out of the loop here in South Africa. I don't know that the supply and demand for routing server software is and that's the main reason why I have spent very little time on the routing back-end during the last 20 months. And most of the work has already been done. An opensource routing engine that can do the job already exists [2]. The routing database can be updated weekly. The hardware requirements are quite reasonable (64-bit processor with tens of GB of RAM). If a new server is needed, I'm sure we'll get the money in fairly quickly. I'm not sure tens of Gb of RAM is that reasonable, though it does rather depend how many tens you mean. It's also rather important to know how fast that memory requirement is likely to grow. Equally I would like to have some idea of how many routes/second you believe this engine to be capable of on that hardware. It depends on how long each route is and how dense the road network is in that area. So I must admit that I can't tell you because I don't have access to a such a machine. The amount of time and RAM used is proportional to the square of the distance between the 'from' and the 'two' marker. So the load can easily be managed by placing an upper limit on it. So when we see an increase in the load (e.g. TV show), a small reduction in that upper limit will solve the problem. No. Whatever solution is adopted should absolutely not involve making a copy of the web site. It should be an addon to the web site, not a copy of it. Like Kai said: A demonstration copy that will be merged with the main site. Updating the database weekly might not be able to meet people expectation like the instant rendering that we have. AFAIK Google Earth takes a month to update their community layer. So weekly updates will be a good start. In case someone would try and do it, it would probably be a good idea to make the frontend code as backend agnostic as possible. Well if somebody can provide a backend that returns a GPX or KML or something then I'm sure somebody will hack up the rails port to be able to display those routes. You can look at the current cgi spec. It's richer than plain GPX, but still quite simple. Using sed or awk to change it to GPX is quite straight forward. (Thanks Lambertus for answering the other questions). ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] How to earn undying fame
On Mon, Jun 21, 2010 at 7:09 PM, Tom Hughes t...@compton.nu wrote: All we need now is a volunteer to run the server in the way that Brian does for the geocoding server. I'll do it. We can start with a small (2GB RAM) server. My ssh key is (also attached): ssh-rsa B3NzaC1yc2EBIwAAAQEArghsg+FgXJgbT+vViUArdtiQy+gUv+fCPQlXHZ5qxwGdOagPy2bQYig9JFxpcmxBLZDKAnQsXFTiE1vLQQZ7rorPBJyKQ1IYB4tKBCMizg1GUmh4EGHEeoksG9ViN9xguHA1SDoxpMfX8kjS0K7Q9lMEFqru+ZxgkHg0zegL+eoHXJphhYfsm5kyZoKp3H2/4fkjOd+6zxUNGXUE9evvVbw+8MAy156OhGnkbfmaJeVuYS6zO5Pkr3w/jK1dwn447+t6iiw1Cd4845eTzaR9eOl8J8O2MhJZUth4tup1Pg//poYvcrVZSa6LoIgwyvaH+hsOpUEag3PdWQG6Kozq8Q== n...@nick id_rsa.pub Description: Binary data ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
On Fri, Mar 26, 2010 at 7:49 AM, Marcus Wolschon marcus.wolsc...@googlemail.com wrote: On Thu, Mar 25, 2010 at 2:28 PM, Nic Roets nro...@gmail.com wrote: Rerouting traffic based on collected track logs is essentially an extension to this: Take the tracklog, divide it into 2 minute intervals (or T seconds). I suggest to use the ways and segments between pairs of nodes on ways instead of and time base. But of cause any such design deccisions are not up to us but the students who´s project this wil become. It´s their project. We merely offer a problem and advising suggestions. Try to filter out the cases where the vehicle was parked or the user was walking around. Ask the router how long it should take to travel from the starting point to the end point. If it's substantially less than T, mark the point (segment) as a penalty point (avoid point) What point? You where talking about start to end a second ago. The points and segments that the router returned. A simple algorithm to choose a representative point or segment somewhere in the middle can be devised. with an appropriate weighting. There is no apropriate weighting as that algorithm does not know IF the delay even happened in that one, random point. You are calculating the delay for a complete route No. I said break it up into short (2 minute) sections. If the penalty isn't attached to exactly the right node, there is still a reasonable chance that it will cause the router to avoid roads that may soon become blocked up. and then suddenly assuming it´s all in single point/segment. Serve these penalty points to clients and routing servers. Then adjust the penalty points according to time of day patterns etc. And in what way do your random penalty-points relate to the completely different route of the client? a) If there is a delay n one direction, there need not be such a delay in both directions. ... My router represents all segments as two directional edges. But a good system will attach some probability to an event that cause delays in both directions, such as road maintenance or a car crash that cause motorists traveling in both directions to slow down to take a peek. I'm not saying it's easy. A lot of trail and error adjustments will be needed. The main point is that the router already does a lot of the work. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
On Fri, Mar 26, 2010 at 6:00 PM, Marcus Wolschon marcus.wolsc...@googlemail.com wrote: Expected, general travel times and delays due to random events are 2 different problems. The two problems are so closely interlinked it makes sense to treat them as one. That there was a car accident last week does not affect the estimate of how long you are likely to travel by the time you react the start of a segment now. Of course it does ! You have to account for the probability that the road is poorly designed at that spot. The most likely source of information about short time traffic obstructions is TMC and it usually tells you the expected delay. So this needs different issue not be addressed now and certainly not here. Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis
On Thu, Mar 25, 2010 at 2:58 PM, John Robert Peterson jrp@gmail.com wrote: If somone were finally to solve the problem of matching up traces to ways, we would be in a position to extend that to identifying some map errors, such as missing ways, or changed road layouts. This has already been done. See the osmunda program that is part of gosmore. Rerouting traffic based on collected track logs is essentially an extension to this: Take the tracklog, divide it into 2 minute intervals (or T seconds). Try to filter out the cases where the vehicle was parked or the user was walking around. Ask the router how long it should take to travel from the starting point to the end point. If it's substantially less than T, mark the point (segment) as a penalty point (avoid point) with an appropriate weighting. Serve these penalty points to clients and routing servers. Then adjust the penalty points according to time of day patterns etc. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Project Proposal - Waze Integration
Hello Graham, AFAIK Waze wants to build their own map of the world and sees OSM as competition. Waze has already imported TIGER, which is PD and I don't think Waze will be a viable concept without TIGER. SteveC reviewed their Iphone app on his blog. There is however scope for building our own Waze infrastructure: 1. Mobile apps for collecting tracklogs. Or libraries that app developers can use. 2. A repository for GPX data that does not have the privacy concerns our repository current have. 3. Algorithms for fixing TIGER data using GPX tracks and / or detecting delays in the road network. Regards, Nic On Wed, Mar 24, 2010 at 9:46 AM, Graham Jones grahamjones...@googlemail.com wrote: Hi Folks, A student has contacted me about a potential Google Summer of Code proposal to provide integration between OSM and Waze (http://wiki.openstreetmap.org/wiki/GSoC_Project_Ideas_2010#Waze_Integration). I had never heard of Waze before, but it seems to be a combination of a user contributed map (like OSM), and traffic information, again user contributed. It sounds sensible to me to use OSM as the base map, and Waze to handle the traffic information. There are a couple of things that I do not understand though, and would appreciate some help so that I can guide the student appropriately: For some reason, Waze seem to be producing their own map rather than using OSM as the basis - does anyone know why? The student says he has contacted Waze and they said it was because of licensing issues - this is a surprise to me, because I think our license is pretty relaxed. Has anyone been in touch with them about it? (Please don't turn this into a 'OSM should be PD' dabate!!). Assuming that the license issue can be sorted, has anyone looked at what Waze are doing and how it could be used with OSM? If so, can you suggest some ideas for a closed-scope student project? Thanks for your help! Regards Graham. -- Dr. Graham Jones Hartlepool, UK email: grahamjones...@gmail.com ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
Andreas, One solution is to add your own tag to the OSM files you generate e.g. smartsoft_id=nnn. And publish the files for review somewhere. Then scan the minutely updates at planet.openstreetmap.de for your id numbers and when they appear you can delete them from your DB. You can also upload data directly to OSM and get the IDs. But if you don't allow us to at least see before hand what you are up to, you run the risk of making us angry. Regards, Nic 2010/3/22 Andreas Höschler ahoe...@smartsoft.de: Hi Tom, However, we want to pass our track data to the OSM project and are looking for a neat way to do that. We can convert our data to any format (some kind of CSV or XML preferred). We just need a TCP interface to post them to or a destination email address where to send the files to. Simply uploading the tracks to some server is no good solution. When a user creates a track in our database it gets a primary key but of course can't be assigned an official OSM ID yet. So the upload process would need to be as follows: Can you clarify whether you're talking about uploading this as GPS trace data (what we would normally recommend for raw GPS data) or as actual OSM map data? OSM!! The data is already rich enough to contain street names, way kind and some other attributes for forrest tracks (e.g. never use when it's raining). Iterate through al newly created tracks - send a track with nodes to the OSM server and receive IDs for the way and the nodes - assign these IDs to our private database objects and thus marking them sent and no longer new For that we need some kind of TCP-based XML interface so that we could send a track in some XML format to a TCP socket and get back the IDs assigned to objects by the public OSM database server. Only that would allow us the assign the returned IDs to our private database objects and thus avoid data duplication. Is there such an API (TCP-socket based)? There are APIs for both, but not necessarily exactly what you want (in particular for GPS trace data). The ideal solution for use would be to send a WAY segment in OSM format to some service and receive a response with the IDs for the created way and the created nodes! But I understand this is not supported due to the obvious merging problem (avoid duplicates)!? Still what is the best option for us? Regards, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Inserting OSM data
On Mon, Mar 22, 2010 at 6:18 PM, Tom Hughes t...@compton.nu wrote: On 22/03/10 15:00, Nic Roets wrote: One solution is to add your own tag to the OSM files you generate e.g. smartsoft_id=nnn. And publish the files for review somewhere. Then scan the minutely updates at planet.openstreetmap.de for your id numbers and when they appear you can delete them from your DB. I'm not sure we really want to encourage the use of our database as a storage space for third party identifiers like that. It could get rather out of hand if everybody else wants to attach their own IDs to our data. Hello Tom, Unfortunately it has already happened many times. Below is a list of the third party identifiers that I have found. Instead of banning them outright, I would be happy if we have guidelines that (a) make it easy for us to identify them as third party (b) make sure they don't clash with other imports and (c) make sure they don't take up unnecessary disk space. Regards, Nic tiger:uuid and others AND_ and AND: kms: LandPro08: NHD: massgis: scgis-shp: KSJ2: geobase: openGeoDB: gnis: CLC: gns: ref:INSEE it:fvg:ctrn: 3dshapes: cladr: BP: chile: // vialidad.cl import navibot: teryt: naptan: asset_ref // naptan qroti: import_uuid sagns_id sangs_id hdop sat pdop fix vdop ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC Suggestion: test.osm suite
Hello Tim, This is my routing test file: http://trac.openstreetmap.org/browser/applications/rendering/gosmore/routingTest.osm It consists of 3 parts (foot, bicycle and motorcar). The best route from West to East will route along the outside. The northern part of the motorcar part tests advanced features, such has trying to avoid busy intersections and sharp turns. There are some renderings test here: http://trac.openstreetmap.org/browser/applications/rendering/osmarender/testdata Regards, Nic On Mon, Mar 15, 2010 at 3:49 PM, Tim Teulings r...@edge.ping.de wrote: Hello! During my work on libosmscout I had the (obvious) idea to test the various implementation aspect of my library against or (or multiple) defined *.osm files. This would have included testing for functionality like rendering things like bridges, tunnels, layers and symbols but also routing correctly in respect of turn restrictions, maxspeed, cross border navigation... For this one or multiple syntetic *.osm file(s) would be optimal, that includes all these features in a stable way (ids of nodes, ways, areas do not change, feature do not disapear or reapear somwhere else). However up to now I had not the time to work on this... My suggestion for GSoC thus would be to let a student develop such test files, resulting in a first version af an ACID-lik test for OSM based map drawing and routing tools. Work would be split into a number of steps: * Collecting the most important features to test (ask the community) * Defining an abstract test suite by categorizing these features and defining small abstract test scenarious. * Design of an environment that allows getting and changing the data, and possibly also not only offers *.osm files but also API instances based on these data * Documentation of environment and change processes * Creation of actual data files Note, that not all step require develpment, in fact the development part is possibly very low, so this might be a problem to get it accepted. Nevertheless this would be possibly of great help for a huge number of existing OSM developers and IMHO testing is a huge and underestimated part of software development. Any opinions about that? Does this make sense? Improvement/addition suggestions? -- Gruß... Tim ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Problem with 2010/03/10 planet...
On Fri, Mar 12, 2010 at 11:55 AM, Frederik Ramm frede...@remote.org wrote: Hi, Jon Burgess wrote: It looks like this was caused by a change made by Frederick back in r19176. The planet dump code used to turn all characters less than 32 into '?' instead of creating these character sequences. I guess he didn't read the bit of the XML spec which says that all characters 32 are invalid except for tab / newline / carriage return[1] Probably right, I didn't read *any* of the XML spec ;-) I cannot remember why I made that change, I guess there must have been some reason but maybe it was a mistake altogether. I am sorry for the inconvenience. If you have an uncompressed version of the planet file, the XML bugs can be fixed using the following three incantations: echo -n ' v='|dd bs=10 seek=6126462692 conv=notrunc of=planet.osm echo -n ' v='|dd bs=10 seek=6533550293 conv=notrunc of=planet.osm echo -n ' backw' | dd bs=10 seek=13047657759 conv=notrunc of=planet.osm Make sure to place the name of your planet file in the of= parameter and make sure that the number of spaces is exactly as written above. If you are streaming the file, then you could use sed to remove any occurrence of #2.; or use grep -v to remove all lines containing #22;Meycauayan City Northbound Entry Point and member type=node ref=494163268 role=backw#27;#27;ard_stop/ I'm using this command egrep -v '#[0-9]*;' because there is also this mistake: tag k=name v=Calle de Luis Rodriguez #15;Ontiveros / They all seem to be created around Sep 2009. Hopefully the bugs (in flash??) has been fixed. And I fixed them (the 3 explicitly mentioned here) on the live server. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Making libxml ignore errors
On Fri, Mar 12, 2010 at 1:48 AM, Bernhard zwischenbrugger b...@datenkueche.com wrote: hi Does anyone perhaps know how to get libxml to either not detect these errors, or how to continue after parsing them ? libxml is correct here. An XML parser MUST stop if there is an encoding error. Just another reason why I should hate XML then. Just another reason why I should implement my own Xml parser [1] [2] Geonames.org has made things very easy for themselves by using CSV files. 1: http://trac.openstreetmap.org/browser/applications/rendering/gosmore/bboxSplit.cpp 2: http://trac.openstreetmap.org/browser/applications/rendering/gosmore/gosmore.cc?rev=4390 ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] First version of long way splitter for gdalcontour output
Ok Mike, I've look at it. It's in an urban area so chances are that it will be useful to someone. But I'm still a bit worried that there is too much detail there, considering what is being imported. One of those segments is shorter than 4m. It's not like a building, where you know someone will be happy if they see their house, or a road where it can influence routing. And you haven't documented your tag (at least not in the obvious place): http://wiki.openstreetmap.org/wiki/Tag:natural%3Dpurple_floodzone http://wiki.openstreetmap.org/wiki/Tag:natural=floodzone http://wiki.openstreetmap.org/wiki/Key:natural Regards, Nic On Wed, Mar 10, 2010 at 10:25 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: I have not done any monster forests, It is just in the beginning. Feel free to try the program on a big file, I would be interested to know how it works. any bug reports will be processed asap. I have updated the blog post, at the bottom you will see two features i extracted and the source. I hope to have some more results soon, mike On Wed, Mar 10, 2010 at 9:21 PM, Nic Roets nro...@gmail.com wrote: On Wed, Mar 10, 2010 at 8:54 PM, jamesmikedup...@googlemail.com jamesmikedup...@googlemail.com wrote: ... Now my code works in a single pass, at least over the entire data. It builds ... This can be optimized further. I have exploited the following optimisations ... I process the entire planet every week (+- 5 passes, +-12 counting compression and decompression). I guess I'll be able to cope if the planet grows by 40% p.a, but you make it sound if that is too slow for you. On a less predicting the future topic: Can you please point me to some of these monster forests or lakes you've uploaded, so that I can check that my software can handle them ? What about Potlatch ? ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Osmosis split of the planet
I'm trying to split the planet into 170 overlapping bboxes like this: http://dev.openstreetmap.de/gosmore/test/ But osmosis keeps running into the 4GB java limit, even after I made a split down the Atlantic. 1. Split the planet into 3 bboxes: Americas, Europe / Africa /Asia /Australia and a bbox that is just large enough to cover all the bboxes that cross the dividing line. 2. Running osmosis for the 95 bboxes in the Americas fails. 3. Running osmosis for the 12 Atlantic bboxes succeeds. 4. Running osmosis for the 60 bboxes in Europe / Africa /Asia /Australia fails: gunzip middle.osm.gz | ionice -c 3 nice -n 19 osmosis --read-xml enableDateParsing=no file=/dev/stdin --tee 60 \ --bb idTrackerType=BitSet left=73.12500 right=180.0 top=9.44906 bottom=-85.05113 --wx 0720048510241024.osm.gz \ --bb idTrackerType=BitSet left=120.58594 right=180.0 top=72.91964 bottom=-25.48295 --wx 0855020310240587.osm.gz \ --bb idTrackerType=BitSet left=98.43750 right=172.61719 top=13.23995 bottom=-85.05113 --wx 0792047410031024.osm.gz \ --bb idTrackerType=BitSet left=100.19531 right=150.82031 top=30.14513 bottom=-75.84517 --wx 0797042209410852.osm.gz \ ... The obvious solution is just to repeat this algorithm until I find something that will work. And the number of candidate splitting latitudes and longitudes is small (4 times the number of bboxes), so evaluating them all in software is feasible (esp. with dynamic programming). Now my question is: Can I tell osmosis to work with intermediate streams ? That would remove the need to gzip / gunzip and write / read from disk. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis split of the planet
Hi Brett, Yes, named pipes may work as intermediate streams. I guess I could try the 64-bit JVM. But I fear that it will become a problem every few months as the planet increases (a larger planet may also mean that I need more bboxes !) As for coding a new idTrackerType that would reduce memory usage: Start with a binary space partitioning (as above) with depth k. Then have k Bitsets to indicate if entities are to the left* the partition and have another k Bitsets to indicate if spans the partition (crosses the boundary). If it neither spans, nor crosses, then it is to the right of the partition. In my case it will reduce the number of bitsets from 190 to +- 16 Bitsets. * Could also be 'up', based on the partition. Thanks for your insights. Regards, Nic On Mon, Feb 8, 2010 at 11:22 AM, Brett Henderson br...@bretth.com wrote: Hi Nic, I'm not entirely sure what you mean by intermediate streams. Do you wish to split the execution across multiple Osmosis instances and stream data between processes? If so, there is no inbuilt support for this other than reading and writing from stdin and stdout respectively. Perhaps there's some way of using named pipes in Linux but that's not something I've ever tried. But the main overhead is typically in XML processing which will be encountered on the boundary point between processes with much the same CPU overhead as reading and writing from temporary files. If the only reason for splitting Osmosis across multiple instances is to allow you to use more than 4GB memory total, then can you switch to a 64-bit JVM? That will let you use as much memory as you have in your system. I assume you're currently setting the -Xmx value to something less than 4GB based on a 32-bit JVM limitation. As an FYI, the BitSet idTrackerType uses a fixed amount of memory per bounding box task dependent on the maximum ids in the planet file. So if you're extracting 60 bounding boxes, it doesn't matter where in the world the bounding box resides or how large it is, the BitSet will consume much the same memory. You need to find how many bounding boxes can fit within the 4GB memory limit and stick within that limit. The limit will decrease over time as the maximum ids in the planet increase. The IdList idTrackerType uses memory proportional to bounding box entity count, but it consumes much more data per entity (at least 32 times as much, maybe 64, haven't investigated this in detail) due to the fact that it holds each id in a sorted list instead of BitSet which stores each id as a single bit in a massive data array. IdList is great for large numbers of small bounding boxes where the total area covers a small portion of the planet. If somebody can code up with a better mechanism for storing these ids it can be plugged in as an alternative idTrackerType. Brett On Mon, Feb 8, 2010 at 7:54 PM, Nic Roets nro...@gmail.com wrote: I'm trying to split the planet into 170 overlapping bboxes like this: http://dev.openstreetmap.de/gosmore/test/ But osmosis keeps running into the 4GB java limit, even after I made a split down the Atlantic. 1. Split the planet into 3 bboxes: Americas, Europe / Africa /Asia /Australia and a bbox that is just large enough to cover all the bboxes that cross the dividing line. 2. Running osmosis for the 95 bboxes in the Americas fails. 3. Running osmosis for the 12 Atlantic bboxes succeeds. 4. Running osmosis for the 60 bboxes in Europe / Africa /Asia /Australia fails: gunzip middle.osm.gz | ionice -c 3 nice -n 19 osmosis --read-xml enableDateParsing=no file=/dev/stdin --tee 60 \ --bb idTrackerType=BitSet left=73.12500 right=180.0 top=9.44906 bottom=-85.05113 --wx 0720048510241024.osm.gz \ --bb idTrackerType=BitSet left=120.58594 right=180.0 top=72.91964 bottom=-25.48295 --wx 0855020310240587.osm.gz \ --bb idTrackerType=BitSet left=98.43750 right=172.61719 top=13.23995 bottom=-85.05113 --wx 0792047410031024.osm.gz \ --bb idTrackerType=BitSet left=100.19531 right=150.82031 top=30.14513 bottom=-75.84517 --wx 0797042209410852.osm.gz \ ... The obvious solution is just to repeat this algorithm until I find something that will work. And the number of candidate splitting latitudes and longitudes is small (4 times the number of bboxes), so evaluating them all in software is feasible (esp. with dynamic programming). Now my question is: Can I tell osmosis to work with intermediate streams ? That would remove the need to gzip / gunzip and write / read from disk. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Osmosis split of the planet
Hello Frederik, I was looking for something like your outPipe=r1, inPipe=r1 trick. But as Brett (and you) point out, doing it in one pass will require lots and lots of RAM. And I've tried it with a dividing line in the US. Except 51 rectangles cross that dividing line and I know 51 BitSets will require more than 4GB RAM. So two passes are also not an option. On Mon, Feb 8, 2010 at 2:06 PM, Frederik Ramm frede...@remote.org wrote: PS: Is it by design that some areas are not covered by any of your rectangles? All areas are covered. Only after clicking on the rectangle will it become a larger yellow rectangle and you can see it's true size. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Mirroring OpenStreetMap
Hello Ian, I have done the sums, but the rendered map it is BIG ! Furthermore, it does not contain everything. (Nightclubs was just recently mentioned on talk) and it does not allow searching or routing. You are much better off with something like Navit, or my program, Gosmore, which is currently under maintenance. The new version of Gosmore will have multipolygon support, 3D rendering and a few other nice features. Regards, Nic On Mon, Feb 8, 2010 at 7:45 PM, Ian Monroe imon...@kde.org wrote: I'm going to talk tomorrow with the director of WiderNet, which has an eGranary Digital Library project that mirrors websites on to hard disks and sends them to schools and hospitals in the developing world (mainly Africa) which have slow or no Internet. So currently the eGranary is 2 terabytes containing Wikipedia, opencourseware, books etc. My thought was that OpenStreetMap would be a great addition to it. And OSM really has more to offer Africa then anywhere else due to lack of commercial maps on the web. I volunteered to perhaps implement this because I have the technical skills in general to be able to set it up, but not because I know much about OSM technology in particular. :) I'm finding the resources on rendering a bit hard to get into. Right now the important question is the feasibility. How big is a fully rendered OpenStreetMap? Or would it make more sense to render on-the-fly? Thanks, Ian Monroe ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] Help needed for processing the planet.osm (for osmdoc.com)
I also had problems with gosmore consuming needing too many disk seeks (or too much memory) and missing the processor cache. I solved these problems by doing less during the xml processing phase. Just categorize the data and writing it into many temporary files. For example nodes with id's in the range 0-10,000,000 in the first file, nodes with id's in the range 10,000,000 to 20,000,000 in another. Strings starting with AA in one file etc. The each of these files are under 200 MB and can be read into memory completely and processed during subsequent phases. So it adds another loop to an already highly nested piece of code. On Mon, Aug 17, 2009 at 9:27 PM, Lars Francke lars.fran...@gmail.comwrote: So please elaborate what combinations you need ;) Since the output is csv /anything/ you imagine is there ;) and if you don't want to code, just add what you want and run uniq -c on it. I'll need output in the following form: tag-key, number of changesets, nodes, relations and ways this key is used on, number of distinct values tag-value, the tag-key this value belongs to, number of changesets, nodes, relations and ways this value is used on Additionally the following information would be nice: key/key combinations and how often these two keys are used in together on changesets, nodes, relations and ways To avoid hitting the database every time a tag is processed I cache the information in a Map in memory. Unfortunately I only have my (t)rusty FreeBSD box with an Dual Core Athlon and 3 GB of RAM. The cache-maps don't fit in memory so I'm looking for a solution to swap this to disk. Additional points for ideas on how to process the daily-diffs to keep the data current :) I need the old and new data for changesets, nodes, relations and ways to process deletions and changes in tags. I think I could write a Osmosis plugin for ths...but that's a problem for another day :) Cheers, Lars ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] A new way to split the planet
On Sun, Aug 16, 2009 at 4:15 AM, Paul Johnson ba...@ursamundi.org wrote: On Fri, 2009-08-14 at 20:07 +0200, Nic Roets wrote: 3. Use an OO spreadsheet to choose the rectangles (density4.ods). OO will makes it quite easy: As you drag to highlight a rectangle, it will display the sum (of nodes) in the status bar. This way you can see how much data will be in the rectangle. I stayed below 14,000,000 which works out at .osm.gz of less than 300MB and gosmore files of less than 400MB. It also works well if you compose expressions like MAX(SUM(A1:K12);SUM(D9:M21)), because you can hit F2 to see how the rectangles overlap and even change them. I ended up with 58 rectangles, which will increase as the planet gets larger. For the sake of simplicity for someone who isn't a spreadsheet nerd, would it be possible to get the relevant cell ranges named? If you look at the spreadsheet from row 1027, you will see that I gave them names. In some cases the same name applies to multiple ranges like US East Midwest. It's difficult to give them all understandable individual names because if they span multiple cities and / or multiple states, but does not cover any city or state completely and you give them a name, it may just mislead people. But the html image map helps to show people what is available. Displaying the rectangles on Open Layers should be quite easy. In the US East Coast the density is high and everything is connected to everything else (no natural or political borders). So a pattern of overlapping rectangles works well. But the spreadsheet also allows you to exploit the natural and especially the political boundaries: If you are in some African country you'll rarely ask your GPS for the shortest route to some European country. On Sat, 15 Aug 2009, Nic Roets wrote: I ended up with 58 rectangles, which will increase as the planet gets larger. Now I'm worried. Fortunately the new rectangles will be in Europe. So I can use the pattern I used in denser US. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] A new way to split the planet
CM and Geofabrik both publish planet extracts by country / region. They are great if you want to answer questions like How many fueling stations are mapped in ... ?. And they are great if you live on an island. But if you want to travel from Brussels in Belgium to Rotterdam in the Netherlands, you want a single extract that covers both, yet is small enough to fit on your GPS. And if your software has access to many of these overlapping extracts, they should preferably be rectangular so that it's easy to programmatically to choose the best one. So here is my process. All the data is at http://nroets.openhost.dk/bbox 1. Use the Mercator projection and divide the planet into 1024x1024 squares. That means each square spans 0.35 degrees in the East West direction and a similar amount North-South. 2. Count the number of OSM nodes in each square (density.csv). 3. Use an OO spreadsheet to choose the rectangles (density4.ods). OO will makes it quite easy: As you drag to highlight a rectangle, it will display the sum (of nodes) in the status bar. This way you can see how much data will be in the rectangle. I stayed below 14,000,000 which works out at .osm.gz of less than 300MB and gosmore files of less than 400MB. It also works well if you compose expressions like MAX(SUM(A1:K12);SUM(D9:M21)), because you can hit F2 to see how the rectangles overlap and even change them. I ended up with 58 rectangles, which will increase as the planet gets larger. 4. Copy and paste all these expressions to a text file and feed them to my density.c program. 5. It generates a script that calls osmosis twice (density.sh). First for Africa, Europe, Asia and Australasia, then for the Americas. This way you can run it with only 4GB RAM. Edit it as desired. 6. It also generates an html image map for choosing a rectangle. http://nroets.openhost.dk/bbox/density.html The rectangles in the default images are smaller than the actual, larger rectangles which are shown after hovering the mouse over them. Clicking on them will download the gosmore files, but you can edit the html as desired. Regards, Nic ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] EMWMS now WMS and YWMS is dead
:( Recently I installed FF2 on my Windows partition so that YWMS will work. It really saves time and money when used with GPRS (compared to potlatch), e.g. traveling to other cities. I can keep on using the currently installed version of JOSM. But 0.6 is just around the corner. Regards, Nic On Sat, Dec 13, 2008 at 7:46 PM, Dirk Stöcker openstreet...@dstoecker.de wrote: Hello, to reduce the confusion a lot I decided that the new enhanced WMS plugin finally replaces the old one. The ewmsplugin is now renamed to wmsplugin again. ewmsplugin and ywms plugin will not be loaded by josm any more beginning with tomorrow morning version. If there should be really a need for old FireFox2 Yahoo support, then this can be implemented in new plugin. But currently I see no sense in supporting very old software. Also I checked in the webkit-image-gtk.c source from WIKI, which uses GTK2+ and Webkit. Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [josm-dev] EMWMS now WMS and YWMS is dead
I guess I misunderstood. As long as EMWMS supports Y! images, that's great. On Sat, Dec 13, 2008 at 11:44 PM, Dirk Stöcker openstreet...@dstoecker.de wrote: On Sat, 13 Dec 2008, Nic Roets wrote: :( Recently I installed FF2 on my Windows partition so that YWMS will work. It really saves time and money when used with GPRS (compared to potlatch), e.g. traveling to other cities. What prevents using the new WMS plugin? Ciao -- http://www.dstoecker.eu/ (PGP key available) ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
[josm-dev] elemstyle.xml
AFAIK, elemstyle.xml was created for the mappaint plugin and now resides in the JOSM SVN. It maps (k,v) tag pairs to line widths, colours, icon names and other attributes. The current version of my 'gosmore' rendering / routing program reads this file. But there are a number of changes that seems logical to me and it would be good to have some agreement before I release gosmore : 1. Add a routing tag, e.g. routing goods=110 hgv=110 motorcar=120 psv=110 oneway=1/ would mean that the goods trucks are allowed on ways of this type and their average speed 110 km/h, while the average speed for a motorcar is 120km/h etc. These are just defaults and can be blocked tagging the way with an approved restriction. You may disagree with my proposal for this tag, but it should not affect JOSM. 2. scale_max : I assume it tells the renderer at what zoom level the way should no longer be rendered. So it seem logical to change tertiary, unclassified, unsurfaced (and others) to at least have the same scale_max as residential. 3. scale_max for citys and towns should be much larger (than mini_roundabout, fuel etc) 4. scale_max for coastline (and rivers) should be much larger. If my interpretation of scale_max is incorrect, I should perhaps start using a different that will not affect JOSM ? Email me if you want to see the actual file or a diff. Regards, Nic ___ josm-dev mailing list josm-dev@openstreetmap.org http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/josm-dev
