Re: [OSM-dev] osm2pgsql bbox filter when applying diffs?
John Smith schrieb: > On 25 March 2010 17:54, Frederik Ramm wrote: >>is the osm2pgsql bbox filter also usable for applying diffs (in a >> situation where somebody wants to keep a country extract in osm2pgsql >> form up to date), or does it only work for full imports? > > I think it only filters nodes, I haven't checked for a while but > previously I had ways with no nodes in the database outside the area I > was filtering... > What would be the best way to keep an individual extent updated? Fetching the planet, extracting the extent of the area of interest and create my own diffs? I could also apply the planet wide diffs and remove everything outside of the extent afterwards. But I don't like this idea. Is there a way to process the diff files in a way that the diff only contains changes in my area of interest? Thanks for any hints. Lars ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
On Fri, Mar 26, 2010 at 7:49 AM, Marcus Wolschon wrote: > On Thu, Mar 25, 2010 at 2:28 PM, Nic Roets wrote: >> Rerouting traffic based on collected track logs is essentially an >> extension to this: Take the tracklog, divide it into 2 minute >> intervals (or T seconds). > > I suggest to use the ways and segments between pairs of nodes > on ways instead of and time base. > But of cause any such design deccisions are not up to us but the > students who´s project this wil become. It´s their project. We merely > offer a problem and advising suggestions. > >> Try to filter out the cases where the >> vehicle was parked or the user was walking around. Ask the router how >> long it should take to travel from the starting point to the end >> point. If it's substantially less than T, mark the point (segment) as >> a penalty point (avoid point) > > What point? > You where talking about start to end a second ago. > The points and segments that the router returned. A simple algorithm to choose a representative point or segment somewhere in the middle can be devised. >> with an appropriate weighting. > > There is no apropriate weighting as that algorithm > does not know IF the delay even happened in that one, random point. > You are calculating the delay for a complete route No. I said break it up into short (2 minute) sections. If the penalty isn't attached to exactly the right node, there is still a reasonable chance that it will cause the router to avoid roads that may soon become blocked up. > and then suddenly assuming it´s all in single point/segment. > >> Serve >> these penalty points to clients and routing servers. Then adjust the >> penalty points according to time of day patterns etc. > > And in what way do your random "penalty-points" relate to the > completely different route of the client? > > a) > If there is a delay n one direction, there need not be such > a delay in both directions. ... My router represents all segments as two directional edges. But a good system will attach some probability to an event that cause delays in both directions, such as road maintenance or a car crash that cause motorists traveling in both directions to slow down to take a peek. I'm not saying it's easy. A lot of trail and error adjustments will be needed. The main point is that the router already does a lot of the work. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql bbox filter when applying diffs?
On 26 March 2010 17:15, Lars Lingner wrote: > Is there a way to process the diff files in a way that the diff only > contains changes in my area of interest? I assume removing any ways without nodes after importing would probably work... ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql bbox filter when applying diffs?
> Is there a way to process the diff files in a way that the diff only > contains changes in my area of interest? If you're fetching the diffs with "osmosis --rrii --wxc", can't you plug a --bb task afzter that? Or does this task not work with changeset-streams? If so, this would be the place to implement this functionality, wouldn't it? Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql bbox filter when applying diffs? - extend: compare with osmosis
Hi Lars, I'm doing some experiments to get a solution for the same (using a bounding polygon instead of bbox) question using osmosis. I did some experiments using osmosis, but ran into problems. I tried the following: 1. update using merged daily-diffs: 1.1 get one-week-old-planet.osm 1.2 cut AOI (--bp) from one-week-old-planet.osm (incl. sorting and completeWays and completeRelations) 1.3 get all the daily-diffs from last week to now 1.4. merge the 7 diffs (sorting at each merge and write task) 1.5 apply the merged diff to my one-week-old-AOI.osm 1.6 cut AOI (--bp) from patched-AOI.osm (incl. sorting and completeWays and completeRelations) 2. compare with recent AOI.osm 2.1 get recent-planet.osm 2.2 cut AOI (--bp) from recent-planet.osm (incl. sorting and completeWays and completeRelations) => expectation: there should be no differences comparing the recent-AOI.osm and the patched-AOI.osm 3. update using applying the diffs step by step 3.1 get one-week-old-planet.osm 3.2 cut AOI (--bp) from one-week-old-planet.osm (incl. sorting and completeWays and completeRelations) 3.3 get all the daily-diffs from last week to now 3.4 apply and cut to one-week-old-AOI.osm for each daily-diff (always incl. sorting and completeWays and completeRelations) => multipatched-AOI.osm => expactation: there should be no differences comparing any pair of recent-AOI.osm, patched-AOI.osm and the multipatched-AOI.osm But this workflow produces three different "*-AOI.osm" For the tests I used osmosis from trunk (rev. 20665), because there was a change focusing the completeRelations feature. Marco Lars Lingner schrieb: > John Smith schrieb: > >> On 25 March 2010 17:54, Frederik Ramm wrote: >> >>>is the osm2pgsql bbox filter also usable for applying diffs (in a >>> situation where somebody wants to keep a country extract in osm2pgsql >>> form up to date), or does it only work for full imports? >>> >> I think it only filters nodes, I haven't checked for a while but >> previously I had ways with no nodes in the database outside the area I >> was filtering... >> >> > > What would be the best way to keep an individual extent updated? > > Fetching the planet, extracting the extent of the area of interest and > create my own diffs? > > I could also apply the planet wide diffs and remove everything outside > of the extent afterwards. But I don't like this idea. > > Is there a way to process the diff files in a way that the diff only > contains changes in my area of interest? > > Thanks for any hints. > > > Lars > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql bbox filter when applying diffs?
Hi, Peter Körner wrote: >> Is there a way to process the diff files in a way that the diff only >> contains changes in my area of interest? > If you're fetching the diffs with "osmosis --rrii --wxc", can't you plug > a --bb task afzter that? Or does this task not work with > changeset-streams? The latter. > If so, this would be the place to implement this > functionality, wouldn't it? No. Imagine a relation with 500 members and someone changes a tag. The relation itself will be contained in the osmChange file, but none of the members will because they are unchanged. Any task handling the osmChange file will not know anything about the spatial extent of the relation. This would only work if the geometry of ways and relations was included in the osmChange file, significantly increasing their size - or if the process which wants to cut a section out of the osmChange file has access to the underlying data. With osmosis you can do a --rrii --rx --ac --bb --wx to read your xml, apply the change, cut out the area of interest (again) and save it. But that is impractical for something like minutely diff application. It would have to be done in osm2pgsql - when applying a change, it would have to do the following: * if change type = modify or delete: apply change as long as object to be modified or deleted is present in local data base (I think it does this already) * if change type = create and object type = node: apply change only if node is within bbox * if change type = create and object type != node: apply change only if referenced members are present in local data base A problem arises (as always) if you have objects that are half in and half out; they will need special treatment. There will be difficult scenarios (nodes A, B created outside bounding box; way C created to connect A and B; all these changes are ignored; later node A is moved inside the bounding box in a different osmChange file; how will you now get hold of way C which suddenly has entered your area of interest?). Bye Frederik -- Frederik Ramm ## eMail frede...@remote.org ## N49°00'09" E008°23'33" ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql bbox filter when applying diffs?
--bb or --bp doesn't work with osc (--wxc) but you can put it after --ac to cut the osm-xml to your AOI before --wx. Marco Peter Körner schrieb: >> Is there a way to process the diff files in a way that the diff only >> contains changes in my area of interest? >> > If you're fetching the diffs with "osmosis --rrii --wxc", can't you plug > a --bb task afzter that? Or does this task not work with > changeset-streams? If so, this would be the place to implement this > functionality, wouldn't it? > > Peter > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] osm2pgsql bbox filter when applying diffs?
Hi > No. Imagine a relation with 500 members and someone changes a tag. The > relation itself will be contained in the osmChange file, but none of the > members will because they are unchanged. Any task handling the osmChange > file will not know anything about the spatial extent of the relation. Hm, sorry, didn't think about this.. > apply the change, cut out the area of interest (again) and save it. But > that is impractical for something like minutely diff application. > > It would have to be done in osm2pgsql - when applying a change, it would > have to do the following Tobias Wendorff recently said, that he implemented sth. like this. Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis
I wrote a draft of my GSoC application. It is available on wiki [1]. I tried to define goals more precisely. Any comments are appreciated. [1] http://wiki.openstreetmap.org/wiki/Google_Summer_of_Code/2010/Student_Applications#Draft_application:_Travel_Time_Analysis -- Lukas Kabrt ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] hstore / db schema Was: Query-Formats
On Thu, Mar 25, 2010 at 3:42 AM, Florian Lohoff wrote: > On Tue, Mar 23, 2010 at 01:30:04PM +0100, Peter Körner wrote: > > Hi > > > > On the osm dev-server we're playing around with the postgres-hstore > > extension which allows us to repond to xapi-like queries, but with > > already assembled LIESTRINGs and such neat things. > > Which is not per se a feature of hstore ... hstore is just another > data type which is neat because it fits into the OSM model concerning > tags ... > > I am currently building most of my applications around the osmosis > db schema but the split into node and node_tags and way and way_tags > is - hum - not very efficient. > The current schema is fairly naive in its implementation. I hadn't heard of hstore when implementing it. Perhaps the current schema needs an upgrade :-) > This is why i would be interested in some tools to import a planet/sub > extract > into a more efficient db scheme which still allows to store all details > of the osm planet (which osm2pgsql does not) but is more efficient > concerning spatial queries. > > E.g. i'd like to drop way_tags and put them into a hstore tags column in > the > way table and then cluster the way table with the geom index on the > linestring > for example. > > Does anyone have speed/place comparisons? > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] hstore / db schema Was: Query-Formats
Brett Henderson schrieb: > I am currently building most of my applications around the osmosis > db schema but the split into node and node_tags and way and way_tags > is - hum - not very efficient. > > The current schema is fairly naive in its implementation. I hadn't > heard of hstore when implementing it. Perhaps the current schema needs > an upgrade :-) I'd suggest collecting more info about hstore-performance before changing anything. We're on the planet.import on dev-server and there's also a ticket open to get hstore-new to the wikimedia toolservers. There you can play with it and test its performance and if it's good then we should definitely give it a try! Peter ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis
> "lk" == Lukas Kabrt writes: lk> I wrote a draft of my GSoC application. It is available on wiki [1]. I lk> tried to define goals more precisely. Any comments are appreciated. Hi Lukas, Your draft looks solid to me. Some points I suggest you consider: - reuse existing libraries like osm4routing[1] for .osm parsing (week 1) - think about how you could obtain representative GPX files and filter out those that don't correspond to the vehicle type/driving pattern you are interested in (many GPX tracks in OSM are likely unrepresentative of normal driving) - perhaps consult potential consumers of your travel time - information (openrouteservice.org, navit, gosmore, etc.) to ensure that your information meets their requirements - (in a later version of your proposal) add some bibliography concerning algorithms for map matching [1] http://github.com/Tristramg/osm4routing -- Eric Marsden ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
Hi, Good to see the talk on Travel Time Analysis i would like you to have a look at http://www.wsdot.wa.gov/projects/completed.htm and also check out hypercube queuing model to have an elevated idea. Cheers, Aditya _ The world in moving pictures http://news.in.msn.com/gallery/archive.aspx___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis
On Fri, Mar 26, 2010 at 11:09 AM, Eric Marsden wrote: >> "lk" == Lukas Kabrt writes: > > lk> I wrote a draft of my GSoC application. It is available on wiki [1]. I > lk> tried to define goals more precisely. Any comments are appreciated. * tool for matching GPS traces to the roads from the OSM * tool for analyzing GPS traces and estimating travel times Are you sure that dividing it this way is reasonable? Can you explain this design decision? What is the output of the first tool? Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
On Fri, Mar 26, 2010 at 12:21 PM, Aditya Vikram Thoomati wrote: > Hi, > > Good to see the talk on Travel Time Analysis i would like you to have a > look at http://www.wsdot.wa.gov/projects/completed.htm and also check out > hypercube queuing model to have an elevated idea. Your links seems to point to a site listing 403 Projects where roads in the US have been improved or repaired. > I'm not saying it's easy. A lot of trail and error adjustments will be needed. > The main point is that the router already does a lot of the work. I still think your aproach is flawed. There should be little trial and error required as you are dealing with exact meassurements to make a a prediction giving certain of these meassurements more weight then others based on how closely they resemble the situation you want to predict. You start with gpx-traces containing many location- and speed- information. You also start with a vector map. But then why divide your traces by timestamp instead of referencing your map? 2 minutes is nothing on a motorway but lets you cross a good part of a city driving a green wave on a major road. >such as road maintenance or a car crash that cause >motorists traveling in both directions to slow down to take a peek. These events should be filtered out from the start if they do not occur every day in the same place. Expected, general travel times and delays due to random events are 2 different problems. That there was a car accident last week does not affect the estimate of how long you are likely to travel by the time you react the start of a segment now. The most likely source of information about short time traffic obstructions is TMC and it usually tells you the expected delay. So this needs different issue not be addressed now and certainly not here. Marcus ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
On Fri, Mar 26, 2010 at 6:00 PM, Marcus Wolschon wrote: > Expected, general travel times and delays due to random events > are 2 different problems. The two problems are so closely interlinked it makes sense to treat them as one. > That there was a car accident last week does not affect the estimate > of how long you are likely to travel by the time you react the start of > a segment now. Of course it does ! You have to account for the probability that the road is poorly designed at that spot. > The most likely source of information about short time traffic obstructions > is TMC and it usually tells you the expected delay. So this needs different > issue not be addressed now and certainly not here. > > Marcus > > ___ > dev mailing list > dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/dev > ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] landuse near roads (WAS: self-intersecting ways)
Let's conclude this discussion here. Thank's all for your input, I'll think about it and let's agree for now that on the long run we won't get away without highway areas. am 26.03.2010 15:36, schrieb ste...@binaervarianz.de: > I'm not sure I can follow you here. I would normally map a road as the axis > than draw the landuse border (or most of the time change the landuse > border) > with the distance to the road axis as observed in the wild. I would end up > with three distinct ways with nodes each. Yes, that's what i meant. Also see below. >>> Points and shapes are defined by coordinates, not by there relation to >>> other nodes or ways. >> Sure, in the database, but is the landuse really defined with its own >> coordinates? > > Is anything not defined by its own coordinates? I don't see the point in > making any difference here. I thought may be the landuse is defined as being next to the road on one side (which of course is an area in real life). But having looked at the Flurkarte, I think you're right: there's one area for the landuse and one *area* for the road sharing the same nodes. >>> You can't rely on the renderer to expand the road and correct your >>> misplaced landuse. >> You have to, thats generalization and it's necessary for any map with a >> scale lesser than say 1:5000. > > 'Scales' belong to paper maps. Generalization was a must back then. > Digital map data has no scale. A renderer may produce an image with a > scale, but the generalization is then part of the renderer, not of the map > data. > 'Don't map for the renderer' is one of the basic principles in OSM. Let the > renderer evolve, don't change the data for it. I didn't think of it as mapping for the renderer but as generalization at the time of survey to not make things complicated (for me). This means I don't have to measure the road width exactly and make sure the landuse has the correct distance. I'm satisfied with knowing "there we have landuse=residential next to the road." >>> What's the logic for mapping then? I map to the border of the landuse if >>> it >>> stands on its on, but to the next road if there is any? >> Yes. > > Ew. Another more general description would be: map to the feature next to the landuse or to the real border of the landuse if there is nothing (mapped) next to it. >>> What if the road gets deleted or moved? Should the landuse be moved > also? >> You mean if the physical road changes? Of course. It has to be re-mapped >> then. >> If the road still exists and the way is deleted it's an error and the >> way should be restored. If the way must be moved because it was not >> precise you simply move it and it stays correct. Whereas with your >> approach you would have to move 3 different nodes at once which becomes >> a problem in bends, because the radius changes. >> I've often seen roughly mapped forest (landsat, ~20m maximum precision >> with calibration) next to a road. Now someone moves the road in >> direction of the forest because it was imprecise, but does not move the >> forest. You now have forest to both sides of the road, which is really >> wrong. >> Had both been glued together it would still be correct. > > I know the problem you describe here. And I acknowledge that it is more > work to move all the nodes. > But all those borders can have a very different level of accuracy and I > would like to leave each of them to be changed seperatly. > Just because I know (through a GPS track) where a road was, I don't > necessarily have paid attention to the landuse around it. > So I may correct the road, but I would leave it to someone with more > knowledge to change the landuse borders around. > Maybe they were far more accurate then the road? Who knows? OK. >>> Wherever I've seen landuse mapped near a road, it was mapped to its >>> physical extends, leaving free space to the single way defining the >>> center line of the road. >>> If anyone decides to map the road as an area later, he don't have to >>> touch the landuse. >> That's a point though, but we're very far from that. > > Maybe it's because someone said "we need this generalization" and "routings > tools / renderers don't know what to do with it". > I'm not a friend of doing something wrong just to have a convenient and > fast solution. In my work this would be called > a 'hack' and it usually explodes in your face half a year later. Maybe it's because someone said "we don't need this degree of precision, so let's just use a way". >>> Or take the practical approach: If you are standing on asphalt, but your >>> maps says you're standing on grass, who is wrong? >> They are both right, but you are using the wrong map for the task. You >> either need a map with generalization or you shouldn't demand such a >> high precision. Remember, we are using comsumer-grade GPS. Precision >> higher than say 3m will require high effort with these devices. > > Again: there is no such thing as 'the wrong map' regarding the data. > You probably still
Re: [OSM-dev] GSoC - Travel Time Analysis (classification aproach of Speedcollector)
On Fri, Mar 26, 2010 at 5:47 PM, Nic Roets wrote: > On Fri, Mar 26, 2010 at 6:00 PM, Marcus Wolschon > wrote: >> Expected, general travel times and delays due to random events >> are 2 different problems. > > The two problems are so closely interlinked it makes sense to treat them as > one. No. One requires statistical data of all of the road-network indexed by the specific road to make predictions about the infuence of this specific piece of road compared to the average road of that kind. The second requires statistical data of popular incidents that can happen everywhere and have such a large impact that the specific road does not matter. A 5Km trafic jam on a motorway of 2 lanes getting narrowed down to 1 lane will have the same impact no matter what the name of the specific motorway is. >> That there was a car accident last week does not affect the estimate >> of how long you are likely to travel by the time you react the start of >> a segment now. > > Of course it does ! How so? There IS no car-accident now and it is unrealistic to assume that there will be by the time you get there. Don´t mistake the average travel time with the statistical expected value for a single, random drive. > You have to account for the probability that the > road is poorly designed at that spot. I don´t think that the road has anything to do with it. To account for the probability you have to 1) detect an instance of a specific type of event 2) know the probability I don´t think 1 is possible with this kind of data. And I don´t think we´ll have the number of tracks for 2 in any meaningful number of places to merit development time. (Remember, we are short of developers everywhere in this project.) >> The most likely source of information about short time traffic obstructions >> is TMC and it usually tells you the expected delay. So this needs different >> issue not be addressed now and certainly not here. ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
Re: [josm-dev] self-intersecting ways
On 26/03/2010 11:57, Frederik Ramm wrote: > 1. Draw areas using same nodes as road - wrong because it suggests the > area is larger than it really is. Sometimes this is the only option. Not all ways are two-lane, asphalt roads; sometimes they're narrow footpaths. If you look here: http://osm.org/go/eurTZdy5p- you'll see a residential area that shares nodes with a highway=footway. The path in question is less than a metre wide, so using separate nodes for the landuse=residential and highway=footway would be messy and just as hard to edit. Besides, the edge of the residential area *is* the footpath. If at some point I resurvey the area and find I got the position of the path wrong and decide to move it, the boundary of the residential area should move too. -- Jonathan (Jonobennett) ___ josm-dev mailing list josm-...@openstreetmap.org http://lists.openstreetmap.org/listinfo/josm-dev
Re: [OSM-dev] Removing Minutely and Hourly Changesets
Hi Peter, >>> The heavies task is 1. Maybe we could set up a database for this >>> task.. >>> Peter >> >> I rather liked Brett's earlier suggestion to have this solved on the >> diff >> server itself. >> >> So you could "wget http://diffserver/state/minute/20100301_00";, >> and >> the web service would find the right state file and return it to you. >> > I'm currently setting up such a service, may be already working > tonight. I just wanted to give this a try. Is this already working? What do I use for diffserver? Thanks, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
[OSM-dev] Hourly/Daily Updates
Hi all, >> I'll leave the daily changesets running for now because they are >> running with a much longer delay and shouldn't miss data. I need to incorporate changes into our map server and am currentlöy building on http://planet.openstreetmap.org/daily and plan down make a wget on wget http://planet.openstreetmap.org/daily/20100325-20100326.osc.gz once a day and then import that changes into our database. Is this save? Will http://planet.openstreetmap.org/daily be maintained in the future. Hourly files would be even better for us. Is this maintained, if so where. It seems to be disabled right now!? Thanks a lot, Andreas ___ dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
