Re: [osmosis-dev] question about --merge
On Wed, May 19, 2010 at 4:29 AM, Christoph Wagner freemaps@googlemail.com wrote: Am 18.05.2010 15:18, schrieb Brett Henderson: To summarise, if you can draw a graph of the data flows in Osmosis, ensure that you never have data being split then recombined at a later point. In your case the only option is to split it into two steps using temporary XML files as you already appear to be doing. It would be nice if Osmosis could detect this situation and throw an error, but it would take a lot of effort and add a lot of complexity that I don't really wish to attempt. Ok, thanks for your reply. I think now I understand the situation better. While reading your answer I got an idea to solve the problem without writing to disk. I use a very big buffer now: osmosis-0.35.1/bin/osmosis --rx input.osm --t outPipe.1=points \ --wk keyList=addr:interpolation,addr:housenumber --un --s --b 1 \ --nk inPipe.0=points keyList=addr:housenumber --s --b 1 \ --m --wx addr.osm This works very well if you have enough RAM and set the bufferCapacity high enough. Yes, that will also work if you're prepared to experiment a bit with buffer sizes. Another possibility would be a better tagfilter that allows me to filter POIs and ways at once. The task would be for example: Give me all ways that match this way-filter and all nodes that match this node-filter or are part of the way that matches the way-filter The --tag-filter task is about the most comprehensive task currently available. To add a full boolean expression language is certainly possible, but not trivial. If somebody wants to take on the challenge of creating a complex yet generic tag analysis task then I'll do my best to help them out. I'll be honest though, it's not likely to be something I'll get around to implementing myself. Thinking about your case a bit more, you could avoid the temp file if you simply read the input file twice so you avoid the --tee. Although that might be slower than creating a temp file and invoking Osmosis twice. If it was me I'd simply create a wrapper shell script to tie several osmosis commands together and do some checking of the Osmosis return code to ensure it hasn't failed at each step. The --buffer approach might work, but the memory consumption is likely to bite you at some point. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] question about --merge
Hello, On 19 May 2010, at 13:42, Brett Henderson wrote: The --tag-filter task is about the most comprehensive task currently available. To add a full boolean expression language is certainly possible, but not trivial. If somebody wants to take on the challenge of creating a complex yet generic tag analysis task then I'll do my best to help them out. I'll be honest though, it's not likely to be something I'll get around to implementing myself. Part of the challenge is that tags and values can contain any character, so your expressions get cluttered with escape sequences, and also must pass through the shell. Another issue is running a used-node filter that would 'know' not to delete the non-way nodes that you want. It's pretty complicated to unambiguously specify this kind of behaviour, unless the used-node filter is integrated into the tagfilter. I'm still open to tagfilter syntax and functionality suggestions. You can do many different things with the current version, but you have to spell it all out explicitly on the command line, which I don't necessarily consider a bad thing. Thinking about your case a bit more, you could avoid the temp file if you simply read the input file twice so you avoid the --tee. Although that might be slower than creating a temp file and invoking Osmosis twice. If it was me I'd simply create a wrapper shell script to tie several osmosis commands together and do some checking of the Osmosis return code to ensure it hasn't failed at each step. The --buffer approach might work, but the memory consumption is likely to bite you at some point. This is the solution I came up with too. I didn't notice any significant slowdown with this approach on a city-sized OSM file. I'm re-including the example command I gave before, because I messed up a parameter the last time: ./osmosis/bin/osmosis \ --rx input.osm \ --tf reject-relations \ --tf accept-nodes amenity=* \ --tf reject-ways outPipe.0=POI \ \ --rx input.osm \ --tf reject-relations \ --tf accept-ways highway=motorway \ --used-node outPipe.0=motorway \ \ --merge inPipe.0=POI inPipe.1=motorway \ --wx test-merge.osm The input file must be sorted for the merge to work right (it should be sorted if it comes from planet.osm, if not you can add a sort task.) -Andrew Byrd ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] question about --merge
Am 18.05.2010 15:18, schrieb Brett Henderson: To summarise, if you can draw a graph of the data flows in Osmosis, ensure that you never have data being split then recombined at a later point. In your case the only option is to split it into two steps using temporary XML files as you already appear to be doing. It would be nice if Osmosis could detect this situation and throw an error, but it would take a lot of effort and add a lot of complexity that I don't really wish to attempt. Ok, thanks for your reply. I think now I understand the situation better. While reading your answer I got an idea to solve the problem without writing to disk. I use a very big buffer now: osmosis-0.35.1/bin/osmosis --rx input.osm --t outPipe.1=points \ --wk keyList=addr:interpolation,addr:housenumber --un --s --b 1 \ --nk inPipe.0=points keyList=addr:housenumber --s --b 1 \ --m --wx addr.osm This works very well if you have enough RAM and set the bufferCapacity high enough. Another possibility would be a better tagfilter that allows me to filter POIs and ways at once. The task would be for example: Give me all ways that match this way-filter and all nodes that match this node-filter or are part of the way that matches the way-filter I don't know if this is difficult to implement but I think this would be usefull. Thanks Christoph signature.asc Description: OpenPGP digital signature ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] question about --merge
Hello list, I am new to this dev-list. My name is Christoph Wagner and I maintain the Garmin All in one Map: http://wiki.openstreetmap.org/wiki/All_in_one_Garmin_Map I want to reduce some server load (I omit details here) and tried to filter OSM-Data with osmosis. For example I have the problem to filter out all kind of adresses in an osm-file. My first guess was that: osmosis-0.35.1/bin/osmosis --rx input.osm --t outPipe.1=points \ --wk keyList=addr:interpolation,addr:housenumber --un --s \ --nk inPipe.0=points keyList=addr:housenumber --s \ --m --wx addr.osm The Problem is, that osmosis never finishes. I wonder why. No error, no writing, no CPU-usage - it just waits and does nothing. There must be something wrong with the --m I guess, but I don't know what. This one here does what I want, but needs to write out and read in again xml files for that: osmosis-0.35.1/bin/osmosis --rx input.osm --t outPipe.1=points \ --wk keyList=addr:interpolation,addr:housenumber --un --s --wx addr_lines.osm \ --nk inPipe.0=points keyList=addr:housenumber --s --wx addr_points.osm osmosis-0.35.1/bin/osmosis --rx addr_lines.osm --rx addr_points.osm --m --wx addr.osm Is this behaviour wanted? Can somebody give me a hint to do it better? I have no idea what is wrong. Thanks Christoph signature.asc Description: OpenPGP digital signature ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
