Re: [osmosis-dev] [OSM-dev] Proposal for a multithreaded PBF reader
Hello, Can anyone provide anecdotes of use cases where multi-threaded PBF reading significantly speeds up processing? Generally I would expect PBF reading to be IO bound rather than processor bound, but I still need to make more accurate measurements. Of course actually processing the OSM data once the PBF is decoded can be quite CPU intensive, but that would imply buffering decoded data and parallelizing geometric operations for example, not the reading. I’d appreciate any data points and example use cases you might have, as I’m currently working on related tooling. Andrew Byrd On 04 Jun 2015, at 05:57, Brett Henderson br...@bretth.com wrote: On 30 April 2015 at 03:27, Paul Norman penor...@mac.com mailto:penor...@mac.com wrote: On 4/29/2015 9:55 AM, Martijn van Exel wrote: If osmosis is the reference implementation, is there a reason why it doesn't seem to leverage this block structure to speed up reading? Or does it? Osmosis has the --read-pbf-fast task which allows multiple worker threads. That's right. I forget how the PBF structure works off the top of my head, but the file is already split into blocks. The main --read-pbf-fast thread simply grabs the outer protobuf blocks from file and then distributes them to worker threads who parse out the OSM entities from within the block. After extraction, the entities within each block are passed to the downstream task in original file order. I'm not sure I see the need to modify the PBF file format. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org https://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Problems with used-way
I just tried the command, and the version of Osmosis you downloaded does provide the used-way filter. This could just be a path issue -- are you sure you are runing the new version you just downloaded and not an older one (pre-0.39) that is installed elsewhere on your system? -Andrew On 12/11/2011 05:17 PM, Janko Mihelić wrote: I found this argument on several places on the net, but it refuses to work for me. Is there something else I should do first? The whole command goes like this: osmosis --read-xml file=croatia.osm --tag-filter accept-relations ref=D1 --used-way --used-node --write-xml output.osm And it says: Task type used-way doesn't exist. It works for other cases, when used-way isn't in. I downloaded osmosis from this link: http://bretth.dev.openstreetmap.org/osmosis-build/osmosis-latest.tgz version 0.40.1 What am I doing wrong? Janko ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] osmosis pipelining troubles
Hello Marcus, Using tee followed by merge can trigger deadlock, and I would guess this is what's happening in your case: http://lists.openstreetmap.org/pipermail/dev/2009-June/015945.html I usually get around the problem by using separate read-xml tasks for each sub-pipeline that will be merged at the end of the pipeline. See the last example in this section of the Osmosis wiki: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage_0.38#--tag-filter_.28--tf.29 -Andrew On 12/07/2011 10:00 AM, MERIGHI Marcus wrote: Hello, while trying to integrate various processing steps into a rather complex pipeline I have run into troubles. Each step alone seems to work, even more complex arrangements, but not all of them. (Go to bottom of message (3) to see an working example with multiple invocations of osmosis.) The osmosis commands below (1) stop at FINE: Waiting for task 1-read-xml to complete., probably forever. CPU utilization goes up first and then slowly down to zero, fstat does not show any read activity on the input file (anymore). I have tried to pin down at what point the processing hangs; everything before --derive-change works, tested with --write-null[-change]. The questions are: 1) Am I getting something fundamentally wrong about the way osmosis pipes work? 2) Is there any obvious mistake in my parameters? 3) Have I hit a bug? (BTW, basicly I am after simplifying ways; but --simplify-ways leaves (now) unused nodes behind; getting rid of them by means of --used-node removes poi-type nodes as well (not used in ways and relations). Therefore I have to go the long way of diffing before- and after --used-node, parse that for used (i.e., poi-type) nodes and merge.) Thanks in advance for any pointers! Marcus ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] osmosis pipelining troubles
Hi again Marcus, Rereading your post, it sounds like you want your output to contain only simplified ways and points of interest. I think you can get by with a much simpler pipeline. Something of this form might do the job: osmosis \ --rx input.osm \ --tf reject-relations \ --simplify-ways epsilonMeters=N \ --used-node outPipe.0=WAYS \ \ --rx input.osm \ --tf reject-relations \ --tf accept-nodes amenity=* \ --tf reject-ways outPipe.0=POI \ \ --merge inPipe.0=WAYS inPipe.1=POI \ --wx merged.osm Hope this helps, Andrew On 12/07/2011 10:00 AM, MERIGHI Marcus wrote: Hello, while trying to integrate various processing steps into a rather complex pipeline I have run into troubles. Each step alone seems to work, even more complex arrangements, but not all of them. (Go to bottom of message (3) to see an working example with multiple invocations of osmosis.) The osmosis commands below (1) stop at FINE: Waiting for task 1-read-xml to complete., probably forever. CPU utilization goes up first and then slowly down to zero, fstat does not show any read activity on the input file (anymore). I have tried to pin down at what point the processing hangs; everything before --derive-change works, tested with --write-null[-change]. The questions are: 1) Am I getting something fundamentally wrong about the way osmosis pipes work? 2) Is there any obvious mistake in my parameters? 3) Have I hit a bug? (BTW, basicly I am after simplifying ways; but --simplify-ways leaves (now) unused nodes behind; getting rid of them by means of --used-node removes poi-type nodes as well (not used in ways and relations). Therefore I have to go the long way of diffing before- and after --used-node, parse that for used (i.e., poi-type) nodes and merge.) Thanks in advance for any pointers! Marcus ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Maven artifacts, using Osmosis as a library, OSMembrane
Brett and Igor, Thanks for your replies last week about using Osmosis as a library and a Maven dependency. I hadn't spotted the Maven branch on the Github repository so I appreciate you pointing that out, and I will be looking to the tests for usage hints as you suggested, Igor. With your comments in mind, some Open Trip Planner developers have discussed this option and we would prefer to eventually add a dependency on Osmosis rather than duplicate functionality. For the moment we will just be using Osmosis as an external tool in scripts, but we would be happy to contribute to the Mavenization effort and may even be able to offer space on a Maven repository to host the artifacts. Just let me know if we can be of assistance. -Andrew On 11/28/2011 02:49 AM, Brett Henderson wrote: On 27 November 2011 13:09, Andrew Byrd and...@fastmail.net mailto:and...@fastmail.net wrote: Hi Brett, Igor, and the rest of the list, Rather than reinvent the wheel I am considering using an Osmosis pipeline for bounding polygon and tag filtering as well as PBF support in the Open Trip Planner project. To do so we would need to add a Maven dependency on (parts of) Osmosis. I see that there is a Maven POM in the core Osmosis module, and a repository at http://oss.sonatype.org/__content/repositories/__openstreetmap-snapshots http://oss.sonatype.org/content/repositories/openstreetmap-snapshots but it contains only version 0.32.1-SNAPSHOT. I find only the Osmosis-core, without the other subprojects. Are artifacts still published somewhere? I didn't know that existed! I suspect Hakan did it a while back. He created the POM file that currently exists in the Osmosis distribution. Unfortunately I broke the Maven build horribly when I split Osmosis into multiple sub-projects in version 0.36. I've been investigating publishing the Osmosis artefacts via Maven but it's probably a long way off. I've re-factored the project structures to follow a more maven-like layout, and I've re-factored tests to eliminate dependencies on Ant-based test data preparation. Those changes are already in the main github repo. https://github.com/openstreetmap/osmosis I've also started experimenting with a Maven based build, but it's early days and I don't know how far I'll get. https://github.com/brettch/osmosis/tree/mvn I see that there has also been some discussion on the list about using Osmosis as a library (quoted message below). Igor, based on a quick look at PipelineExecutor in the head revision of OSMembrane, it looks like you are calling Osmosis as an external program. Are you still planning to tighten the integration with Osmosis? What has been your experience using Osmosis in this way? In theory it shouldn't be difficult to use Osmosis as a library. All tasks can be invoked in isolation without using the entire command-line driven pipeline. The projects could probably be structured more effectively, but worst case you'll end up with some larger binaries. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] question about --merge
Hello, On 19 May 2010, at 13:42, Brett Henderson wrote: The --tag-filter task is about the most comprehensive task currently available. To add a full boolean expression language is certainly possible, but not trivial. If somebody wants to take on the challenge of creating a complex yet generic tag analysis task then I'll do my best to help them out. I'll be honest though, it's not likely to be something I'll get around to implementing myself. Part of the challenge is that tags and values can contain any character, so your expressions get cluttered with escape sequences, and also must pass through the shell. Another issue is running a used-node filter that would 'know' not to delete the non-way nodes that you want. It's pretty complicated to unambiguously specify this kind of behaviour, unless the used-node filter is integrated into the tagfilter. I'm still open to tagfilter syntax and functionality suggestions. You can do many different things with the current version, but you have to spell it all out explicitly on the command line, which I don't necessarily consider a bad thing. Thinking about your case a bit more, you could avoid the temp file if you simply read the input file twice so you avoid the --tee. Although that might be slower than creating a temp file and invoking Osmosis twice. If it was me I'd simply create a wrapper shell script to tie several osmosis commands together and do some checking of the Osmosis return code to ensure it hasn't failed at each step. The --buffer approach might work, but the memory consumption is likely to bite you at some point. This is the solution I came up with too. I didn't notice any significant slowdown with this approach on a city-sized OSM file. I'm re-including the example command I gave before, because I messed up a parameter the last time: ./osmosis/bin/osmosis \ --rx input.osm \ --tf reject-relations \ --tf accept-nodes amenity=* \ --tf reject-ways outPipe.0=POI \ \ --rx input.osm \ --tf reject-relations \ --tf accept-ways highway=motorway \ --used-node outPipe.0=motorway \ \ --merge inPipe.0=POI inPipe.1=motorway \ --wx test-merge.osm The input file must be sorted for the merge to work right (it should be sorted if it comes from planet.osm, if not you can add a sort task.) -Andrew Byrd ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] TagFilter
On 24 Nov 2009, at 12:24, Andrew Byrd wrote: On 24 Nov 2009, at 11:32, Peter Körner wrote: Can I filter for nodes with a specific tag OR nodes in a way with a specific tag or does --used-node throw these POI nodes away? Eg. I want all nodes with the amenity tag set and all ways with the highway tag set plus their nodes. OK, it does work... but you cannot use Tee and Merge tasks simultaneously: http://www.mail-archive.com/d...@openstreetmap.org/msg07991.html The problem is that both inputs for the merge are coming from the same thread. It does deadlock (never finishes) if you use the tee-then- merge approach, I just tried it. One solution is to read the data in two separate tasks: ../osmosis/bin/osmosis \ --rx input.osm \ --tf reject-relations \ --tf accept-nodes amenity=* \ --tf reject-ways outPipe.0=POI \ \ --rx trimet.osm \ --tf reject-relations \ --tf accept-ways highway=motorway \ --used-node outPipe.0=motorway \ \ --merge inPipe.0=POI inPipe.1=motorway \ --wx test-merge.osm The output file does seem to contain only motorways plus their nodes, and amenity nodes. The input file must be sorted for the merge to work right (it should be sorted if it comes from planet.osm, if not you can add a sort task.) I am thinking about how to simplify the syntax, but the problem is that there is no such thing as an OSM highway node versus an amenity node, just nodes which can have any number of tags, so for now it's up to the Osmosis user to clearly state how the entity pipes are to be split and merged. I know the command line is long, but it shows in detail what's happening to your entities, which helps avoid unexpected results. -Andrew ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] TagFilter
On 24 Nov 2009, at 13:13, Andrew Byrd wrote: ../osmosis/bin/osmosis \ --rx input.osm \ --tf reject-relations \ --tf accept-nodes amenity=* \ --tf reject-ways outPipe.0=POI \ \ --rx trimet.osm \ --tf reject-relations \ --tf accept-ways highway=motorway \ --used-node outPipe.0=motorway \ \ --merge inPipe.0=POI inPipe.1=motorway \ --wx test-merge.osm The input file should be the same in both --rx tasks. I changed the first one to make a generic example and missed the second one. -Andrew ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
