I don't think there's any simple rule of thumb when it comes to buffering.
It depends a lot on the use case and the system that you're running on.
Different combinations of tasks, core count, and IO subsystems all make a
big difference.
PBF extraction is very "bursty" where large numbers of entities get pushed
through the pipeline at once, then a pause occurs while the next block is
extracted. For that reason placing a buffer after the PBF task is
advisable. Beyond that it's hard to say. I'd try without a buffer first,
measure, then see if adding additional buffers makes a difference. The
buffer task performs a dual role, it buffers data, but it also de-couples
processing between tasks into separate threads. Most non-read tasks are
passive in that they don't run on their own thread.
The best approach if you have time is to measure individual thread CPU
utilisation within the JVM and determine which threads are maxed out.
There is a plugin for the jconsole application that helps here, but I don't
have a link handy. Adding new threads isn't a free lunch, the overhead of
data moving between threads (and associated CPU caches) is very
noticeable. The --buffer task attempts to make these data movements chunky
to avoid too much synchronisation overhead, but it's not perfect.
On 23 January 2015 at 03:59, dave pitney <mappit...@gmail.com> wrote:
> Hello -
>
> I have the following osmosis command and am wondering if I am using
> --buffer correctly:
>
> osmosis -v \
>
> --read-pbf-fast .../{var}D.osm.pbf workers=7 \
> --buffer bufferCapacity=12000 \
> --bounding-polygon file=".../$var.poly" \
> --buffer bufferCapacity=12000 \
> --write-pbf .../${var}D.osm.pbf omitmetadata=true granularity=10000
>
> I think I am buffering twice -
>
> once from read-pbf to the bounding polygon and second from the
> bounding polygon to write-pbf.
>
> (IMHO a few examples in osmosis detailed usage would be helpful rather
> than just a description)
>
> Thanks, pitney
>
>
>
> --
> pitney
>
> ----------------- end of message ------------------------
>
> _______________________________________________
> osmosis-dev mailing list
> osmosis-dev@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/osmosis-dev
>
_______________________________________________
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/osmosis-dev
