Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
On Thu, May 13, 2010 at 09:22:49AM -0700, Martin Davis wrote: Just tried the buffer-and-union approach in JTS. It works fine - about 70 s for the union, less for the initial buffering. Memory usage was low. I tried buffer-and-union approach in PostGIS. It was very fast ! Some stats: - Original geometry is a MULTILINESTRING with 3272 components - Average number of points per component: 3.3 (from 2 to 93) - Total number of points: 10677 Postgis operations with timing: $ time psql -c 'create table bug_dump2 as select (st_dump(g)).* from bug;' real0m0.330s user0m0.032s sys 0m0.012s $ time psql strk -c 'update bug_dump2 set geom = st_buffer(geom, 0.005)'; UPDATE 3272 real0m0.463s user0m0.040s sys 0m0.008s $ time psql strk -Ac 'select st_npoints(st_union(geom)) from bug_dump2'; st_npoints: 1336 real0m1.181s user0m0.040s sys 0m0.008s This is postgis trunk (to be 2.0.0) with GEOS trunk (to be 3.3.0). I'm surprised it takes 70 seconds with JTS. Am I doing anything wrong here ? --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
On Fri, May 14, 2010 at 03:09:40PM +0200, strk wrote: On Thu, May 13, 2010 at 09:22:49AM -0700, Martin Davis wrote: Just tried the buffer-and-union approach in JTS. It works fine - about 70 s for the union, less for the initial buffering. Memory usage was low. I tried buffer-and-union approach in PostGIS. It was very fast ! Some stats: - Original geometry is a MULTILINESTRING with 3272 components - Average number of points per component: 3.3 (from 2 to 93) - Total number of points: 10677 Postgis operations with timing: $ time psql -c 'create table bug_dump2 as select (st_dump(g)).* from bug;' real0m0.330s user0m0.032s sys 0m0.012s $ time psql strk -c 'update bug_dump2 set geom = st_buffer(geom, 0.005)'; UPDATE 3272 real0m0.463s user0m0.040s sys 0m0.008s $ time psql strk -Ac 'select st_npoints(st_union(geom)) from bug_dump2'; st_npoints: 1336 real0m1.181s user0m0.040s sys 0m0.008s This is postgis trunk (to be 2.0.0) with GEOS trunk (to be 3.3.0). I'm surprised it takes 70 seconds with JTS. Am I doing anything wrong here ? Note that postgis uses the cascaded union... --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
I'm surprise that PostGIS is so much faster too! Can you supply the output, so I can check it against JTS? One thing I notice is that the input I'm using has 5615 elements and 11502 points. This *might* make a big difference, if you are missing some of the bigger linestrings which (a) have large buffer polygons and (b) intersect *all* of the other elements' buffers. strk wrote: On Thu, May 13, 2010 at 09:22:49AM -0700, Martin Davis wrote: Just tried the buffer-and-union approach in JTS. It works fine - about 70 s for the union, less for the initial buffering. Memory usage was low. I tried buffer-and-union approach in PostGIS. It was very fast ! Some stats: - Original geometry is a MULTILINESTRING with 3272 components - Average number of points per component: 3.3 (from 2 to 93) - Total number of points: 10677 Postgis operations with timing: $ time psql -c 'create table bug_dump2 as select (st_dump(g)).* from bug;' real0m0.330s user0m0.032s sys 0m0.012s $ time psql strk -c 'update bug_dump2 set geom = st_buffer(geom, 0.005)'; UPDATE 3272 real0m0.463s user0m0.040s sys 0m0.008s $ time psql strk -Ac 'select st_npoints(st_union(geom)) from bug_dump2'; st_npoints: 1336 real0m1.181s user0m0.040s sys 0m0.008s This is postgis trunk (to be 2.0.0) with GEOS trunk (to be 3.3.0). I'm surprised it takes 70 seconds with JTS. Am I doing anything wrong here ? --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
On Fri, May 14, 2010 at 09:11:30AM -0700, Martin Davis wrote: I'm surprise that PostGIS is so much faster too! Can you supply the output, so I can check it against JTS? One thing I notice is that the input I'm using has 5615 elements and 11502 points. The input I'm using is this: http://trac.osgeo.org/geos/attachment/ticket/344/roads_broken_geom.txt.gz The output I have with it is attached. --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html out.txt.gz Description: application/gunzip ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
Ah, ok. I was using this one. http://trac.osgeo.org/geos/attachment/ticket/344/fail_geom.txt.gz I'll try the other one and see what kind of performance I get. strk wrote: On Fri, May 14, 2010 at 09:11:30AM -0700, Martin Davis wrote: I'm surprise that PostGIS is so much faster too! Can you supply the output, so I can check it against JTS? One thing I notice is that the input I'm using has 5615 elements and 11502 points. The input I'm using is this: http://trac.osgeo.org/geos/attachment/ticket/344/roads_broken_geom.txt.gz The output I have with it is attached. --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
Ok, I tried the roads_broken geometry in JTS using the buffer-and-union approach. The result matches the one you posted, strk. Time was 2.7 s. So the two systems are pretty close. It would be interesting if you tried the other geometry as well. Martin Davis wrote: Ah, ok. I was using this one. http://trac.osgeo.org/geos/attachment/ticket/344/fail_geom.txt.gz I'll try the other one and see what kind of performance I get. strk wrote: On Fri, May 14, 2010 at 09:11:30AM -0700, Martin Davis wrote: I'm surprise that PostGIS is so much faster too! Can you supply the output, so I can check it against JTS? One thing I notice is that the input I'm using has 5615 elements and 11502 points. The input I'm using is this: http://trac.osgeo.org/geos/attachment/ticket/344/roads_broken_geom.txt.gz The output I have with it is attached. --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
On Fri, May 14, 2010 at 11:35:45AM -0700, Martin Davis wrote: Ok, I tried the roads_broken geometry in JTS using the buffer-and-union approach. The result matches the one you posted, strk. Time was 2.7 s. So the two systems are pretty close. It would be interesting if you tried the other geometry as well. Done, but on another (older) machine: Intel(R) Pentium(R) 4 CPU 2.66GHz bogomips: 5333.43 Input points: 11502 Input components: 5615 Output points: 417 $ time psql -c ' select st_npoints(st_union(st_buffer(geom, 0.005))) from ( select (st_dump(g)).geom from bug ) as foo; ' real1m9.838s user0m0.100s sys 0m0.028s Memory is fine, CPU is pretty much fixed at 100%. Interesting enough the buffer itself is pretty quick: $ time psql -c ' select sum(st_npoints(st_buffer(geom, 0.005))) from ( select (st_dump(g)).geom from bug ) as foo;' real0m3.170s user0m0.108s sys 0m0.024s --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
Ok, then that's in the same ballpark as JTS. And yes, the union is much slower than the buffer creation. Interesting that GEOS seems to have caught up to JTS in terms of performance. I seem to remember it used to be slower. (But it's not really any faster, either, in spite of all that C goodness! 8^) strk wrote: On Fri, May 14, 2010 at 11:35:45AM -0700, Martin Davis wrote: Ok, I tried the roads_broken geometry in JTS using the buffer-and-union approach. The result matches the one you posted, strk. Time was 2.7 s. So the two systems are pretty close. It would be interesting if you tried the other geometry as well. Done, but on another (older) machine: Intel(R) Pentium(R) 4 CPU 2.66GHz bogomips: 5333.43 Input points: 11502 Input components: 5615 Output points: 417 $ time psql -c ' select st_npoints(st_union(st_buffer(geom, 0.005))) from ( select (st_dump(g)).geom from bug ) as foo; ' real1m9.838s user0m0.100s sys 0m0.028s Memory is fine, CPU is pretty much fixed at 100%. Interesting enough the buffer itself is pretty quick: $ time psql -c ' select sum(st_npoints(st_buffer(geom, 0.005))) from ( select (st_dump(g)).geom from bug ) as foo;' real0m3.170s user0m0.108s sys 0m0.024s --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
Yes, that would be very interesting. Does GGL provide polygon union now? strk wrote: On Fri, May 14, 2010 at 02:28:39PM -0700, Martin Davis wrote: Interesting that GEOS seems to have caught up to JTS in terms of performance. I seem to remember it used to be slower. (But it's not really any faster, either, in spite of all that C goodness! 8^) That's the price to pay for closely tracking JTS : Would be interesting to see how GGL deals with this dataset (the union part). --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
On Fri, May 14, 2010 at 03:05:29PM -0700, Martin Davis wrote: Yes, that would be very interesting. Does GGL provide polygon union now? Seems so: http://geometrylibrary.geodan.nl/group__union.html --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
Yikes! That's one nasty dataset. JTS has issues with it as well - I wasn't able to get the buffer to complete. As a workaround, you could buffer each line separately and then UnaryUnion the results. GEOS wrote: #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs +--- Reporter: sberna | Owner: geos-devel@lists.osgeo.org Type: defect | Status: new Priority: major | Milestone: Component: Default | Version: 3.2.0 Severity: Unassigned |Keywords: +--- After applying a buffer of 0.005 to the attached geometry in hex wkb the library starts growing in memory usage until the kernel kills it. It never returns the buffered geometry nor any error message. ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
Re: [geos-devel] [GEOS] #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs
Just tried the buffer-and-union approach in JTS. It works fine - about 70 s for the union, less for the initial buffering. Memory usage was low. This geometry is pretty much a poster child for things that defeat the optimizations used in buffering. It has lots of small geometries, thus preventing simplification. And the buffer size chosen is large relative to the geometry size, thus creating lots of overlaps. At the moment I haven't spent much time thinking about how to detect these cases automatically. Of course it would be possible to switch to using an automated buffer-and-union approach if this could be detected. But this seems like such a pathological case I'm not sure it's worth penalizing all other buffer users just to handle it. In any case, it would be worth providing a buffer-and-union algorithm as a builtin class. This could be further optimized by using the same approach as CascadedPolygonUnion to produce a CascadedBuffer class. This would reduce the amount of intermediate memory required (but would not provide any direct performance advantage.) M Martin Davis wrote: Yikes! That's one nasty dataset. JTS has issues with it as well - I wasn't able to get the buffer to complete. As a workaround, you could buffer each line separately and then UnaryUnion the results. GEOS wrote: #343: After applying a buffer over a multiline the library starts growing in memory usage and hungs +--- Reporter: sberna | Owner: geos-devel@lists.osgeo.org Type: defect | Status: new Priority: major | Milestone: Component: Default | Version: 3.2.0 Severity: Unassigned |Keywords: +--- After applying a buffer of 0.005 to the attached geometry in hex wkb the library starts growing in memory usage until the kernel kills it. It never returns the buffered geometry nor any error message. ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel -- Martin Davis Senior Technical Architect Refractions Research, Inc. (250) 383-3022 ___ geos-devel mailing list geos-devel@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/geos-devel
