Hi, I prepared a pull request with the suggested optimizations, if someone
wishes to have a look at it: https://github.com/geotools/geotools/pull/450
Mauro
2014-04-24 11:05 GMT+02:00 Jody Garnett <jody.garn...@gmail.com>:
> All your suggestions sound good, the locking manager is intended to be
> pluggable, so you can implement locking in SQL if your database supports it.
>
>
> Jody Garnett
>
>
> On Thu, Apr 24, 2014 at 6:53 PM, Mauro Bartolomeoli <
> mauro.bartolome...@geo-solutions.it> wrote:
>
>> Hi,
>> recently we hit a performance issue using the JDBCFeatureStore
>> removeFeatures method.
>>
>> The problem is in the JDBCDataStore ensureAuthorization call that is
>> checking if all the features can be removed (no locks on them).
>>
>> We launched a removeFeatures with a Filter.INCLUDE filter on a 1.000.000
>> records table, that should simply remove all the features, so we thought
>> that should be quite fast (if the database is fast enough), but:
>> - the ensureAuthorization executes a sql to extract all the records and
>> check for locks record by record
>> - if prepared statements are not used a setFetchSize is not executed, so
>> on some databases (for example PostGIS) the entire recordset is loaded into
>> memory (and probably throwing an OOM on big datasets)
>>
>> I was thinking of some ways to improve this stuff. My ideas would be:
>> - first of all we should use setFetchSize also on simple Statemenst, not
>> only on PreparedStatements (this would avoid the OOM)
>> - we should try to optimize, when possible the locks check, for example:
>> * if there are no locks on the featuretype the record by record check
>> is not needed
>> * we could try to only extract records that match the given filter
>> combined with a filter from the fids locked on the given featuretype and
>> check only those records
>>
>> Both would require some changes to InProcessLockingManager class, for
>> example making the locks method public (currently is protected).
>>
>> What do you think?
>> Thanks
>> Mauro
>>
>> --
>> ==
>> Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
>> for more information.
>> ==
>>
>> Dott. Mauro Bartolomeoli
>> @mauro_bart
>> Senior Software Engineer
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054 Massarosa (LU)
>> Italy
>> phone: +39 0584 962313
>> fax: +39 0584 1660272
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> -------------------------------------------------------
>>
>>
>> ------------------------------------------------------------------------------
>> Start Your Social Network Today - Download eXo Platform
>> Build your Enterprise Intranet with eXo Platform Software
>> Java Based Open Source Intranet - Social, Extensible, Cloud Ready
>> Get Started Now And Turn Your Intranet Into A Collaboration Platform
>> http://p.sf.net/sfu/ExoPlatform
>> _______________________________________________
>> GeoTools-Devel mailing list
>> GeoTools-Devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/geotools-devel
>>
>>
>
--
==
Meet us at GEO Business 2014! in London! Visit http://goo.gl/fES3aK
for more information.
==
Dott. Mauro Bartolomeoli
@mauro_bart
Senior Software Engineer
GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: +39 0584 962313
fax: +39 0584 1660272
http://www.geo-solutions.it
http://twitter.com/geosolutions_it
-------------------------------------------------------
------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
