You can use the low-level feature writer class directly if you like.
Jody
On 22/12/2009, at 11:11 PM, César Martínez Izquierdo wrote:
> Hello again,
>
> I've tried creating a transaction and I've got good and bad news.
> On the one hand, performance has really improved (which can be mostly
> noticed for small shp sizes).
> On the other hand, if the output is a bit larger (~ 2 MB is enough), I
> can observe a performance degradation:
> - The first 4000 features take 25 seconds to be written
> - The next 4000 features take 75 seconds to be written
> - The next 4000 features take 123 seconds
> - The remaining features (about 3000) take 119 seconds
>
> Note that all the features are rectangles (so the size/complexity is
> the same for all of them).
>
> As a summary, now it takes 5 minutes to create a 2 MB SHP (14896
> features) using a ShapefileDataStore and a single transaction, which
> is the same time I got using the MemoryStore (writing it to disk
> afterwards).
> While this is faster than the initial 63 minutes, I still think it is
> really a lot of time to create a 2MB SHP.
> If I want to create a 10 million records shapefile, it would take (not
> counting performance degradation) more than 2 days.
> I think I would need to use the low-level GeoTools classes to solve this...
>
> For the record, now the code looks like this:
> create method (object initialization):
> transaction = new DefaultTransaction("addFeature");
> transactionFeatures = 0;
> store.setTransaction(transaction);
> m_featureBuilder = new SimpleFeatureBuilder(store.getSchema());
>
>
> addFeature method (which is called repeatedly):
> FeatureCollection<SimpleFeatureType, SimpleFeature> collection =
> FeatureCollections
> .newCollection();
> m_featureBuilder.add(geom); // cached feature builder
> m_featureBuilder.addAll(values);
> SimpleFeature feat = m_featureBuilder.buildFeature(null);
> collection.add(feat);
> store.addFeatures(collection);
>
> postProcess method (called just once after finishing adding features):
> transaction.commit();
> transaction.close();
>
>
> Thanks for all your guidance,
> Best regards,
>
> César
>
>
> El día 22 de diciembre de 2009 12:11, Andrea Aime <[email protected]>
> escribió:
>> César Martínez Izquierdo ha scritto:
>>
>>> That sounds more interesting for me. I'm going to try this, but I'd
>>> like to know the consequences of it.
>>> By doing so, does it mean that features are not written to disk until
>>> the transaction is complete?
>>> This is important for me, as I'll create sometimes shapefiles > 1GB,
>>> so I shouldn't do this on memory.
>>> If that is really the effect, I guess I could create a transaction
>>> every 1000 features or so.
>>
>> They are not kept in memory, we create a separate shapefile for the
>> transaction, and copy it back when the transaction is committed.
>> That is why your code is so slow, you don't set an explicit
>> transaction so an auto-commit one is created every time you do
>> addFeatures, meaning the shapefile is fully copied twice for
>> each of those calls
>>
>> Cheers
>> Andrea
>>
>>
>> --
>> Andrea Aime
>> OpenGeo - http://opengeo.org
>> Expert service straight from the developers.
>>
>
>
>
> --
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> César Martínez Izquierdo
> GIS developer
> - - - - - - - - - - - - - - - - - - - -
> ETC-LUSI: http://etc-lusi.eionet.europa.eu/
> Universitat Autònoma de Barcelona (SPAIN)
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> ------------------------------------------------------------------------------
> This SF.Net email is sponsored by the Verizon Developer Community
> Take advantage of Verizon's best-in-class app development support
> A streamlined, 14 day to market process makes app distribution fast and easy
> Join now and get one step closer to millions of Verizon customers
> http://p.sf.net/sfu/verizon-dev2dev
> _______________________________________________
> Geotools-gt2-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Geotools-gt2-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
