On Fri, Sep 6, 2013 at 1:55 PM, Jody Garnett <[email protected]> wrote: > Yes. > > That code is very similar to how TransactionStateDiff works, it just has to > take some extra steps as it is also used to update existing content. > > The work you copied from content datastore is used by JDBCDataStore to allow > implementations that support FeatureID to have that passed in as a user > property. Shapefile FeatureIDs are based on "row number" so you do not need > all of that code. > > (You can see the docs for details > http://docs.geotools.org/latest/userguide/library/data/featuresource.html#adding-data > ) > > Q: If I put a feature writer example at that location would you of found it? > Or did you look at the shapefile datastore page first? I did look at that and other examples, like the CSV2SHP example in the documentation. IMO, these examples lead to an understanding that transactions were a good way to approach the problem.
In this somewhat outdated page (http://docs.codehaus.org/display/GEOTOOLS/Data+Writing) I did find examples of using FeatureWriter directly. So, an example in the main documentation would be certainly helpful. Thanks, Will > > Jody > > > On Fri, Sep 6, 2013 at 1:20 PM, William Voorsluys <[email protected]> > wrote: >> >> Thanks Jody, >> >> So, I came up with this code, which gets an append writer and doesn't >> use transaction. Can you confirm if that's what you meant to indicate? >> >> ... >> try { >> writer = shpDataStore.getFeatureWriterAppend( >> shpDataStore.getTypeNames()[0], null); >> >> while (jsonIt.hasNext()) { >> >> SimpleFeature feature = jsonIt.next(); >> addFeature(feature, writer, featureStore); >> } >> } finally { >> if (writer != null) { >> writer.close(); >> } >> } >> ... >> >> /** >> * Copied over from {@link ContentFeatureStore} as a way of writing >> features >> * directly into a {@link FeatureWriter} >> */ >> private static FeatureId addFeature(SimpleFeature feature, >> FeatureWriter<SimpleFeatureType, SimpleFeature> writer, >> SimpleFeatureStore featureStore) throws IOException { >> >> SimpleFeature toWrite = writer.next(); >> for (int i = 0; i < toWrite.getType().getAttributeCount(); i++) { >> String name = toWrite.getType().getDescriptor(i).getLocalName(); >> toWrite.setAttribute(name, feature.getAttribute(name)); >> } >> >> // copy over the user data >> if (feature.getUserData().size() > 0) { >> toWrite.getUserData().putAll(feature.getUserData()); >> } >> >> // pass through the fid if the user asked so >> boolean useExisting = Boolean.TRUE.equals(feature.getUserData().get( >> Hints.USE_PROVIDED_FID)); >> if (featureStore.getQueryCapabilities().isUseProvidedFIDSupported() >> && useExisting) { >> ((FeatureIdImpl) toWrite.getIdentifier()).setID(feature.getID()); >> } >> >> // perform the write >> writer.write(); >> >> // copy any metadata from the feature that was actually written >> feature.getUserData().putAll(toWrite.getUserData()); >> >> // add the id to the set of inserted >> FeatureId id = toWrite.getIdentifier(); >> return id; >> } >> >> On Fri, Sep 6, 2013 at 12:12 PM, Jody Garnett <[email protected]> >> wrote: >> > It is more that shapefile does not offer a database session, so we are >> > faking it to make the editing story easier for desktop clients. >> > >> > Using AUTO_COMMIT is a terrible idea as it will involve writing out your >> > file many times (ie each time you add a feature). >> > >> > I tried to indicate a better way in my email, and in the docs, but it is >> > not >> > coming through. >> > >> > >> > On Fri, Sep 6, 2013 at 11:29 AM, William Voorsluys >> > <[email protected]> >> > wrote: >> >> >> >> Hi Jodi, >> >> >> >> Did you mean to reply this to the list? >> >> >> >> It seems clear that transactions are not meant to be used efficiently >> >> on shapefiles. I'm settling on using AUTO_COMMIT and writing a feature >> >> a time using a writer. Do you mean there is no better way of more >> >> efficiently writing features in bulk to the file, say 1000 at a time? >> >> It seems the operation of getting an append writer is the greatest >> >> bottleneck in the operation. >> >> >> >> Will >> >> >> >> On Fri, Sep 6, 2013 at 1:28 AM, Jody Garnett <[email protected]> >> >> wrote: >> >> > I really need a better way to communicate this one, or a special case >> >> > when >> >> > the shapefile is empty or something. >> >> > >> >> > The goal here is to use an append feature writer, directly, and write >> >> > the >> >> > content out as you go in a streaming fashion. >> >> > >> >> > This is what ShapefileDataSource does internally when you call >> >> > transaction.commit(). It goes through the changes that it has >> >> > collected >> >> > in >> >> > memory and writes out a new file. It then renames the old file out of >> >> > the >> >> > way, renames the new file into the correct place, and deletes the old >> >> > file. >> >> > >> >> > >> >> > >> >> > On Thu, Sep 5, 2013 at 8:15 PM, William Voorsluys >> >> > <[email protected]> >> >> > wrote: >> >> >> >> >> >> Dear All, >> >> >> >> >> >> I've been trying a few solutions to efficiently convert GeoJSON into >> >> >> a >> >> >> shapefile without having to store all features in memory. I'm using >> >> >> GeoTools 9.2. >> >> >> >> >> >> The problem is not so much in how to stream the JSON but how to >> >> >> efficiently write the features into the shapefile. I use >> >> >> FeatureJSON#streamFeatureCollection to obtain an iterator. After >> >> >> some >> >> >> googling, I found 3 different ways of writing a shapefile, namely: >> >> >> >> >> >> 1. Repeatedly calling FeatureStore#addFeatures with a collection >> >> >> containing say 1000 features, within a transaction. >> >> >> ----- >> >> >> ListFeatureCollection coll = new ListFeatureCollection(type, >> >> >> features); >> >> >> Transaction transaction = new DefaultTransaction("create"); >> >> >> featureStore.setTransaction(transaction); >> >> >> try { >> >> >> featureStore.addFeatures(coll); >> >> >> transaction.commit(); >> >> >> } catch (IOException e) { >> >> >> transaction.rollback(); >> >> >> throw new IllegalStateException( >> >> >> "Could not write some features to shapefile. Aborting >> >> >> process", e); >> >> >> } finally { >> >> >> transaction.close(); >> >> >> } >> >> >> ----- >> >> >> >> >> >> >> >> >> This option is extremely slow. By profiling a few runs, I noticed >> >> >> that >> >> >> about 50% of CPU time is spent on the method >> >> >> ContentFeatureStore#getWriterAppend, presumably in order to reach >> >> >> the >> >> >> end of the file before each transaction commit. >> >> >> >> >> >> 2. Obtaining an append writer directly from ShapefileDataStore, and >> >> >> write 1000 features at a time within a transaction. >> >> >> >> >> >> This options suffers from the same problems as number one. >> >> >> >> >> >> 3. Obtaining a feature writer from ShapefileDataStore, and write one >> >> >> feature at a time using Transaction.AUTO_COMMIT. >> >> >> >> >> >> ----- >> >> >> FeatureWriter<SimpleFeatureType, SimpleFeature> writer = >> >> >> shpDataStore >> >> >> .getFeatureWriter(shpDataStore.getTypeNames()[0], >> >> >> Transaction.AUTO_COMMIT); >> >> >> >> >> >> while (jsonIt.hasNext()) { >> >> >> >> >> >> SimpleFeature feature = jsonIt.next(); >> >> >> SimpleFeature toWrite = writer.next(); >> >> >> for (int i = 0; i < toWrite.getType().getAttributeCount(); >> >> >> i++) { >> >> >> String name = >> >> >> toWrite.getType().getDescriptor(i).getLocalName(); >> >> >> toWrite.setAttribute(name, feature.getAttribute(name)); >> >> >> } >> >> >> writer.write(); >> >> >> } >> >> >> writer.close(); >> >> >> ---- >> >> >> >> >> >> >> >> >> Option 3 is the fastest, but I feel there would a way of efficiently >> >> >> adding a greater number of features at a time to the shapefile >> >> >> within >> >> >> a transaction. On the other hand, a previous comment in this lists >> >> >> noted: >> >> >> >> >> >> > The above would work for mid-sized data transafers, for massive >> >> >> > ones >> >> >> > against >> >> >> > databases it's better to adopt some sort of batching to avoid >> >> >> > having >> >> >> > a >> >> >> > single >> >> >> > transaction with one million inserts, e.g., insert 1000, commit >> >> >> > the >> >> >> > transaction, >> >> >> > insert another 1000, and so on. >> >> >> > This would work better against databases and against WFS servers, >> >> >> > but not against shapefiles, which instead work better with the >> >> >> > massive >> >> >> > insert... >> >> >> > to each his own. >> >> >> >> >> >> Does this mean that the most efficient way of writing to a shapefile >> >> >> is having all features in memory, rather than being able to append >> >> >> features? >> >> >> I appreciate if someone could suggest a better way of achieving this >> >> >> or point to any documentation that would help me. >> >> >> >> >> >> Best regards, >> >> >> >> >> >> Will >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> >> Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, >> >> >> more! >> >> >> Discover the easy way to master current and previous Microsoft >> >> >> technologies >> >> >> and advance your career. Get an incredible 1,500+ hours of >> >> >> step-by-step >> >> >> tutorial videos with LearnDevNow. Subscribe today and save! >> >> >> >> >> >> >> >> >> >> >> >> http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk >> >> >> _______________________________________________ >> >> >> GeoTools-GT2-Users mailing list >> >> >> [email protected] >> >> >> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users >> >> > >> >> > >> > >> > > > ------------------------------------------------------------------------------ Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk _______________________________________________ GeoTools-GT2-Users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
