Re: [Geotools-devel] [Geotools-gt2-users] Efficiently writing a GeoJSON stream into a Shapefile
Thanks Jody,
So, I came up with this code, which gets an append writer and doesn't
use transaction. Can you confirm if that's what you meant to indicate?
...
try {
writer = shpDataStore.getFeatureWriterAppend(
shpDataStore.getTypeNames()[0], null);
while (jsonIt.hasNext()) {
SimpleFeature feature = jsonIt.next();
addFeature(feature, writer, featureStore);
}
} finally {
if (writer != null) {
writer.close();
}
}
...
/**
* Copied over from {@link ContentFeatureStore} as a way of writing features
* directly into a {@link FeatureWriter}
*/
private static FeatureId addFeature(SimpleFeature feature,
FeatureWriter writer,
SimpleFeatureStore featureStore) throws IOException {
SimpleFeature toWrite = writer.next();
for (int i = 0; i < toWrite.getType().getAttributeCount(); i++) {
String name = toWrite.getType().getDescriptor(i).getLocalName();
toWrite.setAttribute(name, feature.getAttribute(name));
}
// copy over the user data
if (feature.getUserData().size() > 0) {
toWrite.getUserData().putAll(feature.getUserData());
}
// pass through the fid if the user asked so
boolean useExisting = Boolean.TRUE.equals(feature.getUserData().get(
Hints.USE_PROVIDED_FID));
if (featureStore.getQueryCapabilities().isUseProvidedFIDSupported()
&& useExisting) {
((FeatureIdImpl) toWrite.getIdentifier()).setID(feature.getID());
}
// perform the write
writer.write();
// copy any metadata from the feature that was actually written
feature.getUserData().putAll(toWrite.getUserData());
// add the id to the set of inserted
FeatureId id = toWrite.getIdentifier();
return id;
}
On Fri, Sep 6, 2013 at 12:12 PM, Jody Garnett wrote:
> It is more that shapefile does not offer a database session, so we are
> faking it to make the editing story easier for desktop clients.
>
> Using AUTO_COMMIT is a terrible idea as it will involve writing out your
> file many times (ie each time you add a feature).
>
> I tried to indicate a better way in my email, and in the docs, but it is not
> coming through.
>
>
> On Fri, Sep 6, 2013 at 11:29 AM, William Voorsluys
> wrote:
>>
>> Hi Jodi,
>>
>> Did you mean to reply this to the list?
>>
>> It seems clear that transactions are not meant to be used efficiently
>> on shapefiles. I'm settling on using AUTO_COMMIT and writing a feature
>> a time using a writer. Do you mean there is no better way of more
>> efficiently writing features in bulk to the file, say 1000 at a time?
>> It seems the operation of getting an append writer is the greatest
>> bottleneck in the operation.
>>
>> Will
>>
>> On Fri, Sep 6, 2013 at 1:28 AM, Jody Garnett
>> wrote:
>> > I really need a better way to communicate this one, or a special case
>> > when
>> > the shapefile is empty or something.
>> >
>> > The goal here is to use an append feature writer, directly, and write
>> > the
>> > content out as you go in a streaming fashion.
>> >
>> > This is what ShapefileDataSource does internally when you call
>> > transaction.commit(). It goes through the changes that it has collected
>> > in
>> > memory and writes out a new file. It then renames the old file out of
>> > the
>> > way, renames the new file into the correct place, and deletes the old
>> > file.
>> >
>> >
>> >
>> > On Thu, Sep 5, 2013 at 8:15 PM, William Voorsluys
>> >
>> > wrote:
>> >>
>> >> Dear All,
>> >>
>> >> I've been trying a few solutions to efficiently convert GeoJSON into a
>> >> shapefile without having to store all features in memory. I'm using
>> >> GeoTools 9.2.
>> >>
>> >> The problem is not so much in how to stream the JSON but how to
>> >> efficiently write the features into the shapefile. I use
>> >> FeatureJSON#streamFeatureCollection to obtain an iterator. After some
>> >> googling, I found 3 different ways of writing a shapefile, namely:
>> >>
>> >> 1. Repeatedly calling FeatureStore#addFeatures with a collection
>> >> containing say 1000 features, within a transaction.
>> >> -
>> >> ListFeatureCollection coll = new ListFeatureCollection(type,
>> >> features);
>> >> Transaction transaction = new DefaultTransaction("create");
>> >> featureStore.setTransaction(transaction);
>> >> try {
>> >> featureStore.addFeatures(coll);
>> >> transaction.commit();
>> >> } catch (IOException e) {
>> >> transaction.rollback();
>> >> throw new IllegalStateException(
>> >> "Could not write some features to shapefile. Aborting
>> >> process", e);
>> >> } finally {
>> >> transaction.close();
>> >> }
>> >> -
>> >>
>> >>
>> >> This option is extremely slow. By profiling a few runs, I noticed that
>> >> about 50% of CPU time is spent on the method
>> >> ContentFeatureStore#getWriterAppend, presumably in order to reach the
>> >> end of the file befor
Re: [Geotools-devel] MapServerStrategy for WFSDataStore 1.1.0 protocol
done (applied to master and 2.4.x) Nice clean patch, and thanks for fixing the headers On Thu, Sep 5, 2013 at 5:39 PM, Mauro Bartolomeoli < mauro.bartolome...@geo-solutions.it> wrote: > Hi everybody, > I created a pull request (https://github.com/geotools/geotools/pull/262) > to handle some weird behaviours of MapServer with the WFS 1.1.0 regarding > namespaces and field type mappings. > > Thanks > Mauro Bartolomeoli > > > -- > == > Our support, Your Success! Visit http://opensdi.geo-solutions.it for more > information. > == > > Dott. Mauro Bartolomeoli > @mauro_bart > Senior Software Engineer > > GeoSolutions S.A.S. > Via Poggio alle Viti 1187 > 55054 Massarosa (LU) > Italy > phone: +39 0584 962313 > fax: +39 0584 1660272 > > http://www.geo-solutions.it > http://twitter.com/geosolutions_it > > --- > > > -- > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! > Discover the easy way to master current and previous Microsoft technologies > and advance your career. Get an incredible 1,500+ hours of step-by-step > tutorial videos with LearnDevNow. Subscribe today and save! > http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk > ___ > GeoTools-Devel mailing list > GeoTools-Devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/geotools-devel > > -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
Re: [Geotools-devel] [Geotools-gt2-users] Efficiently writing a GeoJSON stream into a Shapefile
It is more that shapefile does not offer a database session, so we are
faking it to make the editing story easier for desktop clients.
Using AUTO_COMMIT is a terrible idea as it will involve writing out your
file many times (ie each time you add a feature).
I tried to indicate a better way in my email, and in the docs, but it is
not coming through.
On Fri, Sep 6, 2013 at 11:29 AM, William Voorsluys wrote:
> Hi Jodi,
>
> Did you mean to reply this to the list?
>
> It seems clear that transactions are not meant to be used efficiently
> on shapefiles. I'm settling on using AUTO_COMMIT and writing a feature
> a time using a writer. Do you mean there is no better way of more
> efficiently writing features in bulk to the file, say 1000 at a time?
> It seems the operation of getting an append writer is the greatest
> bottleneck in the operation.
>
> Will
>
> On Fri, Sep 6, 2013 at 1:28 AM, Jody Garnett
> wrote:
> > I really need a better way to communicate this one, or a special case
> when
> > the shapefile is empty or something.
> >
> > The goal here is to use an append feature writer, directly, and write the
> > content out as you go in a streaming fashion.
> >
> > This is what ShapefileDataSource does internally when you call
> > transaction.commit(). It goes through the changes that it has collected
> in
> > memory and writes out a new file. It then renames the old file out of the
> > way, renames the new file into the correct place, and deletes the old
> file.
> >
> >
> >
> > On Thu, Sep 5, 2013 at 8:15 PM, William Voorsluys >
> > wrote:
> >>
> >> Dear All,
> >>
> >> I've been trying a few solutions to efficiently convert GeoJSON into a
> >> shapefile without having to store all features in memory. I'm using
> >> GeoTools 9.2.
> >>
> >> The problem is not so much in how to stream the JSON but how to
> >> efficiently write the features into the shapefile. I use
> >> FeatureJSON#streamFeatureCollection to obtain an iterator. After some
> >> googling, I found 3 different ways of writing a shapefile, namely:
> >>
> >> 1. Repeatedly calling FeatureStore#addFeatures with a collection
> >> containing say 1000 features, within a transaction.
> >> -
> >> ListFeatureCollection coll = new ListFeatureCollection(type,
> >> features);
> >> Transaction transaction = new DefaultTransaction("create");
> >> featureStore.setTransaction(transaction);
> >> try {
> >> featureStore.addFeatures(coll);
> >> transaction.commit();
> >> } catch (IOException e) {
> >> transaction.rollback();
> >> throw new IllegalStateException(
> >> "Could not write some features to shapefile. Aborting
> >> process", e);
> >> } finally {
> >> transaction.close();
> >> }
> >> -
> >>
> >>
> >> This option is extremely slow. By profiling a few runs, I noticed that
> >> about 50% of CPU time is spent on the method
> >> ContentFeatureStore#getWriterAppend, presumably in order to reach the
> >> end of the file before each transaction commit.
> >>
> >> 2. Obtaining an append writer directly from ShapefileDataStore, and
> >> write 1000 features at a time within a transaction.
> >>
> >> This options suffers from the same problems as number one.
> >>
> >> 3. Obtaining a feature writer from ShapefileDataStore, and write one
> >> feature at a time using Transaction.AUTO_COMMIT.
> >>
> >> -
> >> FeatureWriter writer =
> shpDataStore
> >> .getFeatureWriter(shpDataStore.getTypeNames()[0],
> >> Transaction.AUTO_COMMIT);
> >>
> >> while (jsonIt.hasNext()) {
> >>
> >> SimpleFeature feature = jsonIt.next();
> >> SimpleFeature toWrite = writer.next();
> >> for (int i = 0; i < toWrite.getType().getAttributeCount(); i++) {
> >> String name = toWrite.getType().getDescriptor(i).getLocalName();
> >> toWrite.setAttribute(name, feature.getAttribute(name));
> >> }
> >> writer.write();
> >> }
> >> writer.close();
> >>
> >>
> >>
> >> Option 3 is the fastest, but I feel there would a way of efficiently
> >> adding a greater number of features at a time to the shapefile within
> >> a transaction. On the other hand, a previous comment in this lists
> >> noted:
> >>
> >> > The above would work for mid-sized data transafers, for massive ones
> >> > against
> >> > databases it's better to adopt some sort of batching to avoid having a
> >> > single
> >> > transaction with one million inserts, e.g., insert 1000, commit the
> >> > transaction,
> >> > insert another 1000, and so on.
> >> > This would work better against databases and against WFS servers,
> >> > but not against shapefiles, which instead work better with the massive
> >> > insert...
> >> > to each his own.
> >>
> >> Does this mean that the most efficient way of writing to a shapefile
> >> is having all features in memory, rather than being able to append
> >> features?
> >> I appreciate if someone could sug
[Geotools-devel] [jira] (GEOT-4559) ShapeFileDataStore does not return valid type name for file names starting with . (dot)
Sebastian Graca
created GEOT-4559
ShapeFileDataStore does not return valid type name for file names starting with . (dot)
Issue Type:
Bug
Affects Versions:
10-RC2
Assignee:
Andrea Aime
Components:
shapefile plugin
Created:
05/Sep/13 6:22 AM
Description:
For file names starting with dot (like .index.shp) ShapeFileDataStore returns empty ("") type name instead of file name (".index").
Project:
GeoTools
Priority:
Major
Reporter:
Sebastian Graca
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk___
GeoTools-Devel mailing list
GeoTools-Devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/geotools-devel
[Geotools-devel] [jira] (GEOT-4558) NPE when creating ImageMosaicReader from ImageMosaicDescriptor
Sebastian Graca created GEOT-4558 NPE when creating ImageMosaicReader from ImageMosaicDescriptor Issue Type: Bug Affects Versions: 10-RC2 Assignee: Unassigned Components: imagemosaic plugin Created: 05/Sep/13 5:43 AM Description: When ImageMosaic is constructed from ImageMosaicDescriptor a NPE is thorwn because the method initReaderFromDescriptor does not initialize the granuleCatalog field from passed descriptor. monospaced Caused by: java.lang.NullPointerException: Argument "index" should not be null. at org.geotools.util.Utilities.ensureNonNull(Utilities.java:683) ~[classes/:na] at org.geotools.gce.imagemosaic.ImageMosaicReader.setGridGeometry(ImageMosaicReader.java:381) ~[classes/:na] at org.geotools.gce.imagemosaic.ImageMosaicReader.setGridGeometry(ImageMosaicReader.java:418) ~[classes/:na] at org.geotools.gce.imagemosaic.ImageMosaicReader.initReaderFromDescriptor(ImageMosaicReader.java:246) ~[classes/:na] at org.geotools.gce.imagemosaic.ImageMosaicReader.(ImageMosaicReader.java:205) ~[classes/:na] at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:496) ~[classes/:na] at org.geotools.gce.imagemosaic.ImageMosaicFormat.getReader(ImageMosaicFormat.java:106) ~[classes/:na] monospaced Project: GeoTools Priority: Major Reporter: Sebastian Graca This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
[Geotools-devel] MapServerStrategy for WFSDataStore 1.1.0 protocol
Hi everybody, I created a pull request (https://github.com/geotools/geotools/pull/262) to handle some weird behaviours of MapServer with the WFS 1.1.0 regarding namespaces and field type mappings. Thanks Mauro Bartolomeoli -- == Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information. == Dott. Mauro Bartolomeoli @mauro_bart Senior Software Engineer GeoSolutions S.A.S. Via Poggio alle Viti 1187 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 http://www.geo-solutions.it http://twitter.com/geosolutions_it --- -- Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk___ GeoTools-Devel mailing list GeoTools-Devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/geotools-devel
