Hi Martin,
You can use the StreamingParser rather than just the Parser interface. The
former will allow you to walk through all the features rather than read
them all into memory at once. This does however require you to parse a bit
differently. For instance:
//Parser parser = new Parser(configuration);
StreamingParser parser = new StreamingParser(confguration, in,
SimpleFeature.class);
SimpleFeature next = null;
while ((next = parser.next()) != null) {
//do something with the next feature
}
Hope that helps.
-Justin
On Wed, Nov 21, 2012 at 10:49 PM, Martin Logan
<[email protected]>wrote:
> I would like to know if you can suggest a better way of handling/parsing a
> large GML2 document. At the moment, I'm using the sample code you have
> provided in the
> WFS_1_0_0_ParsingTest under gt-xsd-wfs package.
>
> File tmp = File.createTempFile("geoserver-DescribeFeatureType",
> "xml");
>
> tmp.deleteOnExit();
>
> InputStream in = getClass().getResourceAsStream(
> "geoserver-DescribeFeatureType.xml");
>
> copy(in, tmp);
>
> in = getClass().getResourceAsStream(
> "geoserver-GetFeature-of-a-very-big-file.xml");
>
> DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
>
> dbf.setNamespaceAware(true);
>
> DocumentBuilder db = dbf.newDocumentBuilder();
>
> Document doc = db.parse(in);
>
> String schemaLocation = doc.getDocumentElement().getAttributeNS(
>
> "http://www.w3.org/2001/XMLSchema-instance";,
> "schemaLocation");
>
> String absolutePath =
> DataUtilities.fileToURL(tmp).toExternalForm();
>
> schemaLocation = schemaLocation.replaceAll("
> http://cite.opengeospatial.org/gmlsf .*",
>
> "http://cite.opengeospatial.org/gmlsf " + absolutePath);
>
> doc.getDocumentElement().setAttributeNS("
> http://www.w3.org/2001/XMLSchema-instance";,
>
> "schemaLocation", schemaLocation);
>
> tmp = File.createTempFile("geoserver-GetFeature", "xml");
>
> tmp.deleteOnExit();
>
> Transformer tx = TransformerFactory.newInstance().newTransformer();
>
> tx.transform(new DOMSource(doc), new StreamResult(tmp));
>
> in = new FileInputStream(tmp);
>
> Parser parser = new Parser(configuration);
>
> FeatureCollectionType fc = (FeatureCollectionType)
> parser.parse(in);
>
>
> After running the test against a GML2 document with more than 1000
> features (with geometry included in the returned attribute), it throws an
> OutOfMemoryError. Any hints on what I can replace the Transformer.transform
> call with? I don't think increasing the heap space will be a scalable
> solution as I might have to handle datasets that are even bigger than the
> limit I set for the heap memory.
>
> Nov 20, 2012 12:53:01 PM org.apache.catalina.core.StandardWrapperValve
> invoke
>
> SEVERE: Servlet.service() for servlet [appServlet] in context with path
> [/aurin-data-provider] threw exception [Handler processing failed; nested
> exception is java.lang.OutOfMemoryError: Java heap space] with root cause
>
> java.lang.OutOfMemoryError: Java heap space
>
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
>
> at java.lang.String.<init>(String.java:215)
>
> at java.lang.StringBuffer.toString(StringBuffer.java:585)
>
> at org.apache.xerces.dom.DeferredDocumentImpl.getNodeValueString(Unknown
> Source)
>
> at org.apache.xerces.dom.DeferredDocumentImpl.getNodeValueString(Unknown
> Source)
>
> at org.apache.xerces.dom.DeferredTextImpl.synchronizeData(Unknown Source)
>
> at org.apache.xerces.dom.CharacterDataImpl.getNodeValue(Unknown Source)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:240)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:226)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:132)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(
> DOM2TO.java:94)
>
> at
> com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(
> TransformerImpl.java:679)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(
> TransformerImpl.java:723)
>
> at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(
> TransformerImpl.java:336)
>
>
> Thanks,
> Logan
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Monitor your physical, virtual and cloud infrastructure from a single
> web console. Get in-depth insight into apps, servers, databases, vmware,
> SAP, cloud infrastructure, etc. Download 30-day Free Trial.
> Pricing starts from $795 for 25 servers or applications!
> http://p.sf.net/sfu/zoho_dev2dev_nov
> _______________________________________________
> GeoTools-GT2-Users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
>
>
--
Justin Deoliveira
OpenGeo - http://opengeo.org
Enterprise support for open source geospatial.
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
GeoTools-GT2-Users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
