Thanks Jos,
I have tried moving the tfactory stuff into my specialized cocoon
transformer (com.boeing.transform.SPARQLinTrans), and I've also tried
processing the file in a stand-alone java program with 1 Gig of memory assigned
to it, and in both cases I'm still running out of heap space (see exception
stack below). So, the current problem must lie in the xsl transform libraries
I'm using (i.e. not really a cocoon problem at this point). I know there must
be techniques for doing xsl on very large files, do you have any pointers or
ideas for this problem in general?
Thanks!
Jeff
java.lang.OutOfMemoryError: Java heap space
net.sf.saxon.tinytree.TinyTree.ensureNodeCapacity(TinyTree.java:215)
net.sf.saxon.tinytree.TinyTree.addNode(TinyTree.java:309)
net.sf.saxon.tinytree.TinyBuilder.characters(TinyBuilder.java:306)
net.sf.saxon.event.ReceivingContentHandler.flush(ReceivingContentHandler.java:544)
net.sf.saxon.event.ReceivingContentHandler.startElement(ReceivingContentHandler.java:275)
org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
net.sf.saxon.event.Sender.sendSAXSource(Sender.java:354)
net.sf.saxon.event.Sender.send(Sender.java:184)
net.sf.saxon.event.Sender.send(Sender.java:49)
net.sf.saxon.Controller.transform(Controller.java:1550)
com.boeing.transform.SPARQLinTrans.adapt(SPARQLinTrans.java:143)
Of course this is still a part of the Tomcat process.
> -----Original Message-----
> From: Jos Snellings [mailto:jos.snelli...@pandora.be]
> Sent: Monday, May 10, 2010 10:04 AM
> To: users@cocoon.apache.org
> Subject: RE: Very large Generator file
>
> Yes, Jeff, my guess is that the tfactory stuff will accomplish the job
> for you without complaining about memory shortage! Your tomcat instance
> may have other things around increasing the memory footprint all the
> time.
>
>
> On Mon, 2010-05-10 at 09:43 -0500, Schmitz, Jeffrey A wrote:
> > Yes, the resulting XHTML file is not for online use.
> >
> > And yes, I've given max memory to the tomcat instance. So, will
> having my specialized transformer do the xsl transform directly using a
> SAX transformer not work either? e.g. something like:
> >
> > TransformerFactory tfactory = TransformerFactory.newInstance();
> > Transformer transformer = tfactory.newTransformer(new
> StreamSource(
> > xslID));
> > transformer.transform(new StreamSource(sourceFile), new
> StreamResult(
> > new File(modelFile)));
> >
> >
> > I guess I was under the impression that SAX based xsl transformers
> didn't use much memory regardless file size, but perhaps that's not
> true?
> >
> > If that's not going to work, would moving to a 64 bit machine so I
> can give more memory to Tomcat be an option (is there a 64 bit Tomcat)?
> >
> > > If this is the case a short java program can make that xhtml file
> for
> > > you.
> > I'm not exactly sure what you're suggesting here. Do you mean for
> the tomcat/cocoon process to exec a separate java program that will do
> the xsl transform on the large file? Won't any java program come up
> against the 2 Gig max memory limitation?
> >
> > Thanks,
> > Jeff
> >
> > > -----Original Message-----
> > > From: Jos Snellings [mailto:jos.snelli...@pandora.be]
> > > Sent: Monday, May 10, 2010 9:30 AM
> > > To: users@cocoon.apache.org
> > > Subject: RE: Very large Generator file
> > >
> > > Hi Jeff,
> > >
> > > If you have been trying giving all the available memory to your
> tomcat
> > > instance (JVM options) you are at the end of you tether for the
> current
> > > solutions, unless somebody has a different idea.
> > >
> > > I understood from your previous mail that the resulting xhtml is
> not
> > > for
> > > on-line use, it is a sort of report/summary.
> > > If this is the case a short java program can make that xhtml file
> for
> > > you.
> > >
> > > Cheers,
> > > Jos
> > >
> > >
> > > On Mon, 2010-05-10 at 09:17 -0500, Schmitz, Jeffrey A wrote:
> > > > Hi Jos,
> > > > I've already written my own Transformer (extends
> > > AbstractSAXTransformer) that currently takes the output of the
> default
> > > cocoon xsl transformer (via the sitemap) and performs subsequent,
> non-
> > > xsl transforms, writes the results to file and generates the
> returned
> > > web-page
> > > > e.g.
> > > > <map:match pattern="*/*/*In/*/*In.flow/*" >
> > > > <map:generate src="inputs/{4}" />
> > > > <map:transform src="xsl/{3}InAdapter.xsl">
> > > > <map:parameter name="instanceName" value="{2}"/>
> > > > </map:transform>
> > > > <map:transform type="SPARQLinTrans">
> > > > <map:parameter name="inputFile" value="{4}"/>
> > > > <map:parameter name="SPARQLFlow" value="{5}"/>
> > > > </map:transform>
> > > > <map:serialize type="xhtml"/>
> > > > </map:match>
> > > >
> > > > I'm currently heading in the direction of just adding the initial
> xsl
> > > transformation as part of my own Transformer's duties (I think your
> > > option 3) so that it can hopefully handle the larger files, but I
> was
> > > wondering if there might be a way to easily keep things as they are
> and
> > > continue using the default cocoon xsl tranform as the first step.
> > > Sounds like there's not?
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Jos Snellings [mailto:jos.snelli...@pandora.be]
> > > > > Sent: Monday, May 10, 2010 8:39 AM
> > > > > To: users@cocoon.apache.org
> > > > > Subject: RE: Very large Generator file
> > > > >
> > > > > A SAX pipeline is part of cocoon.
> > > > > Can you explain briefly what you want to do?
> > > > > Consider using a cocoon-3 pipeline.
> > > > > Consider using a SAX transformation:
> > > > > - read xml
> > > > > - do something with it (transforming, for instance)
> > > > > - write output to a file
> > > > >
> > > > > Jos
> > > > >
> > > > > On Mon, 2010-05-10 at 08:23 -0500, Schmitz, Jeffrey A wrote:
> > > > > > Correct, it's not being turned into a web page. Is a SAX
> > > pipeline
> > > > > part of Cocoon? I'm still on Cocoon 2.1, but I've never heard
> that
> > > > > term specifically, I thought the default generator did use SAX.
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jos Snellings [mailto:jos.snelli...@pandora.be]
> > > > > > > Sent: Saturday, May 08, 2010 12:06 AM
> > > > > > > To: users@cocoon.apache.org
> > > > > > > Subject: Re: Very large Generator file
> > > > > > >
> > > > > > > Hi Jeff,
> > > > > > >
> > > > > > > That depends a great deal on what you want to do exactly
> with
> > > the
> > > > > large
> > > > > > > file. I guess you do not want to transform it into one huge
> web
> > > > > page?
> > > > > > > In general, if you have to deal with lengthy inputs, a SAX
> > > pipeline
> > > > > is
> > > > > > > the way to go.
> > > > > > >
> > > > > > > Jos
> > > > > > >
> > > > > > > On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A
> wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I was wondering if there’s a way to handle very large
> > > files as
> > > > > > > > inputs to the default cocoon generator? As it is, I am
> > > getting a
> > > > > > > > memory heap error when trying to process an input file
> > > because
> > > > > it’s
> > > > > > > so
> > > > > > > > large. Am I going to have to process it in some other
> > > manner, or
> > > > > is
> > > > > > > > there a way to make Cocoon perform the transform for me?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jeff
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----------------------------------------------------------
> ----
> > > ----
> > > > > --
> > > > > > > To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
> > > > > > > For additional commands, e-mail: users-
> h...@cocoon.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> ----
> > > --
> > > > > To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
> > > > > For additional commands, e-mail: users-h...@cocoon.apache.org
> > > >
> > >
> > >
> > >
> > > -------------------------------------------------------------------
> --
> > > To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
> > > For additional commands, e-mail: users-h...@cocoon.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
> For additional commands, e-mail: users-h...@cocoon.apache.org
