Re: Very large Generator file
Hi Jeff, You wrote: I guess I was under the impression that SAX based xsl transformers didn't use much memory regardless file size, but perhaps that's not true? It's true. However, when you do a XSLT transformation the entire XML document needs to be available to the XSLT processor, because you can access any node in the document, e.g. /very/large/path/to/some/dark/corner/of/the/XML/node. I'm not sure about your use-case so I don't know if it will help, but you could try using the MultiFragmentTraxTransformer from the cocooncomponents project on Google code. It reduces memory consumption drastically by allowing you to specify a fragment in the XML that you want transformed. If you have a very large document that looks like this: root item/ item/ item/ /root and all you want to do is transform the item/ elements then the MultiFragmentTraxTransformer is your friend. Huib. - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Very large Generator file
Thanks Huib, I'm afraid my document is very interconnected. I'm coming to realize that xslt has pretty much a hard limit on the size of files you can process with it, i.e. if it can't fit into memory and you need the full power of xslt you're pretty much out of luck. What about going to a 64 bit machine and just allocating a huge amount of ram to Tomcat? Can this been done with tomcat/cocoon? -Original Message- From: Huib Verweij [mailto:h...@home.nl] Sent: Tuesday, May 11, 2010 3:51 AM To: users@cocoon.apache.org Cc: Huib Verweij Subject: Re: Very large Generator file Hi Jeff, You wrote: I guess I was under the impression that SAX based xsl transformers didn't use much memory regardless file size, but perhaps that's not true? It's true. However, when you do a XSLT transformation the entire XML document needs to be available to the XSLT processor, because you can access any node in the document, e.g. /very/large/path/to/some/dark/corner/of/the/XML/node. I'm not sure about your use-case so I don't know if it will help, but you could try using the MultiFragmentTraxTransformer from the cocooncomponents project on Google code. It reduces memory consumption drastically by allowing you to specify a fragment in the XML that you want transformed. If you have a very large document that looks like this: root item/ item/ item/ /root and all you want to do is transform the item/ elements then the MultiFragmentTraxTransformer is your friend. Huib. - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
Re: Very large Generator file
On Tue, May 11, 2010 at 07:56:19AM -0500, Schmitz, Jeffrey A wrote: Thanks Huib, I'm afraid my document is very interconnected. I'm coming to realize that xslt has pretty much a hard limit on the size of files you can process with it, i.e. if it can't fit into memory and you need the full power of xslt you're pretty much out of luck. What about going to a 64 bit machine and just allocating a huge amount of ram to Tomcat? Can this been done with tomcat/cocoon? It can be done with a 64-bit JVM. Tomcat won't know and neither will Cocoon. -- Mark H. Wood, Lead System Programmer mw...@iupui.edu Balance your desire for bells and whistles with the reality that only a little more than 2 percent of world population has broadband. -- Ledford and Tyler, _Google Analytics 2.0_ pgpzEqtBX6xps.pgp Description: PGP signature
Re: Very large Generator file
[1] You might look at using Joost/STX which does streaming transforms. It's not XSLT -- it's an xslt-like transformation language designed for one pass processing, so STXPath is more restricted than XPath. http://joost.sourceforge.net/ We've used it within cocoon to extract data which is then may be transformed in a 2nd stage. We've run into a few bugs, but we're not using the latest current version. ( I think there were some incompatibilities with the cocoon 2.1.* stx transformer and the newer joost libraries, but I haven't looked at this in quite a while. ) [2] I believe Saxon can do some transforms in a streaming mode. I think this extension requires one of the paid licensed versions of Saxon. I haven't tried this myself. http://saxonica.blogharbor.com/blog/_archives/2006/7/4/2084120.html -- Steve Majewski On May 11, 2010, at 4:51 AM, Huib Verweij wrote: Hi Jeff, You wrote: I guess I was under the impression that SAX based xsl transformers didn't use much memory regardless file size, but perhaps that's not true? It's true. However, when you do a XSLT transformation the entire XML document needs to be available to the XSLT processor, because you can access any node in the document, e.g. /very/large/path/to/some/ dark/corner/of/the/XML/node. I'm not sure about your use-case so I don't know if it will help, but you could try using the MultiFragmentTraxTransformer from the cocooncomponents project on Google code. It reduces memory consumption drastically by allowing you to specify a fragment in the XML that you want transformed. If you have a very large document that looks like this: root item/ item/ item/ /root and all you want to do is transform the item/ elements then the MultiFragmentTraxTransformer is your friend. - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Very large Generator file
Correct, it's not being turned into a web page. Is a SAX pipeline part of Cocoon? I'm still on Cocoon 2.1, but I've never heard that term specifically, I thought the default generator did use SAX. -Original Message- From: Jos Snellings [mailto:jos.snelli...@pandora.be] Sent: Saturday, May 08, 2010 12:06 AM To: users@cocoon.apache.org Subject: Re: Very large Generator file Hi Jeff, That depends a great deal on what you want to do exactly with the large file. I guess you do not want to transform it into one huge web page? In general, if you have to deal with lengthy inputs, a SAX pipeline is the way to go. Jos On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote: Hello, I was wondering if there’s a way to handle very large files as inputs to the default cocoon generator? As it is, I am getting a memory heap error when trying to process an input file because it’s so large. Am I going to have to process it in some other manner, or is there a way to make Cocoon perform the transform for me? Thanks, Jeff - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Very large Generator file
A SAX pipeline is part of cocoon. Can you explain briefly what you want to do? Consider using a cocoon-3 pipeline. Consider using a SAX transformation: - read xml - do something with it (transforming, for instance) - write output to a file Jos On Mon, 2010-05-10 at 08:23 -0500, Schmitz, Jeffrey A wrote: Correct, it's not being turned into a web page. Is a SAX pipeline part of Cocoon? I'm still on Cocoon 2.1, but I've never heard that term specifically, I thought the default generator did use SAX. -Original Message- From: Jos Snellings [mailto:jos.snelli...@pandora.be] Sent: Saturday, May 08, 2010 12:06 AM To: users@cocoon.apache.org Subject: Re: Very large Generator file Hi Jeff, That depends a great deal on what you want to do exactly with the large file. I guess you do not want to transform it into one huge web page? In general, if you have to deal with lengthy inputs, a SAX pipeline is the way to go. Jos On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote: Hello, I was wondering if there’s a way to handle very large files as inputs to the default cocoon generator? As it is, I am getting a memory heap error when trying to process an input file because it’s so large. Am I going to have to process it in some other manner, or is there a way to make Cocoon perform the transform for me? Thanks, Jeff - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Very large Generator file
Hi Jos,
I've already written my own Transformer (extends AbstractSAXTransformer)
that currently takes the output of the default cocoon xsl transformer (via the
sitemap) and performs subsequent, non-xsl transforms, writes the results to
file and generates the returned web-page
e.g.
map:match pattern=*/*/*In/*/*In.flow/*
map:generate src=inputs/{4} /
map:transform src=xsl/{3}InAdapter.xsl
map:parameter name=instanceName value={2}/
/map:transform
map:transform type=SPARQLinTrans
map:parameter name=inputFile value={4}/
map:parameter name=SPARQLFlow value={5}/
/map:transform
map:serialize type=xhtml/
/map:match
I'm currently heading in the direction of just adding the initial xsl
transformation as part of my own Transformer's duties (I think your option 3)
so that it can hopefully handle the larger files, but I was wondering if there
might be a way to easily keep things as they are and continue using the default
cocoon xsl tranform as the first step. Sounds like there's not?
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Monday, May 10, 2010 8:39 AM
To: users@cocoon.apache.org
Subject: RE: Very large Generator file
A SAX pipeline is part of cocoon.
Can you explain briefly what you want to do?
Consider using a cocoon-3 pipeline.
Consider using a SAX transformation:
- read xml
- do something with it (transforming, for instance)
- write output to a file
Jos
On Mon, 2010-05-10 at 08:23 -0500, Schmitz, Jeffrey A wrote:
Correct, it's not being turned into a web page. Is a SAX pipeline
part of Cocoon? I'm still on Cocoon 2.1, but I've never heard that
term specifically, I thought the default generator did use SAX.
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Saturday, May 08, 2010 12:06 AM
To: users@cocoon.apache.org
Subject: Re: Very large Generator file
Hi Jeff,
That depends a great deal on what you want to do exactly with the
large
file. I guess you do not want to transform it into one huge web
page?
In general, if you have to deal with lengthy inputs, a SAX pipeline
is
the way to go.
Jos
On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote:
Hello,
I was wondering if there’s a way to handle very large files as
inputs to the default cocoon generator? As it is, I am getting a
memory heap error when trying to process an input file because
it’s
so
large. Am I going to have to process it in some other manner, or
is
there a way to make Cocoon perform the transform for me?
Thanks,
Jeff
---
--
To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
For additional commands, e-mail: users-h...@cocoon.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Very large Generator file
Hi Jeff,
If you have been trying giving all the available memory to your tomcat
instance (JVM options) you are at the end of you tether for the current
solutions, unless somebody has a different idea.
I understood from your previous mail that the resulting xhtml is not for
on-line use, it is a sort of report/summary.
If this is the case a short java program can make that xhtml file for
you.
Cheers,
Jos
On Mon, 2010-05-10 at 09:17 -0500, Schmitz, Jeffrey A wrote:
Hi Jos,
I've already written my own Transformer (extends AbstractSAXTransformer)
that currently takes the output of the default cocoon xsl transformer (via
the sitemap) and performs subsequent, non-xsl transforms, writes the results
to file and generates the returned web-page
e.g.
map:match pattern=*/*/*In/*/*In.flow/*
map:generate src=inputs/{4} /
map:transform src=xsl/{3}InAdapter.xsl
map:parameter name=instanceName value={2}/
/map:transform
map:transform type=SPARQLinTrans
map:parameter name=inputFile value={4}/
map:parameter name=SPARQLFlow value={5}/
/map:transform
map:serialize type=xhtml/
/map:match
I'm currently heading in the direction of just adding the initial xsl
transformation as part of my own Transformer's duties (I think your option 3)
so that it can hopefully handle the larger files, but I was wondering if
there might be a way to easily keep things as they are and continue using the
default cocoon xsl tranform as the first step. Sounds like there's not?
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Monday, May 10, 2010 8:39 AM
To: users@cocoon.apache.org
Subject: RE: Very large Generator file
A SAX pipeline is part of cocoon.
Can you explain briefly what you want to do?
Consider using a cocoon-3 pipeline.
Consider using a SAX transformation:
- read xml
- do something with it (transforming, for instance)
- write output to a file
Jos
On Mon, 2010-05-10 at 08:23 -0500, Schmitz, Jeffrey A wrote:
Correct, it's not being turned into a web page. Is a SAX pipeline
part of Cocoon? I'm still on Cocoon 2.1, but I've never heard that
term specifically, I thought the default generator did use SAX.
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Saturday, May 08, 2010 12:06 AM
To: users@cocoon.apache.org
Subject: Re: Very large Generator file
Hi Jeff,
That depends a great deal on what you want to do exactly with the
large
file. I guess you do not want to transform it into one huge web
page?
In general, if you have to deal with lengthy inputs, a SAX pipeline
is
the way to go.
Jos
On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote:
Hello,
I was wondering if there’s a way to handle very large files as
inputs to the default cocoon generator? As it is, I am getting a
memory heap error when trying to process an input file because
it’s
so
large. Am I going to have to process it in some other manner, or
is
there a way to make Cocoon perform the transform for me?
Thanks,
Jeff
---
--
To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
For additional commands, e-mail: users-h...@cocoon.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
For additional commands, e-mail: users-h...@cocoon.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Very large Generator file
Yes, the resulting XHTML file is not for online use.
And yes, I've given max memory to the tomcat instance. So, will having my
specialized transformer do the xsl transform directly using a SAX transformer
not work either? e.g. something like:
TransformerFactory tfactory = TransformerFactory.newInstance();
Transformer transformer = tfactory.newTransformer(new StreamSource(
xslID));
transformer.transform(new StreamSource(sourceFile), new StreamResult(
new File(modelFile)));
I guess I was under the impression that SAX based xsl transformers didn't use
much memory regardless file size, but perhaps that's not true?
If that's not going to work, would moving to a 64 bit machine so I can give
more memory to Tomcat be an option (is there a 64 bit Tomcat)?
If this is the case a short java program can make that xhtml file for
you.
I'm not exactly sure what you're suggesting here. Do you mean for the
tomcat/cocoon process to exec a separate java program that will do the xsl
transform on the large file? Won't any java program come up against the 2 Gig
max memory limitation?
Thanks,
Jeff
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Monday, May 10, 2010 9:30 AM
To: users@cocoon.apache.org
Subject: RE: Very large Generator file
Hi Jeff,
If you have been trying giving all the available memory to your tomcat
instance (JVM options) you are at the end of you tether for the current
solutions, unless somebody has a different idea.
I understood from your previous mail that the resulting xhtml is not
for
on-line use, it is a sort of report/summary.
If this is the case a short java program can make that xhtml file for
you.
Cheers,
Jos
On Mon, 2010-05-10 at 09:17 -0500, Schmitz, Jeffrey A wrote:
Hi Jos,
I've already written my own Transformer (extends
AbstractSAXTransformer) that currently takes the output of the default
cocoon xsl transformer (via the sitemap) and performs subsequent, non-
xsl transforms, writes the results to file and generates the returned
web-page
e.g.
map:match pattern=*/*/*In/*/*In.flow/*
map:generate src=inputs/{4} /
map:transform src=xsl/{3}InAdapter.xsl
map:parameter name=instanceName value={2}/
/map:transform
map:transform type=SPARQLinTrans
map:parameter name=inputFile value={4}/
map:parameter name=SPARQLFlow value={5}/
/map:transform
map:serialize type=xhtml/
/map:match
I'm currently heading in the direction of just adding the initial xsl
transformation as part of my own Transformer's duties (I think your
option 3) so that it can hopefully handle the larger files, but I was
wondering if there might be a way to easily keep things as they are and
continue using the default cocoon xsl tranform as the first step.
Sounds like there's not?
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Monday, May 10, 2010 8:39 AM
To: users@cocoon.apache.org
Subject: RE: Very large Generator file
A SAX pipeline is part of cocoon.
Can you explain briefly what you want to do?
Consider using a cocoon-3 pipeline.
Consider using a SAX transformation:
- read xml
- do something with it (transforming, for instance)
- write output to a file
Jos
On Mon, 2010-05-10 at 08:23 -0500, Schmitz, Jeffrey A wrote:
Correct, it's not being turned into a web page. Is a SAX
pipeline
part of Cocoon? I'm still on Cocoon 2.1, but I've never heard that
term specifically, I thought the default generator did use SAX.
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Saturday, May 08, 2010 12:06 AM
To: users@cocoon.apache.org
Subject: Re: Very large Generator file
Hi Jeff,
That depends a great deal on what you want to do exactly with
the
large
file. I guess you do not want to transform it into one huge web
page?
In general, if you have to deal with lengthy inputs, a SAX
pipeline
is
the way to go.
Jos
On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote:
Hello,
I was wondering if there’s a way to handle very large
files as
inputs to the default cocoon generator? As it is, I am
getting a
memory heap error when trying to process an input file
because
it’s
so
large. Am I going to have to process it in some other
manner, or
is
there a way to make Cocoon perform the transform for me?
Thanks,
Jeff
---
--
To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org
For additional commands, e-mail: users-h
RE: Very large Generator file
Yes, Jeff, my guess is that the tfactory stuff will accomplish the job
for you without complaining about memory shortage! Your tomcat instance
may have other things around increasing the memory footprint all the
time.
On Mon, 2010-05-10 at 09:43 -0500, Schmitz, Jeffrey A wrote:
Yes, the resulting XHTML file is not for online use.
And yes, I've given max memory to the tomcat instance. So, will having my
specialized transformer do the xsl transform directly using a SAX transformer
not work either? e.g. something like:
TransformerFactory tfactory = TransformerFactory.newInstance();
Transformer transformer = tfactory.newTransformer(new StreamSource(
xslID));
transformer.transform(new StreamSource(sourceFile), new StreamResult(
new File(modelFile)));
I guess I was under the impression that SAX based xsl transformers didn't use
much memory regardless file size, but perhaps that's not true?
If that's not going to work, would moving to a 64 bit machine so I can give
more memory to Tomcat be an option (is there a 64 bit Tomcat)?
If this is the case a short java program can make that xhtml file for
you.
I'm not exactly sure what you're suggesting here. Do you mean for the
tomcat/cocoon process to exec a separate java program that will do the xsl
transform on the large file? Won't any java program come up against the 2
Gig max memory limitation?
Thanks,
Jeff
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Monday, May 10, 2010 9:30 AM
To: users@cocoon.apache.org
Subject: RE: Very large Generator file
Hi Jeff,
If you have been trying giving all the available memory to your tomcat
instance (JVM options) you are at the end of you tether for the current
solutions, unless somebody has a different idea.
I understood from your previous mail that the resulting xhtml is not
for
on-line use, it is a sort of report/summary.
If this is the case a short java program can make that xhtml file for
you.
Cheers,
Jos
On Mon, 2010-05-10 at 09:17 -0500, Schmitz, Jeffrey A wrote:
Hi Jos,
I've already written my own Transformer (extends
AbstractSAXTransformer) that currently takes the output of the default
cocoon xsl transformer (via the sitemap) and performs subsequent, non-
xsl transforms, writes the results to file and generates the returned
web-page
e.g.
map:match pattern=*/*/*In/*/*In.flow/*
map:generate src=inputs/{4} /
map:transform src=xsl/{3}InAdapter.xsl
map:parameter name=instanceName value={2}/
/map:transform
map:transform type=SPARQLinTrans
map:parameter name=inputFile value={4}/
map:parameter name=SPARQLFlow value={5}/
/map:transform
map:serialize type=xhtml/
/map:match
I'm currently heading in the direction of just adding the initial xsl
transformation as part of my own Transformer's duties (I think your
option 3) so that it can hopefully handle the larger files, but I was
wondering if there might be a way to easily keep things as they are and
continue using the default cocoon xsl tranform as the first step.
Sounds like there's not?
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Monday, May 10, 2010 8:39 AM
To: users@cocoon.apache.org
Subject: RE: Very large Generator file
A SAX pipeline is part of cocoon.
Can you explain briefly what you want to do?
Consider using a cocoon-3 pipeline.
Consider using a SAX transformation:
- read xml
- do something with it (transforming, for instance)
- write output to a file
Jos
On Mon, 2010-05-10 at 08:23 -0500, Schmitz, Jeffrey A wrote:
Correct, it's not being turned into a web page. Is a SAX
pipeline
part of Cocoon? I'm still on Cocoon 2.1, but I've never heard that
term specifically, I thought the default generator did use SAX.
-Original Message-
From: Jos Snellings [mailto:jos.snelli...@pandora.be]
Sent: Saturday, May 08, 2010 12:06 AM
To: users@cocoon.apache.org
Subject: Re: Very large Generator file
Hi Jeff,
That depends a great deal on what you want to do exactly with
the
large
file. I guess you do not want to transform it into one huge web
page?
In general, if you have to deal with lengthy inputs, a SAX
pipeline
is
the way to go.
Jos
On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote:
Hello,
I was wondering if there’s a way to handle very large
files as
inputs to the default cocoon generator? As it is, I am
getting a
memory heap error when trying to process an input file
because
it’s
so
RE: Very large Generator file
Thanks Jos, I have tried moving the tfactory stuff into my specialized cocoon transformer (com.boeing.transform.SPARQLinTrans), and I've also tried processing the file in a stand-alone java program with 1 Gig of memory assigned to it, and in both cases I'm still running out of heap space (see exception stack below). So, the current problem must lie in the xsl transform libraries I'm using (i.e. not really a cocoon problem at this point). I know there must be techniques for doing xsl on very large files, do you have any pointers or ideas for this problem in general? Thanks! Jeff java.lang.OutOfMemoryError: Java heap space net.sf.saxon.tinytree.TinyTree.ensureNodeCapacity(TinyTree.java:215) net.sf.saxon.tinytree.TinyTree.addNode(TinyTree.java:309) net.sf.saxon.tinytree.TinyBuilder.characters(TinyBuilder.java:306) net.sf.saxon.event.ReceivingContentHandler.flush(ReceivingContentHandler.java:544) net.sf.saxon.event.ReceivingContentHandler.startElement(ReceivingContentHandler.java:275) org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) org.apache.xerces.parsers.XMLParser.parse(Unknown Source) org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) net.sf.saxon.event.Sender.sendSAXSource(Sender.java:354) net.sf.saxon.event.Sender.send(Sender.java:184) net.sf.saxon.event.Sender.send(Sender.java:49) net.sf.saxon.Controller.transform(Controller.java:1550) com.boeing.transform.SPARQLinTrans.adapt(SPARQLinTrans.java:143) Of course this is still a part of the Tomcat process. -Original Message- From: Jos Snellings [mailto:jos.snelli...@pandora.be] Sent: Monday, May 10, 2010 10:04 AM To: users@cocoon.apache.org Subject: RE: Very large Generator file Yes, Jeff, my guess is that the tfactory stuff will accomplish the job for you without complaining about memory shortage! Your tomcat instance may have other things around increasing the memory footprint all the time. On Mon, 2010-05-10 at 09:43 -0500, Schmitz, Jeffrey A wrote: Yes, the resulting XHTML file is not for online use. And yes, I've given max memory to the tomcat instance. So, will having my specialized transformer do the xsl transform directly using a SAX transformer not work either? e.g. something like: TransformerFactory tfactory = TransformerFactory.newInstance(); Transformer transformer = tfactory.newTransformer(new StreamSource( xslID)); transformer.transform(new StreamSource(sourceFile), new StreamResult( new File(modelFile))); I guess I was under the impression that SAX based xsl transformers didn't use much memory regardless file size, but perhaps that's not true? If that's not going to work, would moving to a 64 bit machine so I can give more memory to Tomcat be an option (is there a 64 bit Tomcat)? If this is the case a short java program can make that xhtml file for you. I'm not exactly sure what you're suggesting here. Do you mean for the tomcat/cocoon process to exec a separate java program that will do the xsl transform on the large file? Won't any java program come up against the 2 Gig max memory limitation? Thanks, Jeff -Original Message- From: Jos Snellings [mailto:jos.snelli...@pandora.be] Sent: Monday, May 10, 2010 9:30 AM To: users@cocoon.apache.org Subject: RE: Very large Generator file Hi Jeff, If you have been trying giving all the available memory to your tomcat instance (JVM options) you are at the end of you tether for the current solutions, unless somebody has a different idea. I understood from your previous mail that the resulting xhtml is not for on-line use, it is a sort of report/summary. If this is the case a short java program can make that xhtml file for you. Cheers, Jos On Mon, 2010-05-10 at 09:17 -0500, Schmitz, Jeffrey A wrote: Hi Jos, I've already written my own Transformer (extends AbstractSAXTransformer) that currently takes the output of the default cocoon xsl transformer (via the sitemap) and performs subsequent, non- xsl transforms, writes the results to file and generates the returned web-page e.g
Very large Generator file
Hello, I was wondering if there's a way to handle very large files as inputs to the default cocoon generator? As it is, I am getting a memory heap error when trying to process an input file because it's so large. Am I going to have to process it in some other manner, or is there a way to make Cocoon perform the transform for me? Thanks, Jeff
Re: Very large Generator file
Hi Jeff, That depends a great deal on what you want to do exactly with the large file. I guess you do not want to transform it into one huge web page? In general, if you have to deal with lengthy inputs, a SAX pipeline is the way to go. Jos On Fri, 2010-05-07 at 18:14 -0500, Schmitz, Jeffrey A wrote: Hello, I was wondering if there’s a way to handle very large files as inputs to the default cocoon generator? As it is, I am getting a memory heap error when trying to process an input file because it’s so large. Am I going to have to process it in some other manner, or is there a way to make Cocoon perform the transform for me? Thanks, Jeff - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
