Re: Cocoon 2.2 production server configuration
Hello Jeroen, i've also commented inline: On Tue, Sep 13, 2011 at 10:59 AM, Jeroen Reijn j.re...@onehippo.com wrote: Hi, see my comments inline: On Fri, Sep 9, 2011 at 10:37 AM, Brecht Schoolmeesters brecht.schoolmeest...@zappware.com wrote: Hi, we are currently using cocoon for transforming source xml files into xml files in a different format. The transformations are pretty heavy because data from other xml files is inserted and also transformed. When using cocoon on our test server, we saw that the memory usage was peaking at around 1.1Gb RAM. We were testing with about 10 clients sending requests every 1.5second. Is this normal memory usage for cocoon? That depends. It would be interesting for us to know which version of Cocoon you are using to begin with. Processing XML can always be quite memory intensive depending on the size of the XML being processed and the amount and type of transformations taking place. As the subject of this mail states, we're using cocoon 2.2 ;) The xml files we are transforming are not large (about 400-500lines). We are doing one special thing: we have a pipeline that generates an xslt transformation file which is then used in another pipeline in a transformer: map:transform src=cocoon:/otherpipe / We would also like some more information about the scalability. Will cocoon use a lot more memory when a few more clients connect? I'm not aware of any figures, but every request will of course take more memory, but I'm not sure what is 'a lot'. That mostly depends on the kind of transformations your application is doing. Which configuration files/settings can we change to enable cocoon to handle more requests? We would like to be able to handle about 40requests/second at peak times. I'm not sure, that might depend on the interaction between the client and the cocoon server. 40 requests/second should definitely be possible, but that can depend on what Cocoon is doing. Is for instance the XML processing 'static' in certain situations? Then you might be thinking of caching the result of a certain pipeline or component output. That way you will be able to handle a lot more requests per second. Caching is enabled for the static pipelines. What type of server is recommend for this type of usage? How much RAM, CPU power? Hmm that's a good question. I'm not really an infra guys, so I do not know the specs. And one final question: are there any cocoon components that are optional and therefore can be disabled? If yes, how can we check which components are optional and how can we disable them? Will this help decrease the total memory usage? This might help, but I'm not sure if disabling components will make it use less memory. As far as I know most memory will be taken once a component is triggered in the request cycle. Where can I disable these components and how should this be done? Cheers, Jeroen thanks in advance, best regards, Brecht best regards, Brecht -- Amsterdam - Oosteinde 11, 1017 WT Amsterdam Boston - 1 Broadway, Cambridge, MA 02142 US +1 877 414 4776 (toll free) Europe +31(0)20 522 4466 www.onehippo.com - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
Re: Cocoon 2.2 production server configuration
Hi Robby, thanks for your reply. I think the memory leak is causing the high memory usage. I decompiled the class file we use in our cocoon server and it uses the unpatched PoolableProxyHandler. How can I apply the patch to the cocoon version we are using (stable 2.2.0 version from the site)? I tried checking out the trunk version and compile it, but it gives some missing daisy depency errors. The source files for the stable version are also on the subversion server but there is no build script included. How did you apply the patch? Is it possible to send me your cocoon-sitemap-impl.jar file or should I compile it myself? thanks in advance, best regards, Brecht On Tue, Sep 13, 2011 at 1:52 PM, Robby Pelssers robby.pelss...@nxp.comwrote: Hi Brecht, ** ** Just wondering how large the data is you are processing. We have a cocoon 2.2 app running stable with 512mb of memory. And I’m pretty sure it even uses less memory after we solved a memory leak. Check this thread http://old.nabble.com/memory-leak-in-PoolableProxyHandler-td29023190.html* *** ** ** Can you perhaps isolate a single pipeline execution which would require that much memory? If so it would be interesting to see if I can replicate this in my setup depending on the complexity of course. ** ** Robby ** ** ** ** ** ** ** ** *From:* Brecht Schoolmeesters [mailto:brecht.schoolmeest...@zappware.com] *Sent:* Friday, September 09, 2011 10:38 AM *To:* users@cocoon.apache.org *Subject:* Cocoon 2.2 production server configuration ** ** Hi, ** ** we are currently using cocoon for transforming source xml files into xml files in a different format. The transformations are pretty heavy because data from other xml files is inserted and also transformed. ** ** When using cocoon on our test server, we saw that the memory usage was peaking at around 1.1Gb RAM. We were testing with about 10 clients sending requests every 1.5second. Is this normal memory usage for cocoon? We would also like some more information about the scalability. Will cocoon use a lot more memory when a few more clients connect? ** ** Which configuration files/settings can we change to enable cocoon to handle more requests? We would like to be able to handle about 40requests/second at peak times. ** ** What type of server is recommend for this type of usage? How much RAM, CPU power? And one final question: are there any cocoon components that are optional and therefore can be disabled? If yes, how can we check which components are optional and how can we disable them? Will this help decrease the total memory usage? ** ** ** ** thanks in advance, best regards, Brecht
Re: Cocoon 2.2 production server configuration
Hi Robby, thanks for your reply. I think the memory leak is causing the high memory usage. I decompiled the class file we use in our cocoon server and it uses the unpatched PoolableProxyHandler. How can I apply the patch to the cocoon version we are using (stable 2.2.0 version from the site)? I tried checking out the trunk version and compile it, but it gives some missing daisy depency errors. (see attached errors below) The source files for the stable version are also on the subversion server but there is no build script included. How did you apply the patch? Is it possible to send me your cocoon-sitemap-impl.jar file or should I compile it myself? thanks in advance, best regards, Brecht Missing: -- 1) daisy:daisy-repository-api:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-api -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-api -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-api:jar:1.5-dev 2) daisy:daisy-repository-xmlschema-bindings:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-xmlschema-bindings -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-xmlschema-bindings -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-xmlschema-bindings:jar:1.5-dev 3) daisy:daisy-repository-client-impl:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-client-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-client-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-client-impl:jar:1.5-dev 4) daisy:daisy-repository-common-impl:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-common-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-common-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-common-impl:jar:1.5-dev 5) daisy:daisy-repository-spi:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-spi -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-spi -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-spi:jar:1.5-dev 6) daisy:daisy-jmsclient-api:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-jmsclient-api -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-jmsclient-api -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1)
RE: Cocoon 2.2 production server configuration
Hi Brecht, I checked out the Cocoon sources from SVN and applied the patch locally and installed the module in our repo. I mailed you my patched version of the jar. Kind regards, Robby From: Brecht Schoolmeesters [mailto:brecht.schoolmeest...@zappware.com] Sent: Wednesday, September 14, 2011 12:57 PM To: users@cocoon.apache.org Subject: Re: Cocoon 2.2 production server configuration Hi Robby, thanks for your reply. I think the memory leak is causing the high memory usage. I decompiled the class file we use in our cocoon server and it uses the unpatched PoolableProxyHandler. How can I apply the patch to the cocoon version we are using (stable 2.2.0 version from the site)? I tried checking out the trunk version and compile it, but it gives some missing daisy depency errors. (see attached errors below) The source files for the stable version are also on the subversion server but there is no build script included. How did you apply the patch? Is it possible to send me your cocoon-sitemap-impl.jar file or should I compile it myself? thanks in advance, best regards, Brecht Missing: -- 1) daisy:daisy-repository-api:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-api -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-api -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-api:jar:1.5-dev 2) daisy:daisy-repository-xmlschema-bindings:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-xmlschema-bindings -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-xmlschema-bindings -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-xmlschema-bindings:jar:1.5-dev 3) daisy:daisy-repository-client-impl:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-client-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-client-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-client-impl:jar:1.5-dev 4) daisy:daisy-repository-common-impl:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-common-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-common-impl -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-common-impl:jar:1.5-dev 5) daisy:daisy-repository-spi:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-repository-spi -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=daisy -DartifactId=daisy-repository-spi -Dversion=1.5-dev -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cocoon:cocoon-sitemaptags2daisy-plugin:maven-plugin:1.0.0-SNAPSHOT 2) daisy:daisy-repository-spi:jar:1.5-dev 6) daisy:daisy-jmsclient-api:jar:1.5-dev Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=daisy -DartifactId=daisy-jmsclient-api -Dversion=1.5
Re: Cocoon 2.2 production server configuration
Hi, see my comments inline: On Fri, Sep 9, 2011 at 10:37 AM, Brecht Schoolmeesters brecht.schoolmeest...@zappware.com wrote: Hi, we are currently using cocoon for transforming source xml files into xml files in a different format. The transformations are pretty heavy because data from other xml files is inserted and also transformed. When using cocoon on our test server, we saw that the memory usage was peaking at around 1.1Gb RAM. We were testing with about 10 clients sending requests every 1.5second. Is this normal memory usage for cocoon? That depends. It would be interesting for us to know which version of Cocoon you are using to begin with. Processing XML can always be quite memory intensive depending on the size of the XML being processed and the amount and type of transformations taking place. We would also like some more information about the scalability. Will cocoon use a lot more memory when a few more clients connect? I'm not aware of any figures, but every request will of course take more memory, but I'm not sure what is 'a lot'. That mostly depends on the kind of transformations your application is doing. Which configuration files/settings can we change to enable cocoon to handle more requests? We would like to be able to handle about 40requests/second at peak times. I'm not sure, that might depend on the interaction between the client and the cocoon server. 40 requests/second should definitely be possible, but that can depend on what Cocoon is doing. Is for instance the XML processing 'static' in certain situations? Then you might be thinking of caching the result of a certain pipeline or component output. That way you will be able to handle a lot more requests per second. What type of server is recommend for this type of usage? How much RAM, CPU power? Hmm that's a good question. I'm not really an infra guys, so I do not know the specs. And one final question: are there any cocoon components that are optional and therefore can be disabled? If yes, how can we check which components are optional and how can we disable them? Will this help decrease the total memory usage? This might help, but I'm not sure if disabling components will make it use less memory. As far as I know most memory will be taken once a component is triggered in the request cycle. Cheers, Jeroen thanks in advance, best regards, Brecht -- Amsterdam - Oosteinde 11, 1017 WT Amsterdam Boston - 1 Broadway, Cambridge, MA 02142 US +1 877 414 4776 (toll free) Europe +31(0)20 522 4466 www.onehippo.com - To unsubscribe, e-mail: users-unsubscr...@cocoon.apache.org For additional commands, e-mail: users-h...@cocoon.apache.org
RE: Cocoon 2.2 production server configuration
Hi Brecht, Just wondering how large the data is you are processing. We have a cocoon 2.2 app running stable with 512mb of memory. And I’m pretty sure it even uses less memory after we solved a memory leak. Check this thread http://old.nabble.com/memory-leak-in-PoolableProxyHandler-td29023190.html Can you perhaps isolate a single pipeline execution which would require that much memory? If so it would be interesting to see if I can replicate this in my setup depending on the complexity of course. Robby From: Brecht Schoolmeesters [mailto:brecht.schoolmeest...@zappware.com] Sent: Friday, September 09, 2011 10:38 AM To: users@cocoon.apache.org Subject: Cocoon 2.2 production server configuration Hi, we are currently using cocoon for transforming source xml files into xml files in a different format. The transformations are pretty heavy because data from other xml files is inserted and also transformed. When using cocoon on our test server, we saw that the memory usage was peaking at around 1.1Gb RAM. We were testing with about 10 clients sending requests every 1.5second. Is this normal memory usage for cocoon? We would also like some more information about the scalability. Will cocoon use a lot more memory when a few more clients connect? Which configuration files/settings can we change to enable cocoon to handle more requests? We would like to be able to handle about 40requests/second at peak times. What type of server is recommend for this type of usage? How much RAM, CPU power? And one final question: are there any cocoon components that are optional and therefore can be disabled? If yes, how can we check which components are optional and how can we disable them? Will this help decrease the total memory usage? thanks in advance, best regards, Brecht
