Hi Eric, Now I read carefully your post. I saw you are already using instrumentation.;-)
Can you post the sizes of the returned pages in the thread? I guess this is an important point too. Also, consider to move to cocoon 2.1.6, now here is why: http://issues.apache.org/bugzilla/show_bug.cgi?id=31760 I saw you canged some pools inside tomcat. Did you changed some cocoon pools? Did you adjusted cocoon memory usage inside cocoon.xconf? Best Regards, Antonio Gallardo. On Jue, 17 de Marzo de 2005, 11:30, Eric E. Meyer dijo: > Cocoon Performance Woes, Is it flow? I don't know! > > Hello all, > > My team developed and deployed a web application for a client which is > built on top of Cocoon (www.fivestaralliance.com). > > I have spent over two weeks now attempting to improve the > performance and scalability of the application – with some real > improvements. However, I continue to feel like I'm flying blind – > because the apparent bottlenecks are somewhere outside of my code. I > have read a number of previous debates about javascript flow being slow, > and we are using CForms with Javascript flow, but my main concern is > that I cannot determine where the bottlenecks are. > > Even when the application is bogging down, the components that my team > wrote are performing their tasks (generation, transformation, > flowscripting) at very fast rates (as seen by tracing/logging), but > something else in the framework/process is bogging down the requests. > The analysis tools don't show any obvious resource limitation even at > the highest loading levels. > > Some pages in the site scale quite well, while one in particular does > not scale well at all. While I know that I'm close to having a system > that runs blindingly fast, I'm currently faced with a situation where I > cannot effectively argue that the architecture isn't "fundamentally > flawed" and I'm unable to address a major scalability concern for my > client. I would welcome any concrete suggestions on how to better > determine my bottlenecks and any additional tuning advice. > > Brief description of the application architecture > > CForms, Javascript flow, mix of JX template generators, XInclude > transformer and custom generator transformers. Core application > components implemented in Java. Hibernate persistence. > > Profiling and Monitoring > > My biggest problem is that I've only been able to determine where the > problem isn't at this point. I've used a variety of tools to attempt to > see what's going on, and why the application is bogging down, but I > cannot seem to get a comprehensive picture of what delays/bottlenecks > there are within Cocoon. > > Specifically, it would be extremely helpful to monitor the number of > generators/transformers/other pooled components in use, allocated, freed > while under load. Additionally, it would be useful to see the time taken > up by each of the steps in the process of servicing a request – not just > the set-up and generation/transform times as shown by the Cocoon profiler. > > These are the tools/approaches that I have used: > > Multi-thread load test with JMeter. > Profiling of application code using JProbe (CPU and memory analysis). > Profiling of Cocoon components using Cocoon pipeline profiler. > Monitoring of Cocoon components using the Cocoon instrumentation client. > Monitoring of Cocoon server using Status generator. > Monitored Linux system activity with SAR, iostat, mpstat and vmstat > during load-testing. > Profiling of custom generators, transformers and flowscript using > Jakarata Commons StopWatch and log statements. > > Tweaks made thus far > > Adjusted Java virtual machine parameters > -server -Xms512M -Xmx512M > Adjust logging levels - turned down logging > Adjusted thread pool sizes in Tomcat > 150 -> 350 max > Adjusted database connection pool size up to 50 > Adjusted sitemap component pool sizes up > Optimized some Java code based upon JProbe profiling > Added additional objects to the in-memory cache to reduce database queries > Turned off reloading of sitemaps and javascript files > Replaced default Cocoon JCS cache with Whirlycache > Replaced default Cocoon Xalan XSL transformer with faster Saxon XSL > transformer > Configured Cocoon to reuse XML parsers > Removed Cocoon store janitor > Preloading key OO javascript flowscript at server startup > > Observations > > Windows XP > Pentium 4 1.8Ghz > JDK 1.4.2_06 > Tomcat 5.0.28 > Cocoon 2.1.5.1 > 512MB physical RAM > JVM -server -Xms256M -Xmx256M > > Load test with num users threads each making 5 successive request in a > loop with approximately 3 second think time between requests. No derived > resources – only the main page. > > users overall home search1 search2 search3 detail total > avg ms page num reqs > > 10 486 208 637 588 644 355 500 > 20 1704 378 2684 1837 2875 745 500 > 30 3725 682 5987 4626 6270 1059 450 > 40 19461 1411 36021 23089 34726 2059 600 > 50 72942 3213 130482 90993 131666 8356 500 > > home page: / > search1: /luxury_hotels/europe__france__paris/index.html > search2: /luxury_hotels/bahamas_%26_the_caribbean/beach_resort/index.html > search3: > /luxury_hotels/europe__france__paris/city_centre_location/index.html > detail: /luxury_hotel/new_york,_ny/the_carlyle > > Platform: > > Development > Windows XP > JDK 1.4.2_06 > Tomcat 5.0.28 > Cocoon 2.1.5.1 > > Deployment > Linux 2.6.x > > We see similar degradation on Linux as on Windows. > > The home page has no flowscript or cforms, but does have jxtemplate > generation, xinclude, xslt, and a custom generator. > > The search and detail pages include a cform, and are therefore driven > with flowscript at the top-level matching (and create continuations in > the process of displaying their forms). These pages use jxtemplate > generation, xinclude, xslt, custom generation, custom transformation, > and internal-only sub pipelines. When looking at the pipeline times with > a profiling pipeline, the total times (while under load) are much higher > that the displayed times for the setup and generation steps -- so where > is the time going? > > Regards, > Eric Meyer > > > >
