Hey everyone, An update:
* We installed FusionReactor * Deployed the site on a development box running CF Enterprise * Ran a number of load tests and tests on discrete parts of the application * Did various application and memory monitoring using both FusionReactor and CF's Server Monitor * Tweaked some more code based on the tests, including: ** fixing a few application errors; ** optimizing a few cfc methods and queries that were slow: ** reconfirming that we don't have any improperly scoped variables in the app, and that there wasn't any weird recursion or referencing of objects ** converting some pages that require complex but more or less static database queries into includes that get generated when the data changes in the database when a new data import is performed. ** temporarily commenting out some code that invokes cfx_image3 to generate artifact thumbnails. * Changed CF's Query Caching settings to zero, temporarily. * Retested * Played with various JVM settings that people suggested. No luck. Memory still keeps climbing. In fact, on some of the search results requests that return a large number of records you can watch Jrun grab a good 15-25 megs and not let it go. If you keep hitting the same page you can quickly force Jrun to hit it's buffer and stop responding. You shut this site down and restart Jrun in either dev or production and the server runs beautifully, no memory problems. Here's the weird thing: when you analyze these requests and the server state while running the load test, the total amount of memory being consumed by application and session variables, objects created and destroyed during the request cycle, even the size of the data returned from the queries, is completely insignificant: * The total amount of memory the application's objects that are being stored in the application scope is less than a Meg. * User sessions rarely take up more than a few kilobytes. * There's nothing in the server scope. * The largest query object clocks in at well under a megabyte. Moreover, in the CF Server Monitor it shows that CF is actually destroying the objects as expected and releasing memory back into the pool. However, if you actually look at Jrun in the Task Manager, it's not. And, hitting Run GC to invoke garbage collection in the Server Monitor doesn't get the memory back either. I'm running out of ideas. The last couple of things on our list are to rewrite some of the SQL to use SQL 2005's paging functionality (haven't got to that yet), and to try eliminating one technique we've been using. Some of our application-scoped singletons store a reference to our app's SessionFacade within themselves. This should be fine, because the SessionFacade has NO private/public properties, and when we invoke the SessionFacade's methods inside the other objects we're doing so according to best practices: declaring a variable that is private, i.e., <cfset otherLang = variables.sessionFacade.getOtherLanguage("en") /> Might we have run into a memory leak in the native SQL Server database driver? Or some kind of obscure JVM problem? Or maybe some kind of recursion hidden way in some cfc or view code that isn't causing a request to hang, but is somehow chewing up memory? My next steps after this might be to engage Adobe or maybe Charlie Arehart's company. Anyone have experience dealing with either? Thanks, Sean ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Adobe® ColdFusion® 8 software 8 is the most important and dramatic release to date Get the Free Trial http://ad.doubleclick.net/clk;192386516;25150098;k Archive: http://www.houseoffusion.com/groups/CF-Talk/message.cfm/messageid:307370 Subscription: http://www.houseoffusion.com/groups/CF-Talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.4