Have you tuned the JVM heap size and garbage collection? It's not a "fix" but sometimes, that's what you've gotta do. It might keep things running while you re architect things at least.
I had a long-running task like that in one of my apps and one thing that helped was to break it up into several smaller steps. Each one would run more quickly and require less memory. On 1/30/13 10:19 AM, Nathan Strutz wrote: >>> I see that there are millions of objects in the memory and many of them > are LoopTags and QueryTags > > Is that from a java thread dump or something else? The thread dump should > show you everything the JVM is doing. > > It actually sounds like your server is busy doing a lot of things. Could it > be that you just need to build your infrastructure, split your application > across more servers? At the very least, tune your JVM settings to > accommodate what the application is doing or bring down your active request > threads in the CF admin. > > The improvements you mentioned, while some of them are beneficial (vars and > queryparams), others are probably not (query loops to index loops). Did you > try the VarScoper tool? It's amazing! > > It sounds like you're doing a lot of threading, which makes things faster, > but also makes the server busier. Busier servers mean more things happening > at once, which means you're using more memory, making your problem worse. > Putting loops in cfthread won't make the loops faster; breaking off logical > parts of your application into their own threads could make your > application faster. Too many threads can cause memory leaks depending on > what you are doing, for example, Marc Esher's cfconcurrent project > discovered some big memory leaks when you do certain things with the > cfconcurrent library on CF9. > > > nathan strutz > [www.dopefly.com] [hi.im/nathanstrutz] > > > On Wed, Jan 30, 2013 at 3:27 AM, Helwig, Till Helge > <till.hel...@saxsys.de>wrote: > >> Hi, >> >> I'm working on a very old and very badly coded CF application at the >> moment. It contains servertasks that do a lot of batch processing where >> huge data sets are queried from a database, processed and the results >> written back to the DB. Unfortunately by now the jobs have become so large >> that we frequently get heap space errors. I'm trying to figure out why the >> process takes up so much memory and I've reached a point where I just don't >> know what to look for anymore. When looking at the memory monitor it looks >> to me like in every loop iteration he accumulated about 100 MB of data, >> procceses it and throws it away, but not entirely. A small piece stays in >> the memory every time and in the end it clutters up all the heap space. >> >> What I already did: >> >> - Adding "var" to local variables >> >> - Replacing literals in queries with cfqueryparams >> >> - Replacing StructNew() at the beginning of loops with >> StructClear() >> >> - Replacing query-loops with index-loops >> >> I also started looking at dumps of the Java heap shortly before the >> process crashes, but I'm not really getting any wiser from that. I see that >> there are millions of objects in the memory and many of them are LoopTags >> and QueryTags, but other than that I have no idea what to make of it. >> >> I've read that placing the code of the loop within its own thread might >> help, but that's impossible because our task already runs in its own thread >> and apparently CF can't spawn threads from within threads. >> >> If you have any ideas for me where to look for memory leaks, what else to >> try or an explanation for what might cause the problem, I would be really >> really thankful! >> >> Greetings, >> Till Helge >> >> >> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:354165 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
