RE: Memory Usage Question (HEAPDUMP Included)
One again my memory usage was steady at 20%, something happened and now its up and down from 15% to 90% and back down within minutes. Here is a screen shot of the memory graph in seeFusion. GC seems to be doing its job, but this does not seem normal or does it!? http://screencast.com/t/6SkU3bF9Gsx So I did a heapdump and analyzed it in MAT for eclipse: When I took the dump memory on the server was reported at 90% by seefusion. The JVM has 1 gig allocated to it. The dump doesn't show near that much memory in use, so maybe a GC ran right before I took the dump, I'm not sure. Does anyone see anything that stands out in these reports? Should there be that many jdbc.base.preparedStatement items? http://screencast.com/t/Th4ZiLZw http://screencast.com/t/IdGqa6Uh On another note, its pretty cool that you can see whats stored in the application scope and its memory usage. This screenshot shows the structs/arrays/strings that are application scoped: http://screencast.com/t/XZyRofWTJNN Brook -Original Message- From: Mack [mailto:mrsmith.w...@gmail.com] Sent: December-06-10 4:07 AM To: cf-talk Subject: Re: Memory Usage Question I wish I could see WHAT was taking up the ram, i.e a specific array, cfc instances, etc etc It can be done but it's not easy. Basically you need to do a heap dump of the CF process (you can do that without shutting down the server) and then analyze the heap dump using MAT for Eclipse. MAT can tell you what are the objects on the heap, etc. -- Mack ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339908 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: Memory Usage Question (HEAPDUMP Included)
On Wed, Dec 8, 2010 at 6:09 PM, Brook Davies cft...@logiforms.com wrote: One again my memory usage was steady at 20%, something happened and now its up and down from 15% to 90% and back down within minutes. Here is a screen shot of the memory graph in seeFusion. GC seems to be doing its job, but this does not seem normal or does it!? http://screencast.com/t/6SkU3bF9Gsx Considering that the number of requests is not increasing this shouldn't happen (unless requests after 7:00am retrieve much more records from the database for example) So I did a heapdump and analyzed it in MAT for eclipse: When I took the dump memory on the server was reported at 90% by seefusion. The JVM has 1 gig allocated to it. The dump doesn't show near that much memory in use, so maybe a GC ran right before I took the dump, I'm not sure. Your total memory seems low in the screenshot (aprox 200M) so I suspect that you took the heap-dump right after a gc. Does anyone see anything that stands out in these reports? Should there be that many jdbc.base.preparedStatement items? http://screencast.com/t/Th4ZiLZw http://screencast.com/t/IdGqa6Uh In my experience 2775 is not that big for the number of PreparedStatements. -- Mack ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339914 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: Memory Usage Question
I wish I could see WHAT was taking up the ram, i.e a specific array, cfc instances, etc etc It can be done but it's not easy. Basically you need to do a heap dump of the CF process (you can do that without shutting down the server) and then analyze the heap dump using MAT for Eclipse. MAT can tell you what are the objects on the heap, etc. -- Mack ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339804 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: Memory Usage Question
Java byte code - that is what is going to be in memory. I expect a CFC etc would relate to a particular Java object however while I can dump what is in RAM have not learned how to understand the output and join a CFM to an object. I might be wrong but I don't think you can dump the permanent JVM space so you don't have access to the actual bytecode. I think you can use jmap -permstat to see what classes are loaded and that might give you an idea of what CFC are loaded, etc I wish I could see WHAT was taking up the ram, i.e a specific array, cfc instances, etc etc I think JVM logging or even Java Dev Kit programs like Jconsole and Jvisualvm will perhaps show which Java memory buffer (New/Tenure/Perm) is being consumed. CF monitor will probably help you see what CFM CFC etc is running. So once you know which buffer is being consumed perhaps you can allocate more to that until your able to short out which CFM etc is taking the RAM. Jvisualm is a good suggestion and gives you some of insight into the JVM. -- Mack ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339806 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: Memory Usage Question
Java byte code - that is what is going to be in memory. I expect a CFC etc would relate to a particular Java object however while I can dump what is in RAM have not learned how to understand the output and join a CFM to an object. I wish I could see WHAT was taking up the ram, i.e a specific array, cfc instances, etc etc I think JVM logging or even Java Dev Kit programs like Jconsole and Jvisualvm will perhaps show which Java memory buffer (New/Tenure/Perm) is being consumed. CF monitor will probably help you see what CFM CFC etc is running. So once you know which buffer is being consumed perhaps you can allocate more to that until your able to short out which CFM etc is taking the RAM. Previously you mention SeeFusion. I have no recent SeeFusion experience to comment very much, perhaps you can check the buffers make JVM memory changes to cope and check what CFM CFC is running. Regards, Carl. It would be good to know what CF version and edition and what Java is CF using? Regards, Carl. ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339791 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: Memory Usage Question
Brook, At first glance, I'd say you need a consultant like Charlie Arehart ( http://www.carehart.org/) to see why your application is performing like this. Usually, there is a way to fix the offending memory hole. If you think you can do it, hire Charlie for an hour or so and he will give you some pointers. If your staff doesn't have the talent he can probably fix it completely. There are some alternatives to that plan including hiring Adobe support, or find and hire another expert. Some names that come to mind with deep server experience are Mike Brunt Steven Erat. Ok, back to your problem. 1. Windows task manager reports used memory by default, not allocated memory. Turn on the Virtual Memory Size column in task manager and you should see the full amount. 2. I don't really know about SeeFusion and what it reports on. Maybe ask them about it? 3. It sounds like the request is still running even though the browser request has ended. I think overall what you need to do is fix the offending code. You have a memory leak. It's not ColdFusion, it's not the server, it's your CFML. I would guess that the client storage / array looping operation is a pretty hairy one. Move your client storage into actual database tables and columns and turn off client variables. Think about how you may be able to pre or post-process some of the long-running operations. Run them on a different server. If you have a lot of database operations, cut them down. If you have a lot of components, cut them down. Oh, and good luck. nathan strutz [http://www.dopefly.com/] [http://hi.im/nathanstrutz] On Fri, Dec 3, 2010 at 10:51 AM, Brook Davies br...@logiforms.com wrote: Hello, I have a question about a memory usage pattern that I am trying to figure out. I am using seefusion to watch our server and this is what I am seeing: 1. We have two webservers and a loadbalancer with equal distribution and sticky sessions enabled 2. Memory usage will be constant around 20%-30% in use throughout the day. 3. When I check the taskManager on the machine I see jrun using only 300-400 megs. 4. The arguments to the JVM are -Xms1024m -Xmx1024m -XX:MaxPermSize=384m -XX:+UseParallelGC 5. The app uses client variable storage with a MSSQL database backend 6. The client variable data per client (in some edge cases) can grow to be very large What I am seeing is a specific customer comes along and uses our app and they have some very large strings of data stored in the client scope and invoke some operations that do a lot of array processing on large arrays. These requests are big and slow. We see a big spike in the memory usage. It continues to grow and grow. Then the client appears to disconnect and leave the website (in fact, we verified by banning their IP to test). But the memory never gets released. The task manager now reports jrun up at over 1 gig of memory. We have a script that runs every 30 seconds that checks memory usage and does a full GC when less than 100 megs of memory remain. Prior to implementing that script, we were seeing some out of memory errors. While watching seefusion we can see the memory creep up to 90%+, and then to 10% when the GC script runs. But it immediately begins to climb again. And this process repeats. The traffic on this server is no different than our other server which sits at 19% memory usage. Now, if we restart CF, the problem goes away, and memory usage stays at under 20%. My questions are: 1. I understand that setting -Xms1024m -Xmx1024m should immediately allocate that amount of ram to the JVM. So why does the taskManager report the memory usage at 300 megs (prior to the event)? 2. If a bunch of requests cause memory usage to spike and then those requests die off, should seefusion show memory usage at 90% because thats whats allocated to the JVM and now 'claimed'? So would that be a normal operating condition? 3. What would cause the memory to immediately climb back up after a full GC after 'something' happened but not climb after the service is restarted? Thanks for your insights guys! Brook ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339772 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Re: Memory Usage Question
Hi Brook, You do not say what Cf version and JVm is in use eg CF8.01 Java 1.6.0_04 1. I understand that setting -Xms1024m -Xmx1024m should immediately allocate that amount of ram to the JVM. So why does the taskManager report the memory usage at 300 megs (prior to the event)? Is TASKMAN showing memory committed size? Setting initial size to same as maximum should likely take more memory initially at start-up. Perhaps enable JVM logging and check the log file for memory use (reply if you need more details on how). Is this Server or Enterprise. If Enterprise are you able to run CF Monitor and see the RAM use? 3. What would cause the memory to immediately climb back up after a full GC after 'something' happened but not climb after the service is restarted? A CFM is sill running and Java objects are getting committed in to memory. CF Monitor might assist. HTH, Carl. ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339784 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
RE: Memory Usage Question
I wish I could see WHAT was taking up the ram, i.e a specific array, cfc instances, etc etc Brook -Original Message- From: Carl Meyer [mailto:ca...@tassweb.com.au] Sent: December-04-10 1:00 AM To: cf-talk Subject: Re: Memory Usage Question Hi Brook, You do not say what Cf version and JVm is in use eg CF8.01 Java 1.6.0_04 1. I understand that setting -Xms1024m -Xmx1024m should immediately allocate that amount of ram to the JVM. So why does the taskManager report the memory usage at 300 megs (prior to the event)? Is TASKMAN showing memory committed size? Setting initial size to same as maximum should likely take more memory initially at start-up. Perhaps enable JVM logging and check the log file for memory use (reply if you need more details on how). Is this Server or Enterprise. If Enterprise are you able to run CF Monitor and see the RAM use? 3. What would cause the memory to immediately climb back up after a full GC after 'something' happened but not climb after the service is restarted? A CFM is sill running and Java objects are getting committed in to memory. CF Monitor might assist. HTH, Carl. ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339789 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
Memory Usage Question
Hello, I have a question about a memory usage pattern that I am trying to figure out. I am using seefusion to watch our server and this is what I am seeing: 1. We have two webservers and a loadbalancer with equal distribution and sticky sessions enabled 2. Memory usage will be constant around 20%-30% in use throughout the day. 3. When I check the taskManager on the machine I see jrun using only 300-400 megs. 4. The arguments to the JVM are -Xms1024m -Xmx1024m -XX:MaxPermSize=384m -XX:+UseParallelGC 5. The app uses client variable storage with a MSSQL database backend 6. The client variable data per client (in some edge cases) can grow to be very large What I am seeing is a specific customer comes along and uses our app and they have some very large strings of data stored in the client scope and invoke some operations that do a lot of array processing on large arrays. These requests are big and slow. We see a big spike in the memory usage. It continues to grow and grow. Then the client appears to disconnect and leave the website (in fact, we verified by banning their IP to test). But the memory never gets released. The task manager now reports jrun up at over 1 gig of memory. We have a script that runs every 30 seconds that checks memory usage and does a full GC when less than 100 megs of memory remain. Prior to implementing that script, we were seeing some out of memory errors. While watching seefusion we can see the memory creep up to 90%+, and then to 10% when the GC script runs. But it immediately begins to climb again. And this process repeats. The traffic on this server is no different than our other server which sits at 19% memory usage. Now, if we restart CF, the problem goes away, and memory usage stays at under 20%. My questions are: 1. I understand that setting -Xms1024m -Xmx1024m should immediately allocate that amount of ram to the JVM. So why does the taskManager report the memory usage at 300 megs (prior to the event)? 2. If a bunch of requests cause memory usage to spike and then those requests die off, should seefusion show memory usage at 90% because thats whats allocated to the JVM and now 'claimed'? So would that be a normal operating condition? 3. What would cause the memory to immediately climb back up after a full GC after 'something' happened but not climb after the service is restarted? Thanks for your insights guys! Brook ~| Order the Adobe Coldfusion Anthology now! http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:339758 Subscription: http://www.houseoffusion.com/groups/cf-talk/subscribe.cfm Unsubscribe: http://www.houseoffusion.com/groups/cf-talk/unsubscribe.cfm
