How is it looking now? Any more crashes after the changes? I hope not. :-) By the way lol@elevator example. That was funny.
<Ajas Mohammed /> http://ajashadi.blogspot.com We cannot become what we need to be, remaining what we are. No matter what, find a way. Because thats what winners do. You can't improve what you don't measure. Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many alternatives. On Wed, Dec 7, 2011 at 3:14 PM, Cheyenne Throckmorton < cheyenne.throckmor...@gmail.com> wrote: > Fortunately (weird I know) we just had the server crash again this > afternoon. It at least gave me an excuse to throw some new arguments into > the jvm.config file to restart the cf service > > We went with suggestions from everybody and one from a ugtv webcast with > Carl Meyer that I've been watching to try and wrap my brain around JVM, > memory, permgen stuff. > > Here are the commands we changed > -Xmx1182m -XX:MaxPermSize=512m > > Here is a command we added since it the machine has 4 cores, we decided to > hopefully utilize some multi-threading to improve performance > -XX:ParallelGCThreads=4 > > The good thing about the crash was that I have pretty much ruled out any > one application being the culprit, as with inspection of several distinct > OOM permgen crashes we had different applications running and doing > different things. Over the last month, the hibernate application I think > is getting the finger pointed at it, only because I believe it uses the > most amount of permgen space with all of its dynamic loading. > > Perhaps its kind of like always blaming me (at 6'11 340lbs for those who > don't know me) for setting off the elevator overloaded alarm because I just > so happen to use up the most space, but I could have been on the elevator > just fine until 80 little kids jumped on and caused the problem of maxing > out the load/space. > > In that scenario, kicking me off the elevator isn't the solution, and > perhaps with these new commands we've just gotten a bigger elevator which > may work a little faster even though long-term we may need to investigate > putting the hibernate app on a diet. I will keep monitoring the situation > and update everyone as I learn more, for those of you that are just reading > to learn in case you run into this and for those or you expert server techs > that love peering into the mind of a coder stuck trying to learn server > maintenance problems. > > Cheyenne > > > On Wed, Dec 7, 2011 at 9:43 AM, Ajas Mohammed <ajash...@gmail.com> wrote: > >> On first look, I think you really need to go up on your max memory. On >> our servers, which are still on 32bit windows, we have min setting of 512MB >> and Max of 1182MB. >> >> >> # Arguments to VM >> java.args=-server -Xmx512m >> >> # Arguments to VM >> java.args=-server -Xmx1182m >> >> Having said that, like Charlie says always, its not a final solution to >> the problem but in your case I think its a good starting point.Unless you >> have some kind of other memory constraints, on this server which I dont >> know of, you should go up. >> >> Lot of people say, you should put both the min and max at same setting. >> In my case, that didnt work out, hence min at 512MB and Max 1182MB. >> >> About FusionReactor, I have to say there is nothing to learn. You will >> grasp in one day. I had written a getting started guide which was posted on >> FR community devnet site. >> http://www.fusion-reactor.com/community/developers/autodev_detailed.cfm?article=FRS-238I >> am sure they have more updated articles as mine was written at time of >> version 3.5.1 but still its a good start. >> >> Let us know if you need help with that. In your case, I would see System >> Metrics page, running requests page and also stack trace of the request to >> see what is going on. >> >> Others, correct me, if there is anything, I dont mind at all. :-) >> >> >> <Ajas Mohammed /> >> http://ajashadi.blogspot.com >> We cannot become what we need to be, remaining what we are. >> No matter what, find a way. Because thats what winners do. >> You can't improve what you don't measure. >> Quality is never an accident; it is always the result of high intention, >> sincere effort, intelligent direction and skillful execution; it represents >> the wise choice of many alternatives. >> >> >> On Wed, Dec 7, 2011 at 12:01 AM, Cheyenne Throckmorton < >> cheyenne.throckmor...@gmail.com> wrote: >> >>> You guys rock with all kinds of answers, I'll definitely be trying them >>> out tomorrow when I have a better brain and before whirly. Below is my >>> JVM.config file. >>> >>> Also for others maybe following the thread I found this 1 of 4 article >>> by Charlie helpful, but then for Charlie I was trying to find the other >>> parts after I had determined that I was indeed getting the outofmemory >>> errors in the -out file >>> >>> http://www.carehart.org/blog/client/index.cfm/2010/11/3/when_memory_problems_arent_what_they_seem_part_1 >>> >>> One thing I don't get about the -out files is why -out and -out1 are the >>> only current ones and then -out2 seems to be when the server first started >>> recording and then they only go to -out200 which was before I started >>> seeing this problem. I see in your article I can make them bigger than >>> 200kb, which I may have to do, but it seems weird they stop making at 200 >>> and then I essentially only have the latest 400kb in two files >>> >>> Ajas good point on the 10-day free trials, we have a copy of one or the >>> other that we purchased awhile back, but admittedly in our organization >>> Eric Jones was the main server maintenance guy, until he moved on. He kept >>> track of those licenses and servers. I've been debating which would be >>> longer, to find the license and learn the software or fix the problem :) >>> >>> Finally, revisiting Charlie's idea to look at the last funky thing >>> before the server went down around 16:24 I do see this 2min earlier, and it >>> is in the app that utilizes Hibernate, so I'll definitely need to see if I >>> have a CFC incorrectly set up as an interface or abstract class java style >>> in a way that CF doesn't like and that potentially could be the root of >>> this deal. Unfortunately, since my -out logs don't capture the last outage >>> a few days ago I can't compare, but maybe when I try to re-trigger the >>> event tomorrow I can get this same result. >>> >>> 12/06 16:22:28 Error [jrpp-4004] - Object instantiation exception.An >>> exception occurred while instantiating a Java object. The class must not be >>> an interface or an abstract class. Error: ''. The specific sequence of >>> files included or processed is: >>> F:\.........................\webroot\index.cfm, line: 883 >>> 12/06 16:22:44 Information [scheduler-1] - Session >>> 8430f0f809b2ff4d892f6b78787f5d723575 ended on ip . Length: 2:00:01 Active >>> sessions: -1 >>> 12/06 16:22:52 Error [jrpp-4004] - Object instantiation exception.An >>> exception occurred while instantiating a Java object. The class must not be >>> an interface or an abstract class. Error: ''. The specific sequence of >>> files included or processed is: F:\.....................\webroot\index.cfm, >>> line: 883 >>> 12/06 16:22:56 Error [jrpp-4004] - Exception thrown by error-handling >>> template: >>> 12/06 16:22:59 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen >>> space >>> >>> jvm.config file >>> >>> # >>> # VM configuration >>> # >>> java.home=C:/Program Files/Java/jdk1.6.0_24/jre >>> >>> # Arguments to VM >>> java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false >>> -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch >>> -Dcoldfusion.rootDir={application.home}/../ >>> -Djava.security.policy={application.home}/../lib/coldfusion.policy >>> -Djava.security.auth.policy={application.home}/../lib/neo_jaas.policy >>> -Dcoldfusion.classPath={application.home}/../lib/updates,{application.home}/../lib,{application.home}/../gateway/lib/,{application.home}/../wwwroot/WEB-INF/cfform/jars,{application.home}/../wwwroot/WEB-INF/flex/jars >>> -Dcoldfusion.libPath={application.home}/../lib >>> >>> # >>> # commas will be converted to platform specific separator and the result >>> will be passed >>> # as -Djava.ext.dirs= to the VM >>> java.ext.dirs={jre.home}/lib/ext >>> >>> # >>> # where to find shared libraries >>> >>> java.library.path={application.home}/../lib,{application.home}/../jintegra/bin,{application.home}/../jintegra/bin/international,{application.home}/../lib/oosdk/classes/win >>> system.path.first=false >>> >>> # >>> # set the current working directory - useful for Windows to control >>> # the default search path used when loading DLLs since it comes >>> # before system directory, windows directory and PATH >>> java.user.dir={application.home}/../../lib >>> >>> # JVM classpath >>> >>> java.class.path={application.home}/servers/lib,{application.home}/../lib/macromedia_drivers.jar,{application.home}/lib/cfmx_mbean.jar,{application.home}/../lib/oosdk/classes,{application.home}/../lib/oosdk/lib,{application.home}/lib >>> >>> >>> >>> On Tue, Dec 6, 2011 at 10:58 PM, Charlie Arehart >>> <char...@carehart.org>wrote: >>> >>>> On top of all that’s been said, I’d add this: >>>> >>>> Yes, it’s likely that you simply need to increase the maxpermsize. >>>> Sometimes just another 100-300 can be enough to solve the problem. >>>> >>>> But you also want to make sure that this is really the **first** error >>>> that puts CF into a bad state. It could be that something else precedes it >>>> and precipitates it. >>>> >>>> Look in in the [cf]\runtime\logs\, in its out log for this instance, at >>>> the time of your crash, and look back in that log (before the crash) to see >>>> if any other errors or unusual messages show up. If you see none for 1-20 >>>> minutes before that permgen message, then that alone would seem to be the >>>> problem. >>>> >>>> As to how it could happen, you’re on the right track but not quite >>>> right in what you’ve concluded. The saving of class files is not quite >>>> connected as you think. That’s about compiling, where the permgen is about >>>> loading. CF loads classes into the template cache (and thereby the permgen, >>>> when they’re first requested--and are not in the cache already, or later >>>> when it’s no longer in the cache for any reason, including a restart. A >>>> class can be removed from the template cache also during the life of the >>>> server when the template cache becomes full and it needs to make room for a >>>> more newly requested one, by removing the least recently used one). >>>> >>>> And CF will obtain that class (to load when needed) either by compiling >>>> it from source on the fly, or by pulling it from that cfclasses directory >>>> (if you have enabled that “save class files” option in the Admin). >>>> >>>> So how does the permgen get stressed? Most often by excessive loading >>>> of templates into the template cache. Again, when it gets full, it removes >>>> the oldest to make room for new ones. How could it get full? Because the >>>> number of templates that need to be loaded (CFM files, CFCs, and indeed a >>>> class is created for each METHOD in a CFC) exceeds its size. The default is >>>> 1024. That’s also not a hard cap. There is a soft cache on top of that. >>>> >>>> If you look in the cfclasses directory, see how many files are there. >>>> You might even sort it by date modified and see how many are recently >>>> edited. If that number is at or near 1024, it’s likely your app uses lots >>>> of files and therefore creates lots of classes, which can then cause this >>>> stress. If you increase the size of the template cache, beware that that >>>> cache is stored in the heap, so you may need to raise that, too. (Yes, >>>> loading things into the template cache then affects both the heap and the >>>> permgen, from my observation.) >>>> >>>> >>>> Does that make sense, does that help? You’re right, you don’t find much >>>> out there about it. Hope this is helpful. I may blog it once we have some >>>> back and forth. (I’ve hinted at this info in some entries, but not devoted >>>> one to it.) >>>> >>>> /charlie >>>> >>>> >>>> >>>> *From:* ad...@acfug.org [mailto:ad...@acfug.org] *On Behalf Of *Cheyenne >>>> Throckmorton >>>> *Sent:* Tuesday, December 06, 2011 10:09 PM >>>> *To:* discussion@acfug.org >>>> *Subject:* [ACFUG Discuss] PermGen CF Server Memory Errors >>>> >>>> >>>> >>>> One of our CF servers keeps needing to have the CF service rebooted on >>>> it in order to work and continue serving our sites 1-2x / week. >>>> Fortunately we do have a web service on one of the other box that monitors >>>> this machine to let us know when it starts bugging out again. >>>> >>>> >>>> >>>> The error I am getting on the server has to do with an out of memory >>>> perm gen space issue, something that there is tons of stuff online about >>>> but nothing that I've been able to succinctly tell is a good idea to look >>>> toward resolving the issue. Here are the errors we get before the service >>>> fails. >>>> >>>> >>>> >>>> javax.servlet.ServletException: ROOT CAUSE: >>>> java.lang.OutOfMemoryError: PermGen space >>>> >>>> >>>> >>>> 12/06 16:24:46 Error [jrpp-3954] - PermGen space The specific sequence >>>> of files included or processed is: >>>> F:\...................\case-studies\ENDO-enduring-hot-seat-discussion\index.cfm'' >>>> >>>> 12/06 16:24:46 error ROOT CAUSE: java.lang.OutOfMemoryError: PermGen >>>> space >>>> >>>> >>>> >>>> javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError: >>>> PermGen space >>>> >>>> >>>> >>>> Server: Windows Server 2008 R2 64bit >>>> >>>> ColdFusion 9,0,1,274733 Standard >>>> >>>> Java Version: 1.6.0_24 >>>> >>>> RAM 8GB >>>> >>>> >>>> >>>> JVM Arguments >>>> >>>> -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m >>>> -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../ >>>> -Dcoldfusion.libPath={application.home}/../lib >>>> >>>> Despite my attendance of multiple Charlie lectures and/or because of >>>> such attendance I certainly do not consider myself an expert in the memory >>>> underpinnings of CF. What I do believe is happening through some research >>>> and late night testing is that essentially the server stores information >>>> (in think in cfclasses) that is essentially compiled java code. It does >>>> this to speed the delivery of data generated by CF in a caching like >>>> mechanism. >>>> >>>> >>>> >>>> There is a checkmark in the settings that says "Save Class Files" which >>>> I suspect would solve the problem if indeed this area is getting overflowed >>>> and not properly Garbage Collected (if GC even runs in PermGen). However, >>>> unchecking that is not recommended for a production machine, and it would >>>> obviously slow down all of the sites. Similarly I've seen posts that say >>>> to just increase the MaxPermSize, but I find many posts after those saying >>>> that is not solve and just delaying the problem. I'd rather solve the >>>> problem. >>>> >>>> >>>> >>>> Additionally, when observing the websites when it is acting up we get a >>>> lot of totally blank pages along with 200 OK http response codes. I have >>>> also gotten partial pages at times, but that all leads me to some sort of >>>> partial template loading until the permgen runs out of memory. >>>> >>>> >>>> >>>> Finally, I don't know that it is a 100% deal, but we have launched a >>>> new application on this server that makes use of the new ORM Hibernate >>>> functionality. This happened right around the same time we started >>>> observing these issues. This was my first shot at ORM within CF and I know >>>> I need to go back and re-look at how I have the lazy-loading set up. I >>>> have a feeling that it can definitely be tuned better and that it's >>>> possible this type of code may make heavy use of the permgen space with >>>> dynamically loading data in the way that it works. This heavy banging in >>>> the permgen along with an untuned JVM may be the combo that is causing our >>>> problems, or maybe I'm way off. Figured I'd throw it out there to group to >>>> see what thoughts and solutions might be out there beyond my teams >>>> continued googling of words like "permgen", "coldfusion", "hibernate", >>>> "orm" and "arehart" :) >>>> >>>> >>>> >>>> - Cheyenne Throckmorton >>>> >>>> >>>> >>>> P.S. Looking forward to seeing folks at Whirlyball tomorrow night. >>>> >>>> >>>> -- >>>> Cheyenne Throckmorton - Atlanta, GA >>>> Blog : www.CheyenneJack.com >>>> Twitter : @cheyennejack >>>> Founder : www.AtlantaUserGroups.com >>>> www.TheTallStreetJournal.com >>>> www.MohawksRock.com >>>> >>>> ------------------------------------------------------------- >>>> To unsubscribe from this list, manage your profile @ >>>> http://www.acfug.org?fa=login.edituserform >>>> >>>> For more info, see http://www.acfug.org/mailinglists >>>> Archive @ http://www.mail-archive.com/discussion%40acfug.org/ >>>> List hosted by FusionLink <http://www.fusionlink.com> >>>> ------------------------------------------------------------- >>>> >>> >>> >>> >>> -- >>> Cheyenne Throckmorton - Atlanta, GA >>> Blog : www.CheyenneJack.com >>> Twitter : @cheyennejack >>> Founder : www.AtlantaUserGroups.com >>> www.TheTallStreetJournal.com >>> www.MohawksRock.com >>> >> >> > > > -- > Cheyenne Throckmorton - Atlanta, GA > Blog : www.CheyenneJack.com > Twitter : @cheyennejack > Founder : www.AtlantaUserGroups.com > www.TheTallStreetJournal.com > www.MohawksRock.com >
