Re: placing jars in /server/lib - can cause oome?
har har. glad i could give everyone on this thread a good chuckle. anyway... I didn't want to rattle off more crud, so instead I collected as best I could, information that might elucidate my issue more: and everyone was right, my jar file did NOT live in \server. Application: Reporting application we'll call reporting.war , which unpacks to a directory size of ~ 100MB of jars/classes/static content/etc. # webapp contexts (is that what these are called?): 40 Server: Tomcat 5.0.27 JDK: 1.4.2_13 Tomcat runs as a windows service: Tomcat\bin\tomcat5.exe //RS//reportingTomcat -Xms: 512MB -Xmx: 1024M -Xrs The only thing running on this machine is a Windows apache tomcat service. this application hooks into a sister app server that performs the meat and potatos activities of the application which we've never needed to restart but have done so to see if it was somehow related to web-layer OOMEs (hey, my father used to bang the t.v. set to make the picture more clear and it worked for HIM, so ...) webapp/reporting/web-inf/lib = 93 jar files = 22 Mb webapp/reporting01 webapp/reporting02 . . webapp/reporting30 each of these contexts has a corresponding config file in : \conf\Catalina\localhost\reporting01.xml \conf\Catalina\localhost\reporting02.xml . . \conf\Catalina\localhost\reporting30.xml In this XMl file, I have: Context docBase=E:\tools\portal\reporting.war path=/reporting01 crossContext=false debug=0 reloadable=false trusted=false Logger className=org.apache.catalina.logger.FileLogger prefix=localhost_reporting01_log. suffix=.txt timestamp=true/ Resource name=jdbc/myapp auth=Container type=javax.sql.DataSource / ResourceParams name=jdbc/myapp parameter namefactory/name valueorg.apache.commons.dbcp.BasicDataSourceFactory/value /parameter parameter namedriverClassName/name valuenet.sourceforge.jtds.jdbc.Driver/value /parameter parameter nameurl/name valuejdbc:jtds:sqlserver://sqlserver:1433/reporting01;User=xxx;Password=xxx/value !-- to add DB instance, use ...:port/databaseName;instance=name;User=xx notation -- /parameter parameter namevalidationQuery/name valueselect 1/value /parameter parameter namemaxActive/name value30/value /parameter /ResourceParams /Context The *only* customization we've done is to make a brief call to a database, grab a valid userlogin from our application and perform a handshake to the reporting application. _ custom code lives in \webapps\reportingXX\customcode\ and all apps use jtds-1.2.jar which lives in \Tomcat\common\lib our customization is something like: file db.include, which has: InitialContext initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup(java:comp/env); DataSource ds = (DataSource) envCtx.lookup(jdbc/myapp); ...and we create the connection in: file login.jsp which has: includes db.include Connection con = ds.getConnection; we do some SQL and then close/null any variables using or related to the connection/resultset/etc After about a day, we start seeing these: __ 2007-10-12 12:24:20 ApplicationDispatcher[/reportingXX] Servlet.service() for servlet jsp threw exception java.lang.OutOfMemoryError 2007-10-12 12:24:21 ApplicationDispatcher[[/reportingXX] ] Servlet.service() for servlet jsp threw exception java.lang.OutOfMemoryError 2007-10-12 12:24:21 StandardWrapperValve[action]: Servlet.service() for servlet action threw exception java.lang.OutOfMemoryError __ Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of this application) Start: 9.643: [GC 9.643: [DefNew: 46591K-3981K(52416K), 0.0273929 secs] 46591K-3981K(518464K), 0.0276010 secs] 13.569: [GC 13.569: [DefNew: 50573K-3714K(52416K), 0.0351330 secs] 50573K-7132K(518464K), 0.0352592 secs] 19.409: [GC 19.409: [DefNew: 50306K-4897K(52416K), 0.0262670 secs] 53724K-9800K(518464K), 0.0263980 secs] 22.330: [GC 22.330: [DefNew: 51489K-1120K(52416K), 0.0177011 secs] 56392K-10751K(518464K), 0.0178319 secs] 22.978: [GC 22.978: [DefNew: 47712K-1128K(52416K), 0.0031998 secs] 57343K-10760K(518464K), 0.0033108 secs] 24.976: [GC 24.976: [DefNew: 47720K-3289K(52416K), 0.0163705 secs] 57352K-12920K(518464K), 0.0164999 secs] 38.553: [GC 38.553: [DefNew: 49881K-5047K(52416K), 0.0225587 secs] 59512K-15380K(518464K), 0.0227062 secs] 71.451: [GC 71.451: [DefNew: 51635K-249K(52416K), 0.0224671 secs] 61967K-13712K(518464K), 0.0225193 secs] 97.070: [GC 97.070: [DefNew: 46841K-1806K(52416K), 0.0066059 secs] 60304K-15268K(518464K), 0.0066603 secs] 107.194: [GC 107.194: [DefNew: 48398K-4604K(52416K), 0.0193955 secs] 61860K-18066K(518464K), 0.0194505 secs] 136.409: [GC 136.409: [DefNew: 51195K-251K(52416K), 0.0164256 secs] 64658K-16415K(518464K), 0.0164884 secs] 169.849: [GC 169.849: [DefNew: 46843K-1755K(52416K), 0.0072353 secs] 63007K-17919K(518464K),
Re: placing jars in /server/lib - can cause oome?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A C, A C wrote: Application: Reporting application we'll call reporting.war , which unpacks to a directory size of ~ 100MB of jars/classes/static content/etc. Aah, that's a different story. Certainly, not all 100MB is being loaded into memory and staying there. You probably have images and stuff that are never loaded. # webapp contexts (is that what these are called?): 40 The servlet spec and J2EE interchangeably call them contexts and web applications. I suppose a context only makes sense when the web application is actually running. You could call a WAR file a web application, but you wouldn't call it a context. On the other hand, a deployed web application is running in a context which .. well, IS the web application. Semantics. -Xms: 512MB -Xmx: 1024M Just for kicks, set these to the same value. Then, attach something to the Tomcat process that can read the heap info (free, user, min/max) as well as request a full GC from the JVM. You can use a profiler or something less bulky such as LambdaProbe. Deploy a single web application (and leave the other 39 of them off). Use the web app a bit to exercise it a little bit. Look at the memory usage over time. You should get a mem usage curve that looks like this: \/\/\/\/\/\/\/\/\/\ Nice ASCII art, eh? Seriously, though, your memory should be allocated, then freed, pretty regularly. You should be able to look at the peaks and valleys to determine how much memory a single copy of your webapp requires while running on Tomcat (yes, I know Tomcat is there too, but bear with me). Next, deploy two instances of the webapp and repeat the process (make sure to exercise both copies and not just one). You should be able to see how much memory is used by two instances running on Tomcat. By subtracting the amount you got from test 2 from test 1, you should be able to guess at how much memory a single instance takes. Try your guesses out by repeating the process with 3 or more instances to see if the memory usage increases according to how much you think each instance takes. It's possible that each instance needs maybe 50 or 60MB of heap space to run comfortably. If you've capped the heap at 1GB, that means that you can only have 1024 / 50 ~= 20 instances running comfortably in that amount of memory (and that doesn't even cover the overhead of running Tomcat itself). You either need to increase the available memory (you can probably squeeze up to 1.6GB out of a 32-bit JVM or get yourself a 64-bit OS and JVM) or run some of these instances on another server. The only thing running on this machine is a Windows apache tomcat service. this application hooks into a sister app server that performs the meat and potatos activities of the application which we've never needed to restart but have done so to see if it was somehow related to web-layer OOMEs (hey, my father used to bang the t.v. set to make the picture more clear and it worked for HIM, so ...) Do all 40 instances use that particular resource? What is it, may I ask? webapp/reporting/web-inf/lib = 93 jar files = 22 Mb Note that memory footprint rarely has anything to do with the size of the WAR file that was used to deploy it. Connection con = ds.getConnection; we do some SQL and then close/null any variables using or related to the connection/resultset/etc Do you have this wrapped-up in a try/catch/finally block? Is it possible that you have JDBC resource leaks? You should set: removeAbandoned=true removeAbandonedTimeout=30 logAbandoned=true in your Resource. If your SQL queries take a long time to run, 30 seconds might not be enough timeout time, so feel free to increase that. This configuration will log a stack trace for any connection that does not get returned to the pool within (approximately) removeAbandonedTimeout seconds. You might find that you are leaking connections. Then again, you might not be. It's always good to check. Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of this application) Let's see how your heap looks after a while... 1247.073: [GC 1247.073: [DefNew: 51055K-276K(52480K), 0.0630231 secs] 123930K-75721K(518528K), 0.0630960 secs] You are reclaiming a bunch of memory in each GC. Let's keep going: 1538.764: [Full GC 1538.764: [Tenured: 86527K-59881K(466048K), 5.3928429secs] 133369K-59881K(518528K), [Perm : 32768K-32610K(32768K)], 5.3929301 secs] Looks like you're busting PermGen. You nearly busted it at the last Full GC (had 32768K in it, and freed only about 150K). 2357.129: [Full GC 2357.129: [Tenured: 89508K-71494K(466048K), 7.1956506secs] 105965K-71494K(518528K), [Perm : 49151K-48973K(49152K)], 7.2276270 secs] PermGen size has increased, but you're still very close to filling it up. Note that the full heap is still only 1/2 GB (your initial heap size). 7 seconds is a lng GC. 2485.690: [Full GC 2485.690: [Tenured:
Re: placing jars in /server/lib - can cause oome?
You can determine the anount of remaining RAM and then set MaxRows and FetchSize to the number of Rows e.g. http://www.docjar.com/docs/api/java/lang/Runtime.html //determine how much memory is free for allocating Obects on the heap long remainingRAM = java.lang.Runtime.getRuntime().freeMemory(); int RowsToFetch = remainingRAM/RowSize; http://www.docjar.com/docs/api/java/sql/Statement.html Statement.setFetchSize(RowsToFetch); Statement.setMaxRows(RowsToFetch); Martin-- - Original Message - From: A C [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org Sent: Friday, October 12, 2007 3:47 PM Subject: Re: placing jars in /server/lib - can cause oome? har har. glad i could give everyone on this thread a good chuckle. anyway... I didn't want to rattle off more crud, so instead I collected as best I could, information that might elucidate my issue more: and everyone was right, my jar file did NOT live in \server. Application: Reporting application we'll call reporting.war , which unpacks to a directory size of ~ 100MB of jars/classes/static content/etc. # webapp contexts (is that what these are called?): 40 Server: Tomcat 5.0.27 JDK: 1.4.2_13 Tomcat runs as a windows service: Tomcat\bin\tomcat5.exe //RS//reportingTomcat -Xms: 512MB -Xmx: 1024M -Xrs The only thing running on this machine is a Windows apache tomcat service. this application hooks into a sister app server that performs the meat and potatos activities of the application which we've never needed to restart but have done so to see if it was somehow related to web-layer OOMEs (hey, my father used to bang the t.v. set to make the picture more clear and it worked for HIM, so ...) webapp/reporting/web-inf/lib = 93 jar files = 22 Mb webapp/reporting01 webapp/reporting02 . . webapp/reporting30 each of these contexts has a corresponding config file in : \conf\Catalina\localhost\reporting01.xml \conf\Catalina\localhost\reporting02.xml . . \conf\Catalina\localhost\reporting30.xml In this XMl file, I have: Context docBase=E:\tools\portal\reporting.war path=/reporting01 crossContext=false debug=0 reloadable=false trusted=false Logger className=org.apache.catalina.logger.FileLogger prefix=localhost_reporting01_log. suffix=.txt timestamp=true/ Resource name=jdbc/myapp auth=Container type=javax.sql.DataSource / ResourceParams name=jdbc/myapp parameter namefactory/name valueorg.apache.commons.dbcp.BasicDataSourceFactory/value /parameter parameter namedriverClassName/name valuenet.sourceforge.jtds.jdbc.Driver/value /parameter parameter nameurl/name valuejdbc:jtds:sqlserver://sqlserver:1433/reporting01;User=xxx;Password=xx x/value !-- to add DB instance, use ...:port/databaseName;instance=name;User=xx notation -- /parameter parameter namevalidationQuery/name valueselect 1/value /parameter parameter namemaxActive/name value30/value /parameter /ResourceParams /Context The *only* customization we've done is to make a brief call to a database, grab a valid userlogin from our application and perform a handshake to the reporting application. _ custom code lives in \webapps\reportingXX\customcode\ and all apps use jtds-1.2.jar which lives in \Tomcat\common\lib our customization is something like: file db.include, which has: InitialContext initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup(java:comp/env); DataSource ds = (DataSource) envCtx.lookup(jdbc/myapp); ...and we create the connection in: file login.jsp which has: includes db.include Connection con = ds.getConnection; we do some SQL and then close/null any variables using or related to the connection/resultset/etc After about a day, we start seeing these: __ 2007-10-12 12:24:20 ApplicationDispatcher[/reportingXX] Servlet.service() for servlet jsp threw exception java.lang.OutOfMemoryError 2007-10-12 12:24:21 ApplicationDispatcher[[/reportingXX] ] Servlet.service() for servlet jsp threw exception java.lang.OutOfMemoryError 2007-10-12 12:24:21 StandardWrapperValve[action]: Servlet.service() for servlet action threw exception java.lang.OutOfMemoryError __ Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of this application) Start: 9.643: [GC 9.643: [DefNew: 46591K-3981K(52416K), 0.0273929 secs] 46591K-3981K(518464K), 0.0276010 secs] 13.569: [GC 13.569: [DefNew: 50573K-3714K(52416K), 0.0351330 secs] 50573K-7132K(518464K), 0.0352592 secs] 19.409: [GC 19.409: [DefNew: 50306K-4897K(52416K), 0.0262670 secs] 53724K-9800K(518464K), 0.0263980 secs] 22.330: [GC 22.330: [DefNew: 51489K-1120K(52416K), 0.0177011 secs] 56392K-10751K(518464K), 0.0178319 secs] 22.978: [GC 22.978: [DefNew: 47712K-1128K(52416K
RE: placing jars in /server/lib - can cause oome?
From: A C [mailto:[EMAIL PROTECTED] Subject: placing jars in /server/lib - can cause oome? We've been experiencing horrendous OOMEs recently after deploying 40 clones of a vendor's .war file which encompasses their application/portal. You're a tad short on specifics: Tomcat version, JRE/JDK version, OS platform, actual exception thrown, etc. Then, we have 40 contexts for 40 customers, each one has its own WEB-INF/lib folder with an identical set of class folders and jars etc etc. Each copy of the classes has to be loaded separately, since Tomcat uses one classloader per webapp. You are likely running out of PermGen space, but can't tell for sure without more real information. When we have 30, we need a daily restart, even though we never get more than 25 sessions to the database in that time ... Which indicates you may well have a memory leak in your webapp, and having multiple copies of it just makes things worse. You need a profiler. My question is, does placing the jtds.jar in /server/lib cause an issue? Unless you've done something funny with conf/catalina.properties, that may be completely irrelevant, since none of the webapps can even see classes from that location. Are you sure that jar is even being used? we're planning on refitting the deployment and placing this in common/lib ... Which would make it visible to the webapps, whereas now it's only visible to Tomcat code. Moving it to common/lib should not require any changes. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: placing jars in /server/lib - can cause oome?
Christopher Schultz wrote: A C, A C wrote: We've been experiencing horrendous OOMEs recently after deploying 40 clones of a vendor's .war file which encompasses their application/portal. The context is about 100MB Lesee... 100 MB times 40 instances .. mm... carry the 7 ... hey, that's 4GB! Where have I heard that number before... hmm.. something about ... right! The maximum process memory on a 32-bit OS! lol A C, I have to say this: the clue is in the question... I'll summarise; it works OK when when you deploy one copy of a 100Mb app, but when you deploy FORTY it gets a bit messy and you run out of memory. How is it that you're not thinking about more servers at this point? (I'd be waving a nice diagram at the accountants and saying 'shiny new boxes please') p - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
