Re: placing jars in /server/lib - can cause oome?
You can determine the anount of remaining RAM and then set MaxRows and
FetchSize to the number of Rows e.g.
http://www.docjar.com/docs/api/java/lang/Runtime.html
//determine how much memory is free for allocating Obects on the heap
long remainingRAM = java.lang.Runtime.getRuntime().freeMemory();
int RowsToFetch = remainingRAM/RowSize;
http://www.docjar.com/docs/api/java/sql/Statement.html
Statement.setFetchSize(RowsToFetch);
Statement.setMaxRows(RowsToFetch);
Martin--
- Original Message -
From: "A C" <[EMAIL PROTECTED]>
To: "Tomcat Users List"
Sent: Friday, October 12, 2007 3:47 PM
Subject: Re: placing jars in /server/lib - can cause oome?
> har har. glad i could give everyone on this thread a good chuckle.
>
> anyway... I didn't want to rattle off more crud, so instead I collected as
> best I could, information that might elucidate my issue more:
>
>
> and everyone was right, my jar file did NOT live in \server.
>
>
> Application: Reporting application we'll call "reporting.war" , which
> unpacks to a directory size of ~ 100MB of jars/classes/static content/etc.
> # webapp contexts (is that what these are called?): 40
> Server: Tomcat 5.0.27
> JDK: 1.4.2_13
>
> Tomcat runs as a windows service: Tomcat\bin\tomcat5.exe
> //RS//reportingTomcat
>
> -Xms: 512MB
> -Xmx: 1024M
> -Xrs
>
>
>
>
>
> The only "thing" running on this machine is a Windows apache tomcat
> service. this application "hooks" into a sister app server that performs
> the "meat and potatos" activities of the application which we've never
> needed to restart but have done so to see if it was somehow related to
> web-layer OOMEs (hey, my father used to bang the t.v. set to make the
> picture more clear and it worked for HIM, so ...)
>
> webapp/reporting/web-inf/lib = 93 jar files = 22 Mb
> webapp/reporting01
> webapp/reporting02
> .
> .
> webapp/reporting30
>
> each of these contexts has a corresponding config file in :
>
> \conf\Catalina\localhost\reporting01.xml
> \conf\Catalina\localhost\reporting02.xml
> .
> .
> \conf\Catalina\localhost\reporting30.xml
>
> In this XMl file, I have:
>
> crossContext="false" debug="0" reloadable="false" trusted="false">
> prefix="localhost_reporting01_log." suffix=".txt" timestamp="true"/>
>
>
>
> factory
> org.apache.commons.dbcp.BasicDataSourceFactory
>
>
> driverClassName
> net.sourceforge.jtds.jdbc.Driver
>
>
> url
>
jdbc:jtds:sqlserver://sqlserver:1433/reporting01;User=xxx;Password=xx
x
>
>
>
> validationQuery
> select 1
>
>
> maxActive
> 30
>
>
>
>
>
>
>
>
> The *only* customization we've done is to make a brief call to a database,
> grab a valid userlogin from our application and perform a handshake to the
> reporting application.
>
> _ custom code lives in \webapps\reportingXX\customcode\
>
> and all apps use " jtds-1.2.jar " which lives in \Tomcat\common\lib
>
> our customization is something like:
>
> file > db.include, which has:
>
> InitialContext initCtx = new InitialContext();
> Context envCtx = (Context) initCtx.lookup("java:comp/env");
> DataSource ds = (DataSource) envCtx.lookup("jdbc/myapp");
>
> ...and we create the connection in:
>
> file > login.jsp which has:
>
> includes db.include
>
>
> Connection con = ds.getConnection;
>
> we do some SQL and then close/null any variables using or related to the
> connection/resultset/etc
>
>
> After about a day, we start seeing these:
>
> __
>
> 2007-10-12 12:24:20 ApplicationDispatcher[/reportingXX] Servlet.service()
> for servlet jsp threw exception
> java.lang.OutOfMemoryError
>
> 2007-10-12 12:24:21 ApplicationDispatcher[[/reportingXX] ]
Servlet.service()
> for servlet jsp threw exception
> java.lang.OutOfMemoryError
>
> 2007-10-12 12:24:21 StandardWrapperValve[action]: Servlet.service() for
> servlet action threw exception
> java.lang.OutOfMemoryError
>
> __
>
> Here's the permgen after 1.5 hours of my simulation (35 deployed webapps
of
> this application)
>
> Start:
>
> 9.643: [GC 9.643: [DefNew: 46591K->3981K(52416K), 0.0273929 secs]
> 46591K->3981K(518464K), 0.0276010 secs]
> 13.569: [GC 13.569: [DefNew: 50573K->3714K(52416K), 0.0351330 secs]
> 50573K->7132K(518464K), 0.0352592 secs]
&g
Re: placing jars in /server/lib - can cause oome?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A C, A C wrote: > Application: Reporting application we'll call "reporting.war" , which > unpacks to a directory size of ~ 100MB of jars/classes/static content/etc. Aah, that's a different story. Certainly, not all 100MB is being loaded into memory and staying there. You probably have images and stuff that are never loaded. > # webapp contexts (is that what these are called?): 40 The servlet spec and J2EE interchangeably call them "contexts" and "web applications". I suppose a context only makes sense when the web application is actually running. You could call a WAR file a "web application", but you wouldn't call it a context. On the other hand, a deployed web application is running in a context which .. well, IS the web application. Semantics. > -Xms: 512MB > -Xmx: 1024M Just for kicks, set these to the same value. Then, attach something to the Tomcat process that can read the heap info (free, user, min/max) as well as request a full GC from the JVM. You can use a profiler or something less bulky such as LambdaProbe. Deploy a single web application (and leave the other 39 of them off). Use the web app a bit to exercise it a little bit. Look at the memory usage over time. You should get a mem usage curve that looks like this: \/\/\/\/\/\/\/\/\/\ Nice ASCII art, eh? Seriously, though, your memory should be allocated, then freed, pretty regularly. You should be able to look at the peaks and valleys to determine how much memory a single copy of your webapp requires while running on Tomcat (yes, I know Tomcat is there too, but bear with me). Next, deploy two instances of the webapp and repeat the process (make sure to exercise both copies and not just one). You should be able to see how much memory is used by two instances running on Tomcat. By subtracting the amount you got from test 2 from test 1, you should be able to guess at how much memory a single instance takes. Try your guesses out by repeating the process with 3 or more instances to see if the memory usage increases according to how much you think each instance "takes". It's possible that each instance needs maybe 50 or 60MB of heap space to run comfortably. If you've capped the heap at 1GB, that means that you can only have 1024 / 50 ~= 20 instances running comfortably in that amount of memory (and that doesn't even cover the overhead of running Tomcat itself). You either need to increase the available memory (you can probably squeeze up to 1.6GB out of a 32-bit JVM or get yourself a 64-bit OS and JVM) or run some of these instances on another server. > The only "thing" running on this machine is a Windows apache tomcat > service. this application "hooks" into a sister app server that performs > the "meat and potatos" activities of the application which we've never > needed to restart but have done so to see if it was somehow related to > web-layer OOMEs (hey, my father used to bang the t.v. set to make the > picture more clear and it worked for HIM, so ...) Do all 40 instances use that particular resource? What is it, may I ask? > webapp/reporting/web-inf/lib = 93 jar files = 22 Mb Note that memory footprint rarely has anything to do with the size of the WAR file that was used to deploy it. > Connection con = ds.getConnection; > > we do some SQL and then close/null any variables using or related to the > connection/resultset/etc Do you have this wrapped-up in a try/catch/finally block? Is it possible that you have JDBC resource leaks? You should set: removeAbandoned="true" removeAbandonedTimeout="30" logAbandoned="true" in your . If your SQL queries take a long time to run, 30 seconds might not be enough "timeout" time, so feel free to increase that. This configuration will log a stack trace for any connection that does not get returned to the pool within (approximately) "removeAbandonedTimeout" seconds. You might find that you are leaking connections. Then again, you might not be. It's always good to check. > Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of > this application) Let's see how your heap looks after a while... > 1247.073: [GC 1247.073: [DefNew: 51055K->276K(52480K), 0.0630231 secs] > 123930K->75721K(518528K), 0.0630960 secs] You are reclaiming a bunch of memory in each GC. Let's keep going: > 1538.764: [Full GC 1538.764: [Tenured: 86527K->59881K(466048K), > 5.3928429secs] 133369K->59881K(518528K), [Perm : > 32768K->32610K(32768K)], 5.3929301 secs] Looks like you're busting PermGen. You nearly busted it at the last Full GC (had 32768K in it, and freed only about 150K). > 2357.129: [Full GC 2357.129: [Tenured: 89508K->71494K(466048K), > 7.1956506secs] 105965K->71494K(518528K), [Perm : > 49151K->48973K(49152K)], 7.2276270 secs] PermGen size has increased, but you're still very close to filling it up. Note that the full heap is still only 1/2 GB (your initial heap size). 7 seconds is a lng GC. > 2485
Re: placing jars in /server/lib - can cause oome?
har har. glad i could give everyone on this thread a good chuckle.
anyway... I didn't want to rattle off more crud, so instead I collected as
best I could, information that might elucidate my issue more:
and everyone was right, my jar file did NOT live in \server.
Application: Reporting application we'll call "reporting.war" , which
unpacks to a directory size of ~ 100MB of jars/classes/static content/etc.
# webapp contexts (is that what these are called?): 40
Server: Tomcat 5.0.27
JDK: 1.4.2_13
Tomcat runs as a windows service: Tomcat\bin\tomcat5.exe
//RS//reportingTomcat
-Xms: 512MB
-Xmx: 1024M
-Xrs
The only "thing" running on this machine is a Windows apache tomcat
service. this application "hooks" into a sister app server that performs
the "meat and potatos" activities of the application which we've never
needed to restart but have done so to see if it was somehow related to
web-layer OOMEs (hey, my father used to bang the t.v. set to make the
picture more clear and it worked for HIM, so ...)
webapp/reporting/web-inf/lib = 93 jar files = 22 Mb
webapp/reporting01
webapp/reporting02
.
.
webapp/reporting30
each of these contexts has a corresponding config file in :
\conf\Catalina\localhost\reporting01.xml
\conf\Catalina\localhost\reporting02.xml
.
.
\conf\Catalina\localhost\reporting30.xml
In this XMl file, I have:
factory
org.apache.commons.dbcp.BasicDataSourceFactory
driverClassName
net.sourceforge.jtds.jdbc.Driver
url
jdbc:jtds:sqlserver://sqlserver:1433/reporting01;User=xxx;Password=xxx
validationQuery
select 1
maxActive
30
The *only* customization we've done is to make a brief call to a database,
grab a valid userlogin from our application and perform a handshake to the
reporting application.
_ custom code lives in \webapps\reportingXX\customcode\
and all apps use " jtds-1.2.jar " which lives in \Tomcat\common\lib
our customization is something like:
file > db.include, which has:
InitialContext initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/myapp");
...and we create the connection in:
file > login.jsp which has:
includes db.include
Connection con = ds.getConnection;
we do some SQL and then close/null any variables using or related to the
connection/resultset/etc
After about a day, we start seeing these:
__
2007-10-12 12:24:20 ApplicationDispatcher[/reportingXX] Servlet.service()
for servlet jsp threw exception
java.lang.OutOfMemoryError
2007-10-12 12:24:21 ApplicationDispatcher[[/reportingXX] ] Servlet.service()
for servlet jsp threw exception
java.lang.OutOfMemoryError
2007-10-12 12:24:21 StandardWrapperValve[action]: Servlet.service() for
servlet action threw exception
java.lang.OutOfMemoryError
__
Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of
this application)
Start:
9.643: [GC 9.643: [DefNew: 46591K->3981K(52416K), 0.0273929 secs]
46591K->3981K(518464K), 0.0276010 secs]
13.569: [GC 13.569: [DefNew: 50573K->3714K(52416K), 0.0351330 secs]
50573K->7132K(518464K), 0.0352592 secs]
19.409: [GC 19.409: [DefNew: 50306K->4897K(52416K), 0.0262670 secs]
53724K->9800K(518464K), 0.0263980 secs]
22.330: [GC 22.330: [DefNew: 51489K->1120K(52416K), 0.0177011 secs]
56392K->10751K(518464K), 0.0178319 secs]
22.978: [GC 22.978: [DefNew: 47712K->1128K(52416K), 0.0031998 secs]
57343K->10760K(518464K), 0.0033108 secs]
24.976: [GC 24.976: [DefNew: 47720K->3289K(52416K), 0.0163705 secs]
57352K->12920K(518464K), 0.0164999 secs]
38.553: [GC 38.553: [DefNew: 49881K->5047K(52416K), 0.0225587 secs]
59512K->15380K(518464K), 0.0227062 secs]
71.451: [GC 71.451: [DefNew: 51635K->249K(52416K), 0.0224671 secs]
61967K->13712K(518464K), 0.0225193 secs]
97.070: [GC 97.070: [DefNew: 46841K->1806K(52416K), 0.0066059 secs]
60304K->15268K(518464K), 0.0066603 secs]
107.194: [GC 107.194: [DefNew: 48398K->4604K(52416K), 0.0193955 secs]
61860K->18066K(518464K), 0.0194505 secs]
136.409: [GC 136.409: [DefNew: 51195K->251K(52416K), 0.0164256 secs]
64658K->16415K(518464K), 0.0164884 secs]
169.849: [GC 169.849: [DefNew: 46843K->1755K(52416K), 0.0072353 secs]
63007K->17919K(518464K), 0.0072906 secs]
.
.
.
.
.
.
1247.073: [GC 1247.073: [DefNew: 51055K->276K(52480K), 0.0630231 secs]
123930K->75721K(518528K), 0.0630960 secs]
1279.579: [GC 1279.579: [DefNew: 46932K->1585K(52480K), 0.0256284 secs]
122377K->77031K(518528K), 0.0256938 secs]
1291.668: [GC 1291.668: [DefNew: 48240K->4514K(52480K), 0.0205993 secs]
123686K->79959K(518528K), 0.0206722 secs]
1313.524: [GC 1313.524: [DefNew: 51169K->254K(52480K), 0.0317320 secs]
126615K->78344K(518528K), 0.0317929 secs]
1344.525: [GC 1344.525: [DefNew: 46910K->1729K(52480K), 0.0149128 secs]
125000K->79820K(518528K), 0.0149729 secs]
Re: placing jars in /server/lib - can cause oome?
Christopher Schultz wrote: > A C, > > A C wrote: >> We've been experiencing horrendous OOMEs recently after deploying 40 clones >> of a vendor's .war file which encompasses their application/portal. The >> context is about 100MB > > Lesee... 100 MB times 40 instances .. mm... carry the 7 ... hey, that's > 4GB! Where have I heard that number before... hmm.. something about ... > right! The maximum process memory on a 32-bit OS! lol A C, I have to say this: the clue is in the question... I'll summarise; it works OK when when you deploy one copy of a 100Mb app, but when you deploy FORTY it gets a bit messy and you run out of memory. How is it that you're not thinking about more servers at this point? (I'd be waving a nice diagram at the accountants and saying 'shiny new boxes please') p - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: placing jars in /server/lib - can cause oome?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 A C, A C wrote: > We've been experiencing horrendous OOMEs recently after deploying 40 clones > of a vendor's .war file which encompasses their application/portal. The > context is about 100MB Lesee... 100 MB times 40 instances .. mm... carry the 7 ... hey, that's 4GB! Where have I heard that number before... hmm.. something about ... right! The maximum process memory on a 32-bit OS! Nevermind the fact that a 32-bit JVM can't allocate that much memory in one huge chunk. It looks like your requirements are outstripping your capabilities. If you want to run all these instances on the same machine, you're going to need a 64-bit OS. > When we deploy just ONE of these contexts onto our tomcat server, > eveything's "fine". Probably because it uses 1/40 of the memory that 40 of them require. > When we have 30, we need a daily restart, even though we never get more than > 25 sessions to the database in that time … You should look at what is taking up so much memory. > My question is, does placing the jtds.jar in /server/lib cause an issue? Almost certainly not. It's probably not doing you any good, either, since your application can't use it (only the server can, but maybe you have Tomcat managing your connection pool or something). > We're running a verbose:gc as well and have optimized to the best of our > ability xmx xms settings Like what? An example would be enlightening. > [I] now question some of our deployment mechanics. I'm pretty sure that you just need more memory than you have. If you're right, and each instance of this app takes ~100MB, then you simply cannot deploy 40 of them on a single 32-bit machine. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHDSSV9CaO5/Lv0PARAtljAKCX1LhV8nl+cxnc5pSfRh6NAqNDLQCgkxDU xbMHY6ha/JLZcPRCDB7K5RE= =GBVj -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: placing jars in /server/lib - can cause oome?
> From: A C [mailto:[EMAIL PROTECTED] > Subject: placing jars in /server/lib - can cause oome? > > We've been experiencing horrendous OOMEs recently after > deploying 40 clones of a vendor's .war file which encompasses their > application/portal. You're a tad short on specifics: Tomcat version, JRE/JDK version, OS platform, actual exception thrown, etc. > Then, we have 40 contexts for 40 customers, each one has its > own WEB-INF/lib folder with an identical set of class folders > and jars etc etc. Each copy of the classes has to be loaded separately, since Tomcat uses one classloader per webapp. You are likely running out of PermGen space, but can't tell for sure without more real information. > When we have 30, we need a daily restart, even though we > never get more than 25 sessions to the database in that time ... Which indicates you may well have a memory leak in your webapp, and having multiple copies of it just makes things worse. You need a profiler. > My question is, does placing the jtds.jar in /server/lib > cause an issue? Unless you've done something funny with conf/catalina.properties, that may be completely irrelevant, since none of the webapps can even see classes from that location. Are you sure that jar is even being used? > we're planning on refitting the deployment and > placing this in common/lib ... Which would make it visible to the webapps, whereas now it's only visible to Tomcat code. Moving it to common/lib should not require any changes. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
