har har. glad i could give everyone on this thread a good chuckle.
anyway... I didn't want to rattle off more crud, so instead I collected as
best I could, information that might elucidate my issue more:
and everyone was right, my jar file did NOT live in \server.
Application: Reporting application we'll call "reporting.war" , which
unpacks to a directory size of ~ 100MB of jars/classes/static content/etc.
# webapp contexts (is that what these are called?): 40
Server: Tomcat 5.0.27
JDK: 1.4.2_13
Tomcat runs as a windows service: Tomcat\bin\tomcat5.exe
//RS//reportingTomcat
-Xms: 512MB
-Xmx: 1024M
-Xrs
________________________________________
The only "thing" running on this machine is a Windows apache tomcat
service. this application "hooks" into a sister app server that performs
the "meat and potatos" activities of the application which we've never
needed to restart but have done so to see if it was somehow related to
web-layer OOMEs (hey, my father used to bang the t.v. set to make the
picture more clear and it worked for HIM, so ...)
webapp/reporting/web-inf/lib = 93 jar files = 22 Mb
webapp/reporting01
webapp/reporting02
.
.
webapp/reporting30
each of these contexts has a corresponding config file in :
\conf\Catalina\localhost\reporting01.xml
\conf\Catalina\localhost\reporting02.xml
.
.
\conf\Catalina\localhost\reporting30.xml
In this XMl file, I have:
<Context docBase="E:\tools\portal\reporting.war" path="/reporting01"
crossContext="false" debug="0" reloadable="false" trusted="false">
<Logger className="org.apache.catalina.logger.FileLogger"
prefix="localhost_reporting01_log." suffix=".txt" timestamp="true"/>
<Resource name="jdbc/myapp" auth="Container" type="javax.sql.DataSource" />
<ResourceParams name="jdbc/myapp">
<parameter>
<name>factory</name>
<value>org.apache.commons.dbcp.BasicDataSourceFactory</value>
</parameter>
<parameter>
<name>driverClassName</name>
<value>net.sourceforge.jtds.jdbc.Driver</value>
</parameter>
<parameter>
<name>url</name>
<value>jdbc:jtds:sqlserver://sqlserver:1433/reporting01;User=xxx;Password=xxx</value>
<!-- to add DB instance, use "...:port/databaseName;instance=<name>;User=xx"
notation -->
</parameter>
<parameter>
<name>validationQuery</name>
<value>select 1</value>
</parameter>
<parameter>
<name>maxActive</name>
<value>30</value>
</parameter>
</ResourceParams>
</Context>
________________________________________________
The *only* customization we've done is to make a brief call to a database,
grab a valid userlogin from our application and perform a handshake to the
reporting application.
_________ custom code lives in \webapps\reportingXX\customcode\ ________
and all apps use " jtds-1.2.jar " which lives in \Tomcat\common\lib
our customization is something like:
file > db.include, which has:
InitialContext initCtx = new InitialContext();
Context envCtx = (Context) initCtx.lookup("java:comp/env");
DataSource ds = (DataSource) envCtx.lookup("jdbc/myapp");
...and we create the connection in:
file > login.jsp which has:
includes db.include
Connection con = ds.getConnection;
we do some SQL and then close/null any variables using or related to the
connection/resultset/etc
After about a day, we start seeing these:
__________________________________________________
2007-10-12 12:24:20 ApplicationDispatcher[/reportingXX] Servlet.service()
for servlet jsp threw exception
java.lang.OutOfMemoryError
2007-10-12 12:24:21 ApplicationDispatcher[[/reportingXX] ] Servlet.service()
for servlet jsp threw exception
java.lang.OutOfMemoryError
2007-10-12 12:24:21 StandardWrapperValve[action]: Servlet.service() for
servlet action threw exception
java.lang.OutOfMemoryError
__________________________________________________
Here's the permgen after 1.5 hours of my simulation (35 deployed webapps of
this application)
Start:
9.643: [GC 9.643: [DefNew: 46591K->3981K(52416K), 0.0273929 secs]
46591K->3981K(518464K), 0.0276010 secs]
13.569: [GC 13.569: [DefNew: 50573K->3714K(52416K), 0.0351330 secs]
50573K->7132K(518464K), 0.0352592 secs]
19.409: [GC 19.409: [DefNew: 50306K->4897K(52416K), 0.0262670 secs]
53724K->9800K(518464K), 0.0263980 secs]
22.330: [GC 22.330: [DefNew: 51489K->1120K(52416K), 0.0177011 secs]
56392K->10751K(518464K), 0.0178319 secs]
22.978: [GC 22.978: [DefNew: 47712K->1128K(52416K), 0.0031998 secs]
57343K->10760K(518464K), 0.0033108 secs]
24.976: [GC 24.976: [DefNew: 47720K->3289K(52416K), 0.0163705 secs]
57352K->12920K(518464K), 0.0164999 secs]
38.553: [GC 38.553: [DefNew: 49881K->5047K(52416K), 0.0225587 secs]
59512K->15380K(518464K), 0.0227062 secs]
71.451: [GC 71.451: [DefNew: 51635K->249K(52416K), 0.0224671 secs]
61967K->13712K(518464K), 0.0225193 secs]
97.070: [GC 97.070: [DefNew: 46841K->1806K(52416K), 0.0066059 secs]
60304K->15268K(518464K), 0.0066603 secs]
107.194: [GC 107.194: [DefNew: 48398K->4604K(52416K), 0.0193955 secs]
61860K->18066K(518464K), 0.0194505 secs]
136.409: [GC 136.409: [DefNew: 51195K->251K(52416K), 0.0164256 secs]
64658K->16415K(518464K), 0.0164884 secs]
169.849: [GC 169.849: [DefNew: 46843K->1755K(52416K), 0.0072353 secs]
63007K->17919K(518464K), 0.0072906 secs]
.
.
.
.
.
.
1247.073: [GC 1247.073: [DefNew: 51055K->276K(52480K), 0.0630231 secs]
123930K->75721K(518528K), 0.0630960 secs]
1279.579: [GC 1279.579: [DefNew: 46932K->1585K(52480K), 0.0256284 secs]
122377K->77031K(518528K), 0.0256938 secs]
1291.668: [GC 1291.668: [DefNew: 48240K->4514K(52480K), 0.0205993 secs]
123686K->79959K(518528K), 0.0206722 secs]
1313.524: [GC 1313.524: [DefNew: 51169K->254K(52480K), 0.0317320 secs]
126615K->78344K(518528K), 0.0317929 secs]
1344.525: [GC 1344.525: [DefNew: 46910K->1729K(52480K), 0.0149128 secs]
125000K->79820K(518528K), 0.0149729 secs]
1351.479: [GC 1351.479: [DefNew: 48381K->4582K(52480K), 0.0218740 secs]
126471K->82673K(518528K), 0.0219304 secs]
1356.378: [Full GC 1356.379: [Tenured: 78090K->68675K(466048K),
6.8007345secs] 100262K->68675K(518528K), [Perm :
24575K->24575K(24576K)],
6.8080206 secs]
1368.802: [GC 1368.802: [DefNew: 46653K->4549K(52480K), 0.0286182 secs]
115328K->73225K(518528K), 0.0286805 secs]
1374.837: [GC 1374.837: [DefNew: 51205K->5688K(52480K), 0.0866753 secs]
119881K->76341K(518528K), 0.0867317 secs]
1380.416: [GC 1380.416: [DefNew: 52344K->5824K(52480K), 0.1281856 secs]
122997K->78947K(518528K), 0.1282481 secs]
1387.302: [GC 1387.302: [DefNew: 52479K->5824K(52480K), 0.1806688 secs]
125603K->81752K(518528K), 0.1807336 secs]
1393.171: [GC 1393.171: [DefNew: 52479K->5823K(52480K), 0.0392198 secs]
128408K->84507K(518528K), 0.0392715 secs]
1400.469: [GC 1400.469: [DefNew: 52479K->4437K(52480K), 0.0609195 secs]
131163K->85732K(518528K), 0.0609793 secs]
1403.957: [Full GC 1403.958: [Tenured: 81295K->76875K(466048K),
0.3270033secs] 106066K->76875K(518528K), [Perm :
28671K->28671K(28672K)],
0.3270935 secs]
1410.753: [GC 1410.753: [DefNew: 46656K->4605K(52480K), 0.0423694 secs]
123531K->81480K(518528K), 0.0424263 secs]
1416.358: [GC 1416.358: [DefNew: 51257K->5458K(52480K), 0.0375570 secs]
128133K->86701K(518528K), 0.0376129 secs]
1421.792: [GC 1421.792: [DefNew: 52114K->5339K(52480K), 0.1025094 secs]
133357K->91866K(518528K), 0.1025714 secs]
1538.764: [Full GC 1538.764: [Tenured: 86527K->59881K(466048K),
5.3928429secs] 133369K->59881K(518528K), [Perm :
32768K->32610K(32768K)],
5.3929301 secs]
1551.420: [GC 1551.420: [DefNew: 46655K->1848K(52480K), 0.0192879 secs]
106537K->61729K(518528K), 0.0193421 secs]
1557.712: [Full GC 1557.712: [Tenured: 59881K->64638K(466048K),
0.3245996secs] 99584K->64638K(518528K), [Perm :
36863K->36863K(36864K)],
0.3246770 secs]
1599.653: [GC 1599.653: [DefNew: 46655K->2500K(52480K), 0.0269780 secs]
111294K->67139K(518528K), 0.0270333 secs]
1603.423: [Full GC 1603.423: [Tenured: 64638K->67911K(466048K),
0.3214652secs] 77743K->67911K(518528K), [Perm :
40959K->40959K(40960K)],
0.3215498 secs]
1627.139: [GC 1627.139: [DefNew: 46655K->1289K(52480K), 0.0142188 secs]
114567K->69201K(518528K), 0.0142742 secs]
1670.137: [Full GC 1670.137: [Tenured: 67911K->69866K(466048K),
0.3690838secs] 90651K->69866K(518528K), [Perm :
45054K->45054K(45056K)],
0.3691751 secs]
1866.589: [GC 1866.589: [DefNew: 46418K->1563K(52480K), 0.0130402 secs]
116284K->71430K(518528K), 0.0131005 secs]
2324.705: [GC 2324.705: [DefNew: 48219K->4327K(52480K), 0.1842139 secs]
118086K->74194K(518528K), 0.1842687 secs]
2329.115: [GC 2329.115: [DefNew: 50983K->3916K(52480K), 0.2362021 secs]
120850K->75941K(518528K), 0.2362599 secs]
2332.533: [GC 2332.533: [DefNew: 50572K->1846K(52480K), 2.1846159 secs]
122597K->77692K(518528K), 2.1846729 secs]
2338.547: [GC 2338.547: [DefNew: 48502K->5824K(52480K), 0.4543022 secs]
124348K->84948K(518528K), 0.4543563 secs]
2341.175: [GC 2341.175: [DefNew: 52480K->1973K(52480K), 0.0189354 secs]
131604K->83257K(518528K), 0.0189910 secs]
.
.
.
.
.
2346.742: [GC 2346.742: [DefNew: 52479K->463K(52480K), 0.0085631 secs]
135685K->85193K(518528K), 0.0086173 secs]
2348.384: [GC 2348.384: [DefNew: 47119K->929K(52480K), 0.0053222 secs]
131849K->85658K(518528K), 0.0053758 secs]
2350.048: [GC 2350.048: [DefNew: 47585K->2118K(52480K), 0.0071565 secs]
132314K->86847K(518528K), 0.0072113 secs]
2351.522: [GC 2351.522: [DefNew: 48774K->1679K(52480K), 0.0072613 secs]
133503K->86409K(518528K), 0.0073149 secs]
2353.277: [GC 2353.277: [DefNew: 48335K->5823K(52480K), 0.1749974 secs]
133065K->94648K(518528K), 0.1750549 secs]
2355.637: [GC 2355.637: [DefNew: 52479K->3109K(52480K), 0.0142968 secs]
141304K->92617K(518528K), 0.0143513 secs]
2357.129: [Full GC 2357.129: [Tenured: 89508K->71494K(466048K),
7.1956506secs] 105965K->71494K(518528K), [Perm :
49151K->48973K(49152K)],
7.2276270 secs]
2369.631: [GC 2369.631: [DefNew: 46655K->1199K(52480K), 0.0261916 secs]
118150K->72693K(518528K), 0.0262450 secs]
2452.783: [GC 2452.783: [DefNew: 47855K->2579K(52480K), 0.0135995 secs]
119349K->74073K(518528K), 0.0136626 secs]
2457.584: [GC 2457.584: [DefNew: 49235K->5823K(52480K), 0.0369991 secs]
120729K->78211K(518528K), 0.0370544 secs]
2461.478: [GC 2461.478: [DefNew: 52479K->1589K(52480K), 0.0268260 secs]
124867K->79759K(518528K), 0.0268830 secs]
2464.523: [GC 2464.523: [DefNew: 48245K->1914K(52480K), 0.0117291 secs]
126415K->80083K(518528K), 0.0117836 secs]
2470.175: [GC 2470.175: [DefNew: 48570K->3163K(52480K), 0.0165518 secs]
126739K->81333K(518528K), 0.0166066 secs]
2473.213: [GC 2473.213: [DefNew: 49819K->2853K(52480K), 0.0148212 secs]
127989K->82160K(518528K), 0.0148874 secs]
2474.596: [GC 2474.596: [DefNew: 49509K->3104K(52480K), 0.0218307 secs]
128816K->82411K(518528K), 0.0218863 secs]
2478.481: [GC 2478.481: [DefNew: 49760K->3208K(52480K), 0.0126334 secs]
129067K->82833K(518528K), 0.0126879 secs]
2479.982: [GC 2479.983: [DefNew: 49864K->5824K(52480K), 0.1148272 secs]
129489K->92140K(518528K), 0.1148850 secs]
2481.667: [GC 2481.667: [DefNew: 52479K->3775K(52480K), 0.0376813 secs]
138796K->91694K(518528K), 0.0377389 secs]
2483.403: [GC 2483.403: [DefNew: 50431K->5823K(52480K), 0.1317053 secs]
138350K->96769K(518528K), 0.1317564 secs]
2485.690: [Full GC 2485.690: [Tenured: 90945K->91468K(466048K),
3.2931426secs] 138286K->91468K(518528K), [Perm :
53247K->53247K(53248K)],
3.2932258 secs]
2490.649: [GC 2490.649: [DefNew: 46655K->5824K(52480K), 0.0365206 secs]
138124K->99492K(518528K), 0.0365753 secs].
.
.
.
.
.
2888.080: [GC 2888.080: [DefNew: 47683K->2102K(52480K), 0.0122331 secs]
168795K->123214K(518528K), 0.0122898 secs]
2889.591: [GC 2889.591: [DefNew: 48758K->2383K(52480K), 0.0198578 secs]
169870K->123496K(518528K), 0.0199112 secs]
2891.947: [GC 2891.947: [DefNew: 49039K->3107K(52480K), 0.0112011 secs]
170152K->124219K(518528K), 0.0112553 secs]
2893.455: [GC 2893.455: [DefNew: 49763K->5718K(52480K), 0.0198187 secs]
170875K->127604K(518528K), 0.0198690 secs]
2895.015: [GC 2895.015: [DefNew: 52374K->5824K(52480K), 0.0642596 secs]
174260K->132189K(518528K), 0.0643112 secs]
2896.954: [GC 2896.954: [DefNew: 52480K->3493K(52480K), 0.0117051 secs]
178845K->130019K(518528K), 0.0117596 secs]
2899.660: [GC 2899.660: [DefNew: 50149K->1101K(52480K), 0.0075071 secs]
176675K->127790K(518528K), 0.0075585 secs]
2901.454: [GC 2901.454: [DefNew: 47756K->587K(52480K), 0.0068498 secs]
174445K->127276K(518528K), 0.0069012 secs]
2905.571: [Full GC 2905.571: [Tenured: 126688K->100740K(466048K),
10.5908514secs] 157839K->100740K(518528K), [Perm :
65535K->65063K(65536K)],
10.6232907 secs]
2919.226: [Full GC 2919.226: [Tenured: 100740K->101030K(466048K),
0.5165606secs] 138033K->101030K(518528K), [Perm :
65535K->65535K(65536K)],
0.5166402 secs]
2919.748: [Full GC 2919.748: [Tenured: 101030K->101030K(466048K),
0.5793060secs] 101030K->101030K(518528K), [Perm :
65535K->65535K(65536K)],
0.5794041 secs]
2920.335: [Full GC 2920.335: [Tenured: 101030K->101030K(466048K),
0.4588885secs] 101030K->101030K(518528K), [Perm :
65535K->65535K(65536K)],
0.4589650 secs]
2920.802: [Full GC 2920.802: [Tenured: 101030K->97189K(466048K),
0.6423210secs] 101030K->97189K(518528K), [Perm :
65535K->65497K(65536K)],
0.6424079 secs]
2968.834: [Full GC 2968.834: [Tenured: 97189K->97335K(466048K),
0.6619274secs] 99370K->97335K(518528K), [Perm :
65535K->65535K(65536K)],
0.6620179 secs]
2969.556: [Full GC 2969.556: [Tenured: 97335K->97396K(466048K),
0.4763756secs] 97664K->97396K(518528K), [Perm :
65535K->65535K(65536K)],
0.4764480 secs]
when i remove some 30-odd *.xml deployment files from /conf.../localhost and
have it load just TWO webapps, it lasts much longer
what should i do? is this a PermGen problemo? And if it is, why does my
error message keep saying "ServeletException" and not the OutOfMemory
PermGen message?
thanks - and if anyone needs graphs/files/blood i'll be happy to drop them
somewhere
AC
On 10/10/07, Pid <[EMAIL PROTECTED]> wrote:
>
> Christopher Schultz wrote:
> > A C,
> >
> > A C wrote:
> >> We've been experiencing horrendous OOMEs recently after deploying 40
> clones
> >> of a vendor's .war file which encompasses their
> application/portal. The
> >> context is about 100MB
> >
> > Lesee... 100 MB times 40 instances .. mm... carry the 7 ... hey, that's
> > 4GB! Where have I heard that number before... hmm.. something about ...
> > right! The maximum process memory on a 32-bit OS!
>
> lol
>
>
>
> A C, I have to say this: the clue is in the question...
>
> I'll summarise; it works OK when when you deploy one copy of a 100Mb
> app, but when you deploy FORTY it gets a bit messy and you run out of
> memory.
>
> How is it that you're not thinking about more servers at this point?
>
> (I'd be waving a nice diagram at the accountants and saying 'shiny new
> boxes please')
>
>
> p
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
