Re: Found it - WAS: Memory leak
A bit off-topic here but: Filip Hanik wrote: I'll try the fix since I am still on the eval period of the profilers :) Check my posts on the tomcat users mailing list. I have also found the leak by using free tools: IBM JDK + HeapRoots http://alphaworks.ibm.com/tech/heaproots Might help after your eval period is over ;-) Best regards, David - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Filip Hanik wrote:
Ok, I think I nailed it.
There is a memory leak in Tomcat 5. I downloaded a trial of JProbe (they
should provide free licenses for open source! :)
There is a class called o.a.coyote.RequestGroupInfo.java, in that class
there is a method called addRequestProcessor.
This method stores a reference to a RequestInfo object in an ArrayList, the
RequestInfo object has a reference to a Request object and that eventually
holds the reference to a MessageBytes/ByteChunk.
The nr of instances of ByteChunk and CharChunk keep growing and they never
get garbage collected.
To validate my thoughts, I commented out the line
public void addRequestProcessor( RequestInfo rp ) {
//processor.add( rp );
}
in RequestGroupInfo.java and now the instance count for ByteChunk stay
constant.
I have a feeling that the actual solution is a little bit deeper down, right
now I am eager on getting the clustering to perform the way I want, so if
someone wants to take this on be my guest and let me know. If I don't hear
from anyone, I will commit a solution to this next week.
-1. There is no memory leak with this.
ByteChunks/CharChunks are always reused. If new ones are created, they
will be reused.
Rémy
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Bill Barker wrote: I'd be in favor of just dropping RequestInfo and RequestGroupInfo altogether (and provide RequestListener commented out in web.xml that does the same thing). Then people that want this can still get Request stats via JMX (which I might use on a development site, but never on a production site). And as a bonus, we get to ship an example of a RequestListener :). Removing isn't hard (I've just done it for my downloaded version of 5.0.17). The original code isn't very careful about what operations are atomic, so a replacement Listener that's as good as RequestGroupInfo isn't hard to do either :). -1. Please reconsider. There is no memory leak, please check for yourself. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Remy Maucherat wrote: -1. There is no memory leak with this. ByteChunks/CharChunks are always reused. If new ones are created, they will be reused. Arg, I have to find something. I know ! It's all Costin's fault ! I didn't think about the scaling back feature of the thread pool, to be honest. It should be very easy to avoid the problem, by setting maxSpareThreads to maxThreads. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Found it - WAS: Memory leak
Howdy,
+1 one having a RequestListener -- I've been looking for a non-trivial example to
write and ship along with tomcat!
Yoav Shapira
Millennium ChemInformatics
-Original Message-
From: Bill Barker [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 11:07 PM
To: Tomcat Developers List
Subject: Re: Found it - WAS: Memory leak
I'd be in favor of just dropping RequestInfo and RequestGroupInfo
altogether
(and provide RequestListener commented out in web.xml that does the same
thing). Then people that want this can still get Request stats via JMX
(which I might use on a development site, but never on a production site).
And as a bonus, we get to ship an example of a RequestListener :).
Removing isn't hard (I've just done it for my downloaded version of
5.0.17).
The original code isn't very careful about what operations are atomic, so a
replacement Listener that's as good as RequestGroupInfo isn't hard to do
either :).
- Original Message -
From: Filip Hanik [EMAIL PROTECTED]
To: Tomcat Developers List [EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 6:47 PM
Subject: Found it - WAS: Memory leak
Ok, I think I nailed it.
There is a memory leak in Tomcat 5. I downloaded a trial of JProbe (they
should provide free licenses for open source! :)
There is a class called o.a.coyote.RequestGroupInfo.java, in that class
there is a method called addRequestProcessor.
This method stores a reference to a RequestInfo object in an ArrayList,
the
RequestInfo object has a reference to a Request object and that
eventually
holds the reference to a MessageBytes/ByteChunk.
The nr of instances of ByteChunk and CharChunk keep growing and they
never
get garbage collected.
To validate my thoughts, I commented out the line
public void addRequestProcessor( RequestInfo rp ) {
//processor.add( rp );
}
in RequestGroupInfo.java and now the instance count for ByteChunk stay
constant.
I have a feeling that the actual solution is a little bit deeper down,
right
now I am eager on getting the clustering to perform the way I want, so if
someone wants to take this on be my guest and let me know. If I don't
hear
from anyone, I will commit a solution to this next week.
Filip
-Original Message-
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 5:22 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
RE: Memory leak- yeah I knowhi Michael, thanks for your info. I will look
into that as well and set those settings
The thing I am investigating right now is if the fact that we are
registering a lot of things with JMX causes some references to never be
released.
more info to follow.
Filip
-Original Message-
From: Michael Yates [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 4:43 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Filip,
I spent some time a while back trying to get some reproducibility into
memory use in Tomcat (4.1.24)
I found that even sitting idle tomcats memory use would increase.
Here is my original mail to the tomcat-dev mailing list
4.1.24 secure connector throws a LOT of garbage
After I did some more investigations and figured out a fix the
following
bug was raised
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=19264
To reduce the amount of garbage thrown by tomcat I would recommend two
things, doing these might help you track down other memory issues.
1 - Use -XmxsomeValuem -XmssomeValuem.
The above sizes the heap to whatever value you put in there (note the
value is megabytes). I found (at least on the Solaris JVM) that sizing
the
heap and making the min and max heap sizes the same not only sped up
start
up but also made the garbage collector behave much better.
2- Set the serverSoTimeout value to 0 in the connector configuration in
your server configuration file. This will prevent the sever socket timing
out. Which DOES throw garbage, and if it is a secure connector you are
using
it throws SO much garbage that the garbage collector can not keep up too
well.
Using just number tip 1 above (as the fix to the bug above wasn't in
Tomcat 4.1.x before we needed to go into production. We managed to run at
something like 100 TPS for 48 hours with no falling over and no net
increase
in memory usage.
If you need any more information on the above let me know.
Michael Yates
Software Engineer, Location Center
Nortel Networks Wollongong, Australia.
-Original Message-
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Thursday, 15 January 2004 11:09 AM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Remy, absolutely no intentions of being sarcastic :), if there isn't a
leak, that is great, if there is, will fix it.
I'm trying to figure it out, I didn't ask anyone else to spend time on
it,
or even read my email. but if anyone is doing
Re: Found it - WAS: Memory leak
Shapira, Yoav wrote: Howdy, +1 one having a RequestListener -- I've been looking for a non-trivial example to write and ship along with tomcat! -0. This kind of stuff is already in the servlet API to some extent, and all these event notifications will likely introduce overhead. This looks like a bad idea to me so far. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Found it - WAS: Memory leak
-1. Please reconsider. There is no memory leak, please check for yourself. Rémy dude, I spent hours on this yesterday with a profiler...how about giving in a little bit :) the leak is pretty bad, I ran up to 512MB in hours of heavy load. Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Thursday, January 15, 2004 12:28 AM To: Tomcat Developers List Subject: Re: Found it - WAS: Memory leak Bill Barker wrote: I'd be in favor of just dropping RequestInfo and RequestGroupInfo altogether (and provide RequestListener commented out in web.xml that does the same thing). Then people that want this can still get Request stats via JMX (which I might use on a development site, but never on a production site). And as a bonus, we get to ship an example of a RequestListener :). Removing isn't hard (I've just done it for my downloaded version of 5.0.17). The original code isn't very careful about what operations are atomic, so a replacement Listener that's as good as RequestGroupInfo isn't hard to do either :). -1. Please reconsider. There is no memory leak, please check for yourself. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Filip Hanik wrote: dude, I spent hours on this yesterday with a profiler...how about giving in a little bit :) the leak is pretty bad, I ran up to 512MB in hours of heavy load. For one, I couldn't reproduce the problem. I had to use my imagination and saw that there was an issue when the thread pool was stopping threads. So there is a trivial workaround. OTOH, using HTTP client is better on Windows than ab, which behaves oddly sometimes due to Cygwin's IO. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Found it - WAS: Memory leak
I dont think setting maxSpareThreads==maxThreads is a good solution to a problem as this memory leak. I still have to verify that that would actually solve the problem if we don't want to remove request info, lets fix it. Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Thursday, January 15, 2004 5:24 AM To: Tomcat Developers List Subject: Re: Found it - WAS: Memory leak Remy Maucherat wrote: -1. There is no memory leak with this. ByteChunks/CharChunks are always reused. If new ones are created, they will be reused. Arg, I have to find something. I know ! It's all Costin's fault ! I didn't think about the scaling back feature of the thread pool, to be honest. It should be very easy to avoid the problem, by setting maxSpareThreads to maxThreads. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
This sort of memory leak can't be left unaddressed -- i.e. you shouldn't have to change the settings to avoid an intolerable memory leak. Is this issue new to 5.0.17 or was it in 5.0.16 as well? In either case, it would be good to see this fixed in 5.0.17 -- but it is all the more critical if it was initially broken in 5.0.17! -- Jess Holle Filip Hanik wrote: I dont think setting maxSpareThreads==maxThreads is a good solution to a problem as this memory leak. I still have to verify that that would actually solve the problem if we don't want to remove request info, lets fix it. Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Thursday, January 15, 2004 5:24 AM To: Tomcat Developers List Subject: Re: Found it - WAS: Memory leak Remy Maucherat wrote: -1. There is no memory leak with this. ByteChunks/CharChunks are always reused. If new ones are created, they will be reused. Arg, I have to find something. I know ! It's all Costin's fault ! I didn't think about the scaling back feature of the thread pool, to be honest. It should be very easy to avoid the problem, by setting maxSpareThreads to maxThreads. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Found it - WAS: Memory leak
I ran HTTP client on windows, Tomcat on RH8 AND RH9, both of them have the leak. I ran tomcat through both OptimizeIt and JProbe yesterday, and both of them showed the leak. it should be very easy to reproduce, you just have to be patient...it takes time for the memory to grow, but it does and it doesn't shrink back. Looks like I have to do more digging in, I will try to find out exactly where it is wrong, I just know that skipping addRequestProcessor solves my problem. Be back in a few days with what I hope is a concrete solution, than just commenting out a line Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Thursday, January 15, 2004 9:41 AM To: Tomcat Developers List Subject: Re: Found it - WAS: Memory leak Filip Hanik wrote: dude, I spent hours on this yesterday with a profiler...how about giving in a little bit :) the leak is pretty bad, I ran up to 512MB in hours of heavy load. For one, I couldn't reproduce the problem. I had to use my imagination and saw that there was an issue when the thread pool was stopping threads. So there is a trivial workaround. OTOH, using HTTP client is better on Windows than ab, which behaves oddly sometimes due to Cygwin's IO. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Filip Hanik wrote: I ran HTTP client on windows, Tomcat on RH8 AND RH9, both of them have the leak. I ran tomcat through both OptimizeIt and JProbe yesterday, and both of them showed the leak. it should be very easy to reproduce, you just have to be patient...it takes time for the memory to grow, but it does and it doesn't shrink back. Looks like I have to do more digging in, I will try to find out exactly where it is wrong, I just know that skipping addRequestProcessor solves my problem. Be back in a few days with what I hope is a concrete solution, than just commenting out a line I have fixed the leak, and have provided an easy workaround using the configuration file. I could not reproduce the said leak without finding where it was in the code first, so it will not leak a lot of memory in real world situations. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
I'll try the fix since I am still on the eval period of the profilers :) thanks Filip - Original Message - From: Remy Maucherat [EMAIL PROTECTED] To: Tomcat Developers List [EMAIL PROTECTED] Sent: Thursday, January 15, 2004 9:56 AM Subject: Re: Found it - WAS: Memory leak Filip Hanik wrote: I ran HTTP client on windows, Tomcat on RH8 AND RH9, both of them have the leak. I ran tomcat through both OptimizeIt and JProbe yesterday, and both of them showed the leak. it should be very easy to reproduce, you just have to be patient...it takes time for the memory to grow, but it does and it doesn't shrink back. Looks like I have to do more digging in, I will try to find out exactly where it is wrong, I just know that skipping addRequestProcessor solves my problem. Be back in a few days with what I hope is a concrete solution, than just commenting out a line I have fixed the leak, and have provided an easy workaround using the configuration file. I could not reproduce the said leak without finding where it was in the code first, so it will not leak a lot of memory in real world situations. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Filip Hanik wrote: I'll try the fix since I am still on the eval period of the profilers :) Cool :) (crossing all fingers) You know there's also that tomcat-committers mailing list for the issues that make people nervous (security, etc) ;) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
From: Remy Maucherat [EMAIL PROTECTED] Filip Hanik wrote: I'll try the fix since I am still on the eval period of the profilers :) Cool :) (crossing all fingers) You know there's also that tomcat-committers mailing list for the issues that make people nervous (security, etc) ;) no, I didn't know that, funny me being a committer and all :-)) LOL - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
- Original Message - From: Remy Maucherat [EMAIL PROTECTED] To: Tomcat Developers List [EMAIL PROTECTED] Sent: Thursday, January 15, 2004 5:24 AM Subject: Re: Found it - WAS: Memory leak Remy Maucherat wrote: -1. There is no memory leak with this. ByteChunks/CharChunks are always reused. If new ones are created, they will be reused. Arg, I have to find something. I know ! It's all Costin's fault ! I didn't think about the scaling back feature of the thread pool, to be honest. It should be very easy to avoid the problem, by setting maxSpareThreads to maxThreads. Also, Jk-Coyote doesn't pool threads, so the total number of Requests created will (slowly) grow over time. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL. If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments. In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Bill Barker wrote: Remy Maucherat wrote: -1. There is no memory leak with this. ByteChunks/CharChunks are always reused. If new ones are created, they will be reused. Arg, I have to find something. I know ! It's all Costin's fault ! I didn't think about the scaling back feature of the thread pool, to be honest. It should be very easy to avoid the problem, by setting maxSpareThreads to maxThreads. Also, Jk-Coyote doesn't pool threads, so the total number of Requests created will (slowly) grow over time. I'm almost certain JK doesn't register the RequestInfo objects. It should (and it's a bug), but actually it doesn't. I tried to have the status servlet work with JK also, and had partial fixes, but in the end, I didn't find the feature too worthwhile (and my fixes were hacks), so I forgot about it (good thing I did). Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
- Original Message - From: Remy Maucherat [EMAIL PROTECTED] To: Tomcat Developers List [EMAIL PROTECTED] Sent: Thursday, January 15, 2004 11:41 AM Subject: Re: Found it - WAS: Memory leak Bill Barker wrote: Remy Maucherat wrote: -1. There is no memory leak with this. ByteChunks/CharChunks are always reused. If new ones are created, they will be reused. Arg, I have to find something. I know ! It's all Costin's fault ! I didn't think about the scaling back feature of the thread pool, to be honest. It should be very easy to avoid the problem, by setting maxSpareThreads to maxThreads. Also, Jk-Coyote doesn't pool threads, so the total number of Requests created will (slowly) grow over time. I'm almost certain JK doesn't register the RequestInfo objects. It should (and it's a bug), but actually it doesn't. I tried to have the status servlet work with JK also, and had partial fixes, but in the end, I didn't find the feature too worthwhile (and my fixes were hacks), so I forgot about it (good thing I did). It registers it on line 475 of HandlerRequest. It's true that Jk doesn't do much else with the RequestInfo, but it does register it. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL. If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments. In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
Bill Barker wrote: I'm almost certain JK doesn't register the RequestInfo objects. It should (and it's a bug), but actually it doesn't. I tried to have the status servlet work with JK also, and had partial fixes, but in the end, I didn't find the feature too worthwhile (and my fixes were hacks), so I forgot about it (good thing I did). It registers it on line 475 of HandlerRequest. It's true that Jk doesn't do much else with the RequestInfo, but it does register it. Yes, I know the code exists and would look like it would work, but last time I tried, all the JMX code in JK wasn't executed. For example, the MBean for the ChannelSocket wasn't registered. Did you put traces in there to double check ? I'm not very familar with JK, so I don't know for sure. Even if it's registered, I don't know if it would be a problem or not. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
setting maxSpareThreads==minSpareThreads==maxThreads does take care of the memory leak with the new code, however, going with the default server.xml settings, tomcat is still leaking memory. I will delve into this when I get home tonight, that is when I have access to the profiler. have a good one, Filip - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
- Original Message - From: Remy Maucherat [EMAIL PROTECTED] To: Tomcat Developers List [EMAIL PROTECTED] Sent: Thursday, January 15, 2004 12:33 PM Subject: Re: Found it - WAS: Memory leak Bill Barker wrote: I'm almost certain JK doesn't register the RequestInfo objects. It should (and it's a bug), but actually it doesn't. I tried to have the status servlet work with JK also, and had partial fixes, but in the end, I didn't find the feature too worthwhile (and my fixes were hacks), so I forgot about it (good thing I did). It registers it on line 475 of HandlerRequest. It's true that Jk doesn't do much else with the RequestInfo, but it does register it. Yes, I know the code exists and would look like it would work, but last time I tried, all the JMX code in JK wasn't executed. For example, the MBean for the ChannelSocket wasn't registered. Did you put traces in there to double check ? Yup, it seems that the JMX registration in Jk is broken. HandlerRequest never gets registered, so it never registers the RequestInfo. I'm not very familar with JK, so I don't know for sure. Even if it's registered, I don't know if it would be a problem or not. If it was registered, it most definitely would be a memory leak. You would keep creating Request objects that never get dereferenced. Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This message is intended only for the use of the person(s) listed above as the intended recipient(s), and may contain information that is PRIVILEGED and CONFIDENTIAL. If you are not an intended recipient, you may not read, copy, or distribute this message or any attachment. If you received this communication in error, please notify us immediately by e-mail and then delete all copies of this message and any attachments. In addition you should be aware that ordinary (unencrypted) e-mail sent through the Internet is not secure. Do not send confidential or sensitive information, such as social security numbers, account numbers, personal identification numbers and passwords, to us via ordinary (unencrypted) e-mail. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Found it - WAS: Memory leak
I'd be in favor of just dropping RequestInfo and RequestGroupInfo altogether
(and provide RequestListener commented out in web.xml that does the same
thing). Then people that want this can still get Request stats via JMX
(which I might use on a development site, but never on a production site).
And as a bonus, we get to ship an example of a RequestListener :).
Removing isn't hard (I've just done it for my downloaded version of 5.0.17).
The original code isn't very careful about what operations are atomic, so a
replacement Listener that's as good as RequestGroupInfo isn't hard to do
either :).
- Original Message -
From: Filip Hanik [EMAIL PROTECTED]
To: Tomcat Developers List [EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 6:47 PM
Subject: Found it - WAS: Memory leak
Ok, I think I nailed it.
There is a memory leak in Tomcat 5. I downloaded a trial of JProbe (they
should provide free licenses for open source! :)
There is a class called o.a.coyote.RequestGroupInfo.java, in that class
there is a method called addRequestProcessor.
This method stores a reference to a RequestInfo object in an ArrayList,
the
RequestInfo object has a reference to a Request object and that eventually
holds the reference to a MessageBytes/ByteChunk.
The nr of instances of ByteChunk and CharChunk keep growing and they never
get garbage collected.
To validate my thoughts, I commented out the line
public void addRequestProcessor( RequestInfo rp ) {
//processor.add( rp );
}
in RequestGroupInfo.java and now the instance count for ByteChunk stay
constant.
I have a feeling that the actual solution is a little bit deeper down,
right
now I am eager on getting the clustering to perform the way I want, so if
someone wants to take this on be my guest and let me know. If I don't hear
from anyone, I will commit a solution to this next week.
Filip
-Original Message-
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 5:22 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
RE: Memory leak- yeah I knowhi Michael, thanks for your info. I will look
into that as well and set those settings
The thing I am investigating right now is if the fact that we are
registering a lot of things with JMX causes some references to never be
released.
more info to follow.
Filip
-Original Message-
From: Michael Yates [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 4:43 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Filip,
I spent some time a while back trying to get some reproducibility into
memory use in Tomcat (4.1.24)
I found that even sitting idle tomcats memory use would increase.
Here is my original mail to the tomcat-dev mailing list
4.1.24 secure connector throws a LOT of garbage
After I did some more investigations and figured out a fix the following
bug was raised
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=19264
To reduce the amount of garbage thrown by tomcat I would recommend two
things, doing these might help you track down other memory issues.
1 - Use -XmxsomeValuem -XmssomeValuem.
The above sizes the heap to whatever value you put in there (note the
value is megabytes). I found (at least on the Solaris JVM) that sizing the
heap and making the min and max heap sizes the same not only sped up start
up but also made the garbage collector behave much better.
2- Set the serverSoTimeout value to 0 in the connector configuration in
your server configuration file. This will prevent the sever socket timing
out. Which DOES throw garbage, and if it is a secure connector you are
using
it throws SO much garbage that the garbage collector can not keep up too
well.
Using just number tip 1 above (as the fix to the bug above wasn't in
Tomcat 4.1.x before we needed to go into production. We managed to run at
something like 100 TPS for 48 hours with no falling over and no net
increase
in memory usage.
If you need any more information on the above let me know.
Michael Yates
Software Engineer, Location Center
Nortel Networks Wollongong, Australia.
-Original Message-
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Thursday, 15 January 2004 11:09 AM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Remy, absolutely no intentions of being sarcastic :), if there isn't a
leak, that is great, if there is, will fix it.
I'm trying to figure it out, I didn't ask anyone else to spend time on
it,
or even read my email. but if anyone is doing work on this lets work on it
together.
just got jprobe up and running, once I have ran a few tests, I will
experiment with the garbage collection, the memory is slowly growing, its
been running for an hour.
Filip
-Original Message-
From: Remy Maucherat [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 3:08 PM
RE: Found it - WAS: Memory leak
+0 (would have been +1 but I don't know that I have extra cycles for dev
right now)
I agree on that. The current problem keeps it from being production stable.
With the bug fixed, I am able to run 1000 concurrent users (3 sec sleep
between each request) again 4 clustered tomcat instances on two mediocre
machines. The average response time is about 2 seconds, the cpu is around
40% on each machine, and most important, the memory stays constant. These
tests used to crap out on me after a little while, but now it keeps going
and going. Unfortunately I am saturating my little home network :)
Filip
-Original Message-
From: Bill Barker [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 8:07 PM
To: Tomcat Developers List
Subject: Re: Found it - WAS: Memory leak
I'd be in favor of just dropping RequestInfo and RequestGroupInfo altogether
(and provide RequestListener commented out in web.xml that does the same
thing). Then people that want this can still get Request stats via JMX
(which I might use on a development site, but never on a production site).
And as a bonus, we get to ship an example of a RequestListener :).
Removing isn't hard (I've just done it for my downloaded version of 5.0.17).
The original code isn't very careful about what operations are atomic, so a
replacement Listener that's as good as RequestGroupInfo isn't hard to do
either :).
- Original Message -
From: Filip Hanik [EMAIL PROTECTED]
To: Tomcat Developers List [EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 6:47 PM
Subject: Found it - WAS: Memory leak
Ok, I think I nailed it.
There is a memory leak in Tomcat 5. I downloaded a trial of JProbe (they
should provide free licenses for open source! :)
There is a class called o.a.coyote.RequestGroupInfo.java, in that class
there is a method called addRequestProcessor.
This method stores a reference to a RequestInfo object in an ArrayList,
the
RequestInfo object has a reference to a Request object and that eventually
holds the reference to a MessageBytes/ByteChunk.
The nr of instances of ByteChunk and CharChunk keep growing and they never
get garbage collected.
To validate my thoughts, I commented out the line
public void addRequestProcessor( RequestInfo rp ) {
//processor.add( rp );
}
in RequestGroupInfo.java and now the instance count for ByteChunk stay
constant.
I have a feeling that the actual solution is a little bit deeper down,
right
now I am eager on getting the clustering to perform the way I want, so if
someone wants to take this on be my guest and let me know. If I don't hear
from anyone, I will commit a solution to this next week.
Filip
-Original Message-
From: Filip Hanik [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 5:22 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
RE: Memory leak- yeah I knowhi Michael, thanks for your info. I will look
into that as well and set those settings
The thing I am investigating right now is if the fact that we are
registering a lot of things with JMX causes some references to never be
released.
more info to follow.
Filip
-Original Message-
From: Michael Yates [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 14, 2004 4:43 PM
To: Tomcat Developers List
Subject: RE: Memory leak- yeah I know
Filip,
I spent some time a while back trying to get some reproducibility into
memory use in Tomcat (4.1.24)
I found that even sitting idle tomcats memory use would increase.
Here is my original mail to the tomcat-dev mailing list
4.1.24 secure connector throws a LOT of garbage
After I did some more investigations and figured out a fix the following
bug was raised
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=19264
To reduce the amount of garbage thrown by tomcat I would recommend two
things, doing these might help you track down other memory issues.
1 - Use -XmxsomeValuem -XmssomeValuem.
The above sizes the heap to whatever value you put in there (note the
value is megabytes). I found (at least on the Solaris JVM) that sizing the
heap and making the min and max heap sizes the same not only sped up start
up but also made the garbage collector behave much better.
2- Set the serverSoTimeout value to 0 in the connector configuration in
your server configuration file. This will prevent the sever socket timing
out. Which DOES throw garbage, and if it is a secure connector you are
using
it throws SO much garbage that the garbage collector can not keep up too
well.
Using just number tip 1 above (as the fix to the bug above wasn't in
Tomcat 4.1.x before we needed to go into production. We managed to run at
something like 100 TPS for 48 hours with no falling over and no net
increase
in memory usage.
If you need any more information on the above let me know.
Michael Yates
Software Engineer, Location Center
Nortel Networks Wollongong, Australia
