Memory leak in using SSL with Tomcat 6.0.18

[B. Balakrishna Rao]

Hi,

We are using Tomcat 6.0.18 on 
Linux environment(Red hat Linux) for our production.
We have enabled SSL by deploying SSL certificates. We observed that, over a 
period of time, the memory consumption is increasing and we are facing serious 
performance issues.
I have taken a heap dump and analyzed it using 
Eclipse Memory 
analyzer. What I found was, com.sun.net.ssl.internal.ssl.SSLSocketImpl objects 
are never Garbage collected.
The leak report by Memory analyzer is as follows:
2,996 instances of com.sun.net.ssl.internal.ssl.SSLSocketImpl", loaded by 
"" occupy 219,843,760 (62.76%) bytes
Upon drilling down, these objects are being held by finalizer method of GC.
I have got the following similar link:
http://forums.sun.com/thread.jspa?threadID=5266266

Can anybody tell me if there is any memory leak issues exists with Tomcat 
6.0.18? I read somewhere that, the memory leak issue with SSL was fixed on 
6.0.20.

Balakrishna Rao | Senior Software Engineer | Persistent Systems
balakrishna_...@persistent.co.in  | 
Cell: +91 9704373579 | Tel: +91 (40) 30875030
Innovation in software product design, development and delivery- 
www.persistentsys.com


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

RE: Memory leak in using SSL with Tomcat 6.0.18

[B. Balakrishna Rao]

Hi Mark,

Thanks for your reply.
I tried reading the change log. It appears that in Tomcat 6.0.20 there is a fix 
related to memory leak using SSL.
What I am thinking is that if this is the issue with the Tomcat 6.0.18 or an 
issue with my application.
Can you confirm if there is a memory leak issue exists in using SSL with tomcat 
6.0.18 on Linux environment which similar to my problem? It helps me 
concentrate on upgrading the Tomcat in my production.
Please let me know if you need any further details about my environment.

Thanks,
Bala.


-Original Message-
From: Mark Thomas [mailto:ma...@apache.org]
Sent: Wednesday, August 04, 2010 4:18 PM
To: Tomcat Users List
Subject: Re: Memory leak in using SSL with Tomcat 6.0.18

On 04/08/2010 10:49, B. Balakrishna Rao wrote:
> Hi,
>
> We are using Tomcat<http://www.coderanch.com/forums/f-56/Tomcat> 6.0.18 on 
> Linux environment(Red hat Linux) for our production.
> We have enabled SSL by deploying SSL certificates. We observed that, over a 
> period of time, the memory consumption is increasing and we are facing 
> serious performance issues.
> I have taken a heap dump and analyzed it using 
> Eclipse<http://www.myeclipseide.com/module-htmlpages-display-pid-1.html> 
> Memory analyzer. What I found was, com.sun.net.ssl.internal.ssl.SSLSocketImpl 
> objects are never Garbage collected.
> The leak report by Memory analyzer is as follows:
> 2,996 instances of com.sun.net.ssl.internal.ssl.SSLSocketImpl", loaded by 
> "" occupy 219,843,760 (62.76%) bytes
> Upon drilling down, these objects are being held by finalizer method of GC.
> I have got the following similar link:
> http://forums.sun.com/thread.jspa?threadID=5266266
>
> Can anybody tell me if there is any memory leak issues exists with Tomcat 
> 6.0.18? I read somewhere that, the memory leak issue with SSL was fixed on 
> 6.0.20.

Have you tried reading the changelog?

Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Memory leak in using SSL with Tomcat 6.0.18

[B. Balakrishna Rao]

Hi Mark,

I am trying to apply the patch that is available for the fix below:
https://issues.apache.org/bugzilla/show_bug.cgi?id=47744#c2

However, after giving the below command, putty simply hangs.

patch sslsession-memory-fix.patch

I am using Red hat Linux with Apache Tomcat 6.0.18.
Another issue: Do I need to update the tomcat-coyote.jar along with this patch?
Please refer the below text from the above link:
" If you apply the patch, update tomcat-coyote.jar.."

Can you help me on this please?

Thanks,
Bala.



-Original Message-
From: Mark Thomas [mailto:ma...@apache.org]
Sent: Wednesday, August 04, 2010 4:28 PM
To: Tomcat Users List
Subject: Re: Memory leak in using SSL with Tomcat 6.0.18

On 04/08/2010 11:54, B. Balakrishna Rao wrote:
> Hi Mark,
>
> Thanks for your reply.
> I tried reading the change log. It appears that in Tomcat 6.0.20 there is a 
> fix related to memory leak using SSL.
> What I am thinking is that if this is the issue with the Tomcat 6.0.18 or an 
> issue with my application.
> Can you confirm if there is a memory leak issue exists in using SSL with 
> tomcat 6.0.18 on Linux environment which similar to my problem? It helps me 
> concentrate on upgrading the Tomcat in my production.
> Please let me know if you need any further details about my environment.

I'm pretty sure the fix you are referring to was fixed in 6.0.21, not
6.0.20.

I suggest you read the bug report and compare the details there to what
you are seeing to determine if they are the same.

Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Memory leak in using SSL with Tomcat 6.0.18

[B. Balakrishna Rao]

Hi Mark,

I have implemented your suggestion. I have deployed my application in Tomcat 
6.0.29 version under the same environment as Tomcat 6.0.18(test environment).

After performing the similar operations on both tomcat versions one after 
another, I have taken the heap dumps from both versions.
What I observed is: Number of com.sun.net.ssl.internal.ssl.SSLSocketImpl 
objects are 10 in tomcat 6.0.18 and 7 in tomcat 6.0.29.
I can't able to say that the issue is fixed :(

Well I am attaching my original mail on this issue for your reference.

We are using Tomcat 6.0.18 on Linux environment(Red hat Linux) for our 
production.
We have enabled SSL by deploying SSL certificates. We observed that, over a 
period of time, the memory consumption is increasing and we are facing serious 
performance issues.
I have taken a heap dump and analyzed it using Eclipse Memory analyzer. What I 
found was, com.sun.net.ssl.internal.ssl.SSLSocketImpl objects are never Garbage 
collected.
The leak report by Memory analyzer is as follows:
2,996 instances of com.sun.net.ssl.internal.ssl.SSLSocketImpl", loaded by 
"" occupy 219,843,760 (62.76%) bytes
Upon drilling down, these objects are being held by finalizer method of GC.
I have got the following similar link:
http://forums.sun.com/thread.jspa?threadID=5266266

Can anybody tell me if there is any memory leak issues exists with Tomcat 
6.0.18? I read somewhere that, the memory leak issue with SSL was fixed on 
6.0.20.

Balakrishna Rao | Senior Software Engineer | Persistent Systems
balakrishna_...@persistent.co.in  | Cell: +91 9704373579 | Tel: +91 (40) 
30875030
Innovation in software product design, development and delivery- 
www.persistentsys.com

-Original Message-
From: Mark Thomas [mailto:ma...@apache.org]
Sent: Wednesday, August 04, 2010 6:19 PM
To: Tomcat Users List
Subject: Re: Memory leak in using SSL with Tomcat 6.0.18

On 04/08/2010 13:40, B. Balakrishna Rao wrote:
> Hi Mark,
>
> I am trying to apply the patch that is available for the fix below:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=47744#c2

Why? What makes you think that is the problem you are seeing?

> However, after giving the below command, putty simply hangs.
>
> patch sslsession-memory-fix.patch

Impossible to comment given the lack of context information provided.

> I am using Red hat Linux with Apache Tomcat 6.0.18.
> Another issue: Do I need to update the tomcat-coyote.jar along with this 
> patch?
> Please refer the below text from the above link:
> " If you apply the patch, update tomcat-coyote.jar.."
>
> Can you help me on this please?

Given your difficulties in applying patch, I suspect you will have
difficulties building the source too. Assuming this patch fixes the
issue you are seeing (and I have yet to see any evidence that it will)
why not just use a binary that already includes the fix?

Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Memory leak in using SSL with Tomcat 6.0.18

[B. Balakrishna Rao]

Hi Chris,

Please note that, the 2,996 count is on production environment.
The counts 7 and 10 are on my local environment.

Below is the procedure I am following on my local environment to test this:

Log in -> do some operations -> log out.
I am calling session.invalidate() method upon log out. After that, I am taking 
the heap dump. Eclipse Memory Analyzer tool will do a full GC before it produce 
the results. Hence, com.sun.net.ssl.internal.ssl.SSLSocketImpl should be GCed??


Thanks,
Bala.


-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
Sent: Wednesday, August 04, 2010 7:43 PM
To: Tomcat Users List
Subject: Re: Memory leak in using SSL with Tomcat 6.0.18

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 8/4/2010 10:06 AM, B. Balakrishna Rao wrote:
> I have implemented your suggestion. I have deployed my application 
> in Tomcat 6.0.29 version under the same environment as Tomcat
> 6.0.18(test environment).
> 
> After performing the similar operations on both tomcat versions one 
> after another, I have taken the heap dumps from both versions. What I
> observed is: Number of com.sun.net.ssl.internal.ssl.SSLSocketImpl
> objects are 10 in tomcat 6.0.18 and 7 in tomcat 6.0.29.
>
> I can't able to say that the issue is fixed :(

Neither of these object counts seem unreasonable. Both are much better
than the original report of nearly 3000 object instances.

What is maxThreads set to? I would imagine that there would be a number
SSLSocketImpl objects around for each current connection, plus some that
hadn't yet been GC'd.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxZdVUACgkQ9CaO5/Lv0PAPVgCgljnlorFrcO3FYLY6otoUErxh
M+0Anjo11qs18M5XLOOzQTQlJ5RF/xwY
=iAZ5
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

RE: Memory leak in using SSL with Tomcat 6.0.18

[B. Balakrishna Rao]

Hi Chris,

Attached is the image for incoming references for 
com.sun.net.ssl.internal.ssl.SSLSocketImpl objects.
Please let me know if you want any further details.

Thanks,


-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
Sent: Wednesday, August 04, 2010 10:14 PM
To: Tomcat Users List
Subject: Re: Memory leak in using SSL with Tomcat 6.0.18

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

B.,

On 8/4/2010 10:19 AM, B. Balakrishna Rao wrote:
> Please note that, the 2,996 count is on production environment.
> The counts 7 and 10 are on my local environment.

Ok.

> Below is the procedure I am following on my local environment to test this:
> 
> Log in -> do some operations -> log out.
> I am calling session.invalidate() method upon log out.

Whether you log out of not shouldn't have anything to do with these
objects staying around.

> After that, I am taking the heap dump. Eclipse Memory Analyzer tool
> will do a full GC before it produce the results. Hence, 
> com.sun.net.ssl.internal.ssl.SSLSocketImpl should be GCed??

Most likely.

Since you're using a profiler, can you show us what the stack trace is
of the code that created the SSLSocketImpl objects?

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxZmNEACgkQ9CaO5/Lv0PA3DwCdEpwgPIclWBmmlfM+wD5VX0w4
YPIAn2P5+aVG9u8UswVYPEd5ctXh2jO1
=kV3t
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Prioritizing incoming requests

[B. Balakrishna Rao]

Hi,

I would like to know if we can prioritizing the incoming requests in Tomcat.
Let's say, I have a report module and a cart module in my web application. I 
would like to give more priority to report module than the cart module as 
Report modules seems to take longer period of time. By doing so, What I am 
thinking is that, Tomcat would give high priority to the report requests over 
the cart requests so that reports requests would take lesser time to execute?

Thanks,
Bala.

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

RE: Memory leak in using SSL with Tomcat 6.0.18 - Solved

[B. Balakrishna Rao]

The problem with memory leak using Tomcat 6.0.18 with SSL+JSSE is solved.
I have implemented native SSL(using Apache APR) instead of JSSE SSL and the 
problem went away.
Thanks for your help on this.

Please note that memory leak using with SSL+JSSE is still exist on tomcat 
6.0.29 version.

Thanks,
Bala.

-Original Message-
From: B. Balakrishna Rao 
Sent: Thursday, August 05, 2010 10:17 AM
To: Tomcat Users List
Subject: RE: Memory leak in using SSL with Tomcat 6.0.18

Hi Chris,

Attached is the image for incoming references for 
com.sun.net.ssl.internal.ssl.SSLSocketImpl objects.
Please let me know if you want any further details.

Thanks,


-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Wednesday, August 04, 2010 10:14 PM
To: Tomcat Users List
Subject: Re: Memory leak in using SSL with Tomcat 6.0.18

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

B.,

On 8/4/2010 10:19 AM, B. Balakrishna Rao wrote:
> Please note that, the 2,996 count is on production environment.
> The counts 7 and 10 are on my local environment.

Ok.

> Below is the procedure I am following on my local environment to test this:
> 
> Log in -> do some operations -> log out.
> I am calling session.invalidate() method upon log out.

Whether you log out of not shouldn't have anything to do with these objects 
staying around.

> After that, I am taking the heap dump. Eclipse Memory Analyzer tool 
> will do a full GC before it produce the results. Hence, 
> com.sun.net.ssl.internal.ssl.SSLSocketImpl should be GCed??

Most likely.

Since you're using a profiler, can you show us what the stack trace is of the 
code that created the SSLSocketImpl objects?

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxZmNEACgkQ9CaO5/Lv0PA3DwCdEpwgPIclWBmmlfM+wD5VX0w4
YPIAn2P5+aVG9u8UswVYPEd5ctXh2jO1
=kV3t
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

RE: Tomcat unexpectedly shuts down

[B. Balakrishna Rao]

Hi,

We have enabled the GC logs on our production and our tomcat server crashed 
too!!
Apparently, GC logs takes high CPU and eventually, tomcat server crashes due to 
high CPU utilization of GC logs. Remove them, your problem will be solved.
What I believe is, enabling GC logs are not suggestible for long run. They are 
for observation the system and for short duration only.

Balakrishna Rao | Senior Software Engineer | Persistent Systems
balakrishna_...@persistent.co.in  | Cell: +91 9704373579 | Tel: +91 (40) 
30875030
Innovation in software product design, development and delivery- 
www.persistentsys.com


-Original Message-
From: Gerardo Corro [mailto:rob_gar_...@hotmail.com]
Sent: Monday, August 09, 2010 7:00 PM
To: users@tomcat.apache.org
Subject: RE: Tomcat unexpectedly shuts down



Hi,

Thanks for your email


>
> Do you have enough RAM + swap space to support that large a heap, along with 
> the other memory requirements of the Tomcat process and all other processes 
> on the system?  If not, you may well be getting hit by the Linux OOM-killer, 
> which leaves few traces.

Yes we do, we have 6GB total memory

>
> > However Tomcat fails quite often, no trace information can be
> > found in logs, neither my app logs nor tomcat logs.
>
> What about a core file, or a JVM crash file?
>

No core files neither jvm crash files :(

Best regards


DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org