On Mon, Feb 3, 2014 at 2:10 PM, Daniel Mikusa wrote:
> On Feb 3, 2014, at 3:30 AM, Maor Yosef wrote:
>
> > Hi.
> >
> > 1. We are aware that 6.0.26 is old, but since there is a large
> operational
> > impact, we wont upgrade the tomcat until we will know its definetly an
> > issue in this specific version
>
> >>While I understand what you're saying, I disagree. If you need to sell
> the change to management, you should take a look at the security problems
> that have been fixed since 6.0.26 and >>weigh the cost of upgrading versus
> a security breach or manually applying mitigations, if that's even possible.
>
> >> http://tomcat.apache.org/security-6.html
I agree that an upgrade is needed, but still before we will determine
to which version to upgrade, we need to solve this issue
>
>
> > 2. You are right, it was my mistake, it causes OOM and not stack
> overflow,
> > when we see high sessions count we get exceptions saying "unable to
> create
> > new native thread"
>
> >>This is telling you that there's not enough memory to allocate any more
> threads.
>
> >>1.) Have you tried setting "-Xss"? This will set the thread stack size,
> i.e. how much memory each thread has available for its stack. Often times
> you don't need nearly as much as the >>default allocated by the JVM, so you
> can lower it and get more threads out of the same available memory. Maybe
> try 256k or even lower. You'll know you went too low if you see
> >>StackOverflowErrors.
>
> >>2.) How many threads are being created / used? Perhaps you're creating
> too many threads? You'll probably want to do some monitoring and see how
> many the Tomcat is creating / using >>and how many your application is
> creating / using. Perhaps you've got a problem where too many threads are
> being created or where threads are being created and not properly cleaned
> up. >>Tools that could help here: jconsole / jvisualvm or thread dumps
>
> The memory configuration doesn't seems to be the issue, as mentioned
before we see a direct correlation between the amount of opened sessions
(which can go as high as 100k+)
even for a simple application that is just running a simple jsp file,
which only prints back to the screen "OK", we still see a large number of
sessions for it (also after adding a web.xml with session expiration time =
1)
> 3. Looking at the sessions manager, we see a lot of sessions with a
> > negative TTL - meaning they werent expired, if I manually expire them
> then
> > the sessions count decreases.
>
> >>This doesn't sound related, although it's hard to say. Might be helpful
> if you can include your configuration, minus comments.
>
> > 4. Can you point me to an article on how to configure different
> background
> > thread for each container? is it configured in tomcat or should be
> > implemented in the application?
>
> >>What exactly do you mean here? You can control Tomcat's thread usage
> with an Executor [1] or directly on the connector [2].
>
> >>Dan
>
> >>[1] - http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html
> >>[2] - http://tomcat.apache.org/tomcat-6.0-doc/config/http.html
>
>
> >
> > Thanks,
> >
> >
> > On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko
> > wrote:
> >
> >> 2014-02-02 Maor Yosef :
> >>> Hi,
> >>
> >> 1. 6.0.26 is old.
> >>
> >>> We are facing issues where the sessions are not being expired
> >>> and eventually causing a stack overflow.
> >>
> >> 2. Non-expiring sessions may cause OOM, but they cannot cause a stack
> >> overflow.
> >>
> >> What are your evidences?
> >>
> >>> We see that at some point the sessions get pilled up and we need to
> >>> manually expire those sessions through the manager application, or
> >>> until we will restart tomcat but after a few hours / days, sessions
> >>> will start to get stuck again
> >>
> >> 3. Increasing count of session does not mean that sessions do not
> expire.
> >>
> >> Your evidence = ?
> >>
> >>> We see it in all the applications that are deployed on tomcat,
> >>> including the manager application - this rules out (in my opinion) the
> >>> possibility that its an issue with our application.
> >>
> >> 4. Sessions are expired by a background thread. If the thr
