On Mon, Feb 3, 2014 at 2:10 PM, Daniel Mikusa <dmik...@gopivotal.com> wrote:
> On Feb 3, 2014, at 3:30 AM, Maor Yosef <maoryo...@gmail.com> wrote: > > > Hi. > > > > 1. We are aware that 6.0.26 is old, but since there is a large > operational > > impact, we wont upgrade the tomcat until we will know its definetly an > > issue in this specific version > > >>While I understand what you're saying, I disagree. If you need to sell > the change to management, you should take a look at the security problems > that have been fixed since 6.0.26 and >>weigh the cost of upgrading versus > a security breach or manually applying mitigations, if that's even possible. > > >> http://tomcat.apache.org/security-6.html I agree that an upgrade is needed, but still before we will determine to which version to upgrade, we need to solve this issue > > > > 2. You are right, it was my mistake, it causes OOM and not stack > overflow, > > when we see high sessions count we get exceptions saying "unable to > create > > new native thread" > > >>This is telling you that there's not enough memory to allocate any more > threads. > > >>1.) Have you tried setting "-Xss"? This will set the thread stack size, > i.e. how much memory each thread has available for its stack. Often times > you don't need nearly as much as the >>default allocated by the JVM, so you > can lower it and get more threads out of the same available memory. Maybe > try 256k or even lower. You'll know you went too low if you see > >>StackOverflowErrors. > > >>2.) How many threads are being created / used? Perhaps you're creating > too many threads? You'll probably want to do some monitoring and see how > many the Tomcat is creating / using >>and how many your application is > creating / using. Perhaps you've got a problem where too many threads are > being created or where threads are being created and not properly cleaned > up. >>Tools that could help here: jconsole / jvisualvm or thread dumps > > The memory configuration doesn't seems to be the issue, as mentioned before we see a direct correlation between the amount of opened sessions (which can go as high as 100k+) even for a simple application that is just running a simple jsp file, which only prints back to the screen "OK", we still see a large number of sessions for it (also after adding a web.xml with session expiration time = 1) > 3. Looking at the sessions manager, we see a lot of sessions with a > > negative TTL - meaning they werent expired, if I manually expire them > then > > the sessions count decreases. > > >>This doesn't sound related, although it's hard to say. Might be helpful > if you can include your configuration, minus comments. > > > 4. Can you point me to an article on how to configure different > background > > thread for each container? is it configured in tomcat or should be > > implemented in the application? > > >>What exactly do you mean here? You can control Tomcat's thread usage > with an Executor [1] or directly on the connector [2]. > > >>Dan > > >>[1] - http://tomcat.apache.org/tomcat-6.0-doc/config/executor.html > >>[2] - http://tomcat.apache.org/tomcat-6.0-doc/config/http.html > > > > > > Thanks, > > > > > > On Sun, Feb 2, 2014 at 7:38 PM, Konstantin Kolinko > > <knst.koli...@gmail.com>wrote: > > > >> 2014-02-02 Maor Yosef <maoryo...@gmail.com>: > >>> Hi, > >> > >> 1. 6.0.26 is old. > >> > >>> We are facing issues where the sessions are not being expired > >>> and eventually causing a stack overflow. > >> > >> 2. Non-expiring sessions may cause OOM, but they cannot cause a stack > >> overflow. > >> > >> What are your evidences? > >> > >>> We see that at some point the sessions get pilled up and we need to > >>> manually expire those sessions through the manager application, or > >>> until we will restart tomcat but after a few hours / days, sessions > >>> will start to get stuck again > >> > >> 3. Increasing count of session does not mean that sessions do not > expire. > >> > >> Your evidence = ? > >> > >>> We see it in all the applications that are deployed on tomcat, > >>> including the manager application - this rules out (in my opinion) the > >>> possibility that its an issue with our application. > >> > >> 4. Sessions are expired by a background thread. If the thread is stuck > >> somewhere (e.g. doing auto-deployment work) it will affect expiration > >> of sessions. > >> > >> Your thread dump = ? > >> > >> By default there is one background thread that is shared by all > >> container levels in Tomcat, but you can configure a separate one in > >> each container (container = Context, Host or Engine). > >> > >> 5. Web bots that do not support cookies may generate multiple sessions. > >> > >> See CrawlerSessionManagerValve > >> > >> Best regards, > >> Konstantin Kolinko > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
