Re: SOLVED: Immortal… ehm, frozen instances.

2013-04-03 Thread Chuck Hill
Sounds like it was timing out and throwing an exception that was improperly handled. Seems odd that it would cause zombie sessions even with the try...finally in sleep() and terminate() On 2013-04-03, at 10:32 AM, Altera WO Team wrote: > Hi all, > > I finally solved the problem… well, not re

Re: SOLVED: Immortal… ehm, frozen instances.

2013-04-03 Thread Steve Peery
I strongly agree with the suggestion for a MySQL replication slave server. They are not that hard to set up and then you can dump a backup as often as you like with out effecting the main server and you always have an up to the minute copy incase something happens. I had a problem a couple of mo

Re: SOLVED: Immortal… ehm, frozen instances.

2013-04-03 Thread Kieran Kelleher
Another longer term option is to have a MySQL replication slave server that performs backup snapshots as needed (hourly, daily, weekly). On Apr 3, 2013, at 1:32 PM, Altera WO Team wrote: > Hi all, > > I finally solved the problem… well, not really but I found out the reason. > Turns out that

SOLVED: Immortal… ehm, frozen instances.

2013-04-03 Thread Altera WO Team
Hi all, I finally solved the problem… well, not really but I found out the reason. Turns out that the whole thing is on amazon boxes and the DB machine (MySQL) was being backed up every hour… The backup takes all the cpu (not the backup, the bzip2 of the backup) and locks up MySQL for 2 minutes

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread Altera WO Team
Chuck, I might owe you a lot of beers… Going to remove all the terminate() and see what happens! Thanks, Matteo On 21/mar/2013, at 19:19, Chuck Hill wrote: > > On 2013-03-21, at 10:54 AM, Altera WO Team wrote: > >> >> On 21/mar/2013, at 18:46, Chuck Hill wrote: >> >>> >>> On 2013-03-21

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread Chuck Hill
On 2013-03-21, at 10:54 AM, Altera WO Team wrote: > > On 21/mar/2013, at 18:46, Chuck Hill wrote: > >> >> On 2013-03-21, at 6:47 AM, Altera WO Team wrote: >> >>> I tried anyway and I can confirm. I'm not creating needless sessions. >>> >>> The app locks up anyway though… >> >> Did adding

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread Altera WO Team
On 21/mar/2013, at 15:25, George Domurot wrote: > I've seen this issue before, many years ago too. Where it bit me was where I > touched sessions and contexts in my component without testing to see if I in > fact had either to work with. > > You mentioned creating a context for your email. Ar

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread Altera WO Team
On 21/mar/2013, at 18:46, Chuck Hill wrote: > > On 2013-03-21, at 6:47 AM, Altera WO Team wrote: > >> I tried anyway and I can confirm. I'm not creating needless sessions. >> >> The app locks up anyway though… > > Did adding the finally blocks in your terminate() method help? Is anything

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread Chuck Hill
On 2013-03-21, at 6:47 AM, Altera WO Team wrote: > I tried anyway and I can confirm. I'm not creating needless sessions. > > The app locks up anyway though… Did adding the finally blocks in your terminate() method help? Is anything in your code calling session.terminate() directly? A logou

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread George Domurot
I've seen this issue before, many years ago too. Where it bit me was where I touched sessions and contexts in my component without testing to see if I in fact had either to work with. You mentioned creating a context for your email. Are you then touching context.session()? It will lazy load a s

Re: Immortal… ehm, frozen instances.

2013-03-21 Thread Altera WO Team
I tried anyway and I can confirm. I'm not creating needless sessions. The app locks up anyway though… I lost hope in finding the real reason, for now I would be happy to find a dirty workaround. I need those apps to bounce anyway. I already tried using ERTimeToKill and it's ineffective. The

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Chuck Hill
I think the problem is not that it is creating needless sessions, but that a checked-out session is not getting checked back in. Chuck On 2013-03-19, at 5:08 PM, Simon wrote: > if we have this kind of issue the first thing we do is log out session > creation and check the stack traces. stick

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Simon
if we have this kind of issue the first thing we do is log out session creation and check the stack traces. stick something like this in your session constructor then check the output: public Session() { super(); StringWriter stringWriter = new StringWriter(); PrintWriter printWriter = new Prin

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Altera WO Team
Wow, good hint! In theory I'm not touching a Session but sometimes, when something in an EO changes I trigger e-mail sending. I use ERMailDeliveryHTML using a component instantiated with a brand new wocontext using ERXWOContext.newContext() the component is not referencing a Session in any part

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Simon
...or you are touching the session object outside the RR loop. the classic gotcha is rendering a component in a thread that touches the session in some way e.g. delivering an email in a thread that uses a component to render it's content... been caught by that one s many times! On 19 March 20

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Altera WO Team
So that i catch an exception in ec().lock() basically… Will try! Thanks, Matteo On 19/mar/2013, at 19:24, Chuck Hill wrote: > Hi Matteo, > > I would use the same try...finally block in terminate() as well. And a > finally inside of that to unlock the ec. > > Chuck > > > On 2013-03-19, a

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Chuck Hill
Hi Matteo, I would use the same try...finally block in terminate() as well. And a finally inside of that to unlock the ec. Chuck On 2013-03-19, at 11:20 AM, Altera WO Team wrote: > Thanks Chuck, > > no OutOfMemory errors (at least in the log) > and no override of sleep() > > but we do ove

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Altera WO Team
Thanks Chuck, no OutOfMemory errors (at least in the log) and no override of sleep() but we do override terminate() @Override public void terminate() { log.info("TERMINATE " + sessionID()); ec().lock(); try {

Re: Immortal… ehm, frozen instances.

2013-03-19 Thread Chuck Hill
Hi Matteo, You have one or more Zombie (aka Immortal) Sessions, as shown by stack traces like this: "WorkerThread11" prio=10 tid=0x41848800 nid=0x1010 in Object.wait() [0x7f16f7cfa000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Meth

Immortal… ehm, frozen instances.

2013-03-19 Thread Altera WO Team
Hi all, I'm having a strange issue on a WO installation on EC2 (oracle jvm). Same strange application which had immortal sessions… Sometimes (quite rarely) a bounced application (put in refuse new sessions) never quits and it's not accessible from JavaMonitor. If I look at the logs i see: Mar