SOLVED: Immortal… ehm, frozen instances.

[Altera WO Team]

Hi all,

I finally solved the problem… well, not really but I found out the reason. 
Turns out that the whole thing is on amazon boxes and the DB machine (MySQL) 
was being backed up every hour… The backup takes all the cpu (not the backup, 
the bzip2 of the backup) and locks up MySQL for 2 minutes, if an instance tries 
to access the db in those 2 minutes that instance won't be able to die 
properly. 
Solution? I removed the hourly backup which was useless and scheduled a nightly 
backup with a reniced bzip2. No issues from that day!

Thanks everybody for the help.

Matteo

 
 On 21/mar/2013, at 19:19, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
 
 
 On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is 
 anything in your code calling session.terminate() directly?  A logout link?
 
 the finally blocks didn't help.
 
 Something is calling session.terminate() directly though, happens when a 
 something bad happens in a direct action.
 
 if (existingSession() != null)
 sess().terminate();
 
 It's called only if we can't parse the parameters and should never happen. 
 I'll put some logging in those cases. 
 
 Why should this be a problem?
 
 I thought it was fixed in 5.4.3, but I could be wrong.  In some past version 
 if that same URL can in in rapid succession, you would get deadlocks like 
 you are seeing.  The solution at the time was to not call terminate directly 
 and instead set the session timeout to a couple seconds.  That allowed both 
 requests to clear before terminating the session.
 
 
 Chuck
 
 
 I lost hope in finding the real reason, for now I would be happy to find 
 a dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up 
 the instances running become 9 and in the long run I'll have everything 
 locked up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you 
 are going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but 
 that a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that 
 you can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in 
 an EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand 
 new wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s many 
 times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at java.lang.Object.wait(Object.java:485)
at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
- locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
at 
 

Re: SOLVED: Immortal… ehm, frozen instances.

[Kieran Kelleher]

Another longer term option is to have a MySQL replication slave server that 
performs backup snapshots as needed (hourly, daily, weekly).

On Apr 3, 2013, at 1:32 PM, Altera WO Team webobje...@altera.it wrote:

 Hi all,
 
 I finally solved the problem… well, not really but I found out the reason. 
 Turns out that the whole thing is on amazon boxes and the DB machine (MySQL) 
 was being backed up every hour… The backup takes all the cpu (not the backup, 
 the bzip2 of the backup) and locks up MySQL for 2 minutes, if an instance 
 tries to access the db in those 2 minutes that instance won't be able to die 
 properly. 
 Solution? I removed the hourly backup which was useless and scheduled a 
 nightly backup with a reniced bzip2. No issues from that day!
 
 Thanks everybody for the help.
 
 Matteo
 
 
 On 21/mar/2013, at 19:19, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
 
 
 On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is 
 anything in your code calling session.terminate() directly?  A logout 
 link?
 
 the finally blocks didn't help.
 
 Something is calling session.terminate() directly though, happens when a 
 something bad happens in a direct action.
 
if (existingSession() != null)
sess().terminate();
 
 It's called only if we can't parse the parameters and should never happen. 
 I'll put some logging in those cases. 
 
 Why should this be a problem?
 
 I thought it was fixed in 5.4.3, but I could be wrong.  In some past 
 version if that same URL can in in rapid succession, you would get 
 deadlocks like you are seeing.  The solution at the time was to not call 
 terminate directly and instead set the session timeout to a couple seconds. 
  That allowed both requests to clear before terminating the session.
 
 
 Chuck
 
 
 I lost hope in finding the real reason, for now I would be happy to find 
 a dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 
 6 hours and it happens maybe 2 times a week, but if an instance locks up 
 the instances running become 9 and in the long run I'll have everything 
 locked up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you 
 are going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but 
 that a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that 
 you can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in 
 an EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand 
 new wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s 
 many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by 
 stack traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at java.lang.Object.wait(Object.java:485)
   at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
   - locked 0xd120b328 (a 
 

Re: SOLVED: Immortal… ehm, frozen instances.

[Steve Peery]

I strongly agree with the suggestion for a MySQL replication slave server. They 
are not that hard to set up and then you can dump a backup as often as you like 
with out effecting the main server and you always have an up to the minute copy 
incase something happens. I had a problem a couple of months ago and going to a 
backup up that was just a couple of hours old was a major loss.

Steve

On Apr 3, 2013, at 2:16 PM, Kieran Kelleher kelleh...@gmail.com wrote:

 Another longer term option is to have a MySQL replication slave server that 
 performs backup snapshots as needed (hourly, daily, weekly).
 
 On Apr 3, 2013, at 1:32 PM, Altera WO Team webobje...@altera.it wrote:
 
 Hi all,
 
 I finally solved the problem… well, not really but I found out the reason. 
 Turns out that the whole thing is on amazon boxes and the DB machine (MySQL) 
 was being backed up every hour… The backup takes all the cpu (not the 
 backup, the bzip2 of the backup) and locks up MySQL for 2 minutes, if an 
 instance tries to access the db in those 2 minutes that instance won't be 
 able to die properly. 
 Solution? I removed the hourly backup which was useless and scheduled a 
 nightly backup with a reniced bzip2. No issues from that day!
 
 Thanks everybody for the help.
 
 Matteo
 
 
 On 21/mar/2013, at 19:19, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
 
 
 On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is 
 anything in your code calling session.terminate() directly?  A logout 
 link?
 
 the finally blocks didn't help.
 
 Something is calling session.terminate() directly though, happens when a 
 something bad happens in a direct action.
 
   if (existingSession() != null)
   sess().terminate();
 
 It's called only if we can't parse the parameters and should never 
 happen. I'll put some logging in those cases. 
 
 Why should this be a problem?
 
 I thought it was fixed in 5.4.3, but I could be wrong.  In some past 
 version if that same URL can in in rapid succession, you would get 
 deadlocks like you are seeing.  The solution at the time was to not call 
 terminate directly and instead set the session timeout to a couple 
 seconds.  That allowed both requests to clear before terminating the 
 session.
 
 
 Chuck
 
 
 I lost hope in finding the real reason, for now I would be happy to 
 find a dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 
 6 hours and it happens maybe 2 times a week, but if an instance locks 
 up the instances running become 9 and in the long run I'll have 
 everything locked up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you 
 are going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but 
 that a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out 
 session creation and check the stack traces. stick something like 
 this in your session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that 
 you can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in 
 an EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand 
 new wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s 
 many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by 
 stack traces like this:
 WorkerThread11 prio=10 

Re: SOLVED: Immortal… ehm, frozen instances.

[Chuck Hill]

Sounds like it was timing out and throwing an exception that was improperly 
handled.  Seems odd that it would cause zombie sessions even with the 
try...finally in sleep() and terminate()


On 2013-04-03, at 10:32 AM, Altera WO Team wrote:

 Hi all,
 
 I finally solved the problem… well, not really but I found out the reason. 
 Turns out that the whole thing is on amazon boxes and the DB machine (MySQL) 
 was being backed up every hour… The backup takes all the cpu (not the backup, 
 the bzip2 of the backup) and locks up MySQL for 2 minutes, if an instance 
 tries to access the db in those 2 minutes that instance won't be able to die 
 properly. 
 Solution? I removed the hourly backup which was useless and scheduled a 
 nightly backup with a reniced bzip2. No issues from that day!
 
 Thanks everybody for the help.
 
 Matteo
 
 
 On 21/mar/2013, at 19:19, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
 
 
 On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is 
 anything in your code calling session.terminate() directly?  A logout 
 link?
 
 the finally blocks didn't help.
 
 Something is calling session.terminate() directly though, happens when a 
 something bad happens in a direct action.
 
if (existingSession() != null)
sess().terminate();
 
 It's called only if we can't parse the parameters and should never happen. 
 I'll put some logging in those cases. 
 
 Why should this be a problem?
 
 I thought it was fixed in 5.4.3, but I could be wrong.  In some past 
 version if that same URL can in in rapid succession, you would get 
 deadlocks like you are seeing.  The solution at the time was to not call 
 terminate directly and instead set the session timeout to a couple seconds. 
  That allowed both requests to clear before terminating the session.
 
 
 Chuck
 
 
 I lost hope in finding the real reason, for now I would be happy to find 
 a dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 
 6 hours and it happens maybe 2 times a week, but if an instance locks up 
 the instances running become 9 and in the long run I'll have everything 
 locked up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you 
 are going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but 
 that a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that 
 you can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in 
 an EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand 
 new wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s 
 many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by 
 stack traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at java.lang.Object.wait(Object.java:485)
   at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
   - locked 0xd120b328 (a 
 

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

I tried anyway and I can confirm. I'm not creating needless sessions. 

The app locks up anyway though… 

I lost hope in finding the real reason, for now I would be happy to find a 
dirty workaround. I need those apps to bounce anyway.

I already tried using ERTimeToKill and it's ineffective.

The lockups are quite rare: I have 10 instances running, bouncing every 6 hours 
and it happens maybe 2 times a week, but if an instance locks up the instances 
running become 9 and in the long run I'll have everything locked up.

Any suggestions?


Matteo


On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:

 I think the problem is not that it is creating needless sessions, but that a 
 checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you 
 can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an EO 
 changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the classic 
 gotcha is rendering a component in a thread that touches the session in 
 some way e.g. delivering an email in a thread that uses a component to 
 render it's content... been caught by that one s many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
 [0x7f16f7cfa000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at java.lang.Object.wait(Object.java:485)
at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
- locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
try {
// Your code here!
} finally {
super.sleep();
}
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions…
 
 Sometimes (quite rarely) a bounced application (put in refuse new 
 sessions) never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: refusing new clients and below min active 
 session threshold
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: about to terminate...
 
 The only thing left is to kill the instance… Which is not nice.
 
 I'm not overriding the terminate() method in Application.
 
 I am attaching a stack trace if it helps.
 B2Cjstack.txt
 
 Thanks,
 
 
 
 Matteo Centro
 ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
 Help/Unsubscribe/Update your Subscription:
 https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
 This email sent to ch...@global-village.net
 
 --
 Chuck Hill
 

Re: Immortal… ehm, frozen instances.

[George Domurot]

I've seen this issue before, many years ago too. Where it bit me was where I 
touched sessions and contexts in my component without testing to see if I in 
fact had either to work with.

You mentioned creating a context for your email. Are you then touching 
context.session()? It will lazy load a session. As will component.session().

Search across your project for session() and context(). There's typically a 
culprit out there.

Here's a helper we use in our components for accessing a current session, but 
only if one already exists. It may be overkill, but at the time it helped 
remove our zombie sessions.

public WOSession currentSession()
{
if (hasSession())
{
return session();
}
else if (context() != null)
{
if (context().hasSession())
{
return context().session();
}
else
{
return null;
}
}
else
{
return null;
}
}

-George
On Mar 21, 2013, at 6:47 AM, Altera WO Team webobje...@altera.it wrote:

 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 I lost hope in finding the real reason, for now I would be happy to find a 
 dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up the 
 instances running become 9 and in the long run I'll have everything locked up.
 
 Any suggestions?
 
 
 Matteo
 
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but that a 
 checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you 
 can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an EO 
 changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the classic 
 gotcha is rendering a component in a thread that touches the session in 
 some way e.g. delivering an email in a thread that uses a component to 
 render it's content... been caught by that one s many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
  java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at java.lang.Object.wait(Object.java:485)
   at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
   - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
   at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
   at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
   at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
   at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
   at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
   try {
   // Your code here!
   } finally {
   super.sleep();
   }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 

Re: Immortal… ehm, frozen instances.

[Chuck Hill]

On 2013-03-21, at 6:47 AM, Altera WO Team wrote:

 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 

Did adding the finally blocks in your terminate() method help?  Is anything in 
your code calling session.terminate() directly?  A logout link?


 I lost hope in finding the real reason, for now I would be happy to find a 
 dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up the 
 instances running become 9 and in the long run I'll have everything locked up.
 
 Any suggestions?

I don't recall how ERTimeToKill works, but with those stuck threads you are 
going to need to do

Runtime.getRuntime().exit(1);

and if THAT does not kill it, this should:

Runtime.getRuntime().halt(1);


Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but that a 
 checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you 
 can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an EO 
 changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the classic 
 gotcha is rendering a component in a thread that touches the session in 
 some way e.g. delivering an email in a thread that uses a component to 
 render it's content... been caught by that one s many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
  java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at java.lang.Object.wait(Object.java:485)
   at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
   - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
   at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
   at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
   at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
   at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
   at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
   try {
   // Your code here!
   } finally {
   super.sleep();
   }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions…
 
 Sometimes (quite rarely) a bounced application (put in refuse new 
 sessions) never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: refusing new clients and below min active 
 session threshold
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: about to terminate...
 
 The only thing left is to kill the instance… Which is not nice.
 
 I'm not overriding the terminate() method in Application.
 
 I am attaching a stack trace if it helps.
 B2Cjstack.txt
 
 

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:

 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is anything 
 in your code calling session.terminate() directly?  A logout link?

the finally blocks didn't help.

Something is calling session.terminate() directly though, happens when a 
something bad happens in a direct action.

if (existingSession() != null)
sess().terminate();

It's called only if we can't parse the parameters and should never happen. I'll 
put some logging in those cases. 

Why should this be a problem?


 
 
 I lost hope in finding the real reason, for now I would be happy to find a 
 dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up the 
 instances running become 9 and in the long run I'll have everything locked 
 up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you are 
 going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but that 
 a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you 
 can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an 
 EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s many 
 times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at java.lang.Object.wait(Object.java:485)
  at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
  - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
  at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
  at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
  at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
  at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
  at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
  try {
  // Your code here!
  } finally {
  super.sleep();
  }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions…
 
 Sometimes (quite rarely) a bounced application (put in refuse new 
 sessions) never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

On 21/mar/2013, at 15:25, George Domurot masterm...@knuckleheads.net wrote:

 I've seen this issue before, many years ago too. Where it bit me was where I 
 touched sessions and contexts in my component without testing to see if I in 
 fact had either to work with.
 
 You mentioned creating a context for your email. Are you then touching 
 context.session()? It will lazy load a session. As will component.session().

No, we never touch session() in the e-mail component. 
We create a context because we couldn't figure out how to instantiate a 
Component from outside the R-R loop, the mail component is only used to compose 
the e-mail….

 
 Search across your project for session() and context(). There's typically 
 a culprit out there.
 
 Here's a helper we use in our components for accessing a current session, but 
 only if one already exists. It may be overkill, but at the time it helped 
 remove our zombie sessions.
 
public WOSession currentSession()
{
if (hasSession())
{
return session();
}
else if (context() != null)
{
if (context().hasSession())
{
return context().session();
}
else
{
return null;
}
}
else
{
return null;
}
}

Thanks for that,


Matteo


 
 -George
 On Mar 21, 2013, at 6:47 AM, Altera WO Team webobje...@altera.it wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 I lost hope in finding the real reason, for now I would be happy to find a 
 dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up the 
 instances running become 9 and in the long run I'll have everything locked 
 up.
 
 Any suggestions?
 
 
 Matteo
 
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but that 
 a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you 
 can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an 
 EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s many 
 times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at java.lang.Object.wait(Object.java:485)
  at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
  - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
  at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
  at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
  at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
  at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
  at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state

Re: Immortal… ehm, frozen instances.

[Chuck Hill]

On 2013-03-21, at 10:54 AM, Altera WO Team wrote:

 
 On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is anything 
 in your code calling session.terminate() directly?  A logout link?
 
 the finally blocks didn't help.
 
 Something is calling session.terminate() directly though, happens when a 
 something bad happens in a direct action.
 
   if (existingSession() != null)
   sess().terminate();
 
 It's called only if we can't parse the parameters and should never happen. 
 I'll put some logging in those cases. 
 
 Why should this be a problem?

I thought it was fixed in 5.4.3, but I could be wrong.  In some past version if 
that same URL can in in rapid succession, you would get deadlocks like you are 
seeing.  The solution at the time was to not call terminate directly and 
instead set the session timeout to a couple seconds.  That allowed both 
requests to clear before terminating the session.


Chuck


 I lost hope in finding the real reason, for now I would be happy to find a 
 dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up the 
 instances running become 9 and in the long run I'll have everything locked 
 up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you are 
 going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but that 
 a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you 
 can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an 
 EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s many 
 times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at java.lang.Object.wait(Object.java:485)
  at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
  - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
  at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
  at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
  at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
  at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
  at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
  try {
  // Your code here!
  

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

Chuck, I might owe you a lot of beers… Going to remove all the terminate() and 
see what happens!

Thanks,


Matteo

On 21/mar/2013, at 19:19, Chuck Hill ch...@global-village.net wrote:

 
 On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
 
 
 On 21/mar/2013, at 18:46, Chuck Hill ch...@global-village.net wrote:
 
 
 On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
 
 I tried anyway and I can confirm. I'm not creating needless sessions. 
 
 The app locks up anyway though… 
 
 Did adding the finally blocks in your terminate() method help?  Is anything 
 in your code calling session.terminate() directly?  A logout link?
 
 the finally blocks didn't help.
 
 Something is calling session.terminate() directly though, happens when a 
 something bad happens in a direct action.
 
  if (existingSession() != null)
  sess().terminate();
 
 It's called only if we can't parse the parameters and should never happen. 
 I'll put some logging in those cases. 
 
 Why should this be a problem?
 
 I thought it was fixed in 5.4.3, but I could be wrong.  In some past version 
 if that same URL can in in rapid succession, you would get deadlocks like you 
 are seeing.  The solution at the time was to not call terminate directly and 
 instead set the session timeout to a couple seconds.  That allowed both 
 requests to clear before terminating the session.
 
 
 Chuck
 
 
 I lost hope in finding the real reason, for now I would be happy to find a 
 dirty workaround. I need those apps to bounce anyway.
 
 I already tried using ERTimeToKill and it's ineffective.
 
 The lockups are quite rare: I have 10 instances running, bouncing every 6 
 hours and it happens maybe 2 times a week, but if an instance locks up the 
 instances running become 9 and in the long run I'll have everything locked 
 up.
 
 Any suggestions?
 
 I don't recall how ERTimeToKill works, but with those stuck threads you are 
 going to need to do
 
 Runtime.getRuntime().exit(1);
 
 and if THAT does not kill it, this should:
 
 Runtime.getRuntime().halt(1);
 
 
 Chuck
 
 On 20/mar/2013, at 01:16, Chuck Hill ch...@global-village.net wrote:
 
 I think the problem is not that it is creating needless sessions, but 
 that a checked-out session is not getting checked back in.
 
 Chuck
 
 
 On 2013-03-19, at 5:08 PM, Simon wrote:
 
 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that 
 you can get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an 
 EO changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the 
 classic gotcha is rendering a component in a thread that touches the 
 session in some way e.g. delivering an email in a thread that uses a 
 component to render it's content... been caught by that one s many 
 times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in 
 Object.wait() [0x7f16f7cfa000]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at java.lang.Object.wait(Object.java:485)
 at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
 - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
 at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
 at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
 at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
 at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
 at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is 

Immortal… ehm, frozen instances.

[Altera WO Team]

Hi all,

I'm having a strange issue on a WO installation on EC2 (oracle jvm).
Same strange application which had immortal sessions… 

Sometimes (quite rarely) a bounced application (put in refuse new sessions) 
never quits and it's not accessible from JavaMonitor.
If I look at the logs i see:

Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
com.tla.calendar.Application: refusing new clients and below min active 
session threshold
Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
com.tla.calendar.Application: about to terminate...

The only thing left is to kill the instance… Which is not nice.

I'm not overriding the terminate() method in Application.

I am attaching a stack trace if it helps.
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.7-b02 mixed mode):

Attach Listener daemon prio=10 tid=0x41543800 nid=0x17fd runnable 
[0x]
   java.lang.Thread.State: RUNNABLE

Thread-21 prio=10 tid=0x7f16f8784000 nid=0x1151 waiting on condition 
[0x7f16f75f4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at 
com.webobjects.appserver._private.WOClassicAdaptor.unregisterForEvents(WOClassicAdaptor.java:290)
- locked 0xd043a8e0 (a 
com.webobjects.appserver._private.WODefaultAdaptor)
at com.webobjects.appserver.WOApplication$1.run(WOApplication.java:1258)
at java.lang.Thread.run(Thread.java:662)

Timer-0 daemon prio=10 tid=0x41e93800 nid=0x1150 in Object.wait() 
[0x7f16f77f6000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xc5c898b0 (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Timer.java:483)
- locked 0xc5c898b0 (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)

ERMailSender prio=10 tid=0x41b41000 nid=0x1032 in Object.wait() 
[0x7f16f76f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xd2ea49d0 (a er.javamail.ERQueue)
at er.javamail.ERMailSender.run(ERMailSender.java:356)
- locked 0xd2ea49d0 (a er.javamail.ERQueue)
at java.lang.Thread.run(Thread.java:662)

WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
[0x7f16f7cfa000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xd120b328 (a 
com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at java.lang.Object.wait(Object.java:485)
at 
com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
- locked 0xd120b328 (a 
com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at 
com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
at 
er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
at 
er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
at er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
at com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
com.webobjects.appserver.WODirectAction.performActionNamed(WODirectAction.java:144)
at 
er.extensions.appserver.ERXDirectAction.performActionNamed(ERXDirectAction.java:418)
at 
com.webobjects.appserver._private.WOActionRequestHandler._handleRequest(WOActionRequestHandler.java:259)
at 
com.webobjects.appserver._private.WOActionRequestHandler.handleRequest(WOActionRequestHandler.java:161)
at 
er.extensions.appserver.ERXDirectActionRequestHandler.handleRequest(ERXDirectActionRequestHandler.java:128)
at 
com.webobjects.appserver.WOApplication.dispatchRequest(WOApplication.java:1687)
at 
er.extensions.appserver.ERXApplication.dispatchRequestImmediately(ERXApplication.java:2109)
at 
er.extensions.appserver.ERXApplication.dispatchRequest(ERXApplication.java:2074)
at 
com.webobjects.appserver._private.WOWorkerThread.runOnce(WOWorkerThread.java:144)
at 
com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:226)
at java.lang.Thread.run(Thread.java:662)

WorkerThread9 prio=10 tid=0x41c22000 nid=0x100e in Object.wait() 
[0x7f16f7efc000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 

Re: Immortal… ehm, frozen instances.

[Chuck Hill]

Hi Matteo,

You have one or more Zombie (aka Immortal) Sessions, as shown by stack traces 
like this:
WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
[0x7f16f7cfa000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xd120b328 (a 
com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at java.lang.Object.wait(Object.java:485)
at 
com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
- locked 0xd120b328 (a 
com.webobjects.appserver.WOSessionStore$TimeoutEntry)
at 
com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
at 
er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
at 
er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
at er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
at com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)


This likely has one of two causes:
1. The application is getting OutOfMemory errors, which can leave the session 
store in an insane state
2. The app is throwing an exception from sleep() in Session.  If you overrride 
sleep() it should use a try...finally block

public void sleep() {
try {
// Your code here!
} finally {
super.sleep();
}
}


Chuck



On 2013-03-19, at 9:38 AM, Altera WO Team wrote:

 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions… 
 
 Sometimes (quite rarely) a bounced application (put in refuse new sessions) 
 never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: refusing new clients and below min active 
 session threshold
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: about to terminate...
 
 The only thing left is to kill the instance… Which is not nice.
 
 I'm not overriding the terminate() method in Application.
 
 I am attaching a stack trace if it helps.
 B2Cjstack.txt
 
 Thanks,
 
 
 
 Matteo Centro
 ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
 Help/Unsubscribe/Update your Subscription:
 https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
 This email sent to ch...@global-village.net

-- 
Chuck Hill 
Executive Managing Partner, VP Development and Technical Services

Practical WebObjects - for developers who want to increase their overall 
knowledge of WebObjects or who are trying to solve specific problems.
http://www.global-village.net/gvc/practical_webobjects

Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing 
Companies in B.C! 
Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of 
Canada’s Fastest-Growing Companies by PROFIT Magazine!












 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

Thanks Chuck,

no OutOfMemory errors (at least in the log) 
and no override of sleep()

but we do override terminate()

@Override
public void terminate() {
log.info(TERMINATE  + sessionID());
 
ec().lock();
try {
if (cart != null  shouldDeleteCart) {
log.debug(Deleting cart  + sessionID());
int saveCounter = 0;
do {
if (cart.cleanAll()) {
ec().deleteObject(cart);
}
saveCounter++;
} while (!save()  saveCounter = 
Application.SAVE_ATTEMPTS);
}
} catch (Exception e) {
e.printStackTrace();
}
ec().unlock();

log.info(Terminating session with ID: + sessionID());
super.terminate();
}


ec() simply returns defaultEditingContext() (it's an ERXSession subclass)

and save() is this:

public boolean save() {
boolean hasSaved = false;
EOEditingContext editingContext = ec();
try {
editingContext.saveChanges();
hasSaved = true;
//Thrown for each eo that fails to save.
} catch (EOGeneralAdaptorException saveException) {
log.error(/*** OPTIMISTIC LOCKING FAILURE ***/ 
+sessionID());
log.error(saveException.getMessage());
editingContext.revert();
}
return hasSaved;
}

This method makes me nervous, I don't trust it very much…

Also I'm not sure about the explicit lock and unlock in the terminate method 
but since it's not in a R-R loop it might make sense to explicitly lock the ec, 
or not?


Matteo

On 19/mar/2013, at 18:51, Chuck Hill ch...@global-village.net wrote:

 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack traces 
 like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
 [0x7f16f7cfa000]
   java.lang.Thread.State: WAITING (on object monitor)
   at java.lang.Object.wait(Native Method)
   - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at java.lang.Object.wait(Object.java:485)
   at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
   - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
   at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
   at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
   at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
   at er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
   at com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
   at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the session 
 store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
   try {
   // Your code here!
   } finally {
   super.sleep();
   }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions… 
 
 Sometimes (quite rarely) a bounced application (put in refuse new sessions) 
 never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: refusing new clients and below min active 
 session threshold
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: about to terminate...
 
 The only thing left is to kill the instance… Which is not nice.
 
 I'm not overriding the terminate() method in Application.
 
 I am attaching a stack trace if it helps.
 B2Cjstack.txt
 
 Thanks,
 
 
 
 Matteo Centro
 ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
 Help/Unsubscribe/Update your Subscription:
 https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
 This email sent to ch...@global-village.net
 
 -- 
 Chuck 

Re: Immortal… ehm, frozen instances.

[Chuck Hill]

Hi Matteo,

I would use the same try...finally block in terminate() as well.  And a finally 
inside of that to unlock the ec.

Chuck


On 2013-03-19, at 11:20 AM, Altera WO Team wrote:

 Thanks Chuck,
 
 no OutOfMemory errors (at least in the log) 
 and no override of sleep()
 
 but we do override terminate()
 
   @Override
   public void terminate() {
try {
   log.info(TERMINATE  + sessionID());

   ec().lock();
   try {
   if (cart != null  shouldDeleteCart) {
   log.debug(Deleting cart  + sessionID());
   int saveCounter = 0;
   do {
   if (cart.cleanAll()) {
   ec().deleteObject(cart);
   }
   saveCounter++;
   } while (!save()  saveCounter = 
 Application.SAVE_ATTEMPTS);
   }
   } catch (Exception e) {
   e.printStackTrace();
   }
finally {
   ec().unlock();
}
   
   log.info(Terminating session with ID: + sessionID());
} finally {
   super.terminate();
}
   }
 
 
 ec() simply returns defaultEditingContext() (it's an ERXSession subclass)
 
 and save() is this:
 
   public boolean save() {
   boolean hasSaved = false;
   EOEditingContext editingContext = ec();
   try {
   editingContext.saveChanges();
   hasSaved = true;
   //Thrown for each eo that fails to save.
   } catch (EOGeneralAdaptorException saveException) {
   log.error(/*** OPTIMISTIC LOCKING FAILURE ***/ 
 +sessionID());
   log.error(saveException.getMessage());
   editingContext.revert();
   }
   return hasSaved;
   }
 
 This method makes me nervous, I don't trust it very much…
 
 Also I'm not sure about the explicit lock and unlock in the terminate method 
 but since it's not in a R-R loop it might make sense to explicitly lock the 
 ec, or not?
 
 
 Matteo
 
 On 19/mar/2013, at 18:51, Chuck Hill ch...@global-village.net wrote:
 
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
 [0x7f16f7cfa000]
   java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at java.lang.Object.wait(Object.java:485)
  at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
  - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
  at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
  at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
  at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
  at er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
  at com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
  at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
  try {
  // Your code here!
  } finally {
  super.sleep();
  }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions… 
 
 Sometimes (quite rarely) a bounced application (put in refuse new sessions) 
 never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: refusing new clients and below min active 
 session threshold
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: about to terminate...
 
 The only thing left is to kill the instance… Which is not nice.
 
 I'm not overriding the terminate() method in Application.
 
 I am attaching a stack trace if it helps.
 B2Cjstack.txt
 
 Thanks,
 
 
 
 Matteo Centro
 ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

So that i catch an exception in ec().lock() basically… Will try!

Thanks,


Matteo

On 19/mar/2013, at 19:24, Chuck Hill ch...@global-village.net wrote:

 Hi Matteo,
 
 I would use the same try...finally block in terminate() as well.  And a 
 finally inside of that to unlock the ec.
 
 Chuck
 
 
 On 2013-03-19, at 11:20 AM, Altera WO Team wrote:
 
 Thanks Chuck,
 
 no OutOfMemory errors (at least in the log) 
 and no override of sleep()
 
 but we do override terminate()
 
  @Override
  public void terminate() {
 try {
  log.info(TERMINATE  + sessionID());
   
  ec().lock();
  try {
  if (cart != null  shouldDeleteCart) {
  log.debug(Deleting cart  + sessionID());
  int saveCounter = 0;
  do {
  if (cart.cleanAll()) {
  ec().deleteObject(cart);
  }
  saveCounter++;
  } while (!save()  saveCounter = 
 Application.SAVE_ATTEMPTS);
  }
  } catch (Exception e) {
  e.printStackTrace();
  }
 finally {
  ec().unlock();
 }
  
  log.info(Terminating session with ID: + sessionID());
 } finally {
  super.terminate();
 }
  }
 
 
 ec() simply returns defaultEditingContext() (it's an ERXSession subclass)
 
 and save() is this:
 
  public boolean save() {
  boolean hasSaved = false;
  EOEditingContext editingContext = ec();
  try {
  editingContext.saveChanges();
  hasSaved = true;
  //Thrown for each eo that fails to save.
  } catch (EOGeneralAdaptorException saveException) {
  log.error(/*** OPTIMISTIC LOCKING FAILURE ***/ 
 +sessionID());
  log.error(saveException.getMessage());
  editingContext.revert();
  }
  return hasSaved;
  }
 
 This method makes me nervous, I don't trust it very much…
 
 Also I'm not sure about the explicit lock and unlock in the terminate method 
 but since it's not in a R-R loop it might make sense to explicitly lock the 
 ec, or not?
 
 
 Matteo
 
 On 19/mar/2013, at 18:51, Chuck Hill ch...@global-village.net wrote:
 
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
 [0x7f16f7cfa000]
  java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at java.lang.Object.wait(Object.java:485)
 at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
 - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
 at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
 at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
 at er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
 at com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
 at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
 try {
 // Your code here!
 } finally {
 super.sleep();
 }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
 Hi all,
 
 I'm having a strange issue on a WO installation on EC2 (oracle jvm).
 Same strange application which had immortal sessions… 
 
 Sometimes (quite rarely) a bounced application (put in refuse new 
 sessions) never quits and it's not accessible from JavaMonitor.
 If I look at the logs i see:
 
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: refusing new clients and below min active 
 session threshold
 Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
 com.tla.calendar.Application: about to terminate...
 
 The only thing left is to kill the instance… Which is not nice.
 
 I'm not overriding the terminate() method in Application.
 
 I am attaching a stack trace if it helps.
 B2Cjstack.txt
 
 Thanks,
 
 
 
 Matteo Centro
 ___
 Do not post 

Re: Immortal… ehm, frozen instances.

[Simon]

...or you are touching the session object outside the RR loop. the classic
gotcha is rendering a component in a thread that touches the session in
some way e.g. delivering an email in a thread that uses a component to
render it's content... been caught by that one s many times!


On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:

 Hi Matteo,

 You have one or more Zombie (aka Immortal) Sessions, as shown by stack
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in
 Object.wait() [0x7f16f7cfa000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xd120b328 (a
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at java.lang.Object.wait(Object.java:485)
 at
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
 - locked 0xd120b328 (a
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
 at
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
 at
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
 at
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
 at
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
 at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)


 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you
 overrride sleep() it should use a try...finally block

 public void sleep() {
 try {
 // Your code here!
 } finally {
 super.sleep();
 }
 }


 Chuck



 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:

  Hi all,
 
  I'm having a strange issue on a WO installation on EC2 (oracle jvm).
  Same strange application which had immortal sessions…
 
  Sometimes (quite rarely) a bounced application (put in refuse new
 sessions) never quits and it's not accessible from JavaMonitor.
  If I look at the logs i see:
 
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  -
 com.tla.calendar.Application: refusing new clients and below min active
 session threshold
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  -
 com.tla.calendar.Application: about to terminate...
 
  The only thing left is to kill the instance… Which is not nice.
 
  I'm not overriding the terminate() method in Application.
 
  I am attaching a stack trace if it helps.
  B2Cjstack.txt
 
  Thanks,
 
 
 
  Matteo Centro
  ___
  Do not post admin requests to the list. They will be ignored.
  Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
  Help/Unsubscribe/Update your Subscription:
 
 https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
  This email sent to ch...@global-village.net

 --
 Chuck Hill
 Executive Managing Partner, VP Development and Technical Services

 Practical WebObjects - for developers who want to increase their overall
 knowledge of WebObjects or who are trying to solve specific problems.
 http://www.global-village.net/gvc/practical_webobjects

 Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
 Growing Companies in B.C!
 Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of
 Canada’s Fastest-Growing Companies by PROFIT Magazine!












  ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
 Help/Unsubscribe/Update your Subscription:

 https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk

 This email sent to si...@potwells.co.uk
 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Immortal… ehm, frozen instances.

[Altera WO Team]

Wow, good hint!

In theory I'm not touching a Session but sometimes, when something in an EO 
changes I trigger e-mail sending. 
I use ERMailDeliveryHTML using a component instantiated with a brand new 
wocontext using ERXWOContext.newContext()
the component is not referencing a Session in any part. 

Could it be the cause?

Matteo


On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:

 ...or you are touching the session object outside the RR loop. the classic 
 gotcha is rendering a component in a thread that touches the session in some 
 way e.g. delivering an email in a thread that uses a component to render it's 
 content... been caught by that one s many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack traces 
 like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
 [0x7f16f7cfa000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at java.lang.Object.wait(Object.java:485)
 at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
 - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
 at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
 at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
 at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
 at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
 at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the session 
 store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
 try {
 // Your code here!
 } finally {
 super.sleep();
 }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
  Hi all,
 
  I'm having a strange issue on a WO installation on EC2 (oracle jvm).
  Same strange application which had immortal sessions…
 
  Sometimes (quite rarely) a bounced application (put in refuse new sessions) 
  never quits and it's not accessible from JavaMonitor.
  If I look at the logs i see:
 
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
  com.tla.calendar.Application: refusing new clients and below min active 
  session threshold
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
  com.tla.calendar.Application: about to terminate...
 
  The only thing left is to kill the instance… Which is not nice.
 
  I'm not overriding the terminate() method in Application.
 
  I am attaching a stack trace if it helps.
  B2Cjstack.txt
 
  Thanks,
 
 
 
  Matteo Centro
  ___
  Do not post admin requests to the list. They will be ignored.
  Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
  Help/Unsubscribe/Update your Subscription:
  https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
  This email sent to ch...@global-village.net
 
 --
 Chuck Hill
 Executive Managing Partner, VP Development and Technical Services
 
 Practical WebObjects - for developers who want to increase their overall 
 knowledge of WebObjects or who are trying to solve specific problems.
 http://www.global-village.net/gvc/practical_webobjects
 
 Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest Growing 
 Companies in B.C!
 Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of 
 Canada’s Fastest-Growing Companies by PROFIT Magazine!
 
 
 
 
 
 
 
 
 
 
 
 
  ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
 Help/Unsubscribe/Update your Subscription:
 https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk
 
 This email sent to si...@potwells.co.uk
 

 ___
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com

This email sent to arch...@mail-archive.com

Re: Immortal… ehm, frozen instances.

[Simon]

if we have this kind of issue the first thing we do is log out session
creation and check the stack traces. stick something like this in your
session constructor then check the output:

public Session() {
super();

StringWriter stringWriter = new StringWriter();

 PrintWriter printWriter = new PrintWriter(stringWriter);

 (new Throwable()).printStackTrace(printWriter);

 String trace = stringWriter.toString();  log this somewhere that you
can get easy access to

}




On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:

 Wow, good hint!

 In theory I'm not touching a Session but sometimes, when something in an
 EO changes I trigger e-mail sending.
 I use ERMailDeliveryHTML using a component instantiated with a brand new
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part.

 Could it be the cause?

 Matteo


 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:

 ...or you are touching the session object outside the RR loop. the classic
 gotcha is rendering a component in a thread that touches the session in
 some way e.g. delivering an email in a thread that uses a component to
 render it's content... been caught by that one s many times!


 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:

 Hi Matteo,

 You have one or more Zombie (aka Immortal) Sessions, as shown by stack
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in
 Object.wait() [0x7f16f7cfa000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xd120b328 (a
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at java.lang.Object.wait(Object.java:485)
 at
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
 - locked 0xd120b328 (a
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
 at
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
 at
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
 at
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
 at
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
 at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)


 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you
 overrride sleep() it should use a try...finally block

 public void sleep() {
 try {
 // Your code here!
 } finally {
 super.sleep();
 }
 }


 Chuck



 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:

  Hi all,
 
  I'm having a strange issue on a WO installation on EC2 (oracle jvm).
  Same strange application which had immortal sessions…
 
  Sometimes (quite rarely) a bounced application (put in refuse new
 sessions) never quits and it's not accessible from JavaMonitor.
  If I look at the logs i see:
 
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  -
 com.tla.calendar.Application: refusing new clients and below min active
 session threshold
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  -
 com.tla.calendar.Application: about to terminate...
 
  The only thing left is to kill the instance… Which is not nice.
 
  I'm not overriding the terminate() method in Application.
 
  I am attaching a stack trace if it helps.
  B2Cjstack.txt
 
  Thanks,
 
 
 
  Matteo Centro
  ___
  Do not post admin requests to the list. They will be ignored.
  Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
  Help/Unsubscribe/Update your Subscription:
 
 https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
  This email sent to ch...@global-village.net

 --
 Chuck Hill
 Executive Managing Partner, VP Development and Technical Services

 Practical WebObjects - for developers who want to increase their overall
 knowledge of WebObjects or who are trying to solve specific problems.
 http://www.global-village.net/gvc/practical_webobjects

 Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
 Growing Companies in B.C!
 Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of
 Canada’s Fastest-Growing Companies by PROFIT Magazine!












  ___
 Do not post admin requests to the list. They will be ignored.
 Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
 Help/Unsubscribe/Update your Subscription:

 https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk

 This email sent to 

Re: Immortal… ehm, frozen instances.

[Chuck Hill]

I think the problem is not that it is creating needless sessions, but that a 
checked-out session is not getting checked back in.

Chuck


On 2013-03-19, at 5:08 PM, Simon wrote:

 if we have this kind of issue the first thing we do is log out session 
 creation and check the stack traces. stick something like this in your 
 session constructor then check the output:
 
 public Session() {
 super();
 StringWriter stringWriter = new StringWriter();
 
 PrintWriter printWriter = new PrintWriter(stringWriter);
 
 (new Throwable()).printStackTrace(printWriter);
 
 String trace = stringWriter.toString();  log this somewhere that you can 
 get easy access to
 
 }
 
 
 
 
 
 
 On 19 March 2013 23:15, Altera WO Team webobje...@altera.it wrote:
 Wow, good hint!
 
 In theory I'm not touching a Session but sometimes, when something in an EO 
 changes I trigger e-mail sending. 
 I use ERMailDeliveryHTML using a component instantiated with a brand new 
 wocontext using ERXWOContext.newContext()
 the component is not referencing a Session in any part. 
 
 Could it be the cause?
 
 Matteo
 
 
 On 19/mar/2013, at 21:24, Simon si...@potwells.co.uk wrote:
 
 ...or you are touching the session object outside the RR loop. the classic 
 gotcha is rendering a component in a thread that touches the session in some 
 way e.g. delivering an email in a thread that uses a component to render 
 it's content... been caught by that one s many times!
 
 
 On 19 March 2013 17:51, Chuck Hill ch...@global-village.net wrote:
 Hi Matteo,
 
 You have one or more Zombie (aka Immortal) Sessions, as shown by stack 
 traces like this:
 WorkerThread11 prio=10 tid=0x41848800 nid=0x1010 in Object.wait() 
 [0x7f16f7cfa000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at java.lang.Object.wait(Object.java:485)
 at 
 com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
 - locked 0xd120b328 (a 
 com.webobjects.appserver.WOSessionStore$TimeoutEntry)
 at 
 com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
 at 
 er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
 at 
 er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
 at 
 er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
 at 
 com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
 at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
 
 
 This likely has one of two causes:
 1. The application is getting OutOfMemory errors, which can leave the 
 session store in an insane state
 2. The app is throwing an exception from sleep() in Session.  If you 
 overrride sleep() it should use a try...finally block
 
 public void sleep() {
 try {
 // Your code here!
 } finally {
 super.sleep();
 }
 }
 
 
 Chuck
 
 
 
 On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
 
  Hi all,
 
  I'm having a strange issue on a WO installation on EC2 (oracle jvm).
  Same strange application which had immortal sessions…
 
  Sometimes (quite rarely) a bounced application (put in refuse new 
  sessions) never quits and it's not accessible from JavaMonitor.
  If I look at the logs i see:
 
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
  com.tla.calendar.Application: refusing new clients and below min active 
  session threshold
  Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN  NSLog  - 
  com.tla.calendar.Application: about to terminate...
 
  The only thing left is to kill the instance… Which is not nice.
 
  I'm not overriding the terminate() method in Application.
 
  I am attaching a stack trace if it helps.
  B2Cjstack.txt
 
  Thanks,
 
 
 
  Matteo Centro
  ___
  Do not post admin requests to the list. They will be ignored.
  Webobjects-dev mailing list  (Webobjects-dev@lists.apple.com)
  Help/Unsubscribe/Update your Subscription:
  https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
 
  This email sent to ch...@global-village.net
 
 --
 Chuck Hill
 Executive Managing Partner, VP Development and Technical Services
 
 Practical WebObjects - for developers who want to increase their overall 
 knowledge of WebObjects or who are trying to solve specific problems.
 http://www.global-village.net/gvc/practical_webobjects
 
 Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest 
 Growing Companies in B.C!
 Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of 
 Canada’s Fastest-Growing Companies by PROFIT Magazine!
 
 
 
 
 
 
 
 
 
 
 
 
  ___
 Do not post