Another longer term option is to have a MySQL replication slave server that
performs backup snapshots as needed (hourly, daily, weekly).
On Apr 3, 2013, at 1:32 PM, Altera WO Team <webobje...@altera.it> wrote:
> Hi all,
>
> I finally solved the problem… well, not really but I found out the reason.
> Turns out that the whole thing is on amazon boxes and the DB machine (MySQL)
> was being backed up every hour… The backup takes all the cpu (not the backup,
> the bzip2 of the backup) and locks up MySQL for 2 minutes, if an instance
> tries to access the db in those 2 minutes that instance won't be able to die
> properly.
> Solution? I removed the hourly backup which was useless and scheduled a
> nightly backup with a reniced bzip2. No issues from that day!
>
> Thanks everybody for the help.
>
> Matteo
>
>>
>> On 21/mar/2013, at 19:19, Chuck Hill <ch...@global-village.net> wrote:
>>
>>>
>>> On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
>>>
>>>>
>>>> On 21/mar/2013, at 18:46, Chuck Hill <ch...@global-village.net> wrote:
>>>>
>>>>>
>>>>> On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
>>>>>
>>>>>> I tried anyway and I can confirm. I'm not creating needless sessions.
>>>>>>
>>>>>> The app locks up anyway though…
>>>>>
>>>>> Did adding the finally blocks in your terminate() method help? Is
>>>>> anything in your code calling session.terminate() directly? A logout
>>>>> link?
>>>>
>>>> the finally blocks didn't help.
>>>>
>>>> Something is calling session.terminate() directly though, happens when a
>>>> something bad happens in a direct action.
>>>>
>>>> if (existingSession() != null)
>>>> sess().terminate();
>>>>
>>>> It's called only if we can't parse the parameters and should never happen.
>>>> I'll put some logging in those cases.
>>>>
>>>> Why should this be a problem?
>>>
>>> I thought it was fixed in 5.4.3, but I could be wrong. In some past
>>> version if that same URL can in in rapid succession, you would get
>>> deadlocks like you are seeing. The solution at the time was to not call
>>> terminate directly and instead set the session timeout to a couple seconds.
>>> That allowed both requests to clear before terminating the session.
>>>
>>>
>>> Chuck
>>>
>>>
>>>>>> I lost hope in finding the real reason, for now I would be happy to find
>>>>>> a dirty workaround. I need those apps to bounce anyway.
>>>>>>
>>>>>> I already tried using ERTimeToKill and it's ineffective.
>>>>>>
>>>>>> The lockups are quite rare: I have 10 instances running, bouncing every
>>>>>> 6 hours and it happens maybe 2 times a week, but if an instance locks up
>>>>>> the instances running become 9 and in the long run I'll have everything
>>>>>> locked up.
>>>>>>
>>>>>> Any suggestions?
>>>>>
>>>>> I don't recall how ERTimeToKill works, but with those stuck threads you
>>>>> are going to need to do
>>>>>
>>>>> Runtime.getRuntime().exit(1);
>>>>>
>>>>> and if THAT does not kill it, this should:
>>>>>
>>>>> Runtime.getRuntime().halt(1);
>>>>>
>>>>>
>>>>> Chuck
>>>>>>
>>>>>> On 20/mar/2013, at 01:16, Chuck Hill <ch...@global-village.net> wrote:
>>>>>>
>>>>>>> I think the problem is not that it is creating needless sessions, but
>>>>>>> that a checked-out session is not getting checked back in.
>>>>>>>
>>>>>>> Chuck
>>>>>>>
>>>>>>>
>>>>>>> On 2013-03-19, at 5:08 PM, Simon wrote:
>>>>>>>
>>>>>>>> if we have this kind of issue the first thing we do is log out session
>>>>>>>> creation and check the stack traces. stick something like this in your
>>>>>>>> session constructor then check the output:
>>>>>>>>
>>>>>>>> public Session() {
>>>>>>>> super();
>>>>>>>> StringWriter stringWriter = new StringWriter();
>>>>>>>>
>>>>>>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>>>>>>>
>>>>>>>> (new Throwable()).printStackTrace(printWriter);
>>>>>>>>
>>>>>>>> String trace = stringWriter.toString(); <---- log this somewhere that
>>>>>>>> you can get easy access to
>>>>>>>>
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19 March 2013 23:15, Altera WO Team <webobje...@altera.it> wrote:
>>>>>>>> Wow, good hint!
>>>>>>>>
>>>>>>>> In theory I'm not touching a Session but sometimes, when something in
>>>>>>>> an EO changes I trigger e-mail sending.
>>>>>>>> I use ERMailDeliveryHTML using a component instantiated with a brand
>>>>>>>> new wocontext using ERXWOContext.newContext()
>>>>>>>> the component is not referencing a Session in any part.
>>>>>>>>
>>>>>>>> Could it be the cause?
>>>>>>>>
>>>>>>>> Matteo
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19/mar/2013, at 21:24, Simon <si...@potwells.co.uk> wrote:
>>>>>>>>
>>>>>>>>> ...or you are touching the session object outside the RR loop. the
>>>>>>>>> classic gotcha is rendering a component in a thread that touches the
>>>>>>>>> session in some way e.g. delivering an email in a thread that uses a
>>>>>>>>> component to render it's content... been caught by that one soooo
>>>>>>>>> many times!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 19 March 2013 17:51, Chuck Hill <ch...@global-village.net> wrote:
>>>>>>>>> Hi Matteo,
>>>>>>>>>
>>>>>>>>> You have one or more Zombie (aka Immortal) Sessions, as shown by
>>>>>>>>> stack traces like this:
>>>>>>>>> "WorkerThread11" prio=10 tid=0x0000000041848800 nid=0x1010 in
>>>>>>>>> Object.wait() [0x00007f16f7cfa000]
>>>>>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>>>>> at java.lang.Object.wait(Native Method)
>>>>>>>>> - waiting on <0x00000000d120b328> (a
>>>>>>>>> com.webobjects.appserver.WOSessionStore$TimeoutEntry)
>>>>>>>>> at java.lang.Object.wait(Object.java:485)
>>>>>>>>> at
>>>>>>>>> com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
>>>>>>>>> - locked <0x00000000d120b328> (a
>>>>>>>>> com.webobjects.appserver.WOSessionStore$TimeoutEntry)
>>>>>>>>> at
>>>>>>>>> com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
>>>>>>>>> at
>>>>>>>>> er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
>>>>>>>>> at
>>>>>>>>> er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
>>>>>>>>> at
>>>>>>>>> er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
>>>>>>>>> at
>>>>>>>>> com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
>>>>>>>>> at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This likely has one of two causes:
>>>>>>>>> 1. The application is getting OutOfMemory errors, which can leave the
>>>>>>>>> session store in an insane state
>>>>>>>>> 2. The app is throwing an exception from sleep() in Session. If you
>>>>>>>>> overrride sleep() it should use a try...finally block
>>>>>>>>>
>>>>>>>>> public void sleep() {
>>>>>>>>> try {
>>>>>>>>> // Your code here!
>>>>>>>>> } finally {
>>>>>>>>> super.sleep();
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Chuck
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> I'm having a strange issue on a WO installation on EC2 (oracle jvm).
>>>>>>>>>> Same strange application which had immortal sessions…
>>>>>>>>>>
>>>>>>>>>> Sometimes (quite rarely) a bounced application (put in refuse new
>>>>>>>>>> sessions) never quits and it's not accessible from JavaMonitor.
>>>>>>>>>> If I look at the logs i see:
>>>>>>>>>>
>>>>>>>>>> Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN NSLog
>>>>>>>>>> - <com.tla.calendar.Application>: refusing new clients and below min
>>>>>>>>>> active session threshold
>>>>>>>>>> Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN NSLog
>>>>>>>>>> - <com.tla.calendar.Application>: about to terminate...
>>>>>>>>>>
>>>>>>>>>> The only thing left is to kill the instance… Which is not nice.
>>>>>>>>>>
>>>>>>>>>> I'm not overriding the terminate() method in Application.
>>>>>>>>>>
>>>>>>>>>> I am attaching a stack trace if it helps.
>>>>>>>>>> <B2Cjstack.txt>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Matteo Centro
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
>>>>>>>>>>
>>>>>>>>>> This email sent to ch...@global-village.net
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Chuck Hill
>>>>>>>>> Executive Managing Partner, VP Development and Technical Services
>>>>>>>>>
>>>>>>>>> Practical WebObjects - for developers who want to increase their
>>>>>>>>> overall knowledge of WebObjects or who are trying to solve specific
>>>>>>>>> problems.
>>>>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>>>>
>>>>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>>>>>>>> Growing Companies in B.C!
>>>>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200
>>>>>>>>> ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk
>>>>>>>>>
>>>>>>>>> This email sent to si...@potwells.co.uk
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Chuck Hill
>>>>>>> Executive Managing Partner, VP Development and Technical Services
>>>>>>>
>>>>>>> Practical WebObjects - for developers who want to increase their
>>>>>>> overall knowledge of WebObjects or who are trying to solve specific
>>>>>>> problems.
>>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>>
>>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>>>>>> Growing Companies in B.C!
>>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking
>>>>>>> of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Chuck Hill
>>>>> Executive Managing Partner, VP Development and Technical Services
>>>>>
>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>
>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>>>> Growing Companies in B.C!
>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of
>>>>> Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Chuck Hill
>>> Executive Managing Partner, VP Development and Technical Services
>>>
>>> Practical WebObjects - for developers who want to increase their overall
>>> knowledge of WebObjects or who are trying to solve specific problems.
>>> http://www.global-village.net/gvc/practical_webobjects
>>>
>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>> Growing Companies in B.C!
>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of
>>> Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-dev/kelleherk%40gmail.com
>
> This email sent to kelleh...@gmail.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
