I strongly agree with the suggestion for a MySQL replication slave server. They
are not that hard to set up and then you can dump a backup as often as you like
with out effecting the main server and you always have an up to the minute copy
incase something happens. I had a problem a couple of months ago and going to a
backup up that was just a couple of hours old was a major loss.
Steve
On Apr 3, 2013, at 2:16 PM, Kieran Kelleher <kelleh...@gmail.com> wrote:
> Another longer term option is to have a MySQL replication slave server that
> performs backup snapshots as needed (hourly, daily, weekly).
>
> On Apr 3, 2013, at 1:32 PM, Altera WO Team <webobje...@altera.it> wrote:
>
>> Hi all,
>>
>> I finally solved the problem… well, not really but I found out the reason.
>> Turns out that the whole thing is on amazon boxes and the DB machine (MySQL)
>> was being backed up every hour… The backup takes all the cpu (not the
>> backup, the bzip2 of the backup) and locks up MySQL for 2 minutes, if an
>> instance tries to access the db in those 2 minutes that instance won't be
>> able to die properly.
>> Solution? I removed the hourly backup which was useless and scheduled a
>> nightly backup with a reniced bzip2. No issues from that day!
>>
>> Thanks everybody for the help.
>>
>> Matteo
>>
>>>
>>> On 21/mar/2013, at 19:19, Chuck Hill <ch...@global-village.net> wrote:
>>>
>>>>
>>>> On 2013-03-21, at 10:54 AM, Altera WO Team wrote:
>>>>
>>>>>
>>>>> On 21/mar/2013, at 18:46, Chuck Hill <ch...@global-village.net> wrote:
>>>>>
>>>>>>
>>>>>> On 2013-03-21, at 6:47 AM, Altera WO Team wrote:
>>>>>>
>>>>>>> I tried anyway and I can confirm. I'm not creating needless sessions.
>>>>>>>
>>>>>>> The app locks up anyway though…
>>>>>>
>>>>>> Did adding the finally blocks in your terminate() method help? Is
>>>>>> anything in your code calling session.terminate() directly? A logout
>>>>>> link?
>>>>>
>>>>> the finally blocks didn't help.
>>>>>
>>>>> Something is calling session.terminate() directly though, happens when a
>>>>> something bad happens in a direct action.
>>>>>
>>>>> if (existingSession() != null)
>>>>> sess().terminate();
>>>>>
>>>>> It's called only if we can't parse the parameters and should never
>>>>> happen. I'll put some logging in those cases.
>>>>>
>>>>> Why should this be a problem?
>>>>
>>>> I thought it was fixed in 5.4.3, but I could be wrong. In some past
>>>> version if that same URL can in in rapid succession, you would get
>>>> deadlocks like you are seeing. The solution at the time was to not call
>>>> terminate directly and instead set the session timeout to a couple
>>>> seconds. That allowed both requests to clear before terminating the
>>>> session.
>>>>
>>>>
>>>> Chuck
>>>>
>>>>
>>>>>>> I lost hope in finding the real reason, for now I would be happy to
>>>>>>> find a dirty workaround. I need those apps to bounce anyway.
>>>>>>>
>>>>>>> I already tried using ERTimeToKill and it's ineffective.
>>>>>>>
>>>>>>> The lockups are quite rare: I have 10 instances running, bouncing every
>>>>>>> 6 hours and it happens maybe 2 times a week, but if an instance locks
>>>>>>> up the instances running become 9 and in the long run I'll have
>>>>>>> everything locked up.
>>>>>>>
>>>>>>> Any suggestions?
>>>>>>
>>>>>> I don't recall how ERTimeToKill works, but with those stuck threads you
>>>>>> are going to need to do
>>>>>>
>>>>>> Runtime.getRuntime().exit(1);
>>>>>>
>>>>>> and if THAT does not kill it, this should:
>>>>>>
>>>>>> Runtime.getRuntime().halt(1);
>>>>>>
>>>>>>
>>>>>> Chuck
>>>>>>>
>>>>>>> On 20/mar/2013, at 01:16, Chuck Hill <ch...@global-village.net> wrote:
>>>>>>>
>>>>>>>> I think the problem is not that it is creating needless sessions, but
>>>>>>>> that a checked-out session is not getting checked back in.
>>>>>>>>
>>>>>>>> Chuck
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2013-03-19, at 5:08 PM, Simon wrote:
>>>>>>>>
>>>>>>>>> if we have this kind of issue the first thing we do is log out
>>>>>>>>> session creation and check the stack traces. stick something like
>>>>>>>>> this in your session constructor then check the output:
>>>>>>>>>
>>>>>>>>> public Session() {
>>>>>>>>> super();
>>>>>>>>> StringWriter stringWriter = new StringWriter();
>>>>>>>>>
>>>>>>>>> PrintWriter printWriter = new PrintWriter(stringWriter);
>>>>>>>>>
>>>>>>>>> (new Throwable()).printStackTrace(printWriter);
>>>>>>>>>
>>>>>>>>> String trace = stringWriter.toString(); <---- log this somewhere that
>>>>>>>>> you can get easy access to
>>>>>>>>>
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 19 March 2013 23:15, Altera WO Team <webobje...@altera.it> wrote:
>>>>>>>>> Wow, good hint!
>>>>>>>>>
>>>>>>>>> In theory I'm not touching a Session but sometimes, when something in
>>>>>>>>> an EO changes I trigger e-mail sending.
>>>>>>>>> I use ERMailDeliveryHTML using a component instantiated with a brand
>>>>>>>>> new wocontext using ERXWOContext.newContext()
>>>>>>>>> the component is not referencing a Session in any part.
>>>>>>>>>
>>>>>>>>> Could it be the cause?
>>>>>>>>>
>>>>>>>>> Matteo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 19/mar/2013, at 21:24, Simon <si...@potwells.co.uk> wrote:
>>>>>>>>>
>>>>>>>>>> ...or you are touching the session object outside the RR loop. the
>>>>>>>>>> classic gotcha is rendering a component in a thread that touches the
>>>>>>>>>> session in some way e.g. delivering an email in a thread that uses a
>>>>>>>>>> component to render it's content... been caught by that one soooo
>>>>>>>>>> many times!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 19 March 2013 17:51, Chuck Hill <ch...@global-village.net> wrote:
>>>>>>>>>> Hi Matteo,
>>>>>>>>>>
>>>>>>>>>> You have one or more Zombie (aka Immortal) Sessions, as shown by
>>>>>>>>>> stack traces like this:
>>>>>>>>>> "WorkerThread11" prio=10 tid=0x0000000041848800 nid=0x1010 in
>>>>>>>>>> Object.wait() [0x00007f16f7cfa000]
>>>>>>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>>>>>> at java.lang.Object.wait(Native Method)
>>>>>>>>>> - waiting on <0x00000000d120b328> (a
>>>>>>>>>> com.webobjects.appserver.WOSessionStore$TimeoutEntry)
>>>>>>>>>> at java.lang.Object.wait(Object.java:485)
>>>>>>>>>> at
>>>>>>>>>> com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(WOSessionStore.java:191)
>>>>>>>>>> - locked <0x00000000d120b328> (a
>>>>>>>>>> com.webobjects.appserver.WOSessionStore$TimeoutEntry)
>>>>>>>>>> at
>>>>>>>>>> com.webobjects.appserver.WOApplication.restoreSessionWithID(WOApplication.java:1913)
>>>>>>>>>> at
>>>>>>>>>> er.extensions.appserver.ERXApplication.restoreSessionWithID(ERXApplication.java:2403)
>>>>>>>>>> at
>>>>>>>>>> er.extensions.appserver.ERXWOContext.existingSession(ERXWOContext.java:57)
>>>>>>>>>> at
>>>>>>>>>> er.extensions.appserver.ERXWOContext.hasSession(ERXWOContext.java:69)
>>>>>>>>>> at
>>>>>>>>>> com.webobjects.appserver.WOAction.existingSession(WOAction.java:190)
>>>>>>>>>> at com.tla.calendar.DirectAction.goToAction(DirectAction.java:454)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This likely has one of two causes:
>>>>>>>>>> 1. The application is getting OutOfMemory errors, which can leave
>>>>>>>>>> the session store in an insane state
>>>>>>>>>> 2. The app is throwing an exception from sleep() in Session. If you
>>>>>>>>>> overrride sleep() it should use a try...finally block
>>>>>>>>>>
>>>>>>>>>> public void sleep() {
>>>>>>>>>> try {
>>>>>>>>>> // Your code here!
>>>>>>>>>> } finally {
>>>>>>>>>> super.sleep();
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Chuck
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2013-03-19, at 9:38 AM, Altera WO Team wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I'm having a strange issue on a WO installation on EC2 (oracle jvm).
>>>>>>>>>>> Same strange application which had immortal sessions…
>>>>>>>>>>>
>>>>>>>>>>> Sometimes (quite rarely) a bounced application (put in refuse new
>>>>>>>>>>> sessions) never quits and it's not accessible from JavaMonitor.
>>>>>>>>>>> If I look at the logs i see:
>>>>>>>>>>>
>>>>>>>>>>> Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN NSLog
>>>>>>>>>>> - <com.tla.calendar.Application>: refusing new clients and below
>>>>>>>>>>> min active session threshold
>>>>>>>>>>> Mar 19 12:38:52 B2C[2002] (ERXNSLogLog4jBridge.java:44) WARN NSLog
>>>>>>>>>>> - <com.tla.calendar.Application>: about to terminate...
>>>>>>>>>>>
>>>>>>>>>>> The only thing left is to kill the instance… Which is not nice.
>>>>>>>>>>>
>>>>>>>>>>> I'm not overriding the terminate() method in Application.
>>>>>>>>>>>
>>>>>>>>>>> I am attaching a stack trace if it helps.
>>>>>>>>>>> <B2Cjstack.txt>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Matteo Centro
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
>>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
>>>>>>>>>>>
>>>>>>>>>>> This email sent to ch...@global-village.net
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Chuck Hill
>>>>>>>>>> Executive Managing Partner, VP Development and Technical Services
>>>>>>>>>>
>>>>>>>>>> Practical WebObjects - for developers who want to increase their
>>>>>>>>>> overall knowledge of WebObjects or who are trying to solve specific
>>>>>>>>>> problems.
>>>>>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>>>>>
>>>>>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100
>>>>>>>>>> Fastest Growing Companies in B.C!
>>>>>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200
>>>>>>>>>> ranking of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Do not post admin requests to the list. They will be ignored.
>>>>>>>>>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
>>>>>>>>>> Help/Unsubscribe/Update your Subscription:
>>>>>>>>>> https://lists.apple.com/mailman/options/webobjects-dev/simon%40potwells.co.uk
>>>>>>>>>>
>>>>>>>>>> This email sent to si...@potwells.co.uk
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Chuck Hill
>>>>>>>> Executive Managing Partner, VP Development and Technical Services
>>>>>>>>
>>>>>>>> Practical WebObjects - for developers who want to increase their
>>>>>>>> overall knowledge of WebObjects or who are trying to solve specific
>>>>>>>> problems.
>>>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>>>
>>>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>>>>>>> Growing Companies in B.C!
>>>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking
>>>>>>>> of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Chuck Hill
>>>>>> Executive Managing Partner, VP Development and Technical Services
>>>>>>
>>>>>> Practical WebObjects - for developers who want to increase their overall
>>>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>>>
>>>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>>>>> Growing Companies in B.C!
>>>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking
>>>>>> of Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Chuck Hill
>>>> Executive Managing Partner, VP Development and Technical Services
>>>>
>>>> Practical WebObjects - for developers who want to increase their overall
>>>> knowledge of WebObjects or who are trying to solve specific problems.
>>>> http://www.global-village.net/gvc/practical_webobjects
>>>>
>>>> Global Village Consulting ranks 13th in 2012 in BIV's Top 100 Fastest
>>>> Growing Companies in B.C!
>>>> Global Village Consulting ranks 76th in 24th annual PROFIT 200 ranking of
>>>> Canada’s Fastest-Growing Companies by PROFIT Magazine!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Do not post admin requests to the list. They will be ignored.
>> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
>> Help/Unsubscribe/Update your Subscription:
>> https://lists.apple.com/mailman/options/webobjects-dev/kelleherk%40gmail.com
>>
>> This email sent to kelleh...@gmail.com
>
>
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-dev/speery%40me.com
>
> This email sent to spe...@me.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
