Hi Chris, We are running Java 5 on CentOS 5. We have never seen any thrashing or resource overload on the server through our monitoring system. Good to know that this time shift issue is repeatable by triggering GC. I will build a testing environment with Java 6 and see how it goes. Thanks for your tips.
Regards, Tyler -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Christopher Ekberg Sent: Friday, 9 January 2009 6:26 AM To: Sequoia general mailing list Subject: Re: [Sequoia] Controller Timeout Issue? Ha, we struggled with that too. Still are. 1. Is this on Linux? 2. Are you using Java 5? 3. Do you notice that your system load shoots through the roof? (We saw system load numbers of like 300 on our systems, which became unresponsive for over a minute) We suspect this is a GC bug in the VM; we mostly eliminated this problem by commenting out an explicit call to System.gc() in Sequoia's code (can't remember quite where right now). We still see the problem on rare occasions under load; our backends get trashed and we have to restore the databases from another copy. We were also able to trigger this "load bomb" by hitting the "Perform GC" button in jconsole a few times in rapid succession; it's as if this VM gets very unhappy if something triggers a GC while it's already doing GC. You can also see those warnings if your system clock changed, but I'm hoping you are having the same problem we are. We're hoping that this is a GC bug in Java 5 and are considering switching to Java 6, which apparently rewrote a lot of the GC stuff. Has anyone else had this problem and worked around it? -Chris On Jan 7, 2009, at 10:23 PM, Tyler Shen wrote: > G'day, > > We have been running Sequoia in our production environment for weeks > until this morning our application crashed with the following > messages: > 2009-01-07 10:44:16,490 INFO > [org.karora.moomba.ui.internal.FolderLayout] - Adding view to > layout: com.xbio.nba.abdr.site.ui.view.memberdetailsv...@224d0286 > 25s time shift detected, from Wed Jan 07 10:44:29 EST 2009 to Wed > Jan 07 10:44:56 EST 2009 > Brutal changes of date/time can lead to erroneous controller failure > detections! > 2009-01-07 10:44:59,141 INFO > [org.karora.moomba.ui.internal.FolderLayout] - Adding view to > layout: > com.xbio.nba.abdr.checklist.ui.view.tasktypesetupviewp...@6729246b > 2009-01-07 10:44:59,148 INFO > [org.karora.moomba.ui.internal.PageLayout] - Adding view to layout: > View(com.xbio.nba.abdr.site.ui.view.MemberNavigator) > 2009-01-07 10:44:59,172 WARN > [org.hibernate.util.JDBCExceptionReporter] - SQL Error: 0, SQLState: > null > 2009-01-07 10:44:59,172 ERROR > [org.hibernate.util.JDBCExceptionReporter] - Protocol corruption > while trying to send command: 0. Check the previous command > 18s time shift detected, from Wed Jan 07 10:44:40 EST 2009 to Wed > Jan 07 10:45:00 EST 2009 > Brutal changes of date/time can lead to erroneous controller failure > detections! > 29s time shift detected, from Wed Jan 07 10:44:29 EST 2009 to Wed > Jan 07 10:45:00 EST 2009 > Brutal changes of date/time can lead to erroneous controller failure > detections! > 2009-01-07 10:45:32,600 INFO > [com > .xbio > .nba > .abdr.patienttransition.service.impl.PatientTransitionServiceImpl] - > completeOutstandingTransitions executed > 2009-01-07 10:45:32,713 WARN > [org.hibernate.util.JDBCExceptionReporter] - SQL Error: 0, SQLState: > null > 2009-01-07 10:45:32,713 ERROR > [org.hibernate.util.JDBCExceptionReporter] - Protocol corruption > while trying to send command: 0. Check the previous command > > We have the server running on NTP. How such a time shift occurred is > beyond me. Having briefly gone through the source code, I noticed > that tweaking ControllerTimeout can make the Sequoia more tolerant > once a time shit is detected. > Where can I change ControllerTimeout and is this the proper way to > go around the issues? Thanks! > > Regards, > > Tyler Shen > > > <ATT00001.txt> _______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia _______________________________________________ Sequoia mailing list [email protected] https://forge.continuent.org/mailman/listinfo/sequoia
