Ha, we struggled with that too. Still are.
1. Is this on Linux?
2. Are you using Java 5?
3. Do you notice that your system load shoots through the roof? (We
saw system load numbers of like 300 on our systems, which became
unresponsive for over a minute)
We suspect this is a GC bug in the VM; we mostly eliminated this
problem by commenting out an explicit call to System.gc() in Sequoia's
code (can't remember quite where right now). We still see the problem
on rare occasions under load; our backends get trashed and we have to
restore the databases from another copy. We were also able to trigger
this "load bomb" by hitting the "Perform GC" button in jconsole a few
times in rapid succession; it's as if this VM gets very unhappy if
something triggers a GC while it's already doing GC.
You can also see those warnings if your system clock changed, but I'm
hoping you are having the same problem we are. We're hoping that this
is a GC bug in Java 5 and are considering switching to Java 6, which
apparently rewrote a lot of the GC stuff. Has anyone else had this
problem and worked around it?
-Chris
On Jan 7, 2009, at 10:23 PM, Tyler Shen wrote:
G’day,
We have been running Sequoia in our production environment for weeks
until this morning our application crashed with the following
messages:
2009-01-07 10:44:16,490 INFO
[org.karora.moomba.ui.internal.FolderLayout] - Adding view to
layout: com.xbio.nba.abdr.site.ui.view.memberdetailsv...@224d0286
25s time shift detected, from Wed Jan 07 10:44:29 EST 2009 to Wed
Jan 07 10:44:56 EST 2009
Brutal changes of date/time can lead to erroneous controller failure
detections!
2009-01-07 10:44:59,141 INFO
[org.karora.moomba.ui.internal.FolderLayout] - Adding view to
layout:
com.xbio.nba.abdr.checklist.ui.view.tasktypesetupviewp...@6729246b
2009-01-07 10:44:59,148 INFO
[org.karora.moomba.ui.internal.PageLayout] - Adding view to layout:
View(com.xbio.nba.abdr.site.ui.view.MemberNavigator)
2009-01-07 10:44:59,172 WARN
[org.hibernate.util.JDBCExceptionReporter] - SQL Error: 0, SQLState:
null
2009-01-07 10:44:59,172 ERROR
[org.hibernate.util.JDBCExceptionReporter] - Protocol corruption
while trying to send command: 0. Check the previous command
18s time shift detected, from Wed Jan 07 10:44:40 EST 2009 to Wed
Jan 07 10:45:00 EST 2009
Brutal changes of date/time can lead to erroneous controller failure
detections!
29s time shift detected, from Wed Jan 07 10:44:29 EST 2009 to Wed
Jan 07 10:45:00 EST 2009
Brutal changes of date/time can lead to erroneous controller failure
detections!
2009-01-07 10:45:32,600 INFO
[com
.xbio
.nba
.abdr.patienttransition.service.impl.PatientTransitionServiceImpl] -
completeOutstandingTransitions executed
2009-01-07 10:45:32,713 WARN
[org.hibernate.util.JDBCExceptionReporter] - SQL Error: 0, SQLState:
null
2009-01-07 10:45:32,713 ERROR
[org.hibernate.util.JDBCExceptionReporter] - Protocol corruption
while trying to send command: 0. Check the previous command
We have the server running on NTP. How such a time shift occurred is
beyond me. Having briefly gone through the source code, I noticed
that tweaking ControllerTimeout can make the Sequoia more tolerant
once a time shit is detected.
Where can I change ControllerTimeout and is this the proper way to
go around the issues? Thanks!
Regards,
Tyler Shen
<ATT00001.txt>
_______________________________________________
Sequoia mailing list
[email protected]
https://forge.continuent.org/mailman/listinfo/sequoia
