HELP: Continuing problems under heavy concurrent load!!!

[Simon Harris]

I have two clients accessing a stateless session bean (TX_REQUIRED). We are running Orion 1.2.9 and DB2 on the same machine under Win/NT SP5. Doesn't seem to matter what transaction attributes we use, we always get these errors. Under heavy load, I get the following exception on one client: com.evermind.server.rmi.OrionRemoteException: Transaction was rolled back: Error in transaction: javax.transaction.xa.XAException: The connection was closed: [ DB2Connection {     connectionHandle = 1     SPConnected = false     source = cats     user = cats     conArgs =     closed = false     describeCached = true     describeParam = true     isReadOnly = false     autoClose = false }  - 1, autoCommit: false (false)]         at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:187)         at java.lang.reflect.Method.invoke(Native Method)         at com.evermind.server.rmi.bd.run(JAX)         at com.evermind.server.rmi.bb.hw(JAX)         at com.evermind.server.rmi.bb.run(JAX)         at com.evermind.util.f.run(JAX) at connection to localhost/127.0.0.1 as admin DB2Connection {     connectionHandle = 1     SPConnected = false     source = cats     user = cats     conArgs =     closed = false     describeCached = true     describeParam = true     isReadOnly = false     autoClose = false }  - 1, autoCommit: false (false)]         at com.evermind.server.rmi.bb.invokeMethod(JAX)         at com.evermind.server.rmi.a2.invoke(JAX)         at __Proxy2.handleMessage(Unknown Source)         at cats.gateway.GatewayProxy.handleMessage(GatewayProxy.java:36)         at cats.gateway.external.Loader.process(Loader.java:46)         at cats.gateway.external.DirWatcher$Engine.run(DirWatcher.java:161)           Nested exception is: javax.transaction.xa.XAException: The connection was closed: [ DB2Connection {     connectionHandle = 1     SPConnected = false     source = cats     user = cats     conArgs =     closed = false     describeCached = true     describeParam = true     isReadOnly = false     autoClose = false }  - 1, autoCommit: false (false)]         at com.evermind.sql.ao.prepare(JAX)         at com.evermind.server.go.h3(JAX)         at com.evermind.server.ApplicationServerTransaction.commit(JAX)         at com.evermind.server.ApplicationServerTransaction.end(JAX)         at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)         at java.lang.reflect.Method.invoke(Native Method)         at com.evermind.server.rmi.bd.run(JAX)         at com.evermind.server.rmi.bb.hw(JAX)         at com.evermind.server.rmi.bb.run(JAX)         at com.evermind.util.f.run(JAX) at connection to localhost/127.0.0.1 DB2Connection {     connectionHandle = 1     SPConnected = false     source = cats     user = cats     conArgs =     closed = false     describeCached = true     describeParam = true     isReadOnly = false     autoClose = false }  - 1, autoCommit: false (false)]         at com.evermind.server.rmi.OrionRemoteException.h0(JAX)         at com.evermind.server.rmi.bb.hu(JAX)         at com.evermind.server.rmi.bb.run(JAX)         at java.lang.Thread.run(Unknown Source) and the application log looks like: 19/09/00 12:31 Started 19/09/00 12:33 cats-web: 1.2.9 Started 19/09/00 12:33 Error in bean cats.ejb.Change: Error loading state javax.ejb.NoSuchEntityException: Load failed for 1  at cats.change.ChangeDAO.load(ChangeDAO.java:197)  at cats.change.ChangeEJB.ejbLoad(ChangeEJB.java:1104)  at IChange_EntityBeanWrapper6.loadState(IChange_EntityBeanWrapper6.java:9145)  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) 19/09/00 12:33 Error in bean cats.ejb.Change: Error in ejbPassivate com.evermind.server.DeadlockException: Deadlock detected: thread 7 is waiting for resource entity 1 held by thread 7  at com.evermind.server.ejb.AbstractEJBObject.startCall(JAX)  at com.evermind.server.ejb.EntityEJBObject.removeFromCache(JAX)  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) 19/09/00 12:33 Error in bean cats.ejb.Change: Error while delisting bean from TX com.evermind.transaction.MarshallingXAException  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) Nested exception is: javax.ejb.NoSuchEntityException: Load failed for 1  at cats.change.ChangeDAO.load(ChangeDAO.java:197)  at cats.change.ChangeEJB.ejbLoad(ChangeEJB.java:1104)  at IChange_EntityBeanWrapper6.loadState(IChange_EntityBeanWrapper6.java:9145)  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) 19/09/00 12:33 Error in bean cats.ejb.ChangeEvent: Error loading state javax.ejb.NoSuchEntityException: Load failed for ChangeEvent 1  at cats.changeevent.ChangeEventDAO.load(ChangeEventDAO.java:139)  at cats.changeevent.ChangeEventEJB.ejbLoad(ChangeEventEJB.java:291)  at IChangeEvent_EntityBeanWrapper7.loadState(IChangeEvent_EntityBeanWrapper7.java:740)  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) 19/09/00 12:33 Error in bean cats.ejb.ChangeEvent: Error in ejbPassivate com.evermind.server.DeadlockException: Deadlock detected: thread 7 is waiting for resource entity ChangeEvent 1 held by thread 7  at com.evermind.server.ejb.AbstractEJBObject.startCall(JAX)  at com.evermind.server.ejb.EntityEJBObject.removeFromCache(JAX)  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) 19/09/00 12:33 Error in bean cats.ejb.ChangeEvent: Error while delisting bean from TX com.evermind.transaction.MarshallingXAException  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX) Nested exception is: javax.ejb.NoSuchEntityException: Load failed for ChangeEvent 1  at cats.changeevent.ChangeEventDAO.load(ChangeEventDAO.java:139)  at cats.changeevent.ChangeEventEJB.ejbLoad(ChangeEventEJB.java:291)  at IChangeEvent_EntityBeanWrapper7.loadState(IChangeEvent_EntityBeanWrapper7.java:740)  at com.evermind.server.ejb.EntityEJBObject.endTransaction(JAX)  at com.evermind.server.ApplicationServerTransaction.am6(JAX)  at com.evermind.server.ApplicationServerTransaction.commit(JAX)  at com.evermind.server.ApplicationServerTransaction.end(JAX)  at IGateway_StatelessSessionBeanWrapper10.handleMessage(IGateway_StatelessSessionBeanWrapper10.java:184)  at java.lang.reflect.Method.invoke(Native Method)  at com.evermind.server.rmi.bd.run(JAX)  at com.evermind.server.rmi.bb.hw(JAX)  at com.evermind.server.rmi.bb.run(JAX)  at com.evermind.util.f.run(JAX)