We are using 1.0.1SP4 on JBoss405. The queues are persisted on oracle 10g. JBM has its own non-xa oracle datasource
The multithreaded consumers rely on the XA Connection Factory and are wrapped in XA transactions. One set of connection/session/consumer is created for each sequence of polling, no caching whatsoever, and we proceed with the usual pattern : xaTx.begin con = jmxXACf.createCon ses = con.createSession con.start consumer = ses.createConsumer consumer.receive() consumer.close session.close con.close xaTx.commit they're all cleaned up after the receive call, and the XA transaction is always commited/rollbacked. The jboss instance hosting both JBM and the consumers (bootstraped from a War using JBM loader) runs slowly but steadily to memory and cpu exhaustion, even when it's activity is limited to the constant jms 'polling' (8 consumer threads). As the heap graphs were showing textbook linear memory consumption I ran jmap/jhat after a 2 hours run (with only the 30th first minutes of dense activity). It showed that the number of ServerConsumerEndpoint instances had made the top 5 with approx 200000 : num #instances #bytes class name -------------------------------------- 1: 217640 42500368 [Ljava.util.HashMap$Entry; 2: 223956 25354776 [Ljava.lang.Object; 3: 306257 23970568 [C 4: 468932 22508736 java.util.HashMap$Entry 5: 199658 20764432 org.jboss.jms.server.endpoint.ServerConsumerEndpoint Then I get all kind of stacks bound to the memory shortage like : java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:50:39,420 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] - could not create session | org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() [EMAIL PROTECTED] [EMAIL PROTECTED] handles=0 lastUse=1171014618826 permit=false trackByTx=false [EMAIL PROTECTED] [EMAIL PROTECTED] xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException) | at org.jboss.resource.JBossResourceException.rethrowAsResourceException(JBossResourceException.java:61) | at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:410) | at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842) | at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:389) | at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369) | at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) | at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) | at java.lang.Thread.run(Thread.java:595) | Caused by: java.lang.reflect.UndeclaredThrowableException | at org.jboss.resource.JBossResourceException.process(JBossResourceException.java:204) | at org.jboss.resource.JBossResourceException.<init>(JBossResourceException.java:111) | ... 12 more | Caused by: java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "Timer-1" | 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "TimeoutFactory-0" | 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:24,326 ERROR [org.jboss.jms.util.ExceptionUtil] - ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s] | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:38,514 ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483608] close [a1885-6t7h9a-exxk6yht-1-exyfx05u-3t] | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:41,951 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:55,701 ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u] | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:55,717 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:59,170 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:51:59,170 ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:52:02,623 INFO [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection | 2007-02-09 10:52:06,061 ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:52:19,858 INFO [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection | 2007-02-09 10:52:23,326 ERROR [cms.transcoding.technical] - Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() [EMAIL PROTECTED] [EMAIL PROTECTED] handles=0 lastUse=1171014618826 permit=false trackByTx=false [EMAIL PROTECTED] [EMAIL PROTECTED] xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException) | javax.jms.JMSException: Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() [EMAIL PROTECTED] [EMAIL PROTECTED] handles=0 lastUse=1171014618826 permit=false trackByTx=false [EMAIL PROTECTED] [EMAIL PROTECTED] xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException) | at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:403) | at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369) | at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) | at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) | at java.lang.Thread.run(Thread.java:595) | 2007-02-09 10:52:33,920 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: | java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:52:48,030 ERROR [STDERR] - Exception in thread "JCA PoolFiller" | 2007-02-09 10:52:48,030 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space | 2007-02-09 10:53:06,014 ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s] | org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s] | at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72) | at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.start(ServerConnectionEndpoint.java:254) | at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$start$aop(ConnectionAdvised.java:94) | at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) | at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) | at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723) | at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014) | at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98) | at org.jboss.remoting.Client.invoke(Client.java:607) | at org.jboss.remoting.Client.invoke(Client.java:599) | at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) | at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) | at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) | at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) | at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java) | at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121) | at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:599) | at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:530) | at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:239) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:361) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) | at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) | at java.lang.Thread.run(Thread.java:595) | 2007-02-09 10:54:39,248 ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u] | org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u] | at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72) | at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:287) | at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:90) | at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) | at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:116) | at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source) | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | at java.lang.reflect.Method.invoke(Method.java:585) | at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) | at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) | at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) | at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723) | at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101) | at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) | at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014) | at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98) | at org.jboss.remoting.Client.invoke(Client.java:607) | at org.jboss.remoting.Client.invoke(Client.java:599) | at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) | at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:122) | at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect20.invoke(StateCreationAspect20.java) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) | at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:69) | at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect19.invoke(ConsumerAspect19.java) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) | at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) | at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) | at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) | at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) | at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) | at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) | at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:254) | at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:231) | at org.jboss.resource.adapter.jms.JmsSession.createConsumer(JmsSession.java:473) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createConsumer(DestinationBalancingMessageListenerContainer.java:648) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createListenerConsumer(DestinationBalancingMessageListenerContainer.java:302) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:363) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) | at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) | at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) | at java.lang.Thread.run(Thread.java:595) I saw that some XA related memory leaks issues had been fixed into SP4, so I wondered if these elements might ring a bell before I dig further down... This scenario is very easy to reproduce as-is on production,preprod or dev environments, I didnt have the time yet to create a barebone test, Id rather not write several and Im not sure how much to cut (xa, oracle, spring ?) This not a critical issue as the server handles perfectly the load peaks during the first hour(s) of its (short) life, and we can restart it easily but it's obviously a far-from-ideal situation. Thanks for your help, Best Regards Olivier Cuzacq Digiplug.com View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4019443#4019443 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4019443 _______________________________________________ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
