Hi All,
I've just got hit by an OutOfMemoryError: unable to create new native
thread issue. I've switched UseDedicatedTaskRunner to false which
appears to have resolved the issue for now but I have a feeling I'm
just masking a deeper problem.
I have the following set up:
2 ActiveMQ brokers configured as a network of brokers
Several applications that use
- a Spring JmsTemplate with a PooledConnectionFactory
- a Spring DefaultMessageListenerContainer that uses a regular
ConnectionFactory. The DefaultMessageListenerContainers are set to
CACHE_CONSUMER
- a static failover broker URL for both the JmsTemplate and listener
container to the two brokers
In the applications themselves, I'm getting this:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:455)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:863)
at
org.apache.activemq.thread.PooledTaskRunner.wakeup(PooledTaskRunner.java:76)
at
org.apache.activemq.transport.failover.FailoverTransport.reconnect(FailoverTransport.java:536)
at
org.apache.activemq.transport.failover.FailoverTransport.start(FailoverTransport.java:239)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at
org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
at
org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
at
org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
at
org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:125)
at
org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:110)
at
org.apache.activemq.pool.PooledConnectionFactory.createConnection(PooledConnectionFactory.java:81)
at
org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:461)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:539)
In the brokers I'm getting this:
INFO | jvm 1 | 2009/09/30 12:28:41 | Exception in thread
"ActiveMQ Transport Server Thread Hand
ler: tcp://localhost:41414" java.lang.OutOfMemoryError: unable to
create new native thread
INFO | jvm 1 | 2009/09/30 12:28:41 | at
java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2009/09/30 12:28:41 | at
java.lang.Thread.start(Thread.java:574)
INFO | jvm 1 | 2009/09/30 12:28:41 | at
org.apache.activemq.broker.TransportConnector$1.onAccept(TransportConnector.java:221)
INFO | jvm 1 | 2009/09/30 12:28:41 | at
org.apache.activemq.transport.tcp.TcpTransportServer.handleSocket(TcpTransportServer.java:409)
INFO | jvm 1 | 2009/09/30 12:28:41 | at
org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:341)
INFO | jvm 1 | 2009/09/30 12:28:41 | at
java.lang.Thread.run(Thread.java:595)
INFO | jvm 1 | 2009/09/30 12:59:27 | Exception in thread
"DefaultMessageListenerContainer-80896" java.lang.OutOfMemoryError:
unable to create new native thread
INFO | jvm 1 | 2009/09/30 12:59:27 | at
java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
java.lang.Thread.start(Thread.java:574)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
org.springframework.core.task.SimpleAsyncTaskExecutor.doExecute(SimpleAsyncTaskExecutor.java:157)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:145)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
org.springframework.core.task.SimpleAsyncTaskExecutor.execute(SimpleAsyncTaskExecutor.java:126)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
org.springframework.jms.listener.DefaultMessageListenerContainer.doRescheduleTask(DefaultMessageListenerContainer.java:598)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
org.springframework.jms.listener.AbstractJmsListeningContainer.rescheduleTaskIfNecessary(AbstractJmsListeningContainer.java:509)
INFO | jvm 1 | 2009/09/30 12:59:27 | at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:918)
I'm also getting a lot of these in the logs:
FailoverTransport.doReconnect | Successfully connected to tcp://xxxxxxxx:xxxx
Other information:
We've got a similar set up in our load testing environment and our
other production environment and neither of them appear to have the
same problem (yet?).
I've read the JmsTemplate gotchas and outofmemory pages on the
ActiveMQ site. I think I'm within the guidelines...
We're using Spring 2.5.6
We aren't using a transaction manager
any ideas?
