It might be you are just running out of threads; maybe your setting of
file descriptors per process is too low?
On 4/23/07, Justin C. van Vorst <[EMAIL PROTECTED]> wrote:
On our production server this weekend, activemq-4.1.1 hit a
java.lang.OutOfMemoryError exception and then proceeded to stop functioning.
The process was using about 3GB of RAM.
I've seen some other posts about OOME, but none that quite match the stack
of ours. Does anyone have a place for me to start investigating this issue?
Below are the error logs, command line start-up, and activemq.xml config:
Error log:
2007-04-23 02:43:10,286 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_54813
2007-04-23 02:49:10,415 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_58365
2007-04-23 03:11:10,301 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_42109
2007-04-23 03:15:10,056 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_44595
2007-04-23 03:27:10,120 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51338
2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter
- Failed to checkpoint a message store:
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: unable to create new native thread
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: unable to create new native thread
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at java.util.Timer.<init>(Timer.java:154)
at java.util.Timer.<init>(Timer.java:122)
at com.mysql.jdbc.Connection.<init>(Connection.java:1441)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544)
at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:55)
at
org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:147)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
at
org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter
- Failed to mark the Journal:
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
at
org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
2007-04-23 03:47:10,105 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_34348
2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter
- Failed to checkpoint a message store:
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.io.IOException: Already started.
edu.emory.mathcs.backport.java.util.concurrent.ExecutionException:
java.io.IOException: Already started.
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299)
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
Caused by: java.io.IOException: Already started.
at
org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:145)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189)
at
org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247)
at
org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356)
at
edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter
- Failed to mark the Journal:
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
org.apache.activeio.journal.InvalidRecordLocationException: The location is
less than the last mark.
at
org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403)
at
org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101)
at
org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39)
2007-04-23 03:51:10,188 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_36436
2007-04-23 04:13:10,134 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_49022
2007-04-23 04:17:10,094 [localhost:61613] WARN ManagedTransportConnection
- Failed to register MBean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51068
Command line:
usr/java/current/bin/java -Xmx512M
-Dorg.apache.activemq.UseDedicatedTaskRunner=true
-Dderby.system.home=/opt/activemq-4.1.1/data
-Dderby.storage.fileSyncTransactionLog=true -Dcom.sun.management.jmxremote
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStorePassword=password
-Djavax.net.ssl.keyStore=/opt/activemq-4.1.1/conf/broker.ks
-Djavax.net.ssl.trustStore=/opt/activemq-4.1.1/conf/broker.ts -classpath
-Dactivemq.home=/opt/rhg/activemq-4.1.1 -Dactivemq.base=/opt/activemq-4.1.1
-jar /opt/activemq-4.1.1/bin/run.jar
activemq.xml:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="localhost" useJmx="true"
xmlns="http://activemq.org/config/1.0";>
<destinationPolicy>
<policyMap><policyEntries>
<policyEntry topic="FOO.>">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy />
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries></policyMap>
</destinationPolicy>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data"
dataSource="#mysql-ds"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://localhost:61616"
discoveryUri="multicast://default"/>
<transportConnector name="ssl" uri="ssl://localhost:61617"/>
<transportConnector name="stomp" uri="stomp://localhost:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url"
value="jdbc:mysql://x.x.x.x/activemq?relaxAutoCommit=true"/>
<property name="username" value="username"/>
<property name="password" value="password"/>
<property name="poolPreparedStatements" value="true"/>
</bean>
</beans>
--
View this message in context:
http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10143322
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
James
-------
http://macstrac.blogspot.com/
