On our production server this weekend, activemq-4.1.1 hit a java.lang.OutOfMemoryError exception and then proceeded to stop functioning. The process was using about 3GB of RAM.
I've seen some other posts about OOME, but none that quite match the stack of ours. Does anyone have a place for me to start investigating this issue? Below are the error logs, command line start-up, and activemq.xml config: Error log: 2007-04-23 02:43:10,286 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_54813 2007-04-23 02:49:10,415 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_58365 2007-04-23 03:11:10,301 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_42109 2007-04-23 03:15:10,056 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_44595 2007-04-23 03:27:10,120 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51338 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter - Failed to checkpoint a message store: edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: unable to create new native thread at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299) at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118) at org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386) at org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) at org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) Caused by: java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at java.util.Timer.<init>(Timer.java:154) at java.util.Timer.<init>(Timer.java:122) at com.mysql.jdbc.Connection.<init>(Connection.java:1441) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544) at org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:55) at org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:147) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358) at org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189) at org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41) at org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247) at org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221) at org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356) at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter - Failed to mark the Journal: org.apache.activeio.journal.InvalidRecordLocationException: The location is less than the last mark. org.apache.activeio.journal.InvalidRecordLocationException: The location is less than the last mark. at org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340) at org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403) at org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) at org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) 2007-04-23 03:47:10,105 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_34348 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter - Failed to checkpoint a message store: edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: java.io.IOException: Already started. edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: java.io.IOException: Already started. at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299) at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118) at org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386) at org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) at org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) Caused by: java.io.IOException: Already started. at org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:145) at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358) at org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189) at org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41) at org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247) at org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221) at org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356) at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter - Failed to mark the Journal: org.apache.activeio.journal.InvalidRecordLocationException: The location is less than the last mark. org.apache.activeio.journal.InvalidRecordLocationException: The location is less than the last mark. at org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340) at org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403) at org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) at org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) 2007-04-23 03:51:10,188 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_36436 2007-04-23 04:13:10,134 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_49022 2007-04-23 04:17:10,094 [localhost:61613] WARN ManagedTransportConnection - Failed to register MBean: org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51068 Command line: usr/java/current/bin/java -Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Dderby.system.home=/opt/activemq-4.1.1/data -Dderby.storage.fileSyncTransactionLog=true -Dcom.sun.management.jmxremote -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStorePassword=password -Djavax.net.ssl.keyStore=/opt/activemq-4.1.1/conf/broker.ks -Djavax.net.ssl.trustStore=/opt/activemq-4.1.1/conf/broker.ts -classpath -Dactivemq.home=/opt/rhg/activemq-4.1.1 -Dactivemq.base=/opt/activemq-4.1.1 -jar /opt/activemq-4.1.1/bin/run.jar activemq.xml: <beans> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/> <broker brokerName="localhost" useJmx="true" xmlns="http://activemq.org/config/1.0"> <destinationPolicy> <policyMap><policyEntries> <policyEntry topic="FOO.>"> <dispatchPolicy> <strictOrderDispatchPolicy /> </dispatchPolicy> <subscriptionRecoveryPolicy> <lastImageSubscriptionRecoveryPolicy /> </subscriptionRecoveryPolicy> </policyEntry> </policyEntries></policyMap> </destinationPolicy> <persistenceAdapter> <journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data" dataSource="#mysql-ds"/> </persistenceAdapter> <transportConnectors> <transportConnector name="openwire" uri="tcp://localhost:61616" discoveryUri="multicast://default"/> <transportConnector name="ssl" uri="ssl://localhost:61617"/> <transportConnector name="stomp" uri="stomp://localhost:61613"/> </transportConnectors> <networkConnectors> <networkConnector name="default-nc" uri="multicast://default"/> </networkConnectors> </broker> <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="jdbc:mysql://x.x.x.x/activemq?relaxAutoCommit=true"/> <property name="username" value="username"/> <property name="password" value="password"/> <property name="poolPreparedStatements" value="true"/> </bean> </beans> -- View this message in context: http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10143322 Sent from the ActiveMQ - User mailing list archive at Nabble.com.