What is the correct number of file descriptors necessary to not have ActiveMQ fail in a production environment?
James.Strachan wrote: > > It might be you are just running out of threads; maybe your setting of > file descriptors per process is too low? > > > On 4/23/07, Justin C. van Vorst <[EMAIL PROTECTED]> wrote: >> >> On our production server this weekend, activemq-4.1.1 hit a >> java.lang.OutOfMemoryError exception and then proceeded to stop >> functioning. >> The process was using about 3GB of RAM. >> >> I've seen some other posts about OOME, but none that quite match the >> stack >> of ours. Does anyone have a place for me to start investigating this >> issue? >> >> Below are the error logs, command line start-up, and activemq.xml config: >> >> >> Error log: >> >> 2007-04-23 02:43:10,286 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_54813 >> 2007-04-23 02:49:10,415 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_58365 >> 2007-04-23 03:11:10,301 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_42109 >> 2007-04-23 03:15:10,056 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_44595 >> 2007-04-23 03:27:10,120 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51338 >> 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter >> - Failed to checkpoint a message store: >> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: >> java.lang.OutOfMemoryError: unable to create new native thread >> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: >> java.lang.OutOfMemoryError: unable to create new native thread >> at >> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299) >> at >> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) >> at >> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) >> Caused by: java.lang.OutOfMemoryError: unable to create new native thread >> at java.lang.Thread.start0(Native Method) >> at java.lang.Thread.start(Thread.java:574) >> at java.util.Timer.<init>(Timer.java:154) >> at java.util.Timer.<init>(Timer.java:122) >> at com.mysql.jdbc.Connection.<init>(Connection.java:1441) >> at >> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) >> at >> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37) >> at >> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290) >> at >> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771) >> at >> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) >> at >> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:544) >> at >> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:55) >> at >> org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:147) >> at >> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189) >> at >> org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41) >> at >> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247) >> at >> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356) >> at >> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176) >> at >> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) >> at >> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) >> at java.lang.Thread.run(Thread.java:595) >> 2007-04-23 03:44:31,858 [eckpoint Worker] ERROR JournalPersistenceAdapter >> - Failed to mark the Journal: >> org.apache.activeio.journal.InvalidRecordLocationException: The location >> is >> less than the last mark. >> org.apache.activeio.journal.InvalidRecordLocationException: The location >> is >> less than the last mark. >> at >> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) >> at >> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) >> 2007-04-23 03:47:10,105 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_34348 >> 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter >> - Failed to checkpoint a message store: >> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: >> java.io.IOException: Already started. >> edu.emory.mathcs.backport.java.util.concurrent.ExecutionException: >> java.io.IOException: Already started. >> at >> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.getResult(FutureTask.java:299) >> at >> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.get(FutureTask.java:118) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:386) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) >> at >> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) >> Caused by: java.io.IOException: Already started. >> at >> org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:145) >> at >> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:358) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:189) >> at >> org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41) >> at >> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:247) >> at >> org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:221) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:356) >> at >> edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176) >> at >> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) >> at >> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) >> at java.lang.Thread.run(Thread.java:595) >> 2007-04-23 03:50:01,821 [eckpoint Worker] ERROR JournalPersistenceAdapter >> - Failed to mark the Journal: >> org.apache.activeio.journal.InvalidRecordLocationException: The location >> is >> less than the last mark. >> org.apache.activeio.journal.InvalidRecordLocationException: The location >> is >> less than the last mark. >> at >> org.apache.activeio.journal.active.JournalImpl.setMark(JournalImpl.java:340) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:403) >> at >> org.apache.activemq.store.journal.JournalPersistenceAdapter$2.iterate(JournalPersistenceAdapter.java:129) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:101) >> at >> org.apache.activemq.thread.DedicatedTaskRunner.access$000(DedicatedTaskRunner.java:25) >> at >> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:39) >> 2007-04-23 03:51:10,188 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_36436 >> 2007-04-23 04:13:10,134 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_49022 >> 2007-04-23 04:17:10,094 [localhost:61613] WARN >> ManagedTransportConnection >> - Failed to register MBean: >> org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=stomp,ViewType=address,Name=/x.x.x.x_51068 >> >> >> >> Command line: >> >> usr/java/current/bin/java -Xmx512M >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true >> -Dderby.system.home=/opt/activemq-4.1.1/data >> -Dderby.storage.fileSyncTransactionLog=true >> -Dcom.sun.management.jmxremote >> -Djavax.net.ssl.keyStorePassword=password >> -Djavax.net.ssl.trustStorePassword=password >> -Djavax.net.ssl.keyStore=/opt/activemq-4.1.1/conf/broker.ks >> -Djavax.net.ssl.trustStore=/opt/activemq-4.1.1/conf/broker.ts -classpath >> -Dactivemq.home=/opt/rhg/activemq-4.1.1 >> -Dactivemq.base=/opt/activemq-4.1.1 >> -jar /opt/activemq-4.1.1/bin/run.jar >> >> >> >> activemq.xml: >> >> <beans> >> >> <bean >> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/> >> >> <broker brokerName="localhost" useJmx="true" >> xmlns="http://activemq.org/config/1.0";> >> >> <destinationPolicy> >> <policyMap><policyEntries> >> <policyEntry topic="FOO.>"> >> <dispatchPolicy> >> <strictOrderDispatchPolicy /> >> </dispatchPolicy> >> <subscriptionRecoveryPolicy> >> <lastImageSubscriptionRecoveryPolicy /> >> </subscriptionRecoveryPolicy> >> </policyEntry> >> </policyEntries></policyMap> >> </destinationPolicy> >> >> <persistenceAdapter> >> <journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data" >> dataSource="#mysql-ds"/> >> </persistenceAdapter> >> >> <transportConnectors> >> <transportConnector name="openwire" uri="tcp://localhost:61616" >> discoveryUri="multicast://default"/> >> <transportConnector name="ssl" uri="ssl://localhost:61617"/> >> <transportConnector name="stomp" uri="stomp://localhost:61613"/> >> </transportConnectors> >> >> <networkConnectors> >> <networkConnector name="default-nc" uri="multicast://default"/> >> </networkConnectors> >> >> </broker> >> >> <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" >> destroy-method="close"> >> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> >> <property name="url" >> value="jdbc:mysql://x.x.x.x/activemq?relaxAutoCommit=true"/> >> <property name="username" value="username"/> >> <property name="password" value="password"/> >> <property name="poolPreparedStatements" value="true"/> >> </bean> >> >> </beans> >> -- >> View this message in context: >> http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10143322 >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > > > -- > James > ------- > http://macstrac.blogspot.com/ > > -- View this message in context: http://www.nabble.com/OutOfMemoryError-using-4.1.1-tf3632613s2354.html#a10359823 Sent from the ActiveMQ - User mailing list archive at Nabble.com.
