I shall test with disable multicast dicovery. Thanks. On production we are seeing below exception. Is it also related multicast discovery or Inactity monitor? I am also turning off inactivity monitor thread with maxInactivityDuration=0: <transportConnector name="openwire" uri="tcp://localhost:61616?wireFormat.maxInactivityDuration=0"
JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError". Exception in thread "DefaultMessageListenerContainer-359522" Exception in thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError: Failed to fork OS thread java.lang.OutOfMemoryError: Failed to fork OS thread at java.lang.Thread.startImpl(Native Method) at java.lang.Thread.startImpl(Native Method) at java.lang.Thread.start(Thread.java:970) at java.lang.Thread.start(Thread.java:970) at org.springframework.core.task.SimpleAsyncTaskExecutor.doExecute(SimpleAsyncTaskExecutor.java:157) Gary Tully wrote: > > The networkConnector is using multicast discovery - seeking brokers to > network with. > > The transportConnector is publishing its endpoint over multicast so making > it available to other brokers who wish to connect with it. > > AMQ-2283 is related to the networkConnector, so you should use a static > url > if you still want to create a network. > > 2009/10/17 tejeswara <tejesw...@gmail.com> > >> >> Gary, >> >> In my config file I have multicast in both these entries. Which one you >> think possibly causing this memory leak?. We plan to have failover broker >> so >> we may need multicast discovery?. >> >> <networkConnectors> >> <networkConnector name="default" uri="multicast://default"/> >> </networkConnectors> >> >> <transportConnectors> >> <transportConnector name="default" uri="tcp://localhost:61616" >> discoveryUri="multicast://default"/> >> </transportConnectors> >> >> >> >> >> >> Gary Tully wrote: >> > >> > the config looks ok, just one thought, do you need multicast discovery >> on >> > your transport connector? I wonder if it is possible you are seeing >> > https://issues.apache.org/activemq/browse/AMQ-2283 which resulted in an >> > leak >> > of discovery threads? Removing the discoveryUri will work around it. >> > >> > 2009/10/13 tejeswara <tejesw...@gmail.com> >> > >> >> >> >> Here is our activemq.xml. As you can see it is pretty much same as the >> >> template version we get from activemq package: >> >> <beans >> >> xmlns="http://www.springframework.org/schema/beans"; >> >> xmlns:amq="http://activemq.apache.org/schema/core"; >> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> >> http://activemq.apache.org/schema/core >> >> http://activemq.apache.org/schema/core/activemq-core.xsd >> >> http://activemq.apache.org/camel/schema/spring >> >> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> >> >> <bean >> >> >> >> >> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> >> >> <property name="locations"> >> >> >> >> <value>file:///${activemq.base}/conf/credentials.properties</value> >> >> </property> >> >> </bean> >> >> <broker xmlns="http://activemq.apache.org/schema/core"; >> >> brokerName="localhost" dataDirectory="${activemq.base}/data"> >> >> <destinationPolicy> >> >> <policyMap> >> >> <policyEntries> >> >> <policyEntry queue=">" memoryLimit="50mb"/> >> >> <policyEntry topic=">" memoryLimit="5mb"> >> >> </policyEntry> >> >> </policyEntries> >> >> </policyMap> >> >> </destinationPolicy> >> >> <managementContext> >> >> <managementContext createConnector="true" >> >> connectorPort="11099"/> >> >> </managementContext> >> >> <networkConnectors> >> >> <networkConnector name="default-nc" >> >> uri="multicast://default"/> >> >> </networkConnectors> >> >> <persistenceAdapter> >> >> <amqPersistenceAdapter syncOnWrite="false" >> >> directory="${activemq.base}/data" maxFileLength="20 mb"/> >> >> </persistenceAdapter> >> >> <systemUsage> >> >> <systemUsage> >> >> <memoryUsage> >> >> <memoryUsage limit="200 mb"/> >> >> </memoryUsage> >> >> <storeUsage> >> >> <storeUsage limit="1 gb" name="foo"/> >> >> </storeUsage> >> >> <tempUsage> >> >> <tempUsage limit="100 mb"/> >> >> </tempUsage> >> >> </systemUsage> >> >> </systemUsage> >> >> <transportConnectors> >> >> <transportConnector name="openwire" >> >> uri="tcp://localhost:61616" >> >> discoveryUri="multicast://default"/> >> >> </transportConnectors> >> >> </broker> >> >> <camelContext id="camel" >> >> xmlns="http://activemq.apache.org/camel/schema/spring";> >> >> <package>org.foo.bar</package> >> >> <route> >> >> <from uri="activemq:example.A"/> >> >> <to uri="activemq:example.B"/> >> >> </route> >> >> </camelContext> >> >> <bean id="activemq" >> >> class="org.apache.activemq.camel.component.ActiveMQComponent" > >> >> <property name="connectionFactory"> >> >> <bean class="org.apache.activemq.ActiveMQConnectionFactory"> >> >> <property name="brokerURL" >> >> value="vm://localhost?create=false&waitForStart=10000" /> >> >> <property name="userName" value="${activemq.username}"/> >> >> <property name="password" value="${activemq.password}"/> >> >> </bean> >> >> </property> >> >> </bean> >> >> <jetty xmlns="http://mortbay.com/schemas/jetty/1.0";> >> >> <connectors> >> >> <nioConnector port="11098"/> >> >> </connectors> >> >> <handlers> >> >> <webAppContext contextPath="/admin" >> >> resourceBase="${activemq.base}/webapps/admin" logUrlOnStart="true"/> >> >> <webAppContext contextPath="/demo" >> >> resourceBase="${activemq.base}/webapps/demo" logUrlOnStart="true"/> >> >> <webAppContext contextPath="/fileserver" >> >> resourceBase="${activemq.base}/webapps/fileserver" >> logUrlOnStart="true"/> >> >> </handlers> >> >> </jetty> >> >> </beans> >> >> >> >> >> >> >> >> Gary Tully wrote: >> >> > >> >> > Can you post your activemq.xml? It looks like you need to restrict >> the >> >> > memory usage of the broker and have it spool messages to disk when >> the >> >> > memory limits are reached. See some additional detail at >> >> > >> >> >> http://activemq.apache.org/javalangoutofmemory.html#java.lang.OutOfMemory-SpoolingMessagestoDisk >> >> > * >> >> > >> >> > * >> >> > 2009/10/12 tejeswara <tejesw...@gmail.com> >> >> > >> >> >> >> >> >> Our broker is running on Linux (RHEL4 box) and after 1 or 2 days I >> see >> >> >> below >> >> >> error in logs. >> >> >> Note that, below values are set in bin/activemq in out test lab: >> >> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then >> >> >> ACTIVEMQ_OPTS="-Xmx512M >> >> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=true" >> >> >> fi >> >> >> >> >> >> I tried to stress test broker by creating 200 queues and put 5000 >> >> >> messages >> >> >> (each about 1500 bytes size) >> >> >> in each of these queues. However I could not see any error. Below >> >> error >> >> >> occured when broker is idle. >> >> >> From logs, only activity during this idle period I see is 'DEBUG >> >> >> InactivityMonitor - 30177 ms elapsed since last read >> >> check' >> >> >> >> >> >> Can someone help in pointing out what is going wrong? >> >> >> >> >> >> Note that, OutOfMemoryError is seen in our production even when >> heap >> >> size >> >> >> is >> >> >> bumped up to 1024 and >> >> >> dedicated task runner is set to false. This error usvally occurs >> after >> >> 2 >> >> >> days broker running without restart. >> >> >> if [ -z "$ACTIVEMQ_OPTS" ] ; then >> >> >> ACTIVEMQ_OPTS="-Xmx1024M >> >> >> -Dorg.apache.activemq.UseDedicatedTaskRunner=false" >> >> >> fi >> >> >> >> >> >> >> >> >> Stack Trace:-> >> >> >> >> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail >> >> >> "java/lang/OutOfMemoryError" - Please Wait. >> >> >> JVMDUMP007I JVM Requesting Snap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc' >> >> >> JVMDUMP010I Snap Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0001.20091010.015439.16581.trc >> >> >> JVMDUMP007I JVM Requesting Heap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd' >> >> >> JVMDUMP010I Heap Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.015439.16581.phd >> >> >> JVMDUMP007I JVM Requesting Java Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt' >> >> >> JVMDUMP010I Java Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.015439.16581.txt >> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail >> >> >> "java/lang/OutOfMemoryError". >> >> >> Exception in thread "MulticastDiscovery: null" >> >> >> java.lang.OutOfMemoryError: >> >> >> heap allocation failed >> >> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method) >> >> >> at >> >> >> >> >> >> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181) >> >> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338) >> >> >> at java.lang.Thread.run(Thread.java:803) >> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail >> >> >> "java/lang/OutOfMemoryError" - Please Wait. >> >> >> JVMDUMP007I JVM Requesting Snap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc' >> >> >> JVMDUMP010I Snap Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0002.20091010.034616.16581.trc >> >> >> JVMDUMP007I JVM Requesting Heap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd' >> >> >> JVMDUMP016E Aborting: Cannot create file >> >> >> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034616.16581.phd) >> >> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written >> >> >> JVMDUMP007I JVM Requesting Java Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt' >> >> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer >> >> >> JVMDUMP010I Java Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034616.16581.txt >> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail >> >> >> "java/lang/OutOfMemoryError". >> >> >> Exception in thread "MulticastDiscovery: tcp://stage2sc5464.:61616" >> >> >> java.lang.OutOfMemoryError: heap allocation failed >> >> >> at java.net.PlainDatagramSocketImpl.receive0(Native Method) >> >> >> at >> >> >> >> >> >> java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:181) >> >> >> at java.net.DatagramSocket.receive(DatagramSocket.java:724) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.activemq.transport.discovery.multicast.MulticastDiscoveryAgent.run(MulticastDiscoveryAgent.java:338) >> >> >> at java.lang.Thread.run(Thread.java:803) >> >> >> JVMDUMP006I Processing Dump Event "uncaught", detail >> >> >> "java/lang/OutOfMemoryError" - Please Wait. >> >> >> JVMDUMP007I JVM Requesting Snap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc' >> >> >> JVMDUMP010I Snap Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0003.20091010.034640.16581.trc >> >> >> JVMDUMP007I JVM Requesting Heap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd' >> >> >> JVMDUMP016E Aborting: Cannot create file >> >> >> >> (/x/web/STAGE2SC5464/idimqbroker/heapdump.20091010.034640.16581.phd) >> >> >> JVMDUMP012E Error in Heap Dump: binary dump file cannot be written >> >> >> JVMDUMP007I JVM Requesting Java Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt' >> >> >> UTE503: utsGetTracePointIterator cannot allocate iterator's buffer >> >> >> JVMDUMP010I Java Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/javacore.20091010.034640.16581.txt >> >> >> JVMDUMP013I Processed Dump Event "uncaught", detail >> >> >> "java/lang/OutOfMemoryError". >> >> >> Exception in thread "InactivityMonitor WriteCheck" >> >> >> java.lang.OutOfMemoryError >> >> >> at >> java.security.AccessController.getProtectionDomains(Native >> >> >> Method) >> >> >> at >> >> >> >> java.security.AccessController.getContext(AccessController.java:132) >> >> >> at java.lang.Thread.initialize(Thread.java:347) >> >> >> at java.lang.Thread.<init>(Thread.java:278) >> >> >> at java.lang.Thread.<init>(Thread.java:170) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.activemq.transport.InactivityMonitor$5.newThread(InactivityMonitor.java:303) >> >> >> at >> >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:411) >> >> >> at >> >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:463) >> >> >> at >> >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:878) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:128) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:103) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) >> >> >> at java.util.TimerThread.mainLoop(Timer.java:527) >> >> >> at java.util.TimerThread.run(Timer.java:477) >> >> >> Exception in thread "RMI TCP Connection(454)-10.15.26.210" >> >> >> java.lang.NoClassDefFoundError: java.io.StreamCorruptedException >> >> >> at >> >> >> >> >> >> >> >> >> sun.rmi.transport.StreamRemoteCall.getResultStream(StreamRemoteCall.java:177) >> >> >> at >> >> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:335) >> >> >> at sun.rmi.transport.Transport$1.run(Transport.java:168) >> >> >> at >> >> >> >> java.security.AccessController.doPrivileged(AccessController.java:275) >> >> >> java.lang.NoClassDefFoundError: java.rmi.UnmarshalException >> >> >> at >> >> >> >> >> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506) >> >> >> at >> >> >> >> >> >> >> >> >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838) >> >> >> at >> >> >> >> >> >> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233) >> >> >> at >> >> >> >> >> >> >> >> >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912) >> >> >> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:358) >> >> >> at java.lang.Thread.run(Thread.java:803) >> >> >> at >> sun.rmi.transport.DGCImpl_Stub.dirty(DGCImpl_Stub.java:67) >> >> >> at >> >> >> >> >> >> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:344) >> >> >> at >> >> >> >> >> >> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:160) >> >> >> at >> >> >> >> >> >> >> >> >> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555) >> >> >> at java.lang.Thread.run(Thread.java:803) >> >> >> Exception in thread "RMI LeaseChecker" >> java.lang.NoClassDefFoundError: >> >> >> java.rmi.server.Unreferenced >> >> >> at sun.rmi.transport.Target.refSetRemove(Target.java:320) >> >> >> at sun.rmi.transport.Target.vmidDead(Target.java:444) >> >> >> at sun.rmi.transport.DGCImpl.checkLeases(DGCImpl.java:265) >> >> >> at sun.rmi.transport.DGCImpl.access$400(DGCImpl.java:47) >> >> >> at >> sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:329) >> >> >> at java.lang.Thread.run(Thread.java:803) >> >> >> Unhandled exception >> >> >> Type=Segmentation error vmState=0x00053fff >> >> >> J9Generic_Signal_Number=00000004 Signal_Number=0000000b >> >> >> Error_Value=00000000 >> >> >> Signal_Code=00000001 >> >> >> Handler1=008F64F0 Handler2=00142477 InaccessibleAddress=00000000 >> >> >> EDI=00000000 ESI=27E112E0 EAX=00000000 EBX=00000000 >> >> >> ECX=08B7DF40 EDX=00000000 >> >> >> EIP=00F97DB4 ES=0000007B DS=C02D007B ESP=00855BB8 >> >> >> EFlags=00210206 CS=00000073 SS=0000007B EBP=00855BB8 >> >> >> Module=/x/web/STAGE2SC5464/idimqbroker/java/jre/bin/libj9jit23.so >> >> >> Module_base_address=00F09000 >> >> >> >> >> >> >> >> >> >> >> >> Method_being_compiled=org/springframework/jms/listener/DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run()V >> >> >> Target=2_30_20070524_12771_lHdSMR (Linux 2.6.9-55.ELsmp) >> >> >> CPU=x86 (8 logical CPUs) (0x47621d000 RAM) >> >> >> JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait. >> >> >> JVMDUMP007I JVM Requesting System Dump using >> >> >> '/x/web/STAGE2SC5464/idimqbroker/core.20091011.220932.16581.dmp' >> >> >> JVMDUMP012E Error in System Dump: cannot find core file, check >> "ulimit >> >> >> -c" >> >> >> is set high enough >> >> >> JVMDUMP007I JVM Requesting Snap Dump using >> >> >> >> '/x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc' >> >> >> JVMDUMP010I Snap Dump written to >> >> >> /x/web/STAGE2SC5464/idimqbroker/Snap0004.20091011.220932.16581.trc >> >> >> JVMDUMP007I JVM Requesting Java Dump using >> >> >> '/tmp/javacore.20091011.220932.16581.txt' >> >> >> NULL >> >> >> >> >> >> ------------------------------------------------------------------------ >> >> >> 0SECTION TITLE subcomponent dump routine >> >> >> NULL =============================== >> >> >> 1TISIGINFO Dump Event "gpf" (00002000) received >> >> >> 1TIDATETIME Date: 2009/10/11 at 22:10:17 >> >> >> 1TIFILENAME Javacore filename: >> >> >> /tmp/javacore.20091011.220932.16581.txt >> >> >> NULL >> >> >> >> >> >> ------------------------------------------------------------------------ >> >> >> 0SECTION GPINFO subcomponent dump routine >> >> >> NULL ================================ >> >> >> 2XHOSLEVEL OS Level : Linux 2.6.9-55.ELsmp >> >> >> 2XHCPUS Processors - >> >> >> 3XHCPUARCH Architecture : x86 >> >> >> 3XHNUMCPUS How Many : 8 >> >> >> NULL >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> View this message in context: >> >> >> >> >> >> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25864372.html >> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >> >> >> >> >> >> > >> >> > >> >> > -- >> >> > http://blog.garytully.com >> >> > >> >> > Open Source Integration >> >> > http://fusesource.com >> >> > >> >> > >> >> >> >> -- >> >> View this message in context: >> >> >> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25881255.html >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> >> >> >> > >> > >> > -- >> > http://blog.garytully.com >> > >> > Open Source Integration >> > http://fusesource.com >> > >> > >> >> -- >> View this message in context: >> http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25935327.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > > > -- > http://blog.garytully.com > > Open Source Integration > http://fusesource.com > > -- View this message in context: http://www.nabble.com/java-lang-OutOfMemoryError-tp25864372p25959853.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
