I ran my 5.3 test with the following <systemUsage> <systemUsage sendFailIfNoSpace="true"> <memoryUsage> <memoryUsage limit="100mb"/> </memoryUsage> <storeUsage> <storeUsage limit="500 mb" name="foo"/> </storeUsage> <tempUsage> <tempUsage limit="100mb"/> </tempUsage> </systemUsage> </systemUsage>
...and had my producer fill up the store; it tried to pump 400k messages each at 2k in size. Producer flow control was also enabled. The broker did not hurl any OOM exceptions, "just javax.jms.ResourceAllocationException: Usage Manager Store is Full. Stopping producer..." exceptions as expected. The producer got kicked off its connection. The only time I got OOMs was when I used this. <systemUsage> <systemUsage sendFailIfNoSpace="true"> </systemUsage> </systemUsage> Joe Daniel Kluesing-2 wrote: > > So I'm not sure if that's what Rob is talking about being fixed in 5.4 > (And I'll try the snapshot as soon as it's ready) but if I don't have the > sendFailIfNoSpace then my understanding is the producers send calls > block/wait/timeout - as opposed to fail - so it's more difficult to get > into a HA configuration. It's a minor point, not having an OOM is much > more important, but I definitely want the send calls to fail for the > producer if the broker ever does anything funny. > > Thanks for the feedback on the config, very helpful. > > -----Original Message----- > From: Joe Fernandez [mailto:joe.fernan...@ttmsolutions.com] > Sent: Thursday, January 21, 2010 1:01 PM > To: users@activemq.apache.org > Subject: RE: OOM with high KahaDB index time > > > Just for grins, I threw your cfg file into our 5.3 testbed and sure > enough, > we got the OOMs; I pumped 200k messages with each being 2k in size. FWIW, > taking this out of the cfg file made things run a lot better. > > <systemUsage> > <systemUsage sendFailIfNoSpace="true"> > </systemUsage> > </systemUsage> > > With the above taken out of the cfg file, I was able to pump 400k messages > into the broker, no OOMs and memory utilization looked much better. I also > gave a fully-defined systemUsage a try and that also appeared to do the > trick. > > <systemUsage> > <systemUsage> > <memoryUsage> > <memoryUsage limit="100 mb"/> > </memoryUsage> > <storeUsage> > <storeUsage limit="1 gb" name="foo"/> > </storeUsage> > <tempUsage> > <tempUsage limit="100 mb"/> > </tempUsage> > </systemUsage> > </systemUsage> > > So may be worth giving it a whirl if you can't scoot over to the trunk and > ride Rob's patch. > > Joe > > > Daniel Kluesing-2 wrote: >> >> I tried the suggestion of going with the default cursor, but I still get >> OOM errors. I've included my full config file below, I think I'm running >> fairly vanilla/default. >> >> After about 350k persistent messages, the logs start to look like: >> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took >> 3118 ms >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took >> 5118 ms >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took >> 2736 ms >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took >> 2945 ms >> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took >> 2654 ms >> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took >> 3174 ms >> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took >> 5891 ms >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took >> 2906 ms >> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took >> 7619 ms >> Exception in thread "InactivityMonitor WriteCheck" >> java.lang.OutOfMemoryError: Java heap space >> at java.util.jar.Attributes.read(Attributes.java:377) >> at java.util.jar.Manifest.read(Manifest.java:182) >> at java.util.jar.Manifest.<init>(Manifest.java:52) >> at >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165) >> at java.util.jar.JarFile.getManifest(JarFile.java:146) >> at >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:221) >> at java.net.URLClassLoader.access$000(URLClassLoader.java:56) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:195) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:251) >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) >> at >> org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132) >> at >> org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106) >> at >> org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) >> at java.util.TimerThread.mainLoop(Timer.java:512) >> at java.util.TimerThread.run(Timer.java:462) >> >> Config file: >> >> <beans >> xmlns="http://www.springframework.org/schema/beans" >> xmlns:amq="http://activemq.apache.org/schema/core" >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >> xsi:schemaLocation="http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd >> http://activemq.apache.org/schema/core >> http://activemq.apache.org/schema/core/activemq-core.xsd"> >> >> <bean >> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> >> <property name="locations"> >> >> <value>file:${activemq.base}/conf/credentials.properties</value> >> </property> >> </bean> >> >> <broker xmlns="http://activemq.apache.org/schema/core" >> brokerName="sub01chi" dataDirectory="${activemq.base}/data"> >> >> <managementContext> >> <managementContext createConnector="true"/> >> </managementContext> >> >> <persistenceAdapter> >> <kahaDB directory="${activemq.base}/data/kahadb"/> >> </persistenceAdapter> >> >> <destinationPolicy> >> <policyMap> >> <policyEntries> >> <policyEntry queue="P>" >> producerFlowControl="true" >> memoryLimit="10mb"></policyEntry> >> </policyEntries> >> </policyMap> >> </destinationPolicy> >> <systemUsage> >> <systemUsage sendFailIfNoSpace="true"> >> </systemUsage> >> </systemUsage> >> <transportConnectors> >> <transportConnector name="openwire" >> uri="tcp://0.0.0.0:61616"/> >> </transportConnectors> >> </broker> >> <import resource="jetty.xml"/> >> </beans> >> >> -----Original Message----- >> From: Rob Davies [mailto:rajdav...@gmail.com] >> Sent: Monday, January 18, 2010 10:42 PM >> To: users@activemq.apache.org >> Subject: Re: OOM with high KahaDB index time >> >> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote: >> >>> Hi, >>> >>> I'm running the 5.3 release as a standalone broker. In one case, a >>> producer is running without a consumer, producing small, persistent >>> messages, with the FileCursor pendingQueuePolicy (per >>> https://issues.apache.org/activemq/browse/AMQ-2512) >>> option and flow control memoryLimit set to 100mb for the queue in >>> question. (Through a policy entry) >>> >>> As the queue grows above 300k messages, KahaDB indexing starts >>> climbing above 1 second. At around 350k messages, the indexing is >>> taking over 8 seconds. At this point, I start getting java out of >>> heap space errors in essentially random parts of the code. After a >>> while, the producers timeout with a channel inactive for too long >>> error, and the entire broker basically wedges itself. At this point, >>> consumers are generally unable to bind to the broker quitting with >>> timeout errors. When they can connect, consuming a single message >>> triggers an index re-build, which takes 2-8seconds. Turning on >>> verbose garbage collection, the jvm is collecting like mad but >>> reclaiming no space. >>> >>> If I restart the broker, it comes back up, I can consume the old >>> messages, and can handle another 350k messages until it wedges. >>> >>> I can reproduce under both default gc and incremental gc. >>> >>> Two questions: >>> - It seems like someone is holding onto a handle to the messages >>> after they have been persisted to disk - is this a known issue? >>> Should I open a JIRA for it? (Or is there another explanation?) >>> >>> - Is there any documentation about the internals of KahaDB - the >>> kind of indices etc? I'd like to get a better understanding of the >>> index performance and in general how KahaDB compares to something >>> like BerkeleyDB. >>> >>> Thanks >>> >>> >>> >> >> >> There's is some confusion over naming of our persistence options that >> doesn't help. There is Kaha - which uses multiple log files and a Hash >> based index - this is currently used by the FileCursor - whilst KahaDB >> is a newer implementation, which is more robust and typically uses a >> BTreeIndex. There is currently a new implementation of the Filecursor >> btw - but that's a different matter. You can't currently configure the >> HashIndex via the FileCursor - but it looks like this is the problem >> you are encountering - as it looks like you need to increase the max >> hash buckets. >> >> >> So I would recommend the following >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor >> for non-persistent messages - and uses the underlying database for >> persistent messages >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the >> problems you are seeing with the Filecursor >> >> or - increase the maximum number of hash buckets for the FileCursor >> index - by setting a Java system property - maximumCapacity to 65536 >> (the default is 16384) >> >> cheers, >> >> Rob >> >> http://twitter.com/rajdavies >> I work here: http://fusesource.com >> My Blog: http://rajdavies.blogspot.com/ >> I'm writing this: http://www.manning.com/snyder/ >> >> >> >> >> >> >> > > -- > View this message in context: > http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > -- View this message in context: http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.