I tried the suggestion of going with the default cursor, but I still get OOM
errors. I've included my full config file below, I think I'm running fairly
vanilla/default.
After about 350k persistent messages, the logs start to look like:
INFO | Slow KahaDB access: Journal append took: 10 ms, Index update took 3118 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 5118 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 2736 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 2945 ms
INFO | Slow KahaDB access: Journal append took: 33 ms, Index update took 2654
ms
INFO | Slow KahaDB access: Journal append took: 82 ms, Index update took 3174
ms
INFO | Slow KahaDB access: Journal append took: 1 ms, Index update took 5891 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update took 2906 ms
INFO | Slow KahaDB access: Journal append took: 60 ms, Index update took 7619
ms
Exception in thread "InactivityMonitor WriteCheck" java.lang.OutOfMemoryError:
Java heap space
at java.util.jar.Attributes.read(Attributes.java:377)
at java.util.jar.Manifest.read(Manifest.java:182)
at java.util.jar.Manifest.<init>(Manifest.java:52)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
at java.util.jar.JarFile.getManifest(JarFile.java:146)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:221)
at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at
org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132)
at
org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106)
at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Config file:
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core";
brokerName="sub01chi" dataDirectory="${activemq.base}/data">
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue="P>"
producerFlowControl="true" memoryLimit="10mb"></policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
-----Original Message-----
From: Rob Davies [mailto:rajdav...@gmail.com]
Sent: Monday, January 18, 2010 10:42 PM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
> Hi,
>
> I'm running the 5.3 release as a standalone broker. In one case, a
> producer is running without a consumer, producing small, persistent
> messages, with the FileCursor pendingQueuePolicy (per
> https://issues.apache.org/activemq/browse/AMQ-2512)
> option and flow control memoryLimit set to 100mb for the queue in
> question. (Through a policy entry)
>
> As the queue grows above 300k messages, KahaDB indexing starts
> climbing above 1 second. At around 350k messages, the indexing is
> taking over 8 seconds. At this point, I start getting java out of
> heap space errors in essentially random parts of the code. After a
> while, the producers timeout with a channel inactive for too long
> error, and the entire broker basically wedges itself. At this point,
> consumers are generally unable to bind to the broker quitting with
> timeout errors. When they can connect, consuming a single message
> triggers an index re-build, which takes 2-8seconds. Turning on
> verbose garbage collection, the jvm is collecting like mad but
> reclaiming no space.
>
> If I restart the broker, it comes back up, I can consume the old
> messages, and can handle another 350k messages until it wedges.
>
> I can reproduce under both default gc and incremental gc.
>
> Two questions:
> - It seems like someone is holding onto a handle to the messages
> after they have been persisted to disk - is this a known issue?
> Should I open a JIRA for it? (Or is there another explanation?)
>
> - Is there any documentation about the internals of KahaDB - the
> kind of indices etc? I'd like to get a better understanding of the
> index performance and in general how KahaDB compares to something
> like BerkeleyDB.
>
> Thanks
>
>
>
There's is some confusion over naming of our persistence options that
doesn't help. There is Kaha - which uses multiple log files and a Hash
based index - this is currently used by the FileCursor - whilst KahaDB
is a newer implementation, which is more robust and typically uses a
BTreeIndex. There is currently a new implementation of the Filecursor
btw - but that's a different matter. You can't currently configure the
HashIndex via the FileCursor - but it looks like this is the problem
you are encountering - as it looks like you need to increase the max
hash buckets.
So I would recommend the following
1. Use the default pendingQueuePolicy (which only uses a FileCursor
for non-persistent messages - and uses the underlying database for
persistent messages
2. Try KahaDB - which - with the BTreeIndex - will not hit the
problems you are seeing with the Filecursor
or - increase the maximum number of hash buckets for the FileCursor
index - by setting a Java system property - maximumCapacity to 65536
(the default is 16384)
cheers,
Rob
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
