I tried the suggestion of going with the default cursor, but I
still get
OOM errors. I've included my full config file below, I think I'm
running
fairly vanilla/default.
After about 350k persistent messages, the logs start to look like:
INFO | Slow KahaDB access: Journal append took: 10 ms, Index
update took
3118 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
took
5118 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
took
2736 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
took
2945 ms
INFO | Slow KahaDB access: Journal append took: 33 ms, Index
update took
2654 ms
INFO | Slow KahaDB access: Journal append took: 82 ms, Index
update took
3174 ms
INFO | Slow KahaDB access: Journal append took: 1 ms, Index update
took
5891 ms
INFO | Slow KahaDB access: Journal append took: 0 ms, Index update
took
2906 ms
INFO | Slow KahaDB access: Journal append took: 60 ms, Index
update took
7619 ms
Exception in thread "InactivityMonitor WriteCheck"
java.lang.OutOfMemoryError: Java heap space
at java.util.jar.Attributes.read(Attributes.java:377)
at java.util.jar.Manifest.read(Manifest.java:182)
at java.util.jar.Manifest.<init>(Manifest.java:52)
at
java.util.jar.JarFile.getManifestFromReference(JarFile.java:165)
at java.util.jar.JarFile.getManifest(JarFile.java:146)
at
sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:
693)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:
221)
at java.net.URLClassLoader.access$000(URLClassLoader.java:
56)
at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
at java.security.AccessController.doPrivileged(Native
Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:
188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:
319)
at
org
.apache
.activemq
.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:
132)
at
org.apache.activemq.transport.InactivityMonitor
$2.run(InactivityMonitor.java:106)
at
org
.apache
.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:
33)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Config file:
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:amq="http://activemq.apache.org/schema/core";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
<bean
class
=
"org
.springframework
.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core";
brokerName="sub01chi" dataDirectory="${activemq.base}/data">
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue="P>"
producerFlowControl="true"
memoryLimit="10mb"></policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
-----Original Message-----
From: Rob Davies [mailto:rajdav...@gmail.com]
Sent: Monday, January 18, 2010 10:42 PM
To: users@activemq.apache.org
Subject: Re: OOM with high KahaDB index time
On 18 Jan 2010, at 22:14, Daniel Kluesing wrote:
Hi,
I'm running the 5.3 release as a standalone broker. In one
case, a
producer is running without a consumer, producing small,
persistent
messages, with the FileCursor pendingQueuePolicy (per
https://issues.apache.org/activemq/browse/AMQ-2512)
option and flow control memoryLimit set to 100mb for the queue in
question. (Through a policy entry)
As the queue grows above 300k messages, KahaDB indexing starts
climbing above 1 second. At around 350k messages, the indexing is
taking over 8 seconds. At this point, I start getting java out of
heap space errors in essentially random parts of the code.
After a
while, the producers timeout with a channel inactive for too long
error, and the entire broker basically wedges itself. At this
point,
consumers are generally unable to bind to the broker quitting
with
timeout errors. When they can connect, consuming a single message
triggers an index re-build, which takes 2-8seconds. Turning on
verbose garbage collection, the jvm is collecting like mad but
reclaiming no space.
If I restart the broker, it comes back up, I can consume the old
messages, and can handle another 350k messages until it wedges.
I can reproduce under both default gc and incremental gc.
Two questions:
- It seems like someone is holding onto a handle to the messages
after they have been persisted to disk - is this a known issue?
Should I open a JIRA for it? (Or is there another explanation?)
- Is there any documentation about the internals of KahaDB - the
kind of indices etc? I'd like to get a better understanding of
the
index performance and in general how KahaDB compares to something
like BerkeleyDB.
Thanks
There's is some confusion over naming of our persistence options
that
doesn't help. There is Kaha - which uses multiple log files and a
Hash
based index - this is currently used by the FileCursor - whilst
KahaDB
is a newer implementation, which is more robust and typically
uses a
BTreeIndex. There is currently a new implementation of the
Filecursor
btw - but that's a different matter. You can't currently configure
the
HashIndex via the FileCursor - but it looks like this is the
problem
you are encountering - as it looks like you need to increase the
max
hash buckets.
So I would recommend the following
1. Use the default pendingQueuePolicy (which only uses a
FileCursor
for non-persistent messages - and uses the underlying database for
persistent messages
2. Try KahaDB - which - with the BTreeIndex - will not hit the
problems you are seeing with the Filecursor
or - increase the maximum number of hash buckets for the
FileCursor
index - by setting a Java system property - maximumCapacity to
65536
(the default is 16384)
cheers,
Rob
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
