Thanks Joe... no I'm still on 5.3 (cannot go 5.4 SNAPSHOT right now), and wanted to get the best out of the currently available version.
I'll give it a try. Cheers, F. On Mon, Jan 25, 2010 at 4:14 PM, Joe Fernandez < [email protected]> wrote: > > Here it is - its the xml that Dan posted with some slight mod's. Are you > using the latest trunk and not the Jan 20 5.4 SNAPSHOT? > > <beans > xmlns="http://www.springframework.org/schema/beans"; > xmlns:amq="http://activemq.apache.org/schema/core"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > http://activemq.apache.org/schema/core > http://activemq.apache.org/schema/core/activemq-core.xsd";> > > <bean > > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > <property name="locations"> > <value>file:${activemq.base}/conf/credentials.properties</value> > </property> > </bean> > > <broker xmlns="http://activemq.apache.org/schema/core"; > brokerName="danbroker" dataDirectory="${activemq.base}/data"> > > <managementContext> > <managementContext createConnector="true"/> > </managementContext> > > <persistenceAdapter> > <kahaDB directory="${activemq.base}/data/kahadb"/> > </persistenceAdapter> > > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" producerFlowControl="true" > memoryLimit="10mb"> > </policyEntry> > </policyEntries> > </policyMap> > </destinationPolicy> > > > <systemUsage> > <systemUsage sendFailIfNoSpace="true"> > </systemUsage> > </systemUsage> > > > <transportConnectors> > <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/> > </transportConnectors> > </broker> > <import resource="jetty.xml"/> > </beans> > > > > > Fred Moore-3 wrote: > > > > Hi Joe, > > > > can you post your whole activemq.xml for this "good enough" test on 5.3? > > > > ...I'm still missing something and getting OOMs. > > TIA, > > F. > > > > > > On Fri, Jan 22, 2010 at 9:07 PM, Joe Fernandez < > > [email protected]> wrote: > > > >> > >> I ran my 5.3 test with the following > >> > >> <systemUsage> > >> <systemUsage sendFailIfNoSpace="true"> > >> <memoryUsage> > >> <memoryUsage limit="100mb"/> > >> </memoryUsage> > >> <storeUsage> > >> <storeUsage limit="500 mb" name="foo"/> > >> </storeUsage> > >> <tempUsage> > >> <tempUsage limit="100mb"/> > >> </tempUsage> > >> </systemUsage> > >> </systemUsage> > >> > >> ...and had my producer fill up the store; it tried to pump 400k messages > >> each at 2k in size. Producer flow control was also enabled. The broker > >> did > >> not hurl any OOM exceptions, "just > javax.jms.ResourceAllocationException: > >> Usage Manager Store is Full. Stopping producer..." exceptions as > >> expected. > >> The producer got kicked off its connection. > >> > >> The only time I got OOMs was when I used this. > >> > >> <systemUsage> > >> <systemUsage sendFailIfNoSpace="true"> > >> </systemUsage> > >> </systemUsage> > >> > >> Joe > >> > >> > >> > >> Daniel Kluesing-2 wrote: > >> > > >> > So I'm not sure if that's what Rob is talking about being fixed in 5.4 > >> > (And I'll try the snapshot as soon as it's ready) but if I don't have > >> the > >> > sendFailIfNoSpace then my understanding is the producers send calls > >> > block/wait/timeout - as opposed to fail - so it's more difficult to > get > >> > into a HA configuration. It's a minor point, not having an OOM is much > >> > more important, but I definitely want the send calls to fail for the > >> > producer if the broker ever does anything funny. > >> > > >> > Thanks for the feedback on the config, very helpful. > >> > > >> > -----Original Message----- > >> > From: Joe Fernandez [mailto:[email protected]] > >> > Sent: Thursday, January 21, 2010 1:01 PM > >> > To: [email protected] > >> > Subject: RE: OOM with high KahaDB index time > >> > > >> > > >> > Just for grins, I threw your cfg file into our 5.3 testbed and sure > >> > enough, > >> > we got the OOMs; I pumped 200k messages with each being 2k in size. > >> FWIW, > >> > taking this out of the cfg file made things run a lot better. > >> > > >> > <systemUsage> > >> > <systemUsage sendFailIfNoSpace="true"> > >> > </systemUsage> > >> > </systemUsage> > >> > > >> > With the above taken out of the cfg file, I was able to pump 400k > >> messages > >> > into the broker, no OOMs and memory utilization looked much better. I > >> also > >> > gave a fully-defined systemUsage a try and that also appeared to do > the > >> > trick. > >> > > >> > <systemUsage> > >> > <systemUsage> > >> > <memoryUsage> > >> > <memoryUsage limit="100 mb"/> > >> > </memoryUsage> > >> > <storeUsage> > >> > <storeUsage limit="1 gb" name="foo"/> > >> > </storeUsage> > >> > <tempUsage> > >> > <tempUsage limit="100 mb"/> > >> > </tempUsage> > >> > </systemUsage> > >> > </systemUsage> > >> > > >> > So may be worth giving it a whirl if you can't scoot over to the trunk > >> and > >> > ride Rob's patch. > >> > > >> > Joe > >> > > >> > > >> > Daniel Kluesing-2 wrote: > >> >> > >> >> I tried the suggestion of going with the default cursor, but I still > >> get > >> >> OOM errors. I've included my full config file below, I think I'm > >> running > >> >> fairly vanilla/default. > >> >> > >> >> After about 350k persistent messages, the logs start to look like: > >> >> > >> >> INFO | Slow KahaDB access: Journal append took: 10 ms, Index update > >> took > >> >> 3118 ms > >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update > >> took > >> >> 5118 ms > >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update > >> took > >> >> 2736 ms > >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update > >> took > >> >> 2945 ms > >> >> INFO | Slow KahaDB access: Journal append took: 33 ms, Index update > >> took > >> >> 2654 ms > >> >> INFO | Slow KahaDB access: Journal append took: 82 ms, Index update > >> took > >> >> 3174 ms > >> >> INFO | Slow KahaDB access: Journal append took: 1 ms, Index update > >> took > >> >> 5891 ms > >> >> INFO | Slow KahaDB access: Journal append took: 0 ms, Index update > >> took > >> >> 2906 ms > >> >> INFO | Slow KahaDB access: Journal append took: 60 ms, Index update > >> took > >> >> 7619 ms > >> >> Exception in thread "InactivityMonitor WriteCheck" > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> at java.util.jar.Attributes.read(Attributes.java:377) > >> >> at java.util.jar.Manifest.read(Manifest.java:182) > >> >> at java.util.jar.Manifest.<init>(Manifest.java:52) > >> >> at > >> >> java.util.jar.JarFile.getManifestFromReference(JarFile.java:165) > >> >> at java.util.jar.JarFile.getManifest(JarFile.java:146) > >> >> at > >> >> sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:693) > >> >> at > >> java.net.URLClassLoader.defineClass(URLClassLoader.java:221) > >> >> at java.net.URLClassLoader.access$000(URLClassLoader.java:56) > >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:195) > >> >> at java.security.AccessController.doPrivileged(Native Method) > >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > >> >> at > >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > >> >> at > >> >> > >> > org.apache.activemq.transport.InactivityMonitor.writeCheck(InactivityMonitor.java:132) > >> >> at > >> >> > >> > org.apache.activemq.transport.InactivityMonitor$2.run(InactivityMonitor.java:106) > >> >> at > >> >> > >> > org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) > >> >> at java.util.TimerThread.mainLoop(Timer.java:512) > >> >> at java.util.TimerThread.run(Timer.java:462) > >> >> > >> >> Config file: > >> >> > >> >> <beans > >> >> xmlns="http://www.springframework.org/schema/beans"; > >> >> xmlns:amq="http://activemq.apache.org/schema/core"; > >> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > >> >> xsi:schemaLocation="http://www.springframework.org/schema/beans > >> >> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd > >> >> http://activemq.apache.org/schema/core > >> >> http://activemq.apache.org/schema/core/activemq-core.xsd";> > >> >> > >> >> <bean > >> >> > >> > class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> > >> >> <property name="locations"> > >> >> > >> >> <value>file:${activemq.base}/conf/credentials.properties</value> > >> >> </property> > >> >> </bean> > >> >> > >> >> <broker xmlns="http://activemq.apache.org/schema/core"; > >> >> brokerName="sub01chi" dataDirectory="${activemq.base}/data"> > >> >> > >> >> <managementContext> > >> >> <managementContext createConnector="true"/> > >> >> </managementContext> > >> >> > >> >> <persistenceAdapter> > >> >> <kahaDB directory="${activemq.base}/data/kahadb"/> > >> >> </persistenceAdapter> > >> >> > >> >> <destinationPolicy> > >> >> <policyMap> > >> >> <policyEntries> > >> >> <policyEntry queue="P>" > >> producerFlowControl="true" > >> >> memoryLimit="10mb"></policyEntry> > >> >> </policyEntries> > >> >> </policyMap> > >> >> </destinationPolicy> > >> >> <systemUsage> > >> >> <systemUsage sendFailIfNoSpace="true"> > >> >> </systemUsage> > >> >> </systemUsage> > >> >> <transportConnectors> > >> >> <transportConnector name="openwire" > >> >> uri="tcp://0.0.0.0:61616"/> > >> >> </transportConnectors> > >> >> </broker> > >> >> <import resource="jetty.xml"/> > >> >> </beans> > >> >> > >> >> -----Original Message----- > >> >> From: Rob Davies [mailto:[email protected]] > >> >> Sent: Monday, January 18, 2010 10:42 PM > >> >> To: [email protected] > >> >> Subject: Re: OOM with high KahaDB index time > >> >> > >> >> > >> >> On 18 Jan 2010, at 22:14, Daniel Kluesing wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> I'm running the 5.3 release as a standalone broker. In one case, a > >> >>> producer is running without a consumer, producing small, persistent > >> >>> messages, with the FileCursor pendingQueuePolicy (per > >> >>> https://issues.apache.org/activemq/browse/AMQ-2512) > >> >>> option and flow control memoryLimit set to 100mb for the queue in > >> >>> question. (Through a policy entry) > >> >>> > >> >>> As the queue grows above 300k messages, KahaDB indexing starts > >> >>> climbing above 1 second. At around 350k messages, the indexing is > >> >>> taking over 8 seconds. At this point, I start getting java out of > >> >>> heap space errors in essentially random parts of the code. After a > >> >>> while, the producers timeout with a channel inactive for too long > >> >>> error, and the entire broker basically wedges itself. At this point, > >> >>> consumers are generally unable to bind to the broker quitting with > >> >>> timeout errors. When they can connect, consuming a single message > >> >>> triggers an index re-build, which takes 2-8seconds. Turning on > >> >>> verbose garbage collection, the jvm is collecting like mad but > >> >>> reclaiming no space. > >> >>> > >> >>> If I restart the broker, it comes back up, I can consume the old > >> >>> messages, and can handle another 350k messages until it wedges. > >> >>> > >> >>> I can reproduce under both default gc and incremental gc. > >> >>> > >> >>> Two questions: > >> >>> - It seems like someone is holding onto a handle to the messages > >> >>> after they have been persisted to disk - is this a known issue? > >> >>> Should I open a JIRA for it? (Or is there another explanation?) > >> >>> > >> >>> - Is there any documentation about the internals of KahaDB - the > >> >>> kind of indices etc? I'd like to get a better understanding of the > >> >>> index performance and in general how KahaDB compares to something > >> >>> like BerkeleyDB. > >> >>> > >> >>> Thanks > >> >>> > >> >>> > >> >>> > >> >> > >> >> > >> >> There's is some confusion over naming of our persistence options that > >> >> doesn't help. There is Kaha - which uses multiple log files and a > Hash > >> >> based index - this is currently used by the FileCursor - whilst > KahaDB > >> >> is a newer implementation, which is more robust and typically uses a > >> >> BTreeIndex. There is currently a new implementation of the Filecursor > >> >> btw - but that's a different matter. You can't currently configure > the > >> >> HashIndex via the FileCursor - but it looks like this is the problem > >> >> you are encountering - as it looks like you need to increase the max > >> >> hash buckets. > >> >> > >> >> > >> >> So I would recommend the following > >> >> 1. Use the default pendingQueuePolicy (which only uses a FileCursor > >> >> for non-persistent messages - and uses the underlying database for > >> >> persistent messages > >> >> 2. Try KahaDB - which - with the BTreeIndex - will not hit the > >> >> problems you are seeing with the Filecursor > >> >> > >> >> or - increase the maximum number of hash buckets for the FileCursor > >> >> index - by setting a Java system property - maximumCapacity to 65536 > >> >> (the default is 16384) > >> >> > >> >> cheers, > >> >> > >> >> Rob > >> >> > >> >> http://twitter.com/rajdavies > >> >> I work here: http://fusesource.com > >> >> My Blog: http://rajdavies.blogspot.com/ > >> >> I'm writing this: http://www.manning.com/snyder/ > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > > >> > -- > >> > View this message in context: > >> > > >> > http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27264475.html > >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27279301.html > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://old.nabble.com/OOM-with-high-KahaDB-index-time-tp27217704p27307957.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > >
