I'm not sure, if this is all related to the Memoryleak I found in AMQ 5.2
see
http://www.nabble.com/OutOfMemoryError-after-9505925-persistent-messages-to21722796.html
I just made a new test run last night and my PS Old Gen is about to explode.
GC has no effect any more.
Cybexion wrote:
>
> Hi,
>
> I tried all this without success.
> The more messages are in the queue the slower it gets. My system was
> running during the whole weekend now. The first hour I had about 900
> messages per second (JMS sending took about 0ms). Then after about 1-2
> hours JMS sending suddently takes 1 ore more seconds.
>
> My config looks like this (just the interesting stuff):
> <bean id="store"
> class="org.apache.activemq.store.amq.AMQPersistenceAdapter">
> <property name="directory" value="${activemq.base}/data"/>
> <!-- 1GB = 1073741824 -->
> <property name="maxFileLength"
> value="1073741824"/>
> <property name="maxReferenceFileLength"
> value="104857600"/>
> <property name="checkpointInterval" value="60000"/>
> <property name="cleanupInterval" value="30000"/>
> <property name="persistentIndex" value="true"/>
> <property name="indexMaxBinSize" value="10485760"/>
> <property name="indexBinSize" value="1048576"/>
> <property name="indexPageSize" value="65536"/>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core";
> persistent="true"
> advisorySupport="false"
> deleteAllMessagesOnStartup="false"
> useJmx="true"
> monitorConnectionSplits="false"
> splitSystemUsageForProducersConsumers="false"
> brokerName="localhost"
> dataDirectory="${activemq.base}/data"
> persistenceAdapter="#store"
> >
>
>
> <!-- The maximum about of space the broker will use before slowing
> down producers -->
> <systemUsage>
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="1gb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="30gb" store="#store"/>
> </storeUsage>
>
> <!-- temp usage is used with non persistent messages only
> -->
> <tempUsage>
> <tempUsage limit="64mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> Anyone else having an idea?
> Next thing I try is to limit the queue size to lets say on GB and see if
> performance is fine if the size is limited.
>
>
>
>
>
>
> Gary Tully wrote:
>>
>>> Well, the main question is now:
>>> 1. Why is the queue getting slower when there are a lot of messages
>>> inside?
>> So with X million messages, enqueue times are 4-7 seconds?
>> This is probabally because the hashIndex is limited, I think you need
>> to make the indexMaxBinSize quite large and use the same value for
>> indexBinSize so that no resizing takes place at runtime. This should
>> speed up index insertion.
>> Also increasing the indexPageSize may help by increasing the quantity
>> of data persisted in one write.
>>
>> After that, this may be indicative of a problem with the single large
>> hash index file.
>>
>>
>>
>>> 2. How can I change the size of the data-queue-data-959 files?
>> The size of the files in the reference store are controlled by the
>> maxReferenceFileLength attribute.
>>
>>> Would it make sense to have a few big of those files instead of a lot of
>>> small one?
>> Not sure, I think this is only relevant to cleanup, a large file with
>> lots of references may take a long time to be unferenced and ready for
>> cleanup.
>>
>>> 3. What is this file for: hash-index-queue-data_queue#3a#2f#2ftracking
>> This is the persistent part of the persistent hashIndex
>>
>> The following link gives a overview of the amq persistence adapter
>> explaining the layout etc:
>> http://activemq.apache.org/amq-message-store.html
>>
>> 2009/1/23 Cybexion <cybex...@email.de>:
>>>
>>> Hello,
>>>
>>> I changed my AMQPersistenceAdapter to write 1 GB files (total store size
>>> is
>>> 30GB)
>>> <bean id="store"
>>> class="org.apache.activemq.store.amq.AMQPersistenceAdapter">
>>> <property name="directory"
>>> value="${activemq.base}/data"/>
>>> <!-- 1GB = 1073741824 -->
>>> <property name="maxFileLength" value="1073741824"/>
>>> <property name="checkpointInterval" value="60000"/>
>>> <property name="cleanupInterval" value="30000"/>
>>> <property name="persistentIndex" value="true"/>
>>> </bean>
>>>
>>> After that I created messages during the whole night into the queue.
>>> The result looks like this:
>>>
>>> ls -l data/journal
>>> -rw-r--r-- 1 activemq java 1073741777 2009-01-22 21:06 data-2
>>> -rw-r--r-- 1 activemq java 1073741637 2009-01-22 23:17 data-3
>>> -rw-r--r-- 1 activemq java 1073741573 2009-01-23 01:17 data-4
>>> -rw-r--r-- 1 activemq java 1073741815 2009-01-23 03:40 data-5
>>> -rw-r--r-- 1 activemq java 1073741674 2009-01-23 06:48 data-6
>>> -rw-r--r-- 1 activemq java 1073217536 2009-01-23 11:22 data-7
>>> -rw-r--r-- 1 activemq java 2088 2009-01-23 11:22 data-control
>>>
>>> About 5.2 GB of data has been created.
>>>
>>> These are the hundreds of files containing the keys to the messages.
>>> ls -l data/kr-store/data
>>> -rw-r--r-- 1 activemq java 2097142 2009-01-23 11:04
>>> data-queue-data-957
>>> -rw-r--r-- 1 activemq java 2097117 2009-01-23 11:11
>>> data-queue-data-958
>>> -rw-r--r-- 1 activemq java 2068877 2009-01-23 11:21
>>> data-queue-data-959
>>> -rw-r--r-- 1 activemq java 1462429697 2009-01-23 11:24
>>> hash-index-queue-data_queue#3a#2f#2ftracking
>>> -rw-r--r-- 1 activemq java 153 2009-01-22 17:10
>>> index-container-roots
>>> -rw-r--r-- 1 activemq java 705609480 2009-01-23 11:24 index-queue-data
>>> -rw-r--r-- 1 activemq java 0 2009-01-22 17:09 lock
>>>
>>> ls -l | wc
>>> 721 5762 54026
>>>
>>> Here are some additional JMX infos about the broker
>>>
>>> ActiveMQ Broker Memory percent Usage : 0,00
>>> ActiveMQ Broker Store percent Usage : 26,00
>>> ActiveMQ Broker Temp percent Usage : 0,00
>>> ActiveMQ Broker Consumer Count : 1,00
>>> ActiveMQ Broker Total Dequeue Count : 3.629.262,00
>>> ActiveMQ Broker Total Enqueue Count : 16.132.693,00
>>> ActiveMQ Broker Total Message Count : 12.503.432,00
>>>
>>> Now to the bad news:
>>> It looks like the more messages are in the queue, the slower it gets to
>>> send
>>> new messages into it (using JmsTemplate.send() method).
>>> When I currently run I java programm that inserts messages into the
>>> Queue I
>>> have response times of 4-7 seconds !!!!
>>> This is not good.
>>> Here is an extract of the AMQ log file:
>>> 2009-01-22 18:08:11,229 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 1024 to 2048
>>> 2009-01-22 18:10:24,642 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 2048 to 4096
>>> 2009-01-22 18:13:35,224 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 4096 to 8192
>>> 2009-01-22 18:18:12,564 [ueue://tracking] INFO
>>> kaha.impl.index.hash.HashIndex - Resize hash bins
>>> queue-data_queue#3a#2f#2ftracking from 8192 to 16384
>>>
>>> Well, the main question is now:
>>> 1. Why is the queue getting slower when there are a lot of messages
>>> inside?
>>> 2. How can I change the size of the data-queue-data-959 files?
>>> Would it make sense to have a few big of those files instead of a lot of
>>> small one?
>>> 3. What is this file for: hash-index-queue-data_queue#3a#2f#2ftracking
>>>
>>> Thanks and regards
>>>
>>> Jörg
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Performance-in-relation-to-queue-size-tp21601585p21622121.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> http://blog.garytully.com
>>
>> Open Source SOA
>> http://FUSESource.com
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Performance-in-relation-to-queue-size-tp21601585p21743840.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
