Hello,
I just made another test:
What happens, if the Queue is full. Will the performance grow again?
I Changed my Queue size to GB (<storeUsage limit="2gb" store="#store"/> ).
Now I restarted my test, having 900 JMS Producers sending messages into the
Queue.
The first few minutes it is very fast. After a while JMS sendings goes
slower and slower.
Whenn the Queue is about to reach her maximum size I can only send about 100
JMS Requests per second (if at all).
But when the Queue has reached the maximum size, and
ResourceAllocationException are thrown on the server side like this:
2009-01-26 22:09:40,797 [127.0.0.1:48276] ERROR
emq.broker.TransportConnection - Async error occurred:
javax.jms.ResourceAllocationE
xception: Usage Manager Store is Full
javax.jms.ResourceAllocationException: Usage Manager Store is Full
Then the performance goes up again.
Currently I can see again about 900 Messages per second.
An interesting point here is, that the Queue size is shrinking but the
Broker Store percent usage is increating. I'm now at about 104% store usage.
Btw. I would be nice if one could change the exception above from ERROR to
warning. In my case this is not an ERROR and I would like to not see it in
my logs.
Concerning the performance I have the following theory:
While the Queue is growing there is a file that is constantly resizing (or
copied?) This causes more and more delay th bigger the queue gets.
I don't think it is one of the data file from the journal directory.
In my case they look like this:
ls -l data/journal
-rw-r--r-- 1 activemq java 1073741680 2009-01-26 19:56 data-1
-rw-r--r-- 1 activemq java 1073217536 2009-01-26 22:14 data-2
-rw-r--r-- 1 activemq java 2088 2009-01-26 22:14 data-control
Personally I don't trust the "hash-index-queue-data_queue" file in the
kr-store/data directory:
ls -l data/kr-store/data
-rw-r--r-- 1 activemq java 271 2009-01-26 18:35
data-container-roots-1
-rw-r--r-- 1 activemq java 104857602 2009-01-26 19:23 data-queue-data-2
-rw-r--r-- 1 activemq java 104857570 2009-01-26 20:01 data-queue-data-3
-rw-r--r-- 1 activemq java 104857570 2009-01-26 21:11 data-queue-data-4
-rw-r--r-- 1 activemq java 13787718 2009-01-26 21:25 data-queue-data-5
-rw-r--r-- 1 activemq java 17086873601 2009-01-26 22:18
hash-index-queue-data_queue#3a#2f#2ftracking
-rw-r--r-- 1 activemq java 153 2009-01-26 18:35
index-container-roots
-rw-r--r-- 1 activemq java 121943448 2009-01-26 22:18 index-queue-data
-rw-r--r-- 1 activemq java 0 2009-01-26 18:34 lock
How does it come that this file is respectable 16 GB big???
Could it be that maintaining the key does take a lot of time?
Is the Kahastore really faster than using a e.g. mysql DB?
I let my test continue until tomorrow, lets see if something new happens.
Thanks and best regards
Jörg
Cybexion wrote:
>
> Hi,
>
> I have a queue that should hold up to 30 GB of small persistent messages:
> <systemUsage sendFailIfNoSpace="true">
> <memoryUsage>
> <memoryUsage limit="1gb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="30gb" store="#store"/>
> </storeUsage>
>
> The queue consists of 10 MB files and are specified like this:
> <bean id="store"
> class="org.apache.activemq.store.amq.AMQPersistenceAdapter">
> <property name="directory" value="${activemq.base}/data"/>
> <property name="maxFileLength" value="10000000"/>
> <property name="checkpointInterval" value="60000"/>
> <property name="cleanupInterval" value="30000"/>
> <property name="persistentIndex" value="true"/>
> </bean>
>
> This night I made a test with lots of produceds an just 5 consumer. I
> wanted to fill the queue to its limit.
> Well, so far I managed to send 18 Million messages into the queue. I have
> 20 % Store usage so far.
> What I noticed is, that mostly the messages are send within 0-20
> milliseconds. However sometimes I have a lot of messages which take 10+
> seconds to be send. This is not good in my scenario. I need a producer
> that should be very very fast in sending the messages.
> Question now:
> Could it be that the number of data files or the size of the datafiles has
> impact on the performance?
> Would it be better to have one queue big file with 30GB of size instead of
> hundreds of files having 1 MB?
>
> The key index ist persistent. I tested this and it seems to be the only
> way of NOT getting an out of memory error.
> Does it make sense to change the indexkeysize or the indexpagesize in the
> store? Would this maybe gice additional constant performance?
>
> Btw:
> I'm using AMQ 5.2
>
> Thanks and best regards
>
> Jörg
>
>
>
>
>
>
>
--
View this message in context:
http://www.nabble.com/Performance-in-relation-to-queue-size-tp21601585p21673976.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
