> Well, the main question is now: > 1. Why is the queue getting slower when there are a lot of messages inside? So with X million messages, enqueue times are 4-7 seconds? This is probabally because the hashIndex is limited, I think you need to make the indexMaxBinSize quite large and use the same value for indexBinSize so that no resizing takes place at runtime. This should speed up index insertion. Also increasing the indexPageSize may help by increasing the quantity of data persisted in one write.
After that, this may be indicative of a problem with the single large hash index file. > 2. How can I change the size of the data-queue-data-959 files? The size of the files in the reference store are controlled by the maxReferenceFileLength attribute. > Would it make sense to have a few big of those files instead of a lot of > small one? Not sure, I think this is only relevant to cleanup, a large file with lots of references may take a long time to be unferenced and ready for cleanup. > 3. What is this file for: hash-index-queue-data_queue#3a#2f#2ftracking This is the persistent part of the persistent hashIndex The following link gives a overview of the amq persistence adapter explaining the layout etc: http://activemq.apache.org/amq-message-store.html 2009/1/23 Cybexion <[email protected]>: > > Hello, > > I changed my AMQPersistenceAdapter to write 1 GB files (total store size is > 30GB) > <bean id="store" > class="org.apache.activemq.store.amq.AMQPersistenceAdapter"> > <property name="directory" value="${activemq.base}/data"/> > <!-- 1GB = 1073741824 --> > <property name="maxFileLength" value="1073741824"/> > <property name="checkpointInterval" value="60000"/> > <property name="cleanupInterval" value="30000"/> > <property name="persistentIndex" value="true"/> > </bean> > > After that I created messages during the whole night into the queue. > The result looks like this: > > ls -l data/journal > -rw-r--r-- 1 activemq java 1073741777 2009-01-22 21:06 data-2 > -rw-r--r-- 1 activemq java 1073741637 2009-01-22 23:17 data-3 > -rw-r--r-- 1 activemq java 1073741573 2009-01-23 01:17 data-4 > -rw-r--r-- 1 activemq java 1073741815 2009-01-23 03:40 data-5 > -rw-r--r-- 1 activemq java 1073741674 2009-01-23 06:48 data-6 > -rw-r--r-- 1 activemq java 1073217536 2009-01-23 11:22 data-7 > -rw-r--r-- 1 activemq java 2088 2009-01-23 11:22 data-control > > About 5.2 GB of data has been created. > > These are the hundreds of files containing the keys to the messages. > ls -l data/kr-store/data > -rw-r--r-- 1 activemq java 2097142 2009-01-23 11:04 data-queue-data-957 > -rw-r--r-- 1 activemq java 2097117 2009-01-23 11:11 data-queue-data-958 > -rw-r--r-- 1 activemq java 2068877 2009-01-23 11:21 data-queue-data-959 > -rw-r--r-- 1 activemq java 1462429697 2009-01-23 11:24 > hash-index-queue-data_queue#3a#2f#2ftracking > -rw-r--r-- 1 activemq java 153 2009-01-22 17:10 index-container-roots > -rw-r--r-- 1 activemq java 705609480 2009-01-23 11:24 index-queue-data > -rw-r--r-- 1 activemq java 0 2009-01-22 17:09 lock > > ls -l | wc > 721 5762 54026 > > Here are some additional JMX infos about the broker > > ActiveMQ Broker Memory percent Usage : 0,00 > ActiveMQ Broker Store percent Usage : 26,00 > ActiveMQ Broker Temp percent Usage : 0,00 > ActiveMQ Broker Consumer Count : 1,00 > ActiveMQ Broker Total Dequeue Count : 3.629.262,00 > ActiveMQ Broker Total Enqueue Count : 16.132.693,00 > ActiveMQ Broker Total Message Count : 12.503.432,00 > > Now to the bad news: > It looks like the more messages are in the queue, the slower it gets to send > new messages into it (using JmsTemplate.send() method). > When I currently run I java programm that inserts messages into the Queue I > have response times of 4-7 seconds !!!! > This is not good. > Here is an extract of the AMQ log file: > 2009-01-22 18:08:11,229 [ueue://tracking] INFO > kaha.impl.index.hash.HashIndex - Resize hash bins > queue-data_queue#3a#2f#2ftracking from 1024 to 2048 > 2009-01-22 18:10:24,642 [ueue://tracking] INFO > kaha.impl.index.hash.HashIndex - Resize hash bins > queue-data_queue#3a#2f#2ftracking from 2048 to 4096 > 2009-01-22 18:13:35,224 [ueue://tracking] INFO > kaha.impl.index.hash.HashIndex - Resize hash bins > queue-data_queue#3a#2f#2ftracking from 4096 to 8192 > 2009-01-22 18:18:12,564 [ueue://tracking] INFO > kaha.impl.index.hash.HashIndex - Resize hash bins > queue-data_queue#3a#2f#2ftracking from 8192 to 16384 > > Well, the main question is now: > 1. Why is the queue getting slower when there are a lot of messages inside? > 2. How can I change the size of the data-queue-data-959 files? > Would it make sense to have a few big of those files instead of a lot of > small one? > 3. What is this file for: hash-index-queue-data_queue#3a#2f#2ftracking > > Thanks and regards > > Jörg > > -- > View this message in context: > http://www.nabble.com/Performance-in-relation-to-queue-size-tp21601585p21622121.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > -- http://blog.garytully.com Open Source SOA http://FUSESource.com
