Queue browsing just involves HTTP interactions with the Jetty web server, and can be scripted if the volume of queues makes manual browsing impossible.
After you browsed those queues, did any of the very old KahaDB data files get deleted? That's the second question you're trying to answer by browsing the queues: are there any messages that are past their expiration date but haven't yet been expired by the broker, such that the pure act of attempting to browse them triggers their deletion? (And note that you can't answer that question until you've browsed all queues that have unconsumed messages, because a KahaDB data file can't be deleted until every single message in it is deleted, so browsing "a lot of queues" might trigger a file deletion or might not, but it doesn't indicate anything about whether that's our problem or not.) I recommend you make a copy of your KahaDB data files and put them somewhere you can keep coming back them, then make a copy of those files and use the second copy to stand up a new broker in a dev/test environment, with no consumers and no producers, so you can do some experiments without affecting your production broker. Then purge every queue (again, this can be scripted, with a little work) and see if all of your KahaDB files (except the current one) get deleted. (Keep in mind that the deletion thread runs periodically, so there may be a small delay, depending on your configuration.) I expect that everything will get deleted once there are no messages left in any queue, which would mean that KahaDB itself is working correctly, and your problem is that you have old messages that aren't being deleted. Next, use your archived copy of the KahaDB data files to restore the test broker to a state where all of those messages are present again, and browse every queue. As you go, note whether any message has either no JMSExpiration header set, or a JMSExpiration header that's more than your intended timeout (3 weeks?) after the creation date (JMSExpiration - JMSTimestamp > 3 weeks), which will tell you whether any of your assumptions about the behavior of your producers are invalid and that your perceived problem may in fact be caused by invalid expectations. When you're done, see if any of the earliest KahaDB data files get deleted; if it did, then your problem is that the broker isn't deleting expired messages when there is no consumer for them. Running those experiments should help to zero in on what's causing the behavior you're seeing, and whether it's a bug in ActiveMQ or problems in your software. Also, keep in mind that the raw number of files is irrelevant; what should be constant (after a warm-up period equal to the expiration interval all producers are using) is the number of files *created* since the oldest live file, of which some may have been deleted since then. KahaDB can delete any file it's done with, and there's a lot of randomness about what messages a given file contains and therefore whether it can be deleted, so expecting a constant rate of file deletions isn't realistic. But since the filenames have a one-up number in them, you can tell easily how many files were created between the oldest file and the newest (current) file. All of that assumes a constant rate of message production and consumption; in a real-world application such as your own, there may be variability of data file creation rates due to the variability of message production/consumption rates, so the "constant" rate of file creation is very much an average, not a fixed value. The other thing that should be constant is that the oldest data file should never be substantially older than your largest expiration interval (if your producers don't all use the same value). All messages in the file should be older than the last-modified timestamp of the file, so they should all be candidates for expiration when the file passes the expiration interval. If your oldest data file is not older than the max expiration interval, then you don't have a problem. What is the modification date/time of your oldest file? Please make sure you answer this question now, before you go running the experiments I recommended, since your answer could eliminate the need for you to run those experiments. Tim On Wed, Dec 23, 2015 at 1:59 PM, Shine <activ...@club.webhop.org> wrote: > Hi Tim, > > there are about 2500 queues and it is not so easy to browse all queues and > unconsumed messages. > > I took a look to a lot of queues with unconsumed messages and all messages > has an (valid = future date) expiration date. > > > Now 561 files / 17,9 GB > > regards > Shine > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705414.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
