Any chance we can get a hold of your test code. 1. If I can test against a known configuration that is known to work I can find what I'm doing wrong faster 2. I see that most most posts about working setups are NOT Centos, in fact I have seen two posts that use Ubuntu successfully. Am I to change my linux distro? 3. How many is "many" connections or machines that connect to the server? At more than fifty we see issues. 4. How would having a total of 256k ram effect this?
Jose Luna-2 wrote: > > > Hello, > > We have the requirement of creating a very large number of queues (tens of > thousands) in a single broker. My searches turned up several others > trying to do something similar, but with no results. We were recently able > to accomplish this, so I decided to write a little howto. All of this > information can be found in the mailing list, wiki, or XML reference, but > it wasn't easy to pull it all together. Hopefully this will help someone. > Also, please let me know if there are any inaccuracies, or if anything can > be added. > > == Thread Count == > === "QueueThread" === > One of the issues that you'll come across is receiving the out of memory > error "unable to create new native thread" when you create thousands of > queues. A "QueueThread" thread is created for every new queue. The > optimizedDispatch property for the queue policyEntry seemed to address our > needs, the description states "don't use a separate thread for dispatching > from a Queue" (http://activemq.apache.org/per-destination-policies.html). > However, turning this on just seemed to create "TempQueue" threads > instead, still creating a thread for each queue. Digging a little deeper, > we find that using OptimizedDispatch creates the thread using a > TaskRunner, which should pool the threads. After further research, we > found that the property org.apache.activemq.UseDedicatedTaskRunner should > be set to false so that the task runner actually pools the threads > (http://activemq.apache.org/javalangoutofmemory.html --- this document > also provides other useful info, like > how to lower the stack size for each thread). > > Example Policy Entry in INSTALL_DIR/conf/activemq.conf: > <policyEntries> > <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/> > </policyEntries> > > You can either change the 'UseDedicatedTaskRunner' property in your > startup script, INSTALL_DIR/bin/activemq: > ACTIVEMQ_OPTS="-Xmx512M > -Dorg.apache.activemq.UseDedicatedTaskRunner=false" > Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf . > > Note: It would be very useful it the documentation listed above referenced > each other, as they are most useful when used together. > > === "Checkpoint" thread === > If you're using persistent storage, a "Checkpoint" thread is also created > for every queue that is created. These are used to write persistent > messages to the store. Fortunately this is already created with a task > runner, so setting the UseDedicatedTaskRunner property to false will > prevent these thread from overhwelming your system. > > === Connection related threads === > If you're also dealing with high thread count due to a very high number of > connections, see > http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ. > > Also, consider reducing the stack size for each thread (the JVM option > -Xss). > > == "too many open files" == > ActiveMQ uses the amqPersistenceAdapter by default for persistent > messages. Unfortunately, this persistence adapter (as well as the > kahaPersistenceAdapter) opens a file descriptor for each queue. When > creating large numbers of queues, you'll quickly run into the limit for > your OS. There are two possible solutions to this: 1.) raise the > per-process open file limit for your OS 2.) Use a jdbc persistent adapter. > The first can simply be googled for your OS. The second uses a connection > pool to your DB of choice, and no longer requires an open file for each > queue created. > > An unrelated issue is running out of file descriptors because there are > too many connections. (In linux, and other OSes, a file descriptor is > used for open sockets). I don't know of any other way to handle that, > other than increasing the limit for your process. > > == Conclusion == > We were able to create more than 20k queues using ~30 threads in a test > environment with a 512MB heap. Environment: ActiveMQ 5.2.0, SUN Java > 1.6.0, Ubuntu 9.04 . Our testing hasn't been exhaustive, but everything > is running smoothly so far. > > JLuna > > > > > -- View this message in context: http://www.nabble.com/Large-number-of-queues-%28HowTo%29-tp23602928p23682395.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
