Hi Mitch, yeah, I said in thread I was referring to, that it is working with "regular" stomp connector. I started investigating AMQ-2440 patch the other day, should be have something soon.
Cheers -- Dejan Bosanac - http://twitter.com/dejanb Open Source Integration - http://fusesource.com/ ActiveMQ in Action - http://www.manning.com/snyder/ Blog - http://www.nighttale.net On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger <mitch.gran...@sophos.com>wrote: > So we turned off stomp+nio and went back to plain old stomp and so far it's > working fine. New(IO) isn't always better, I guess :-) > > Seems like maybe it's this issue -> > https://issues.apache.org/activemq/browse/AMQ-2440 > > > afei wrote: > >> i have same problem >> >> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >> >> >> themitchy wrote: >> >>> This is what we've done to tune so far: >>> >>> - UseDedicatedTaskRunner=false >>> - flow control is off >>> - stomp transport uses transport.closeAsync=false >>> >>> I agree that it is because of the high number of open/close connections >>> from Stomp. When we monitor through JConsole we can see more threads >>> starting up for each new connection. The problem is that these threads >>> don't get let go. Even though the stomp clients are disconnecting the >>> number of threads that get released is less than the number created. So >>> the thread count goes up and up until it fails. All of the above >>> settings/tuning only delay when it will hit the wall. >>> >>> Dejan Bosanac wrote: >>> >>>> Hi Mitch, >>>> >>>> I think the root cause of this problem is that you probably have Stomp >>>> clients that open/close connection at a high rate. I simulated this >>>> problem >>>> on OSX with a StompLoadTest ( >>>> >>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>> ), >>>> while trying to reproduce "too many open files" problem. You can find >>>> some >>>> of my findings (and workaround) in this thread. >>>> >>>> >>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>> >>>> (BTW. it is producing "too many open files" problem on linux) Basically, >>>> the >>>> problem with stomp is that every send is done in separate connection and >>>> thus considered to be a new producer for every message. So when producer >>>> flow control is hit, the producers are piling up and probably not >>>> releasing >>>> connections. Thus you can observe large number of tcp connections on the >>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the system >>>> limit is hit at one point. In the above thread, you can find a >>>> workaround >>>> that worked for me for that test. I started investigating this more and >>>> hopefully I'll have some more findings in the near future. >>>> >>>> Cheers >>>> -- >>>> Dejan Bosanac - http://twitter.com/dejanb >>>> >>>> Open Source Integration - http://fusesource.com/ >>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>> Blog - http://www.nighttale.net >>>> >>>> >>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>> <mitch.gran...@sophos.com>wrote: >>>> >>>> Update: We've [nearly] proven that this only happens with AMQ running >>>>> on >>>>> openVZ. What exactly is causing it, we're still not sure. After >>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>> OutOfMemoryError. >>>>> >>>>> It works just fine on regular hardware; We're going to try VMWare >>>>> tomorrow. >>>>> >>>>> One thing really worth mentioning is that by using the fileCursor we >>>>> actually started seeing it use the Temp Store. When reading about >>>>> systemUsage it is NOT intuitive that the Temp Store does not come into >>>>> play >>>>> with the default cursor. Anyone keeping a significant volume of >>>>> messages on >>>>> their queues should be well served by changing the cursor. >>>>> >>>>> >>>>> Mitch Granger wrote: >>>>> >>>>> Config is attached. We have also tried the activemq-scalability.xml >>>>>> with >>>>>> the only change being adding a stomp connector. >>>>>> >>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>> consumers >>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>> >>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>> subscribe >>>>>> there's no chance of recovering from this state. >>>>>> >>>>>> Rob Davies wrote: >>>>>> >>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>> >>>>>>> We're using only persistent messages and heap size is set to 2GB yet >>>>>>> we >>>>>>> >>>>>>>> hit >>>>>>>> the memoryUsage limit quite quickly (system usage config below). >>>>>>>> This >>>>>>>> is >>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new native >>>>>>>> thread" >>>>>>>> as the process quickly reaches the 2GB of heap we gave it. How are >>>>>>>> we >>>>>>>> getting to that point with the memoryUsage limit set far below it? >>>>>>>> >>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage? >>>>>>>> >>>>>>>> <systemUsage> >>>>>>>> <systemUsage> >>>>>>>> <memoryUsage> >>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>> </memoryUsage> >>>>>>>> <storeUsage> >>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>> </storeUsage> >>>>>>>> <tempUsage> >>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>> </tempUsage> >>>>>>>> </systemUsage> >>>>>>>> </systemUsage> >>>>>>>> >>>>>>>> -- >>>>>>>> View this message in context: >>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>>>>> >>>>>>>> >>>>>>>> Can you send the rest of your config ? >>>>>>> >>>>>>> Rob Davies >>>>>>> http://twitter.com/rajdavies >>>>>>> I work here: http://fusesource.com >>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>> >>
