i test r831258,the problem still exist
Gary Tully wrote: > > fyi: you can disable periodic message expiry processing using a > destination policy entry that sets expireMessagesPeriod = 0 > > 2009/11/9 afei <afei1...@126.com>: >> >> >> when a large of messages in queue,and no consumer or the consumer is very >> slow, the OOM problem occur, because : >> in org.apache.activemq.broker.region.Queue,the 588 line is : >> doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); >> ,transform to : >> doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); >> is ok. >> >> >> Dejan Bosanac wrote: >>> >>> Hi Mitch, >>> >>> yeah, I said in thread I was referring to, that it is working with >>> "regular" >>> stomp connector. I started investigating AMQ-2440 patch the other day, >>> should be have something soon. >>> >>> Cheers >>> -- >>> Dejan Bosanac - http://twitter.com/dejanb >>> >>> Open Source Integration - http://fusesource.com/ >>> ActiveMQ in Action - http://www.manning.com/snyder/ >>> Blog - http://www.nighttale.net >>> >>> >>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger >>> <mitch.gran...@sophos.com>wrote: >>> >>>> So we turned off stomp+nio and went back to plain old stomp and so far >>>> it's >>>> working fine. New(IO) isn't always better, I guess :-) >>>> >>>> Seems like maybe it's this issue -> >>>> https://issues.apache.org/activemq/browse/AMQ-2440 >>>> >>>> >>>> afei wrote: >>>> >>>>> i have same problem >>>>> >>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>>>> >>>>> >>>>> themitchy wrote: >>>>> >>>>>> This is what we've done to tune so far: >>>>>> >>>>>> - UseDedicatedTaskRunner=false >>>>>> - flow control is off >>>>>> - stomp transport uses transport.closeAsync=false >>>>>> >>>>>> I agree that it is because of the high number of open/close >>>>>> connections >>>>>> from Stomp. When we monitor through JConsole we can see more threads >>>>>> starting up for each new connection. The problem is that these >>>>>> threads >>>>>> don't get let go. Even though the stomp clients are disconnecting >>>>>> the >>>>>> number of threads that get released is less than the number created. >>>>>> So >>>>>> the thread count goes up and up until it fails. All of the above >>>>>> settings/tuning only delay when it will hit the wall. >>>>>> >>>>>> Dejan Bosanac wrote: >>>>>> >>>>>>> Hi Mitch, >>>>>>> >>>>>>> I think the root cause of this problem is that you probably have >>>>>>> Stomp >>>>>>> clients that open/close connection at a high rate. I simulated this >>>>>>> problem >>>>>>> on OSX with a StompLoadTest ( >>>>>>> >>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>>>>> ), >>>>>>> while trying to reproduce "too many open files" problem. You can >>>>>>> find >>>>>>> some >>>>>>> of my findings (and workaround) in this thread. >>>>>>> >>>>>>> >>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>>>>> >>>>>>> (BTW. it is producing "too many open files" problem on linux) >>>>>>> Basically, >>>>>>> the >>>>>>> problem with stomp is that every send is done in separate connection >>>>>>> and >>>>>>> thus considered to be a new producer for every message. So when >>>>>>> producer >>>>>>> flow control is hit, the producers are piling up and probably not >>>>>>> releasing >>>>>>> connections. Thus you can observe large number of tcp connections on >>>>>>> the >>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>>>> system >>>>>>> limit is hit at one point. In the above thread, you can find a >>>>>>> workaround >>>>>>> that worked for me for that test. I started investigating this more >>>>>>> and >>>>>>> hopefully I'll have some more findings in the near future. >>>>>>> >>>>>>> Cheers >>>>>>> -- >>>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>>> >>>>>>> Open Source Integration - http://fusesource.com/ >>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>>> Blog - http://www.nighttale.net >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>>>> <mitch.gran...@sophos.com>wrote: >>>>>>> >>>>>>> Update: We've [nearly] proven that this only happens with AMQ >>>>>>> running >>>>>>>> on >>>>>>>> openVZ. What exactly is causing it, we're still not sure. After >>>>>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>>>>> OutOfMemoryError. >>>>>>>> >>>>>>>> It works just fine on regular hardware; We're going to try VMWare >>>>>>>> tomorrow. >>>>>>>> >>>>>>>> One thing really worth mentioning is that by using the fileCursor >>>>>>>> we >>>>>>>> actually started seeing it use the Temp Store. When reading about >>>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come >>>>>>>> into >>>>>>>> play >>>>>>>> with the default cursor. Anyone keeping a significant volume of >>>>>>>> messages on >>>>>>>> their queues should be well served by changing the cursor. >>>>>>>> >>>>>>>> >>>>>>>> Mitch Granger wrote: >>>>>>>> >>>>>>>> Config is attached. We have also tried the >>>>>>>> activemq-scalability.xml >>>>>>>>> with >>>>>>>>> the only change being adding a stomp connector. >>>>>>>>> >>>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>>>>> consumers >>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>>>> >>>>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>>>> subscribe >>>>>>>>> there's no chance of recovering from this state. >>>>>>>>> >>>>>>>>> Rob Davies wrote: >>>>>>>>> >>>>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>>>> >>>>>>>>>> We're using only persistent messages and heap size is set to 2GB >>>>>>>>>> yet >>>>>>>>>> we >>>>>>>>>> >>>>>>>>>>> hit >>>>>>>>>>> the memoryUsage limit quite quickly (system usage config below). >>>>>>>>>>> This >>>>>>>>>>> is >>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>>>> native >>>>>>>>>>> thread" >>>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. How >>>>>>>>>>> are >>>>>>>>>>> we >>>>>>>>>>> getting to that point with the memoryUsage limit set far below >>>>>>>>>>> it? >>>>>>>>>>> >>>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory >>>>>>>>>>> usage? >>>>>>>>>>> >>>>>>>>>>> <systemUsage> >>>>>>>>>>> <systemUsage> >>>>>>>>>>> <memoryUsage> >>>>>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>>>>> </memoryUsage> >>>>>>>>>>> <storeUsage> >>>>>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>>>>> </storeUsage> >>>>>>>>>>> <tempUsage> >>>>>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>>>>> </tempUsage> >>>>>>>>>>> </systemUsage> >>>>>>>>>>> </systemUsage> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> View this message in context: >>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at >>>>>>>>>>> Nabble.com. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Can you send the rest of your config ? >>>>>>>>>> >>>>>>>>>> Rob Davies >>>>>>>>>> http://twitter.com/rajdavies >>>>>>>>>> I work here: http://fusesource.com >>>>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>> >>>>> >>> >>> >>> ----- >>> Dejan Bosanac >>> >>> Open Source Integration - http://fusesource.com/ >>> ActiveMQ in Action - http://www.manning.com/snyder/ >>> Blog - http://www.nighttale.net >>> >> http://old.nabble.com/file/p26264779/dump.jpg >> -- >> View this message in context: >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > > > > -- > http://blog.garytully.com > > Open Source Integration > http://fusesource.com > > -- View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26278185.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
