when consuming a large of messages,the method:asyncWakeup() is invoked crazily,so the executor has a great deal of runnable that callback Queue.iterate(), but Queue.iterate() is much slower than the increasing of runnable in the executor. this result in OOM.
avoid to invoke the method:asyncWakeup() frequently??? Gary Tully wrote: > > iterate actually does a dispatch. when consumers change etc, we again > try and dispatch to ensure prefetch buffers are maintained. > > Do a svn up to r834579 as I committed a fix to trunk for this today > resolving https://issues.apache.org/activemq/browse/AMQ-2481 > > 2009/11/10 afei <afei1...@126.com>: >> >> in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(), >> what is the method:iterate() doing? >> >> >> afei wrote: >>> >>> in addition,another problem of OOM. >>> >>> >>> >>> Gary Tully wrote: >>>> >>>> fyi: you can disable periodic message expiry processing using a >>>> destination policy entry that sets expireMessagesPeriod = 0 >>>> >>>> 2009/11/9 afei <afei1...@126.com>: >>>>> >>>>> >>>>> when a large of messages in queue,and no consumer or the consumer is >>>>> very >>>>> slow, the OOM problem occur, because : >>>>> in org.apache.activemq.broker.region.Queue,the 588 line is : >>>>> doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); >>>>> ,transform to : >>>>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); >>>>> is ok. >>>>> >>>>> >>>>> Dejan Bosanac wrote: >>>>>> >>>>>> Hi Mitch, >>>>>> >>>>>> yeah, I said in thread I was referring to, that it is working with >>>>>> "regular" >>>>>> stomp connector. I started investigating AMQ-2440 patch the other >>>>>> day, >>>>>> should be have something soon. >>>>>> >>>>>> Cheers >>>>>> -- >>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>> >>>>>> Open Source Integration - http://fusesource.com/ >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>> Blog - http://www.nighttale.net >>>>>> >>>>>> >>>>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger >>>>>> <mitch.gran...@sophos.com>wrote: >>>>>> >>>>>>> So we turned off stomp+nio and went back to plain old stomp and so >>>>>>> far >>>>>>> it's >>>>>>> working fine. New(IO) isn't always better, I guess :-) >>>>>>> >>>>>>> Seems like maybe it's this issue -> >>>>>>> https://issues.apache.org/activemq/browse/AMQ-2440 >>>>>>> >>>>>>> >>>>>>> afei wrote: >>>>>>> >>>>>>>> i have same problem >>>>>>>> >>>>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>>>>>>> >>>>>>>> >>>>>>>> themitchy wrote: >>>>>>>> >>>>>>>>> This is what we've done to tune so far: >>>>>>>>> >>>>>>>>> - UseDedicatedTaskRunner=false >>>>>>>>> - flow control is off >>>>>>>>> - stomp transport uses transport.closeAsync=false >>>>>>>>> >>>>>>>>> I agree that it is because of the high number of open/close >>>>>>>>> connections >>>>>>>>> from Stomp. When we monitor through JConsole we can see more >>>>>>>>> threads >>>>>>>>> starting up for each new connection. The problem is that these >>>>>>>>> threads >>>>>>>>> don't get let go. Even though the stomp clients are disconnecting >>>>>>>>> the >>>>>>>>> number of threads that get released is less than the number >>>>>>>>> created. >>>>>>>>> So >>>>>>>>> the thread count goes up and up until it fails. All of the above >>>>>>>>> settings/tuning only delay when it will hit the wall. >>>>>>>>> >>>>>>>>> Dejan Bosanac wrote: >>>>>>>>> >>>>>>>>>> Hi Mitch, >>>>>>>>>> >>>>>>>>>> I think the root cause of this problem is that you probably have >>>>>>>>>> Stomp >>>>>>>>>> clients that open/close connection at a high rate. I simulated >>>>>>>>>> this >>>>>>>>>> problem >>>>>>>>>> on OSX with a StompLoadTest ( >>>>>>>>>> >>>>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>>>>>>>> ), >>>>>>>>>> while trying to reproduce "too many open files" problem. You can >>>>>>>>>> find >>>>>>>>>> some >>>>>>>>>> of my findings (and workaround) in this thread. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>>>>>>>> >>>>>>>>>> (BTW. it is producing "too many open files" problem on linux) >>>>>>>>>> Basically, >>>>>>>>>> the >>>>>>>>>> problem with stomp is that every send is done in separate >>>>>>>>>> connection >>>>>>>>>> and >>>>>>>>>> thus considered to be a new producer for every message. So when >>>>>>>>>> producer >>>>>>>>>> flow control is hit, the producers are piling up and probably not >>>>>>>>>> releasing >>>>>>>>>> connections. Thus you can observe large number of tcp connections >>>>>>>>>> on >>>>>>>>>> the >>>>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>>>>>>> system >>>>>>>>>> limit is hit at one point. In the above thread, you can find a >>>>>>>>>> workaround >>>>>>>>>> that worked for me for that test. I started investigating this >>>>>>>>>> more >>>>>>>>>> and >>>>>>>>>> hopefully I'll have some more findings in the near future. >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> -- >>>>>>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>>>>>> >>>>>>>>>> Open Source Integration - http://fusesource.com/ >>>>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>>>>>> Blog - http://www.nighttale.net >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>>>>>>> <mitch.gran...@sophos.com>wrote: >>>>>>>>>> >>>>>>>>>> Update: We've [nearly] proven that this only happens with AMQ >>>>>>>>>> running >>>>>>>>>>> on >>>>>>>>>>> openVZ. What exactly is causing it, we're still not sure. >>>>>>>>>>> After >>>>>>>>>>> memoryUsage is met, the number of threads skyrockets until we >>>>>>>>>>> get >>>>>>>>>>> OutOfMemoryError. >>>>>>>>>>> >>>>>>>>>>> It works just fine on regular hardware; We're going to try >>>>>>>>>>> VMWare >>>>>>>>>>> tomorrow. >>>>>>>>>>> >>>>>>>>>>> One thing really worth mentioning is that by using the >>>>>>>>>>> fileCursor >>>>>>>>>>> we >>>>>>>>>>> actually started seeing it use the Temp Store. When reading >>>>>>>>>>> about >>>>>>>>>>> systemUsage it is NOT intuitive that the Temp Store does not >>>>>>>>>>> come >>>>>>>>>>> into >>>>>>>>>>> play >>>>>>>>>>> with the default cursor. Anyone keeping a significant volume of >>>>>>>>>>> messages on >>>>>>>>>>> their queues should be well served by changing the cursor. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Mitch Granger wrote: >>>>>>>>>>> >>>>>>>>>>> Config is attached. We have also tried the >>>>>>>>>>> activemq-scalability.xml >>>>>>>>>>>> with >>>>>>>>>>>> the only change being adding a stomp connector. >>>>>>>>>>>> >>>>>>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect >>>>>>>>>>>> new >>>>>>>>>>>> consumers >>>>>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>>>>>>> >>>>>>>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>>>>>>> subscribe >>>>>>>>>>>> there's no chance of recovering from this state. >>>>>>>>>>>> >>>>>>>>>>>> Rob Davies wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> We're using only persistent messages and heap size is set to >>>>>>>>>>>>> 2GB >>>>>>>>>>>>> yet >>>>>>>>>>>>> we >>>>>>>>>>>>> >>>>>>>>>>>>>> hit >>>>>>>>>>>>>> the memoryUsage limit quite quickly (system usage config >>>>>>>>>>>>>> below). >>>>>>>>>>>>>> This >>>>>>>>>>>>>> is >>>>>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>>>>>>> native >>>>>>>>>>>>>> thread" >>>>>>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. >>>>>>>>>>>>>> How >>>>>>>>>>>>>> are >>>>>>>>>>>>>> we >>>>>>>>>>>>>> getting to that point with the memoryUsage limit set far >>>>>>>>>>>>>> below >>>>>>>>>>>>>> it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory >>>>>>>>>>>>>> usage? >>>>>>>>>>>>>> >>>>>>>>>>>>>> <systemUsage> >>>>>>>>>>>>>> <systemUsage> >>>>>>>>>>>>>> <memoryUsage> >>>>>>>>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>>>>>>>> </memoryUsage> >>>>>>>>>>>>>> <storeUsage> >>>>>>>>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>>>>>>>> </storeUsage> >>>>>>>>>>>>>> <tempUsage> >>>>>>>>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>>>>>>>> </tempUsage> >>>>>>>>>>>>>> </systemUsage> >>>>>>>>>>>>>> </systemUsage> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> View this message in context: >>>>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at >>>>>>>>>>>>>> Nabble.com. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you send the rest of your config ? >>>>>>>>>>>>> >>>>>>>>>>>>> Rob Davies >>>>>>>>>>>>> http://twitter.com/rajdavies >>>>>>>>>>>>> I work here: http://fusesource.com >>>>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> ----- >>>>>> Dejan Bosanac >>>>>> >>>>>> Open Source Integration - http://fusesource.com/ >>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>> Blog - http://www.nighttale.net >>>>>> >>>>> http://old.nabble.com/file/p26264779/dump.jpg >>>>> -- >>>>> View this message in context: >>>>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html >>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> http://blog.garytully.com >>>> >>>> Open Source Integration >>>> http://fusesource.com >>>> >>>> >>> http://old.nabble.com/file/p26278228/oom.jpg >>> >> >> -- >> View this message in context: >> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > > > > -- > http://blog.garytully.com > > Open Source Integration > http://fusesource.com > > -- View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26297020.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
