in org.apache.activemq.broker.region ,why so many invoke asyncWakeup(), what is the method:iterate() doing?
afei wrote: > > in addition,another problem of OOM. > > > > Gary Tully wrote: >> >> fyi: you can disable periodic message expiry processing using a >> destination policy entry that sets expireMessagesPeriod = 0 >> >> 2009/11/9 afei <afei1...@126.com>: >>> >>> >>> when a large of messages in queue,and no consumer or the consumer is >>> very >>> slow, the OOM problem occur, because : >>> in org.apache.activemq.broker.region.Queue,the 588 line is : >>> doBrowse(true, browsedMessages, this.getMaxExpirePageSize()); >>> ,transform to : >>> doBrowse(false, browsedMessages, this.getMaxExpirePageSize()); >>> is ok. >>> >>> >>> Dejan Bosanac wrote: >>>> >>>> Hi Mitch, >>>> >>>> yeah, I said in thread I was referring to, that it is working with >>>> "regular" >>>> stomp connector. I started investigating AMQ-2440 patch the other day, >>>> should be have something soon. >>>> >>>> Cheers >>>> -- >>>> Dejan Bosanac - http://twitter.com/dejanb >>>> >>>> Open Source Integration - http://fusesource.com/ >>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>> Blog - http://www.nighttale.net >>>> >>>> >>>> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger >>>> <mitch.gran...@sophos.com>wrote: >>>> >>>>> So we turned off stomp+nio and went back to plain old stomp and so far >>>>> it's >>>>> working fine. New(IO) isn't always better, I guess :-) >>>>> >>>>> Seems like maybe it's this issue -> >>>>> https://issues.apache.org/activemq/browse/AMQ-2440 >>>>> >>>>> >>>>> afei wrote: >>>>> >>>>>> i have same problem >>>>>> >>>>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg >>>>>> >>>>>> >>>>>> themitchy wrote: >>>>>> >>>>>>> This is what we've done to tune so far: >>>>>>> >>>>>>> - UseDedicatedTaskRunner=false >>>>>>> - flow control is off >>>>>>> - stomp transport uses transport.closeAsync=false >>>>>>> >>>>>>> I agree that it is because of the high number of open/close >>>>>>> connections >>>>>>> from Stomp. When we monitor through JConsole we can see more >>>>>>> threads >>>>>>> starting up for each new connection. The problem is that these >>>>>>> threads >>>>>>> don't get let go. Even though the stomp clients are disconnecting >>>>>>> the >>>>>>> number of threads that get released is less than the number created. >>>>>>> So >>>>>>> the thread count goes up and up until it fails. All of the above >>>>>>> settings/tuning only delay when it will hit the wall. >>>>>>> >>>>>>> Dejan Bosanac wrote: >>>>>>> >>>>>>>> Hi Mitch, >>>>>>>> >>>>>>>> I think the root cause of this problem is that you probably have >>>>>>>> Stomp >>>>>>>> clients that open/close connection at a high rate. I simulated this >>>>>>>> problem >>>>>>>> on OSX with a StompLoadTest ( >>>>>>>> >>>>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log >>>>>>>> ), >>>>>>>> while trying to reproduce "too many open files" problem. You can >>>>>>>> find >>>>>>>> some >>>>>>>> of my findings (and workaround) in this thread. >>>>>>>> >>>>>>>> >>>>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080 >>>>>>>> >>>>>>>> (BTW. it is producing "too many open files" problem on linux) >>>>>>>> Basically, >>>>>>>> the >>>>>>>> problem with stomp is that every send is done in separate >>>>>>>> connection >>>>>>>> and >>>>>>>> thus considered to be a new producer for every message. So when >>>>>>>> producer >>>>>>>> flow control is hit, the producers are piling up and probably not >>>>>>>> releasing >>>>>>>> connections. Thus you can observe large number of tcp connections >>>>>>>> on >>>>>>>> the >>>>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the >>>>>>>> system >>>>>>>> limit is hit at one point. In the above thread, you can find a >>>>>>>> workaround >>>>>>>> that worked for me for that test. I started investigating this more >>>>>>>> and >>>>>>>> hopefully I'll have some more findings in the near future. >>>>>>>> >>>>>>>> Cheers >>>>>>>> -- >>>>>>>> Dejan Bosanac - http://twitter.com/dejanb >>>>>>>> >>>>>>>> Open Source Integration - http://fusesource.com/ >>>>>>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>>>>>> Blog - http://www.nighttale.net >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger >>>>>>>> <mitch.gran...@sophos.com>wrote: >>>>>>>> >>>>>>>> Update: We've [nearly] proven that this only happens with AMQ >>>>>>>> running >>>>>>>>> on >>>>>>>>> openVZ. What exactly is causing it, we're still not sure. After >>>>>>>>> memoryUsage is met, the number of threads skyrockets until we get >>>>>>>>> OutOfMemoryError. >>>>>>>>> >>>>>>>>> It works just fine on regular hardware; We're going to try VMWare >>>>>>>>> tomorrow. >>>>>>>>> >>>>>>>>> One thing really worth mentioning is that by using the fileCursor >>>>>>>>> we >>>>>>>>> actually started seeing it use the Temp Store. When reading about >>>>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come >>>>>>>>> into >>>>>>>>> play >>>>>>>>> with the default cursor. Anyone keeping a significant volume of >>>>>>>>> messages on >>>>>>>>> their queues should be well served by changing the cursor. >>>>>>>>> >>>>>>>>> >>>>>>>>> Mitch Granger wrote: >>>>>>>>> >>>>>>>>> Config is attached. We have also tried the >>>>>>>>> activemq-scalability.xml >>>>>>>>>> with >>>>>>>>>> the only change being adding a stomp connector. >>>>>>>>>> >>>>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new >>>>>>>>>> consumers >>>>>>>>>> but nothing comes back after we send the SUBSCRIBE frame. >>>>>>>>>> >>>>>>>>>> I expect sending to fail when we hit this limit but if we can't >>>>>>>>>> subscribe >>>>>>>>>> there's no chance of recovering from this state. >>>>>>>>>> >>>>>>>>>> Rob Davies wrote: >>>>>>>>>> >>>>>>>>>> On 26 Oct 2009, at 17:38, themitchy wrote: >>>>>>>>>>> >>>>>>>>>>> We're using only persistent messages and heap size is set to >>>>>>>>>>> 2GB >>>>>>>>>>> yet >>>>>>>>>>> we >>>>>>>>>>> >>>>>>>>>>>> hit >>>>>>>>>>>> the memoryUsage limit quite quickly (system usage config >>>>>>>>>>>> below). >>>>>>>>>>>> This >>>>>>>>>>>> is >>>>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new >>>>>>>>>>>> native >>>>>>>>>>>> thread" >>>>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it. How >>>>>>>>>>>> are >>>>>>>>>>>> we >>>>>>>>>>>> getting to that point with the memoryUsage limit set far below >>>>>>>>>>>> it? >>>>>>>>>>>> >>>>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory >>>>>>>>>>>> usage? >>>>>>>>>>>> >>>>>>>>>>>> <systemUsage> >>>>>>>>>>>> <systemUsage> >>>>>>>>>>>> <memoryUsage> >>>>>>>>>>>> <memoryUsage limit="256 mb"/> >>>>>>>>>>>> </memoryUsage> >>>>>>>>>>>> <storeUsage> >>>>>>>>>>>> <storeUsage limit="60 gb" name="foo"/> >>>>>>>>>>>> </storeUsage> >>>>>>>>>>>> <tempUsage> >>>>>>>>>>>> <tempUsage limit="60 gb"/> >>>>>>>>>>>> </tempUsage> >>>>>>>>>>>> </systemUsage> >>>>>>>>>>>> </systemUsage> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> View this message in context: >>>>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html >>>>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at >>>>>>>>>>>> Nabble.com. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Can you send the rest of your config ? >>>>>>>>>>> >>>>>>>>>>> Rob Davies >>>>>>>>>>> http://twitter.com/rajdavies >>>>>>>>>>> I work here: http://fusesource.com >>>>>>>>>>> My Blog: http://rajdavies.blogspot.com/ >>>>>>>>>>> I'm writing this: http://www.manning.com/snyder/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> ----- >>>> Dejan Bosanac >>>> >>>> Open Source Integration - http://fusesource.com/ >>>> ActiveMQ in Action - http://www.manning.com/snyder/ >>>> Blog - http://www.nighttale.net >>>> >>> http://old.nabble.com/file/p26264779/dump.jpg >>> -- >>> View this message in context: >>> http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> http://blog.garytully.com >> >> Open Source Integration >> http://fusesource.com >> >> > http://old.nabble.com/file/p26278228/oom.jpg > -- View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26279671.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
