Ultimately I'm pretty confident this problem is an NFS problem - and as Johan has already let the cat out of the bag ;) - let me ask the following:
Which version of NFS 4 are you using and which environment? Have you checked the system logs for NFS errors on all the machines running ActiveMQ brokers ? thanks, Rob On 29 May 2013, at 00:46, Christian Posta <christian.po...@gmail.com> wrote: > I can make two recommendations. > > #1, being the preferred, create a test case that shows this... that will > give us the best chance of finding out what's going on... take a look at > the following test cases in the activemq source code to give you an idea > about how to go about doing it... > > http://svn.apache.org/viewvc/activemq/trunk/activemq-unit-tests/src/test/java/org/apache/activemq/usecases/ > > http://svn.apache.org/viewvc/activemq/trunk/activemq-unit-tests/src/test/java/org/apache/activemq/bugs/ > > http://svn.apache.org/viewvc/activemq/trunk/activemq-unit-tests/src/test/java/org/apache/activemq/test/JmsTopicSendReceiveTest.java?view=markup > > > #2, if creating a test case doesn't sound like something you want to get > into.. i guess, give us the exact configs of broker, clients, number of > consumers, number of topics, message sizes, etc, etc all details and if one > of us gets the urge we can try it out on our boxes. this will not be nearly > as good as #1, and will provide a higher barrier to entry because we spend > our spare time doing this and like to spend that time debugging and fixing, > and not setting up environments and usecases which may not even show a bug > :) > > > > > On Tue, May 28, 2013 at 4:34 PM, fenbers <mark.fenb...@noaa.gov> wrote: > >> >> >> >> >> >> I'm getting the Sync exception on both, local and NFS. >> Originally, >> I was only using a local disk, but there wasn't much disk space for >> the ever growing list of 33MB enumerated .log files that weren't >> cleaned up. So I reconfigured ActiveMQ to put these db files on >> an >> NFS mount. But the sync exceptions occurred either way. >> >> I've changed *all* my consumers to AUTO_ACKNOWLEDGE, thinking that >> maybe an ACKNOWLEDGEment leak was causing the undeleted files. >> That >> didn't help... The TRACE level logging points to only two of my 5 >> topics that accumulate these undeleted db files. So I've >> concentrated by scrutiny over consumers of these two topics. But >> have not found anything out of the ordinary. >> >> What is puzzling me still, is that the frequency of the log file >> build-up and the frequency of exceptions continues to increase even >> though the amount of messages sent per day by the producers remains >> nearly constant... >> Mark >> >> On 5/28/2013 6:06 PM, ceposta [via >> ActiveMQ] wrote: >> >> Sounds like there's multiple issues... >> >> You're journal files aren't being cleaned up, AND you're getting >> the Sync >> >> exception? >> >> You get the sync exception on local disk mount? Or just NFS? >> >> >> If the journals aren't being cleaned up, are your consumers >> properly >> >> ack'ing messages? >> >> >> >> On Tue, May 28, 2013 at 2:42 PM, fenbers < [hidden email] > >> wrote: >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > I would LOVE to help you help me!&nbsp; But >> I have >> no idea how to go >> >> > about making a test case.&nbsp; If you >> could drop >> some hints in this >> >> > regard, I might be able to produce one. >> >> > >> >> > My ActiveMQ issues seem to be related to network >> slowness, which we >> >> > are diagnosing separately.&nbsp; Or maybe >> it is the >> other way around, >> >> > where ActiveMQ problems are causing network >> sluggishness.&nbsp; Either >> >> > way, there seems to be a correlation, except >> that when >> network >> >> > responsiveness improves, ActiveMQ does not. >> >> > >> >> > The problem I'm having with AMQ is progressive, >> which >> is even more >> >> > puzzling, because we are not adding to the >> number of >> messages that >> >> > AMQ has to handle.&nbsp; Today, we were up >> to 191 >> undeleted db-NNN.log >> >> > files in the database directory before I >> stopped AMQ >> and deleted >> >> > them.&nbsp;&nbsp; NNN was up to 451, so >> 260 >> files had been cleaned up >> >> > by AMQ's >> >> > automatic processes... >> >> > >> >> > Will log files assist you in helping >> me?&nbsp; I >> have TRACE level >> >> > messages turned on, so they are quite large. >> >> > >> >> > Mark >> >> > >> >> > On 5/28/2013 5:22 PM, rajdavies [via >> >> > ActiveMQ] wrote: >> >> > >> >> > Hi Mark, >> >> > >> >> > >> >> > could you produce a test case for your >> problem - it >> would help us >> >> > identify the problem a lot quicker >> >> > >> >> > >> >> > thanks, >> >> > >> >> > >> >> > Rob >> >> > >> >> > On 30 Apr 2013, at 16:40, fenbers >> &lt; [hidden >> email] &gt; >> >> > wrote: >> >> > >> >> > >> >> > &gt; Zagan wrote >> >> > >> >> > &gt;&gt; Can you please >> check if your .log >> files in the /data >> >> > directory are cleaned >> >> > >> >> > &gt;&gt; up? On basis of >> the information I >> suppose this >> >> > behaviour is due to a >> >> > >> >> > &gt;&gt; misconfiguration >> of your clients. >> >> > >> >> > &gt;&gt; If this is the >> case often broken >> log file cleanup is a >> >> > symptom. >> >> > >> >> > &gt; >> >> > &gt; I get the same error as >> brought up in this >> thread (KahaDB >> >> > failed to store to >> >> > >> >> > &gt; Journal). &nbsp;And >> yes, I also have a >> problem with the >> >> > numbered >> >> > .log files not >> >> > >> >> > &gt; all getting cleaned up >> (most files are >> removed >> >> > appropriately). &nbsp;I have >> >> > >> >> > &gt; suspected a client >> configuration problem >> for a long time, >> >> > but can't figure >> >> > >> >> > &gt; out what's wrong -- even >> with TRACE >> logging turned on. >> >> > &nbsp;In >> >> > the meantime, I >> >> > >> >> > &gt; have to cope with >> ActiveMQ crashing (i.e., >> shutting itself >> >> > down) about every >> >> > >> >> > &gt; two days. &nbsp;The >> logs point to a >> disk storage problem, but >> >> > I >> >> > have plenty of >> >> > >> >> > &gt; space, so that's not the >> issue! >> &nbsp;I've tried a couple of >> >> > different Linux >> >> > >> >> > &gt; boxes and both local and >> NFS mounts, and >> this issue occurs >> >> > on both of them. >> >> > >> >> > &gt; >> >> > &gt; I'm at a loss!! >> &nbsp;I'm running >> 5.8.0... >> >> > >> >> > &gt; >> >> > &gt; Mark >> >> > >> >> > &gt; >> >> > &gt; >> >> > &gt; >> >> > &gt; -- >> >> > >> >> > &gt; View this message in >> context: >> >> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-crashes-frequently-tp4305407p4666469.html >> > &gt; Sent from the ActiveMQ - >> User mailing list >> archive at >> >> > Nabble.com. >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > If you reply to this email, your >> >> > message will be added to >> the discussion below: >> >> > >> >> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-crashes-frequently-tp4305407p4667572.html >> > >> >> > >> >> > To unsubscribe from ActiveMQ >> crashes frequently, >> click >> >> > here . >> >> > NAML >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > mark_fenbers.vcf (360 bytes) < >> >> > >> http://activemq.2283324.n4.nabble.com/attachment/4667574/0/mark_fenbers.vcf >> > > >> >> > >> >> > >> >> > >> >> > >> >> > -- >> >> > View this message in context: >> >> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-crashes-frequently-tp4305407p4667574.html >> > Sent from the ActiveMQ - User mailing list archive at >> Nabble.com. >> >> > >> >> >> >> >> -- >> *Christian Posta* >> >> http://www.christianposta.com/blog >> twitter: @christianposta >> >> http://www.christianposta.com/blog >> >> >> >> >> >> If you reply to this email, your >> message will be added to the discussion below: >> >> http://activemq.2283324.n4.nabble.com/ActiveMQ-crashes-frequently-tp4305407p4667575.html >> >> >> To unsubscribe from ActiveMQ crashes frequently, click >> here . >> NAML >> >> >> >> >> >> >> mark_fenbers.vcf (360 bytes) < >> http://activemq.2283324.n4.nabble.com/attachment/4667583/0/mark_fenbers.vcf >>> >> >> >> >> >> -- >> View this message in context: >> http://activemq.2283324.n4.nabble.com/ActiveMQ-crashes-frequently-tp4305407p4667583.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > > > > -- > *Christian Posta* > http://www.christianposta.com/blog > twitter: @christianposta
