Gary, Agreed about the test case - my familiarity with the AMQ test suite is not that good (excuses, excuses :) ) so what I tried to do, unsuccessfully, was to play with the ConsumerTool without much luck on my dev machine which is a WinXP running on a Centrino Duo, and which is different from our target boxes - SunFire 420s. Scheduling is possibly different?
In any case, I'm glad you've been able to reproduce this. Is the fix going to be linked to the JIRA item? If so, I would be able to apply it (in case it really is localised) to our version of AMQ and continue with out tests. Cheers, Bonny Gary Tully wrote: > > Bonnie, > in CLIENT_ACK, the list is used on close to deal with duplicate > suppression, these messages will > get redelivered so they need to be removed from audit that tracks > their receipt. That logic is at fault. > > Cheers, > Gary. > > 2009/2/9 bonnyr <bon...@optusnet.com.au>: >> >> Gary, >> >> Why would the collection be used in CLIENT_ACK mode then? >> >> In our application we've got to work in this mode since there's a >> processing >> chain >> that may take a while between delivery of the message and its ack, so >> chaging this >> mode is not possible for our use case. >> >> We're aware of the fact that CLIENT_ACK actually acks all the messages in >> a >> session and >> it's not really related to the message itself, but that's a story for >> another day and another >> version of AMQ... >> >> However, wouldn't static analysis of the code be sufficient in this case? >> The code segment >> above is the only thing that is not protected by synchronization, whilst >> every other use of >> this collection is. >> >> Cheers, >> >> Bonny >> >> >> Gary Tully wrote: >>> >>>> I'm not quite able to cause this to happen in a simple test case. >>>> Perhaps >>>> this is to do with some >>>> thread timing/scheduling issues in the simple test case vs our >>>> application. >>>> >>> Yea, that is fairly typical. >>> >>>> However, are you in a >>>> position to explain what the deliveredMessage collection is used for? >>>> Is >>>> it >>>> not used in all connection/ >>>> ack modes? >>>> >>> Sure, the deliveredMessage list is used to generate acks in the Auto >>> ack mode. In addition, >>> with the consumer optimizeAcknowledge option, there is more likely to >>> be some outstanding >>> delivered messages that require acks. On close, any outstanding acks >>> will require delivery so >>> you may run into contention with concurrent delivery and close >>> execution as you suggest. >>> >>> May I suggest using AUTO_ACK and additionally using >>> optimizeAcknowledge if needed. >>> >>>> Perhaps if I knew what the behaviour should be like, I'd be able >>>> to construct a test >>>> case that simulate our environment. >>>> >>> Hopefully :-) >>> >>> Gary. >>> >>> >>>> Cheers, >>>> >>>> Bonny >>>> >>>> >>>> Gary Tully wrote: >>>>> >>>>> Hi Bonny, >>>>> that looks like a bug indeed, should be easy to replicate in a Junit >>>>> tests case I think. Could you raise a jira issue for this and if you >>>>> have some tests code that demonstrates, please include it. >>>>> >>>>> for more info see: http://activemq.apache.org/contributing.html >>>>> >>>>> Thanks, >>>>> Gary. >>>>> >>>>> 2009/2/6 bonnyr <bon...@optusnet.com.au>: >>>>>> >>>>>> AMQ 5.1 (but problem exists in the sources of AMQ-5.2 as of today) >>>>>> >>>>>> My setup: >>>>>> * Broker is configured with a single queue, full with messages, on a >>>>>> host >>>>>> accessible via the network. >>>>>> * Application configured with a single consumer, connected to a >>>>>> single >>>>>> sesssion, running in its own thread. >>>>>> * ActiveMQ delivers lots of messages using one of the >>>>>> ActiveMQSessionTask >>>>>> threads. >>>>>> * Session configured as CLIENT_ACKNOLEDGE >>>>>> >>>>>> >>>>>> Since the queue is full of messages, delivery of these messages >>>>>> happen >>>>>> as >>>>>> fast as the network >>>>>> can deliver and the AMQ thread is invoking the onMessage with each >>>>>> new >>>>>> message. These messages >>>>>> are then processed by the consumer thread. The consumer thread then >>>>>> decides >>>>>> to close the connection >>>>>> and the following ensues: >>>>>> [code] >>>>>> java.util.ConcurrentModificationException >>>>>> at >>>>>> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:617) >>>>>> at java.util.LinkedList$ListItr.next(LinkedList.java:552) >>>>>> at >>>>>> org.apache.activemq.ActiveMQMessageConsumer.dispose(ActiveMQMessageConsumer.java:663) >>>>>> at >>>>>> org.apache.activemq.ActiveMQMessageConsumer.close(ActiveMQMessageConsumer.java:583) >>>>>> at >>>>>> com.xxx.app..AMsgQueueConsumer.doClose(AMsgQueueConsumer.java:351) >>>>>> at com.xxx.app.AMsgQueueConsumer.suspend >>>>>> >>>>>> ... snip ... >>>>>> >>>>>> [/code] >>>>>> >>>>>> This happens because AMQ is busy delivering messages and there is a >>>>>> collection [b]deliveredMessages[/b] that is >>>>>> not protected by synchronisation in exactly one place (everywhere >>>>>> else >>>>>> it >>>>>> is...) which is executing the >>>>>> following code (in the dispose method): >>>>>> [code] >>>>>> ... snip ... >>>>>> if (session.isClientAcknowledge()) { >>>>>> if (!this.info.isBrowser()) { >>>>>> // rollback duplicates that aren't acknowledged >>>>>> for (MessageDispatch old : deliveredMessages) { >>>>>> session.connection.rollbackDuplicate(this, >>>>>> old.getMessage()); >>>>>> } >>>>>> } >>>>>> } >>>>>> ... snip ... >>>>>> [/code] >>>>>> >>>>>> Since the code iterates over the collection, and the iterator checks >>>>>> for >>>>>> modifications to the underlying collection, and since the AMQ message >>>>>> delivery thread has managed to sneak in a couple more messages >>>>>> while the consumer thread attempted to close the connection, the >>>>>> problem >>>>>> occurs (it could be that >>>>>> the for syntax hides the explicit iterator calls, but...) >>>>>> >>>>>> >>>>>> Is this an ommission or is there a reason for not synchronising >>>>>> access >>>>>> to >>>>>> this collection? If it's not a bug >>>>>> then how should the consumer be disconnected? >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Bonny >>>>>> -- >>>>>> View this message in context: >>>>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21867250.html >>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> http://blog.garytully.com >>>>> >>>>> Open Source SOA >>>>> http://FUSESource.com >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21907669.html >>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> http://blog.garytully.com >>> >>> Open Source SOA >>> http://FUSESource.com >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21924323.html >> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> >> > > > > -- > http://blog.garytully.com > > Open Source SOA > http://FUSESource.com > > -- View this message in context: http://www.nabble.com/ConcurrentModificationException-while-closing-consumer-tp21867250p21945059.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
