Here's what I suspect is happening. Broker 1 delivers the message, and you process it but don't acknowledge it. Broker 1 goes down, Broker 2 becomes master, and you reconnect to Broker 2. Broker 2 sees that the message is in the queue (and has no way to know that it's been delivered but never acked, so all it knows is that it's supposed to deliver the message), and dispatches it to your client. Your client receives the message, says "that's a duplicate, I've already seen that one" and rejects it. This delivery-duplicate cycle happens 6 times in rapid succession, and Broker 2 moves the message to the DLQ.
Can you please turn up ActiveMQ-related logging to DEBUG on both the broker and the client and see if the logs seem to confirm that theory? If that's in fact what's going on, we'll probably need to modify the client logic to treat the first redelivery of an unacked message following a failover as a non-duplicate delivery that we should ignore but not tell the broker is a dupe. Tim On Sun, Apr 10, 2016 at 12:34 PM, Natarajan, Rajeswari < rajeswari.natara...@sap.com> wrote: > Hdp132 - master > > I see the Exception message 2 times. > > URI > failover:(tcp://hostname1:61616,tcp://hostname2:61616) > > -----Original Message----- > From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain > Sent: Saturday, April 09, 2016 9:44 PM > To: ActiveMQ Users <users@activemq.apache.org> > Subject: RE: MasterSlave config ActiveMQ > > Is hdp132 the master or the slave? And what URI does your client use to > connect? > > Do you see that message once, or six times? > On Apr 8, 2016 12:32 PM, "Natarajan, Rajeswari" < > rajeswari.natara...@sap.com> > wrote: > > > Thanks Tim for the detailed email. When I do failover I see below > > exception in our app logs. > > > > org.apache.activemq.transport.failover.FailoverTransport > > Transport (tcp://hdp132.lab1.ariba.com:61616) failed, attempting to > > automatically reconnect java.io.EOFException: null > > at java.io.DataInputStream.readInt(DataInputStream.java:392) > > at > > > org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:267) > > at > > > org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) > > at > > > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) > > at > > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) > > > > The message redelivery attempt is default in the activemq conf, so it is > > 6 times. > > > > > > I just send 1 message to the queue ,receive it and don't ack it ,then do > > failover . There is a thread which listens for receiving messages in the > > queue ,I don't see any exception in there . So not sure from where the > > transport exception is thrown. > > In the produce send , receive and ack code the JMS > > Is there a way to track failover happened. May be some sort of listener > > ,then the session can be recovered. > > > > Regards, > > Rajeswari > > > > -----Original Message----- > > From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim Bain > > Sent: Thursday, April 07, 2016 8:53 PM > > To: ActiveMQ Users <users@activemq.apache.org> > > Subject: Re: MasterSlave config ActiveMQ > > > > I apologize, I misunderstood what you meant by "I made sure the messages > > will not be acknowledged." > > > > When either the broker or a consumer goes down while a consumer is > actively > > consuming a message, the broker will consider that a filed delivery and > try > > to redeliver the message. The maximumRedeliveries property of the > > Redelivery Policy (http://activemq.apache.org/redelivery-policy.html) > > controls how many times the message will be redelivered before being put > > into the DLQ. The default value is 6; is this the value you're using, or > > have you explicitly set a different value? With the default value of 6, > > your consumer should fail over to the new master, the message should be > > redelivered (because this is redelivery attempt #1 and that's less than > 6), > > and your consumer should start processing the message for the second > time. > > As long as that doesn't throw any exceptions or fail in any other way > (have > > you confirmed this?), the second redelivery attempt should eventually > > result in a successful processing of the message. > > > > However, it sounds like that's not happening for you, and > > > > > http://stackoverflow.com/questions/8576821/cant-get-activemq-to-resend-my-messages#comment35122357_8576821 > > seems to indicate that when using INDIVIDUAL_ACKNOWLEDGE mode, if you > don't > > close the Connection or recover() the Session, the messages will be > > considered duplicates and ignored by the consumer. I haven't worked with > > INDIVIDUAL_ACKNOWLEDGE mode myself; can anyone else on the list shed more > > light on whether that comment is accurate? > > > > If so, it would sound like clients who use INDIVIDUAL_ACKNOWLEDGE mode > > shouldn't use the failover transport, because they need the connection to > > die so they can recreate it. If that were true, I'd expect we'd have a > > warning somewhere on the wiki saying that those two features aren't > > compatible with one another, but I haven't seen one in my searching this > > evening... > > > > Tim > > > > On Thu, Apr 7, 2016 at 12:02 PM, Natarajan, Rajeswari < > > rajeswari.natara...@sap.com> wrote: > > > > > I am trying to understand, sorry if I sound , I don't believe you or > > > skeptic. I have an application with the broker url defined as below > > (with > > > shared file system) > > > > > > failover:(tcp://<hostname1>:61616,tcp://<hostname2>:61616. > > > > > > When I failover I see the web console and see one consumer on the fail > > > over host ,as the application is having a consumer and got a message > ,but > > > did not acknowledge (delaying it on purpose for a use case) during > > > failover.. In our use case, there will be messagse sent to the queue > and > > we > > > do some processing ,but not acknowledging the messages ,unless we get a > > > condition satisfied ,which might take some time. So when a failover > > happens > > > within that time ,all such messages will be moved to DLQ? > > > Is there any way to have them in the same state as before in the fail > > > over host. > > > > > > Thank you, > > > Rajeswari > > > > > > -----Original Message----- > > > From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim > Bain > > > Sent: Thursday, April 07, 2016 5:41 AM > > > To: ActiveMQ Users <users@activemq.apache.org> > > > Subject: RE: MasterSlave config ActiveMQ > > > > > > Quoting from what I originally wrote you: "or if there's a consumer > > > on the failover host that tries and fails to consume them." Consuming > a > > > message but not acking it == failing to consume it. The failover > > > functionality works, despite your apparent skepticism; retry your test > > with > > > no consumers if you don't believe me. > > > > > > Tim > > > On Apr 7, 2016 5:22 AM, "Natarajan, Rajeswari" < > > > rajeswari.natara...@sap.com> > > > wrote: > > > > > > > FYI using non-transactional session with > > > > ActiveMQSession.INDIVIDUAL_ACKNOWLEDGE > > > > > > > > Rajeswari > > > > > > > > -----Original Message----- > > > > From: Natarajan, Rajeswari [mailto:rajeswari.natara...@sap.com] > > > > Sent: Thursday, April 07, 2016 3:57 AM > > > > To: users@activemq.apache.org > > > > Subject: RE: MasterSlave config ActiveMQ > > > > > > > > Thank you. But in my case I am sure the messages have not reached > > > > JMSExpiration date. > > > > > > > > I assume there will be a consumer on the failover host ,once the > > failover > > > > happens. But when I tested, I made sure the messages will not be > > > > acknowledged . Still I see consistently the pending/enqueued > messages > > go > > > > DLQ upon failover and don't appear in the original Queue as enqueued. > > > > Am I missing something > > > > > > > > Rajeswari > > > > > > > > -----Original Message----- > > > > From: tbai...@gmail.com [mailto:tbai...@gmail.com] On Behalf Of Tim > > Bain > > > > Sent: Wednesday, April 06, 2016 10:26 PM > > > > To: ActiveMQ Users <users@activemq.apache.org> > > > > Subject: Re: MasterSlave config ActiveMQ > > > > > > > > Only if they had reached their JMSExpiration date, or if there's a > > > consumer > > > > on the failover host that tries and fails to consume them. > Otherwise, > > > they > > > > will continue to be available on the queue. > > > > > > > > On Wed, Apr 6, 2016 at 5:03 PM, Natarajan, Rajeswari < > > > > rajeswari.natara...@sap.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > I am testing activemq master slave config using shared file system. > > > > > > > > > > 1) I issued 1 message to a queue and I saw that in both pending > > message > > > > > count and enqueued count. The message was not dequeued. > > > > > > > > > > 2)Killed the master ,the secondary took over ,in secondary's > > > webconsole I > > > > > see that > > > > > > > > > > Messages dequeued as 1 > > > > > > > > > > And in the ActiveMQ.DLQ I see 1 pending message and 1 enqueued > > > message. > > > > > > > > > > > > > > > So in case of failover ,the pending/unacknowledged messages go to > DLQ > > > and > > > > > never get processed ? > > > > > > > > > > > > > > > Thank you, > > > > > Rajeswari > > > > > > > > > > > > > > >
