Hi Ramayan, this is exception indicates that a message has been deleted from the store, but the in-memory queue still references it. Unfortunately the exception doesn't really tell us anything about how the broker will have got to this state.
Having looked at the code I have an idea about what may be happening - do your messages have a TTL set? I *think* that the AMQP 0-10 message path may be vulnerable to a race condition if a message expires at precisely the same time that the message is picked up to be sent to a consumer (essentially the message has to be available to consume and then three lines of code later it must have been deleted). The 0-9-1 codepath is not vulnerable to this as it caches the size of the message (as you can see in the stack trace, the consumer is checking the size of the message to make sure that the consumer has enough credit to receive it). We're unlikely to do a patch release for 0.32, but we will likely be putting out a new 6.0.x release soon, and soon after a 6.1 release. Would you be able to upgrade to one of these, or would you prefer me to send you a patch file that you could apply to the 0.32 source to test? -- Rob On 6 October 2016 at 21:27, Ramayan Tiwari <[email protected]> wrote: > Hi, > > > We are ran into this StoreException in our production environment multiple > times on different brokers, which caused broker shutdown. We are running > 0.32 Java broker with 0.16 client. I see that this was reported and fixed > here: > https://issues.apache.org/jira/browse/QPID-4012 > > This is still happening, I don't have enough context to reproduce this > locally. Any help is appreciated! > > Thanks > Ramayan > > *Exception* > > Uncaught exception, shutting down. > org.apache.qpid.server.store.StoreException: Metadata not found for message > with id 1762118451 > at > org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.getMessageMetaData(AbstractBDBMessageStore.java:343) > at > org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.getMetaData(AbstractBDBMessageStore.java:1224) > at > org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.getMetaData(MessageTransferMessage.java:41) > at > org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.getSize(MessageTransferMessage.java:56) > at > org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.allocateCredit(ConsumerTarget_0_10.java:494) > at > org.apache.qpid.server.queue.QueueConsumerImpl.wouldSuspend(QueueConsumerImpl.java:278) > at > org.apache.qpid.server.queue.AbstractQueue.attemptDelivery(AbstractQueue.java:2059) > at > org.apache.qpid.server.queue.AbstractQueue.flushConsumer(AbstractQueue.java:1981) > at > org.apache.qpid.server.queue.AbstractQueue.flushConsumer(AbstractQueue.java:1957) > at > org.apache.qpid.server.queue.QueueConsumerImpl.flush(QueueConsumerImpl.java:318) > at > org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.flush(ConsumerTarget_0_10.java:605) > at > org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageFlush(ServerSessionDelegate.java:521) > at > org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageFlush(ServerSessionDelegate.java:82) > at org.apache.qpid.transport.MessageFlush.dispatch(MessageFlush.java:87) > at > org.apache.qpid.transport.SessionDelegate.command(SessionDelegate.java:55) > at > org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:99) > at > org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:82) > at org.apache.qpid.transport.Method.delegate(Method.java:159) > at org.apache.qpid.transport.Session.received(Session.java:596) > at org.apache.qpid.transport.Connection.dispatch(Connection.java:452) > at > org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:64) > at > org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:40) > at > org.apache.qpid.transport.MethodDelegate.messageFlush(MethodDelegate.java:143) > at org.apache.qpid.transport.MessageFlush.dispatch(MessageFlush.java:87) > at > org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:54) > at > org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:40) > at org.apache.qpid.transport.Method.delegate(Method.java:159) > at org.apache.qpid.transport.Connection.received(Connection.java:405) > at > org.apache.qpid.server.protocol.v0_10.ServerConnection.access$001(ServerConnection.java:64) > at > org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run(ServerConnection.java:316) > at > org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run(ServerConnection.java:312) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:311) > at > org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:64) > at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) > at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:198) > at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) > at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) > at org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) > at org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) > at > org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:199) > at > org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:114) > at > org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.received(ProtocolEngine_0_10.java:179) > at > org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.received(ProtocolEngine_0_10.java:43) > at > org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:153) > at > org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:51) > at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) > at java.lang.Thread.run(Thread.java:745) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
