I've added a patch file which should apply to 0.32 on the following JIRA: https://issues.apache.org/jira/browse/QPID-7451
Hope this helps, Rob On 6 October 2016 at 22:04, Rob Godfrey <[email protected]> wrote: > Hi Ramayan, > > this is exception indicates that a message has been deleted from the > store, but the in-memory queue still references it. Unfortunately the > exception doesn't really tell us anything about how the broker will > have got to this state. > > Having looked at the code I have an idea about what may be happening - > do your messages have a TTL set? I *think* that the AMQP 0-10 message > path may be vulnerable to a race condition if a message expires at > precisely the same time that the message is picked up to be sent to a > consumer (essentially the message has to be available to consume and > then three lines of code later it must have been deleted). The 0-9-1 > codepath is not vulnerable to this as it caches the size of the > message (as you can see in the stack trace, the consumer is checking > the size of the message to make sure that the consumer has enough > credit to receive it). > > We're unlikely to do a patch release for 0.32, but we will likely be > putting out a new 6.0.x release soon, and soon after a 6.1 release. > Would you be able to upgrade to one of these, or would you prefer me > to send you a patch file that you could apply to the 0.32 source to > test? > > -- Rob > > On 6 October 2016 at 21:27, Ramayan Tiwari <[email protected]> wrote: >> Hi, >> >> >> We are ran into this StoreException in our production environment multiple >> times on different brokers, which caused broker shutdown. We are running >> 0.32 Java broker with 0.16 client. I see that this was reported and fixed >> here: >> https://issues.apache.org/jira/browse/QPID-4012 >> >> This is still happening, I don't have enough context to reproduce this >> locally. Any help is appreciated! >> >> Thanks >> Ramayan >> >> *Exception* >> >> Uncaught exception, shutting down. >> org.apache.qpid.server.store.StoreException: Metadata not found for message >> with id 1762118451 >> at >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.getMessageMetaData(AbstractBDBMessageStore.java:343) >> at >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.getMetaData(AbstractBDBMessageStore.java:1224) >> at >> org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.getMetaData(MessageTransferMessage.java:41) >> at >> org.apache.qpid.server.protocol.v0_10.MessageTransferMessage.getSize(MessageTransferMessage.java:56) >> at >> org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.allocateCredit(ConsumerTarget_0_10.java:494) >> at >> org.apache.qpid.server.queue.QueueConsumerImpl.wouldSuspend(QueueConsumerImpl.java:278) >> at >> org.apache.qpid.server.queue.AbstractQueue.attemptDelivery(AbstractQueue.java:2059) >> at >> org.apache.qpid.server.queue.AbstractQueue.flushConsumer(AbstractQueue.java:1981) >> at >> org.apache.qpid.server.queue.AbstractQueue.flushConsumer(AbstractQueue.java:1957) >> at >> org.apache.qpid.server.queue.QueueConsumerImpl.flush(QueueConsumerImpl.java:318) >> at >> org.apache.qpid.server.protocol.v0_10.ConsumerTarget_0_10.flush(ConsumerTarget_0_10.java:605) >> at >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageFlush(ServerSessionDelegate.java:521) >> at >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.messageFlush(ServerSessionDelegate.java:82) >> at org.apache.qpid.transport.MessageFlush.dispatch(MessageFlush.java:87) >> at >> org.apache.qpid.transport.SessionDelegate.command(SessionDelegate.java:55) >> at >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:99) >> at >> org.apache.qpid.server.protocol.v0_10.ServerSessionDelegate.command(ServerSessionDelegate.java:82) >> at org.apache.qpid.transport.Method.delegate(Method.java:159) >> at org.apache.qpid.transport.Session.received(Session.java:596) >> at org.apache.qpid.transport.Connection.dispatch(Connection.java:452) >> at >> org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:64) >> at >> org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:40) >> at >> org.apache.qpid.transport.MethodDelegate.messageFlush(MethodDelegate.java:143) >> at org.apache.qpid.transport.MessageFlush.dispatch(MessageFlush.java:87) >> at >> org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:54) >> at >> org.apache.qpid.transport.ConnectionDelegate.command(ConnectionDelegate.java:40) >> at org.apache.qpid.transport.Method.delegate(Method.java:159) >> at org.apache.qpid.transport.Connection.received(Connection.java:405) >> at >> org.apache.qpid.server.protocol.v0_10.ServerConnection.access$001(ServerConnection.java:64) >> at >> org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run(ServerConnection.java:316) >> at >> org.apache.qpid.server.protocol.v0_10.ServerConnection$1.run(ServerConnection.java:312) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:360) >> at >> org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:311) >> at >> org.apache.qpid.server.protocol.v0_10.ServerConnection.received(ServerConnection.java:64) >> at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:97) >> at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:198) >> at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:131) >> at org.apache.qpid.transport.network.Frame.delegate(Frame.java:128) >> at org.apache.qpid.transport.network.Assembler.received(Assembler.java:102) >> at org.apache.qpid.transport.network.Assembler.received(Assembler.java:44) >> at >> org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:199) >> at >> org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:114) >> at >> org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.received(ProtocolEngine_0_10.java:179) >> at >> org.apache.qpid.server.protocol.v0_10.ProtocolEngine_0_10.received(ProtocolEngine_0_10.java:43) >> at >> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:153) >> at >> org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:51) >> at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) >> at java.lang.Thread.run(Thread.java:745) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
