On 09/02/16 12:52, Gordon Sim wrote:
On 02/09/2016 10:37 AM, Toralf Lund wrote:
On 08/02/16 18:07, Gordon Sim wrote:
When enqueing a message, the store attaches its own identifier to it
for use when dequeueing the same message. The error message indicates
that this identifier could not be obtained from the message when it
was trying to dequeue.
OK.
I must say I'm struggling to understand why there is dequeuing from the
queue in question at the spot I'm getting the error message, though.
What the application actually does at the time, is to set up a receiver
for a completely different queue.
Is the same session used for subscribing to the queue for which the
error is reported? (or sending to it if it is indeed an LVQ?)
The process in question does not subscribe to the queue at all, but it
does send to it - indirectly through an exchange. And the same session
is used.
The system can keep sending to the exchange and otherwise work just fine
as long as nothing is bound to the message address(es) used, but the
problems start occurring as soon as my receiver queue is set up - which
is done by a different (client) application.
With AMQP 0-10, once a session hits an error, that is reported for all
subsequent attempts to use the session. With asynchronous operations
the error may not be detected immediately when issuing the triggering
command.
Oh. Right. I guess that might explain it. I knew that the session would
keep the error state, but didn't quite realise that the reporting might
be "delayed" in this manner.
Something else I noticed just now, is that Connection::isOpen() starts
to return "false" somewhere in the sequence leading up to the exception.
This is consistent with a message I'm getting if enabling trace in the
application (as opposed to the broker):
2016-02-09 13:44:29 [Client] warning Broker closed connection: 501,
Queue "pgs.compass.out": Dequeuing message with null persistence Id.
(/work/build/build/qpid-0.22/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1374)
I didn't check the connection object at this stage before, but only
dealt with the session.
It is essentially some kind of internal bug. There have been issues
fixed with similar symptoms when using an LVQ[1], or a paged queue[2],
or federation (though this should have been fixed a very long time
ago) [3]. There could be other causes as well.
Right. LVQ is definitely involved.
And it is a durable LVQ?
Queue option "--durable" is used.
The broker is also running in a cluster configuration.
If possible, I would suggest upgrading as there have been a lot of
changes and fixes since 0.22.
OK. I will certainly consider that.
I'm wondering if there is any way at all to get temporarily out of this
situation, though. Like I said, I didn't use to get this error, and the
application also seems to work when using a different broker, so I'm
assuming it's somehow related to some kind of unexpected state change. I
tried moving away /root/.qpidd and restarting, but it didn't help...
That is strange. Restarting should get rid of all state. How easy is
it to reproduce on that broker? Would generating a protocol trace be
feasible? (The brokers are all the same version? and that hasn't
changed since it worked? Is the store enabled for all of them?)
I can reproduce the problem every time I try now. The other broker is
using the same QPid version, and I think it's supposed to be set up the
same way. It's managed by someone else (who didn't understand the error
message either when I asked), though, so I guess I should check closer.
I don't think anything has changed since it all worked, but I may of
course be missing something (as I tend to suspect when this kind of
thing happens.)
I tried enabling some trace in the broker, too, but didn't notice
anything obvious there, except that the same error message is included a
little section is included below.
I guess I have to look into installing a newer version...
- Toralf
2016-02-09 13:38:49 [Protocol] trace SENT
[qpid.10.30.15.211:5672-10.30.15.212:40182]: Frame[BEbe; channel=1;
{ExecutionSyncBody: }]
2016-02-09 13:38:49 [Protocol] trace RECV
[qpid.10.30.15.215:5672-10.30.39.189:55220]: Frame[Bbe; channel=1;
{MessageTransferBody: destination=DATA.topic.exch; accept-mode=1;
acquire-mode=0; }]
2016-02-09 13:38:49 [Protocol] trace RECV
[qpid.10.30.15.215:5672-10.30.39.189:55220]: Frame[be; channel=1; header
(222 bytes); properties={{MessageProperties: content-length=2;
correlation-id=; content-type=application/x-protobuf; user-id=;
application-headers={message_type:V2:21:vbin16(Compass.Out.Heartbeat),qpid.subject:V2:21:str16(compass.out.heartbeat),timestamp:F8:int64(1455021843),timestamp_usec:F8:int64(73421),version:V2:13:vbin16(not
specified)}; }{DeliveryProperties: routing-key=compass.out.heartbeat; }}]
2016-02-09 13:38:49 [Protocol] trace RECV
[qpid.10.30.15.215:5672-10.30.39.189:55220]: Frame[Ebe; channel=1;
content (2 bytes) \x08\x00]
2016-02-09 13:38:49 [Protocol] trace RECV
[qpid.10.30.15.211:5672-10.30.15.212:40182]: Frame[BEbe; channel=1;
{SessionCompletedBody: commands={ [0,3259] }; }]
2016-02-09 13:38:49 [Broker] warning Exchange DATA.topic.exch cannot
deliver to queue pgs.compass.out: Queue "pgs.compass.out": Dequeuing
message with null persistence Id.
(/work/build/build/qpid-0.22/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1374)
2016-02-09 13:38:49 [Protocol] trace SENT
[qpid.10.30.15.215:5672-10.30.39.189:55054]: Frame[Bbe; channel=3;
{MessageTransferBody: destination=pgs.compass.out; accept-mode=0;
acquire-mode=1; }]
2016-02-09 13:38:49 [Protocol] error Unexpected exception: Queue
"pgs.compass.out": Dequeuing message with null persistence Id.
(/work/build/build/qpid-0.22/cpp/src/qpid/legacystore/MessageStoreImpl.cpp:1374)
2016-02-09 13:38:49 [Protocol] trace SENT
[qpid.10.30.15.211:5672-10.30.15.212:40182]: Frame[Bbe; channel=12;
{MessageTransferBody: destination=qpid.replicator-pgs.compass.out;
accept-mode=0; acquire-mode=1; }]
2016-02-09 13:38:49 [Protocol] trace SENT
[qpid.10.30.15.215:5672-10.30.39.189:55054]: Frame[be; channel=3; header
(246 bytes); properties={{MessageProperties: content-length=2;
correlation-id=; content-type=application/x-protobuf; user-id=;
application-headers={message_type:V2:21:vbin16(Compass.Out.Heartbeat),qpid.subject:V2:21:str16(compass.out.heartbeat),timestamp:F8:int64(1455021843),timestamp_usec:F8:int64(73421),version:V2:13:vbin16(not
specified)}; }{DeliveryProperties: timestamp=1455021529;
exchange=DATA.topic.exch; routing-key=compass.out.heartbeat; }}]
2016-02-09 13:38:49 [Protocol] trace SENT
[qpid.10.30.15.211:5672-10.30.15.212:40182]: Frame[be; channel=12;
header (24 bytes); properties={{DeliveryProperties:
routing-key=qpid.ha-dequeue; }}]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[http://www.pgs.com/mediaFiles/Exclaimer%20graphics/PGS_LOGO_RGB_42x53px.jpg]<http://www.pgs.com>
Toralf Lund
Senior Software Engineer
Imaging & Engineering | Geoscience & Engineering
Telephone: +47 67 52 64 00
Direct: +47 67 51 57 78
VOIP: 74715778
Mobile: +47 91 31 66 91
Email: [email protected]<mailto:[email protected]>
A Clearer Image | www.pgs.com<http://www.pgs.com>
[http://www.pgs.com/mediaFiles/Exclaimer%20graphics/150731-ColourWave-660x14px.jpg]
Address: Lilleakerveien 4C, 0283 Oslo, Norway
Postal Address: P.O.Box 251 Lilleaker, 0216 Oslo, Norway
This e-mail, including any attachments and response string, may contain
proprietary information which is confidential and may be legally privileged. It
is for the intended recipient only. If you are not the intended recipient or
transmission error has misdirected this e-mail, please notify the author by
return e-mail and delete this message and any attachment immediately. If you
are not the intended recipient you must not use, disclose, distribute, forward,
copy, print or rely on this e-mail in any way except as permitted by the author.
