Hi all,
I am running a three node fully connected mesh of dispatch routers with
10000 attached clients and I am seeing some unstable inter-router
connections (I am sending around 1000 message per second through the
network). The inter-router connections fail every so many seconds with the
message: Connection to router-2:55672 failed: amqp:session:invalid-field
sequencing error, expected delivery-id 7, got 6
I am not sure if this is to be expected / unexpected or that I have a
faulty configuration.
The routers are connected as follows: router-0 -> router-1, router-0 ->
router-2, router-1 -> router-2
Attached are my qdrouter.conf (from one of the routers) and a log snippet
from router-0 at debug level from connection drop to connection
re-established to connection drop again.
Versions: qpid-dispatch@1.0.1-rc1, qpid-proton@0.20.0
Hope someone can help.
Cheers,
Marcel
2018-04-11 18:47:04.801720 +0000 ROUTER_LS (info) Link to Neighbor Router Lost - link_tag=0
2018-04-11 18:47:05.785203 +0000 ROUTER_LS (info) Computed next hops: {}
2018-04-11 18:47:05.785281 +0000 ROUTER_LS (info) Computed costs: {}
2018-04-11 18:47:05.785307 +0000 ROUTER_LS (info) Computed valid origins: {}
2018-04-11 18:47:05.785337 +0000 ROUTER_LS (info) Computed radius: 0
2018-04-11 18:47:09.908855 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac158850 rc:1 qdr_forward_deliver_CT - add to undelivered list
2018-04-11 18:47:09.908977 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e02e50 rc:1 qdr_forward_deliver_CT - add to undelivered list
2018-04-11 18:47:09.909109 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0bac158850 rc:0 qdr_link_process_deliveries - remove from undelivered list
2018-04-11 18:47:09.909246 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0ba7e02e50 rc:0 qdr_link_process_deliveries - remove from undelivered list
2018-04-11 18:47:10.470544 +0000 ROUTER_CORE (debug) Delivery incref: dlv:55b497cb1c10 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:10.470618 +0000 ROUTER_CORE (debug) Delivery incref: dlv:55b497cb1c10 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:10.470635 +0000 ROUTER_CORE (debug) Delivery decref: dlv:55b497cb1c10 rc:1 release protection of return from deliver
2018-04-11 18:47:10.470723 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e03dd0 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:10.470734 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e03dd0 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:10.470759 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0ba7e03dd0 rc:1 release protection of return from deliver
2018-04-11 18:47:10.470790 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e05cd0 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:10.470798 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e05cd0 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:10.470806 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0ba7e05cd0 rc:1 release protection of return from deliver
2018-04-11 18:47:10.470948 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:55b497cb1c10 rc:0 qdr_link_forward_CT - removed from action (2)
2018-04-11 18:47:10.476199 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:7f0ba7e03dd0 rc:0 qdr_link_forward_CT - removed from action (2)
2018-04-11 18:47:10.476262 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:7f0ba7e05cd0 rc:0 qdr_link_forward_CT - removed from action (2)
2018-04-11 18:47:12.472096 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e05e50 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:12.472168 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e05e50 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:12.472225 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:7f0ba7e05e50 rc:1 qdr_link_forward_CT - removed from action (2)
2018-04-11 18:47:12.472241 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0ba7e05e50 rc:0 release protection of return from deliver
2018-04-11 18:47:14.471905 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac15a610 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:14.471968 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac15a610 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:14.472001 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:7f0bac15a610 rc:1 qdr_link_forward_CT - removed from action (2)
2018-04-11 18:47:14.472015 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0bac15a610 rc:0 release protection of return from deliver
2018-04-11 18:47:14.921856 +0000 ROUTER_LS (info) Computed next hops: {'router-2': 'router-2', 'router-1': 'router-2'}
2018-04-11 18:47:14.921941 +0000 ROUTER_LS (info) Computed costs: {'router-2': 1, 'router-1': 2}
2018-04-11 18:47:14.921970 +0000 ROUTER_LS (info) Computed valid origins: {'router-2': [], 'router-1': []}
2018-04-11 18:47:14.922009 +0000 ROUTER_LS (info) Computed radius: 2
2018-04-11 18:47:14.922396 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac157350 rc:1 qdr_forward_deliver_CT - add to undelivered list
2018-04-11 18:47:14.922548 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0ba7e05090 rc:1 qdr_forward_deliver_CT - add to undelivered list
2018-04-11 18:47:14.922599 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac158a10 rc:1 qdr_forward_deliver_CT - add to undelivered list
2018-04-11 18:47:14.922669 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0bac157350 rc:0 qdr_link_process_deliveries - remove from undelivered list
2018-04-11 18:47:14.922718 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0ba7e05090 rc:0 qdr_link_process_deliveries - remove from undelivered list
2018-04-11 18:47:14.922786 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0bac158a10 rc:0 qdr_link_process_deliveries - remove from undelivered list
2018-04-11 18:47:14.926454 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac15a7d0 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:14.926515 +0000 ROUTER_CORE (debug) Delivery incref: dlv:7f0bac15a7d0 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:14.926549 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:7f0bac15a7d0 rc:1 qdr_link_forward_CT - removed from action (2)
2018-04-11 18:47:14.926889 +0000 ROUTER_CORE (debug) Delivery decref: dlv:7f0bac15a7d0 rc:0 release protection of return from deliver
2018-04-11 18:47:14.927129 +0000 ROUTER_CORE (debug) Delivery incref: dlv:55b497caded0 rc:1 qdr_link_deliver_to - newly created delivery, add to action list
2018-04-11 18:47:14.927150 +0000 ROUTER_CORE (debug) Delivery incref: dlv:55b497caded0 rc:2 qdr_link_deliver_to - protect returned value
2018-04-11 18:47:14.927179 +0000 ROUTER_CORE (debug) Delivery incref: dlv:55b497caded0 rc:3 qdr_delivery_push_CT - add to updated list
2018-04-11 18:47:14.927191 +0000 ROUTER_CORE (debug) Delivery decref_CT: dlv:55b497caded0 rc:2 qdr_link_forward_CT - removed from action (1)
2018-04-11 18:47:14.927205 +0000 ROUTER_CORE (debug) Delivery incref: dlv:55b497caded0 rc:3 referenced by a pn_delivery
2018-04-11 18:47:14.927215 +0000 ROUTER_CORE (debug) Delivery decref: dlv:55b497caded0 rc:2 release protection of return from deliver
2018-04-11 18:47:14.927361 +0000 ROUTER_CORE (debug) Delivery decref: dlv:55b497caded0 rc:1 removed reference from pn_delivery
2018-04-11 18:47:14.927377 +0000 ROUTER_CORE (debug) Delivery decref: dlv:55b497caded0 rc:0 qdr_connection_process - remove from updated list
2018-04-11 18:47:14.927527 +0000 SERVER (info) Connection to router-2:55672 failed: amqp:session:invalid-field sequencing error, expected delivery-id 7, got 6
2018-04-11 18:47:14.927588 +0000 CONTAINER (debug) Aborting link 'qdlink.7PnbBMEtc8I3OOe' due to parent connection end
2018-04-11 18:47:14.927633 +0000 CONTAINER (debug) Aborting link 'qdlink.2iu8_6MdqqVi+fn' due to parent connection end
2018-04-11 18:47:14.927684 +0000 CONTAINER (debug) Aborting link 'qdlink.+i1yRegGR42y3bW' due to parent connection end
2018-04-11 18:47:14.927723 +0000 CONTAINER (debug) Aborting link 'qdlink.LrpaARzYvN+5ZlA' due to parent connection end
2018-04-11 18:47:14.928124 +0000 ROUTER_LS (info) Link to Neighbor Router Lost - link_tag=0
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
