Hello,
Thanks for the prompt responses.
Just for context:
The version we are using in production is the Qpid dispatch 1.5.0.
Our architecture consists of a dispatch router which is a facade for all
consumers and producers. Behind the dispatch router, we have one or
multiple brokers.
The producers connect to the dispatch router only. They publish their
messages in a topic on the broker (via an autolink on the dispatch).
The broker routes internally the messages (through binding keys) to
specific queues.
The dispatch router prefetches the messages from those queues on a
different autolink, to deliver them later in multicast mode to the
consumers.
Then the consumers connect to the dispatch to receive the messages.
The problem is:
When there are no more consumers attached to the dispatch and we have still
messages on the broker queue, we have a cycle: - issuing credit - message
transfer - then a release. This is done thousands of times per second
between the broker-j and the dispatch, thus saturating CPU and network
ressources. We attached a sample of the dispatch logs in trace mode where
you can see clearly 4 cycles in approximately 1.2 milliseconds.
>From what we understood from the doc and your emails, this problem also
exists on the latest version (we haven't tested it yet).
The dispatch router will always release the message then issue a credit if
there are no consumers connected (the deadly loop).
Is our understanding correct?
We are trying to do a patch to break the cycle: we do not issue credits
when we release a message if there are no consumers connected.
But apparently this is not aligned with your vision on how multicast should
work (DISPATCH-779).
Any suggestions of what can be done?
Thanks,
Jeremy and Rabih
On Wed, Oct 30, 2019 at 3:41 PM Ted Ross <tr...@redhat.com> wrote:
> On Tue, Oct 29, 2019 at 6:23 PM VERMEULEN Olivier <
> olivier.vermeu...@murex.com> wrote:
>
> > Hello,
> >
> > Yes the waypoint address (in from broker) is using a multicast
> > distribution.
> > Unfortunately skipping the broker is not an option for us right now.
> > Our whole architecture relies on the broker to guarantee that no messages
> > will ever be lost...
> > For information we're asking for a quick workaround because we're facing
> > this problem on a client production environment...
> >
>
> Are you looking for a patch you can apply locally to work around your
> issue?
>
>
> >
> > Thanks,
> > Olivier
> >
> > -----Original Message-----
> > From: Ken Giusti <kgiu...@redhat.com>
> > Sent: mardi 29 octobre 2019 18:07
> > To: users <users@qpid.apache.org>
> > Subject: Re: multicast without consumers
> >
> > On Tue, Oct 29, 2019 at 11:54 AM jeremy <jeremyao...@gmail.com> wrote:
> >
> > > Hello Gordon,
> > >
> > > We debugged the dispatch router, and fell on the code which releases
> > > undeliverable messages(
> > > https://github.com/apache/qpid-dispatch/blob/1.5.0/src/router_core/tra
> > > nsfer.c#L869
> > > ).
> > >
> > > Check the comment on line 879. It states that if the distribution is
> > > multicast, the credit will be replenished after the release. The issue
> > > that introduced this behavior is:
> > > https://issues.apache.org/jira/browse/DISPATCH-1012
> > >
> > >
> > Is the waypoint address (in from broker) using multicast distribution?
> >
> > The router treats multicast addresses like topics - you can publish to a
> > multicast address (topic) regardless of the presence of consumers.
> That's
> > the reason credit is being replenished even when no consumers are
> present.
> >
> > That's probably what's happening here - broker sends first queued message
> > to the router, which attempts to send it to the topic. Since there are
> no
> > consumers (and the message is sent from the broker as unsettled) the
> > router cannot deliver it so it returns the released status. The released
> > status causes the broker to redeliver the message. Repeat.
> >
> >
> >
> >
> > > In fact, we need an urgent fix/workaround for this. Perhaps there is a
> > > quick workaround, awaiting the full analysis of this problem?
> > >
> > >
> > As a work around can you avoid sending these multicast messages to the
> > broker queue? In other words send them directly to the router instead of
> > using a waypoint?
> >
> >
> >
> > > Thanks
> > >
> > >
> > >
> > >
> > > -----
> > > Cheers,
> > > Jeremy
> > > --
> > > Sent from:
> > > http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For
> > > additional commands, e-mail: users-h...@qpid.apache.org
> > >
> > >
> >
> > --
> > -K
> > *******************************
> > This e-mail contains information for the intended recipient only. It may
> > contain proprietary material or confidential information. If you are not
> > the intended recipient you are not authorized to distribute, copy or use
> > this e-mail or any attachment to it. Murex cannot guarantee that it is
> > virus free and accepts no responsibility for any loss or damage arising
> > from its use. If you have received this e-mail in error please notify
> > immediately the sender and delete the original email received, any
> > attachments and all copies from your system.
> >
>
2019-10-30 12:01:28.392390 +0100 SERVER (trace) [2]:0 -> @flow(19)
[next-incoming-id=622374, incoming-window=2147483647, next-outgoing-id=1,
outgoing-window=2147483647, handle=20, delivery-count=622374, link-credit=250,
drain=false]
2019-10-30 12:01:28.392407 +0100 SERVER (trace) [2]:0 -> @disposition(21)
[role=true, first=622373, settled=true, state=@released(38) []]
2019-10-30 12:01:28.392525 +0100 SERVER (trace) [2]:0 <- @transfer(20)
[handle=20, delivery-id=622374, delivery-tag=b"\x00\x00\x00\x00\x00\x09\x7f&",
message-format=0] (220)
"\x00Sp\xc0\x02\x01A\x00Sr\xd1\x00\x00\x00,\x00\x00\x00\x04\xa3\x0ex-opt-jms-destQ\x01\xa3\x12x-opt-jms-msg-typeQ\x05\x00Ss\xd0\x00\x00\x00\x88\x00\x00\x00\x0a\xa1/ID:088c07dc-33d4-4f91-be98-fa2efae05e4a:1:1:1-1@\xa1Av1.portfolio-management.full-revaluation.business-action-response@@@@@@\x83\x00\x00\x01n\x1cQ*\x8e\x00Sw\xa1\x0cyour
message"
2019-10-30 12:01:28.392555 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf012fd18 rc:1 qdr_link_deliver - newly created delivery, add to action
list
2019-10-30 12:01:28.392565 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf012fd18 rc:2 qdr_link_deliver - protect returned value
2019-10-30 12:01:28.392574 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf012fd18 rc:3 referenced by a pn_delivery
2019-10-30 12:01:28.392583 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bf012fd18 rc:2 release protection of return from deliver
2019-10-30 12:01:28.392594 +0100 ROUTER_CORE (trace) Core action 'link_deliver'
2019-10-30 12:01:28.392618 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf012fd18 rc:3 qdr_delivery_push_CT - add to updated list
2019-10-30 12:01:28.392627 +0100 ROUTER_CORE (debug) Delivery decref_CT:
dlv:7f6bf012fd18 rc:2 qdr_link_forward_CT - removed from action (no path)
2019-10-30 12:01:28.392644 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bf012fd18 rc:1 removed reference from pn_delivery
2019-10-30 12:01:28.392654 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bf012fd18 rc:0 qdr_connection_process - remove from updated list
2019-10-30 12:01:28.392666 +0100 ROUTER_CORE (trace) Core action
'delete_delivery'
2019-10-30 12:01:28.392685 +0100 SERVER (trace) [2]:0 -> @flow(19)
[next-incoming-id=622375, incoming-window=2147483647, next-outgoing-id=1,
outgoing-window=2147483647, handle=20, delivery-count=622375, link-credit=250,
drain=false]
2019-10-30 12:01:28.392702 +0100 SERVER (trace) [2]:0 -> @disposition(21)
[role=true, first=622374, settled=true, state=@released(38) []]
2019-10-30 12:01:28.392805 +0100 SERVER (trace) [2]:0 <- @transfer(20)
[handle=20, delivery-id=622375, delivery-tag=b"\x00\x00\x00\x00\x00\x09\x7f'",
message-format=0] (220)
"\x00Sp\xc0\x02\x01A\x00Sr\xd1\x00\x00\x00,\x00\x00\x00\x04\xa3\x0ex-opt-jms-destQ\x01\xa3\x12x-opt-jms-msg-typeQ\x05\x00Ss\xd0\x00\x00\x00\x88\x00\x00\x00\x0a\xa1/ID:088c07dc-33d4-4f91-be98-fa2efae05e4a:1:1:1-1@\xa1Av1.portfolio-management.full-revaluation.business-action-response@@@@@@\x83\x00\x00\x01n\x1cQ*\x8e\x00Sw\xa1\x0cyour
message"
2019-10-30 12:01:28.392835 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6be401e258 rc:1 qdr_link_deliver - newly created delivery, add to action
list
2019-10-30 12:01:28.392845 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6be401e258 rc:2 qdr_link_deliver - protect returned value
2019-10-30 12:01:28.392855 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6be401e258 rc:3 referenced by a pn_delivery
2019-10-30 12:01:28.392863 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6be401e258 rc:2 release protection of return from deliver
2019-10-30 12:01:28.392873 +0100 ROUTER_CORE (trace) Core action 'link_deliver'
2019-10-30 12:01:28.392887 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6be401e258 rc:3 qdr_delivery_push_CT - add to updated list
2019-10-30 12:01:28.392895 +0100 ROUTER_CORE (debug) Delivery decref_CT:
dlv:7f6be401e258 rc:2 qdr_link_forward_CT - removed from action (no path)
2019-10-30 12:01:28.392916 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6be401e258 rc:1 removed reference from pn_delivery
2019-10-30 12:01:28.392927 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6be401e258 rc:0 qdr_connection_process - remove from updated list
2019-10-30 12:01:28.392938 +0100 ROUTER_CORE (trace) Core action
'delete_delivery'
2019-10-30 12:01:28.392950 +0100 SERVER (trace) [2]:0 -> @flow(19)
[next-incoming-id=622376, incoming-window=2147483647, next-outgoing-id=1,
outgoing-window=2147483647, handle=20, delivery-count=622376, link-credit=250,
drain=false]
2019-10-30 12:01:28.392967 +0100 SERVER (trace) [2]:0 -> @disposition(21)
[role=true, first=622375, settled=true, state=@released(38) []]
2019-10-30 12:01:28.393086 +0100 SERVER (trace) [2]:0 <- @transfer(20)
[handle=20, delivery-id=622376, delivery-tag=b"\x00\x00\x00\x00\x00\x09\x7f(",
message-format=0] (220)
"\x00Sp\xc0\x02\x01A\x00Sr\xd1\x00\x00\x00,\x00\x00\x00\x04\xa3\x0ex-opt-jms-destQ\x01\xa3\x12x-opt-jms-msg-typeQ\x05\x00Ss\xd0\x00\x00\x00\x88\x00\x00\x00\x0a\xa1/ID:088c07dc-33d4-4f91-be98-fa2efae05e4a:1:1:1-1@\xa1Av1.portfolio-management.full-revaluation.business-action-response@@@@@@\x83\x00\x00\x01n\x1cQ*\x8e\x00Sw\xa1\x0cyour
message"
2019-10-30 12:01:28.393116 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bec0d5958 rc:1 qdr_link_deliver - newly created delivery, add to action
list
2019-10-30 12:01:28.393126 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bec0d5958 rc:2 qdr_link_deliver - protect returned value
2019-10-30 12:01:28.393136 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bec0d5958 rc:3 referenced by a pn_delivery
2019-10-30 12:01:28.393144 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bec0d5958 rc:2 release protection of return from deliver
2019-10-30 12:01:28.393154 +0100 ROUTER_CORE (trace) Core action 'link_deliver'
2019-10-30 12:01:28.393170 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bec0d5958 rc:3 qdr_delivery_push_CT - add to updated list
2019-10-30 12:01:28.393178 +0100 ROUTER_CORE (debug) Delivery decref_CT:
dlv:7f6bec0d5958 rc:2 qdr_link_forward_CT - removed from action (no path)
2019-10-30 12:01:28.393195 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bec0d5958 rc:1 removed reference from pn_delivery
2019-10-30 12:01:28.393209 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bec0d5958 rc:0 qdr_connection_process - remove from updated list
2019-10-30 12:01:28.393221 +0100 ROUTER_CORE (trace) Core action
'delete_delivery'
2019-10-30 12:01:28.393233 +0100 SERVER (trace) [2]:0 -> @flow(19)
[next-incoming-id=622377, incoming-window=2147483647, next-outgoing-id=1,
outgoing-window=2147483647, handle=20, delivery-count=622377, link-credit=250,
drain=false]
2019-10-30 12:01:28.393250 +0100 SERVER (trace) [2]:0 -> @disposition(21)
[role=true, first=622376, settled=true, state=@released(38) []]
2019-10-30 12:01:28.393369 +0100 SERVER (trace) [2]:0 <- @transfer(20)
[handle=20, delivery-id=622377, delivery-tag=b"\x00\x00\x00\x00\x00\x09\x7f)",
message-format=0] (220)
"\x00Sp\xc0\x02\x01A\x00Sr\xd1\x00\x00\x00,\x00\x00\x00\x04\xa3\x0ex-opt-jms-destQ\x01\xa3\x12x-opt-jms-msg-typeQ\x05\x00Ss\xd0\x00\x00\x00\x88\x00\x00\x00\x0a\xa1/ID:088c07dc-33d4-4f91-be98-fa2efae05e4a:1:1:1-1@\xa1Av1.portfolio-management.full-revaluation.business-action-response@@@@@@\x83\x00\x00\x01n\x1cQ*\x8e\x00Sw\xa1\x0cyour
message"
2019-10-30 12:01:28.393408 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf0049b18 rc:1 qdr_link_deliver - newly created delivery, add to action
list
2019-10-30 12:01:28.393419 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf0049b18 rc:2 qdr_link_deliver - protect returned value
2019-10-30 12:01:28.393428 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf0049b18 rc:3 referenced by a pn_delivery
2019-10-30 12:01:28.393437 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bf0049b18 rc:2 release protection of return from deliver
2019-10-30 12:01:28.393446 +0100 ROUTER_CORE (trace) Core action 'link_deliver'
2019-10-30 12:01:28.393457 +0100 ROUTER_CORE (debug) Delivery incref:
dlv:7f6bf0049b18 rc:3 qdr_delivery_push_CT - add to updated list
2019-10-30 12:01:28.393464 +0100 ROUTER_CORE (debug) Delivery decref_CT:
dlv:7f6bf0049b18 rc:2 qdr_link_forward_CT - removed from action (no path)
2019-10-30 12:01:28.393481 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bf0049b18 rc:1 removed reference from pn_delivery
2019-10-30 12:01:28.393491 +0100 ROUTER_CORE (debug) Delivery decref:
dlv:7f6bf0049b18 rc:0 qdr_connection_process - remove from updated list
2019-10-30 12:01:28.393503 +0100 ROUTER_CORE (trace) Core action
'delete_delivery'
2019-10-30 12:01:28.393515 +0100 SERVER (trace) [2]:0 -> @flow(19)
[next-incoming-id=622378, incoming-window=2147483647, next-outgoing-id=1,
outgoing-window=2147483647, handle=20, delivery-count=622378, link-credit=250,
drain=false]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
