On Fri, Nov 1, 2019 at 10:46 AM Robbie Gemmell <robbie.gemm...@gmail.com> wrote:
> On Thu, 31 Oct 2019 at 17:31, Ted Ross <tr...@redhat.com> wrote: > > > > On Thu, Oct 31, 2019 at 6:57 AM Robbie Gemmell <robbie.gemm...@gmail.com > > > > wrote: > > > > > With the below, are you saying that previously the router would always > > > immediately accept an unsettled message sent to a multicast address, > > > and then either send it on (pre-settled) or drop it if there was > > > nowhere to direct it > > > > > > Yes. > > > > > > > ....but in the latter case, now it would release > > > it instead? > > > > > > Yes. > > > > > > > If so, is it possible to configure the old behaviour, for > > > folks that actually wanted that? > > > > > > > Perhaps. Does anyone want it? It _would_ prevent the looping behavior. > > > > Who knows, did we ask? I dont think it is at all out of the realms of > possibility that some do, especially given it is essentially how a > brokered topic might often work, and how dispatch behaved until now. > > My main point is jsut that it was a fairly large change in behaviour > given it worked the way it had for some time deliberately (as opposed > to some bug), and now all of a sudden it isnt possible to get that > behaviour at all. I dont think its a very nice change to have made in > a minor release without providing any option to keep things working as > they had. As suggested, sending pre-settled is the closest equivalent > now (though not quite identical), but that unfortunately may require > updating all your senders too. > I agree with you in general about the change in behavior, however the old behavior in this case was never a desirable behavior. A little history on this feature: When we first supported multicast distribution, the default behavior was to reject all unsettled deliveries to multicast addresses. This was technically correct in that multicast was only appropriate for pre-settled QoS. In practice, however, this was a pretty serious problem. Not reading the fine print, developers would send unsettled deliveries to multicast addresses and get rejections and not understand why their communication didn't work. Because of this, we changed the behavior to accept unsettled deliveries even if they were not delivered to any consumer. I believe there was a detailed discussion of this on this email list. Now, we have a much more proper way to handle unsettled deliveries on multicast addresses. The question that arose was how to handle the situation when there are no consumers. We could behave like anycast addresses and throttle the senders, or we could behave more like a topic and allow the sending of messages, but release the deliveries if there are no consumers. In light of the current conversation, I would propose that the behavior be made the same as anycast distribution without any configurable options. I don't think that releasing deliveries is all that "topic-like" and accepting them is incorrect and misleading. What would a broker do if messages were sent to a non-persistent topic for which there were no subscribers? > > > > > > > > > Or does it still accept in that case and you just meant that the end > > > receiver outcomes are now interpreted to decide the response to the > > > sender? What if the different multicast points return different > > > outcomes? > > > > > > If there are different dispositions, there is a priority list that > > determines which of the provided dispositions will be returned to the > > sender. > > > > > > > What if one recipient doesnt provide an outcome for ages? > > > > > > Then that delivery will not be settled for ages. > > > > > > > Or > > > goes away without providing one? > > > > > > > Then the outcome from that recipient will be MODIFIED at the time the > > receiver detaches. > > > > > > > > > > Or maybe it was both of the above? :) > > > > > > In some ways either of these seem like odd changes for a minor release > > > unless its possible to toggle the previous long standing behaviour, > > > and not say have to switch all your senders to pre-settled to mimmick > > > but still not quite match it (since at least with the unsettled case > > > before, youd at least know whether a [first, if multiple hops] router > > > processed the message at all). > > > > > > Robbie > > > > > > On Wed, 30 Oct 2019 at 13:54, Ken Giusti <kgiu...@redhat.com> wrote: > > > > > > > > On Tue, Oct 29, 2019 at 6:23 PM VERMEULEN Olivier < > > > > olivier.vermeu...@murex.com> wrote: > > > > > > > > > Hello, > > > > > > > > > > Yes the waypoint address (in from broker) is using a multicast > > > > > distribution. > > > > > Unfortunately skipping the broker is not an option for us right > now. > > > > > Our whole architecture relies on the broker to guarantee that no > > > messages > > > > > will ever be lost... > > > > > > > > > > > > > That won't be the case for multicast actually. Prior to release > 1.9.0 of > > > > the router multicast messages would be dropped without notification > when > > > > under load. > > > > > > > > This relates to the issue you're experiencing now I believe. In > 1.9.0 we > > > > fixed this via > > > > > > > > https://issues.apache.org/jira/browse/DISPATCH-1266 > > > > > > > > Previously multicast messages were marked a pre-settled on entry to > the > > > > router and an "accepted" status was returned to the sender _before_ > the > > > > multicast was forwarded at all. Since the message was marked > pre-settled > > > > the router mesh will be more likely to drop it should congestion > occur. > > > > (Note this not the case with unsettled anycast messages - the router > will > > > > send a "release" status should the message need be discarded). > > > > > > > > This auto-settle behavior was undesirable for a number of reasons as > you > > > > can imagine, so in 1.9.0 we changed the behavior of multicast > deliveries: > > > > > > > > *unsettled* messages sent to multicast addresses are no longer > > > pre-settled > > > > by the router. The router will send back the final acknowledgement > > > > (accepted, release, etc) once all *present* subscribers for the > multicast > > > > address return acknowledgements. > > > > > > > > The behavior of pre-settled multicast did not change. > > > > > > > > So you can probably restore the original behavior by reverted back > to a > > > > pre-1.9.0 release, however be aware that even then there's no > guarantee > > > the > > > > message will be dropped. In fact it's _more_ likely to be dropped > (and > > > > signalled as accepted) in pre-1.9.0 releases. > > > > > > > > > > > > > > > > > For information we're asking for a quick workaround because we're > > > facing > > > > > this problem on a client production environment... > > > > > > > > > > Thanks, > > > > > Olivier > > > > > > > > > > -----Original Message----- > > > > > From: Ken Giusti <kgiu...@redhat.com> > > > > > Sent: mardi 29 octobre 2019 18:07 > > > > > To: users <users@qpid.apache.org> > > > > > Subject: Re: multicast without consumers > > > > > > > > > > On Tue, Oct 29, 2019 at 11:54 AM jeremy <jeremyao...@gmail.com> > wrote: > > > > > > > > > > > Hello Gordon, > > > > > > > > > > > > We debugged the dispatch router, and fell on the code which > releases > > > > > > undeliverable messages( > > > > > > > > > https://github.com/apache/qpid-dispatch/blob/1.5.0/src/router_core/tra > > > > > > nsfer.c#L869 > > > > > > ). > > > > > > > > > > > > Check the comment on line 879. It states that if the > distribution is > > > > > > multicast, the credit will be replenished after the release. The > > > issue > > > > > > that introduced this behavior is: > > > > > > https://issues.apache.org/jira/browse/DISPATCH-1012 > > > > > > > > > > > > > > > > > Is the waypoint address (in from broker) using multicast > distribution? > > > > > > > > > > The router treats multicast addresses like topics - you can > publish to > > > a > > > > > multicast address (topic) regardless of the presence of consumers. > > > That's > > > > > the reason credit is being replenished even when no consumers are > > > present. > > > > > > > > > > That's probably what's happening here - broker sends first queued > > > message > > > > > to the router, which attempts to send it to the topic. Since > there > > > are no > > > > > consumers (and the message is sent from the broker as unsettled) > the > > > > > router cannot deliver it so it returns the released status. The > > > released > > > > > status causes the broker to redeliver the message. Repeat. > > > > > > > > > > > > > > > > > > > > > > > > > > In fact, we need an urgent fix/workaround for this. Perhaps > there is > > > a > > > > > > quick workaround, awaiting the full analysis of this problem? > > > > > > > > > > > > > > > > > As a work around can you avoid sending these multicast messages to > the > > > > > broker queue? In other words send them directly to the router > instead > > > of > > > > > using a waypoint? > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ----- > > > > > > Cheers, > > > > > > Jeremy > > > > > > -- > > > > > > Sent from: > > > > > > > http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For > > > > > > additional commands, e-mail: users-h...@qpid.apache.org > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -K > > > > > ******************************* > > > > > This e-mail contains information for the intended recipient only. > It > > > may > > > > > contain proprietary material or confidential information. If you > are > > > not > > > > > the intended recipient you are not authorized to distribute, copy > or > > > use > > > > > this e-mail or any attachment to it. Murex cannot guarantee that > it is > > > > > virus free and accepts no responsibility for any loss or damage > arising > > > > > from its use. If you have received this e-mail in error please > notify > > > > > immediately the sender and delete the original email received, any > > > > > attachments and all copies from your system. > > > > > > > > > > > > > > > > > -- > > > > -K > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > > > For additional commands, e-mail: users-h...@qpid.apache.org > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
