Hello,
We are testing with the trace+ log. We will brief you of the results when
they are ready.
Our goal is not to load balance equally the messages between the consumers
but we would like the dispatch router to send the message to the free
consumer.
Specially if the consumer does a long calculation, we do not want to block
the message in the outbound queue knowing there is another idle consumer.
Is there any special configuration for this?
Best regards,
Rabih
On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <gmur...@redhat.com> wrote:
> On Fri, Jul 12, 2019 at 11:29 AM Rabih M <rabih.prom...@gmail.com> wrote:
>
> > Hello,
> >
> > Thank you Ganesh for your answer.
> > To answer your first question yes the consumers we have, issue one credit
> > per call.
> > For the the second question, yes we are attaching consumers before
> sending
> > the messages.
> >
> > After setting the dispatch "link capacity" on the listeners to 1, we
> > managed to have an equal load-balancing of the messages between consumer
> 1
> > and 2.
> > Our understanding is that the problem comes from the prefetch at the
> level
> > of the "normal" listener, which is preventing the "inter-router" listener
> > to pass messages to the second dispatch.
> >
> Just want to note here that the linkCapacity on the inter-router listeners
> are ignored. So setting linkCapacity to 1 on that listener has no effect.
>
> This is what I think is happening
> 1. The Router 1 (R1) and Router 2 (R2) are started and each of them connect
> to both Brokers, B1 and B2 and establish the autolinks. As soon as inbound
> autoLinks (inbound to the router from the broker) are established on
> address myQueue, the broker(s) immediately provides initial credit on both
> autoLinks. This initial credit is usually 500 or 1000 depending on the
> broker you are using.
> 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects to R2
> providing one credit each respectively to each router.
> 3. Now the Producer connects to R1 and sends one message M1 (it cannot send
> more than one message because the linkCapacity is set to 1). Assuming M1
> goes to B1 (it has an equal chance of ending up in B2 as well), B1 sees
> that there are 2 consumers (autolinks) for that message.Say B1 sends M1 to
> R1, M1 will be sent to C1.
> 4. Producer produces M2 and it goes to B2. B2 might send it to R1 or R2.
> 4a. If B2 sends M2 to R1, the message might go over the
> inter-router link to R2 to C2 because there is no credit to send to C1.
> 4b. If B2 sends M2 to R2, then C2 gets M2 because routers prefer
> local consumers to remote consumers.
> 4c. If C1 is fast enough to consume M1 and replenish the credit
> and if B2 ends up sending M2 to R1, C1 might end up getting the second
> message as well.
> 4d. You need to also find out if the brokers load balances across
> autolinks.
>
> In conclusion, you are lucky that the messages are being load balanced
> across C1 and C2 when you changed linkCapacity to 1 but it is making it
> more likely. This might not always happen.
>
> Turn on trace logging on the routers to see which route the messages are
> taking. Try the scenario over and over again.
>
> To turn on trace logging add the following to the router config file -
>
> log {
> module: DEFAULT
> enable: trace+
> output: path/to/qdrouterd.log
> }
>
> If you want true load balancing between the consumers, put a router R3 in
> front of R1 and R2 and connect both consumers to R3. The producer can
> connect to any router.
>
> >
> > You can find attached the new config files.
> >
> > Is this the correct way to resolve this kind of problem? Does it sound
> > reasonable to you?
> >
> > Best regards,
> > Rabih and Ali
> >
> > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <gmur...@redhat.com>
> wrote:
> >
> >>
> >>
> >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <rabih.prom...@gmail.com>
> wrote:
> >>
> >>> Hello,
> >>>
> >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on redhat
> linux
> >>> rhel 6.4.
> >>>
> >>> Use case description:
> >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded queue
> >>> "myQueue" on the 2 brokers.
> >>> Here is an illustration:
> >>>
> >>> [image: Diagram.jpg]
> >>>
> >>> The producer produces 2 messages, the messages are load balanced on the
> >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
> >>>
> >> Does your receive() function issue one credit per call?
> >>
> >>>
> >>> Our observation is that the consumer 1 consumes the first message and
> >>> the consumer 2 is never getting the second message.
> >>> We are aware that the first dispatch router will do a prefetch for the
> 2
> >>> messages but what is weird is that the prefetched message 2 is never
> routed
> >>> to the second dispatch router and consumer 2.
> >>> I attached the dispatch routers config.
> >>>
> >>> This is what is going on in my view -
> >> 1. The producer sends two messages - Message 1 goes to Broker 1 and
> >> Message 2 goes to Broker 2.
> >> 2. Now Consumer 1 attaches to Router 1 and calls receive() issuing one
> >> credit. (The Consumer 2 has not yet attached to Router 2). The autolinks
> >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a prefetch
> of
> >> 250 credits each. So,Message 1 and Message 2
> >> both come down to Router 1. Message 1 is sent to Consumer 1. Message 2
> is
> >> waiting on the outbound queue of Consumer 1 waiting to receive a credit
> >> from Consumer 1.
> >> 3. Now Consumer 2 shows up and wants to receive Message 2 but that
> >> message is already queued up to go to Consumer 1 who is still keeping
> his
> >> connection open
> >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED back to
> >> its broker and that message will be sent to Consumer 2.
> >>
> >> Have you tried attaching Consumer 1 and Consumer 2 first and then
> >> subsequently bringing up the Producer 1 and send two messages ? In that
> >> case, I think, both consumers will receive one message.
> >>
> >> Thanks.
> >>
> >>> Do you have any idea how we can make consumer 2 receive the message?
> >>>
> >>> Thanks for your help,
> >>> Rabih
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> >>> For additional commands, e-mail: users-h...@qpid.apache.org
> >>
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> > For additional commands, e-mail: users-h...@qpid.apache.org
>
