On Fri, Jul 19, 2019 at 7:50 AM Rabih M <rabih.prom...@gmail.com> wrote:
> Hello,
>
> Yes this is one of our production use cases, where we have low throughput
> but the processing of the messages can take from seconds to minutes
> (financial calculations). This is why it was bothering us to have an idle
> consumer while another is overloaded.
>
You should consider using link-routes for your consumers. Link-routes
trade off the load balancing for tight control of credit end-to-end. If a
consumer is link-routed to a broker, the broker won't deliver a message on
that link until the consumer issues a credit, and then it will only
delivery as many messages as credits were issued.
>
> I understand that for the performance, we need to prefetch messages from
> broker to make them available in the dispatch, but what i did not
> understand is why inside a router we need to assign the prefetched messages
> to a consumer connection and not waiting until the connected consumer
> issues a credit, knowing that from a performance point of view, the costly
> action of "prefetching the messages through IO calls" was made.
> Is it because the complexity of the routing algorithm and the
> communications between the dispatch routers will increase?
>
>
When messages are sent into the router network, they are immediately routed
to a destination. The routers don't hold messages for later routing.
Also, the synchronization of real-time credit state across a network for
all addresses is not practical or scalable.
> Last point, we did the following test:
> A dispatch router have a connector with a LinkCapacity=250 connected to a
> broker and a listener with a LinkCapacity=1 connected to a consumer.
>
The important link capacity in this scenario is the 250 as it controls the
router pre-fetch. The consumer's link capacity of 1 is not relevant to
this case.
>
> [image: Untitled Diagram.jpg]
>
I can't see the diagram, but I think I get the idea.
>
>
> 1- the router 1 prefetches 250 message from the broker
> 2- the consumer issues a credit
> 3- the consumer receives a message from the router but does not acknowledge
> 4- the consumer issues another credit
> 5- the consumer receives a message from the router but does not
> acknowledge again
> Steps 4 and 5 can be repeated until the 250 msgs are all transferred to
> the consumer
>
This is consistent with there being only one consumer for the address on
the network at the time the broker sent the 250 messages.
>
> Is this an expected behavior? consumer 1 should not acknowledge before he
> can receive another message knowing that the link capacity of the listener
> is 1?
>
Best practice for acknowledgement is for the consumer to acknowledge
(settle) immediately after finishing the processing of the message (i.e.
once that message is no longer consuming memory or compute resources on the
host). This causes the settlement state of deliveries to be directly
related to consumer resources. Again, the link capacity of 1 is not having
any effect on the behavior of this scenario.
>
> Thanks for your explanations and help,
>
Am I to understand that your case is this? You have a distributed work
queue in which the time-to-process is highly variable. Some messages are
processed quickly and others take much longer. You don't want to incur the
longer latency on messages that can be handled quickly if there are many
more fast messages than slow messages.
Is it possible to know beforehand which messages are going to take long?
Could you put these on a different queue with a different address?
> Best regards,
> Rabih
>
>
> On Wed, Jul 17, 2019 at 6:46 PM Ted Ross <tr...@redhat.com> wrote:
>
>>
>>
>> On Wed, Jul 17, 2019 at 12:00 PM Rabih M <rabih.prom...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> We tested with LinkCapacity equal to 1 on the "normal" listener with
>>> debug level trace+, here are our findings:
>>> Our Cluster:
>>> [image: Diagram.jpg]
>>> We are using the broker-j, the consumer are connected to the dispatch
>>> routers before we start sending.
>>>
>>> For use case 1:
>>> 1- the producer sends a message.
>>> 2- Consumer 1 issues one credit, receives the message without
>>> acknowledging.
>>> 3- the producer sends another message.
>>> 4- Consumer 2 in auto-ack mode issues one credit and receives the
>>> message.
>>> 5- we repeated steps 3 and 4 ten times.
>>> 6- Consumer 1 acknowledges.
>>> The results were correct: all the messages were correctly distributed to
>>> the idle consumer.
>>>
>>> For use case 2:
>>> 1- the producer send 10 messages while no credits were issued yet by the
>>> consumers.
>>> 2- Consumer 1 issues one credit, receives a message without
>>> acknowledging.
>>> 3- Consumer 2 in auto-ack mode issues one credit and times out after 5
>>> seconds if nothing is received.
>>> 4- we repeated step 3 eight times.
>>> 5- Consumer 1 acknowledges.
>>> The results were not as expected: 4 messages were blocked in the
>>> outbound queue of the consumer 1 and consumer 2 was able to receive only 5
>>> messages.
>>> We analysed the traces to follow the messages. We found that 4 messages
>>> were blocked in the dispatch 1.
>>> Conclusion: if no consumers are issuing credits (are busy) then the
>>> incoming messages will be pre-assigned automatically by the dispatch router
>>> to the listeners (in a round robin way?).
>>>
>>> Is it an expected behavior in the dispatch router? is it not supposed to
>>> wait for a credit to be issued before binding the message to an outbound
>>> queue?
>>>
>>
>> Yes, this is the expected behavior. The router does not propagate each
>> individual credit from receiver to sender. It would be impractical to do
>> so, would not scale well, and probably still wouldn't provide the behavior
>> you expect. What the router does is to use delivery settlement as the way
>> to control credit flow to producers. If the link capacity is 250, each
>> producer will be limited to 250 unsettled deliveries at any time. As the
>> deliveries are settled, more credit is issued. This scales well in large
>> networks, keeps a limit on the memory consumed by deliveries, and allows
>> for high delivery rates.
>>
>> Is this a real use case or are you experimenting to learn how the router
>> works?
>>
>> Under steady state flow, the messages will be delivered to the consumers
>> in proportion to the rate at which the consumers acknowledge (settle)
>> deliveries. If a consumer attaches a receiving link but withholds credit,
>> the router network will route deliveries to that consumer in anticipation
>> of credit being issued. It is an anti-pattern to attach a receiving link
>> and stop issuing credit. Credit should be used to control the rate of
>> delivery. If you want to stop delivery, detach the receiver.
>>
>> If you really want a completely synchronous transfer across your network,
>> you can set the link capacity on the broker connections to 1. This will
>> limit the number of in-flight unsettled deliveries on each incoming
>> auto-link to 1. It will be slow and will be prone to stalling, especially
>> if your consumers withhold acknowledgement.
>>
>> If you want to check the paths of the messages, I attached the routers
>>> logs of the use case 2.
>>>
>>> Best regards,
>>> Rabih
>>>
>>> On Mon, Jul 15, 2019 at 8:14 PM Ganesh Murthy <gmur...@redhat.com>
>>> wrote:
>>>
>>>> On Mon, Jul 15, 2019 at 1:26 PM Rabih M <rabih.prom...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hello,
>>>> >
>>>> > We are testing with the trace+ log. We will brief you of the results
>>>> when
>>>> > they are ready.
>>>> >
>>>> > Our goal is not to load balance equally the messages between the
>>>> consumers
>>>> > but we would like the dispatch router to send the message to the free
>>>> > consumer.
>>>> >
>>>>
>>>> What you did (setting the linkCapacity on the producer and consumer
>>>> listeners to 1) is the right thing to do if you want the router
>>>> to send the message to *any* free consumer.
>>>>
>>>> If Consumer C1 is processing the first message and does not issue the
>>>> next
>>>> credit until it finishes processing the first message and if the second
>>>> message arrives to Router 1 when C1 is
>>>> still processing the message, then Router 1 will definitely forward the
>>>> message to Router 2. BUT if C1 is fast enough and ends up processing the
>>>> first message and immediately issues credit, the second message if it
>>>> arrives in Router 1 will also
>>>> be sent to C1 (because the router prefers local consumers).
>>>>
>>>> Remember that the key here is which Router ends up getting the message
>>>> since each broker has two autoLinks to both routers and we don't know
>>>> which
>>>> autoLink the broker will choose.
>>>>
>>>> But overall, with your new configuration, the router will send the
>>>> message
>>>> to a consumer that is not busy no further configuration is necessary.
>>>>
>>>> Specially if the consumer does a long calculation, we do not want to
>>>> block
>>>> > the message in the outbound queue knowing there is another idle
>>>> consumer.
>>>> >
>>>> > Is there any special configuration for this?
>>>> >
>>>> > Best regards,
>>>> > Rabih
>>>> >
>>>> >
>>>> > On Fri, Jul 12, 2019 at 7:26 PM Ganesh Murthy <gmur...@redhat.com>
>>>> wrote:
>>>> >
>>>> > > On Fri, Jul 12, 2019 at 11:29 AM Rabih M <rabih.prom...@gmail.com>
>>>> > wrote:
>>>> > >
>>>> > > > Hello,
>>>> > > >
>>>> > > > Thank you Ganesh for your answer.
>>>> > > > To answer your first question yes the consumers we have, issue one
>>>> > credit
>>>> > > > per call.
>>>> > > > For the the second question, yes we are attaching consumers before
>>>> > > sending
>>>> > > > the messages.
>>>> > > >
>>>> > > > After setting the dispatch "link capacity" on the listeners to 1,
>>>> we
>>>> > > > managed to have an equal load-balancing of the messages between
>>>> > consumer
>>>> > > 1
>>>> > > > and 2.
>>>> > > > Our understanding is that the problem comes from the prefetch at
>>>> the
>>>> > > level
>>>> > > > of the "normal" listener, which is preventing the "inter-router"
>>>> > listener
>>>> > > > to pass messages to the second dispatch.
>>>> > > >
>>>> > > Just want to note here that the linkCapacity on the inter-router
>>>> > listeners
>>>> > > are ignored. So setting linkCapacity to 1 on that listener has no
>>>> effect.
>>>> > >
>>>> > > This is what I think is happening
>>>> > > 1. The Router 1 (R1) and Router 2 (R2) are started and each of them
>>>> > connect
>>>> > > to both Brokers, B1 and B2 and establish the autolinks. As soon as
>>>> > inbound
>>>> > > autoLinks (inbound to the router from the broker) are established on
>>>> > > address myQueue, the broker(s) immediately provides initial credit
>>>> on
>>>> > both
>>>> > > autoLinks. This initial credit is usually 500 or 1000 depending on
>>>> the
>>>> > > broker you are using.
>>>> > > 2. Now Consumer 1 (C1) connects to R1 and Consumer 2 (C2) connects
>>>> to R2
>>>> > > providing one credit each respectively to each router.
>>>> > > 3. Now the Producer connects to R1 and sends one message M1 (it
>>>> cannot
>>>> > send
>>>> > > more than one message because the linkCapacity is set to 1).
>>>> Assuming M1
>>>> > > goes to B1 (it has an equal chance of ending up in B2 as well), B1
>>>> sees
>>>> > > that there are 2 consumers (autolinks) for that message.Say B1
>>>> sends M1
>>>> > to
>>>> > > R1, M1 will be sent to C1.
>>>> > > 4. Producer produces M2 and it goes to B2. B2 might send it to R1
>>>> or R2.
>>>> > > 4a. If B2 sends M2 to R1, the message might go over the
>>>> > > inter-router link to R2 to C2 because there is no credit to send
>>>> to C1.
>>>> > > 4b. If B2 sends M2 to R2, then C2 gets M2 because routers
>>>> > prefer
>>>> > > local consumers to remote consumers.
>>>> > > 4c. If C1 is fast enough to consume M1 and replenish the
>>>> credit
>>>> > > and if B2 ends up sending M2 to R1, C1 might end up getting the
>>>> second
>>>> > > message as well.
>>>> > > 4d. You need to also find out if the brokers load balances
>>>> > across
>>>> > > autolinks.
>>>> > >
>>>> > > In conclusion, you are lucky that the messages are being load
>>>> balanced
>>>> > > across C1 and C2 when you changed linkCapacity to 1 but it is
>>>> making it
>>>> > > more likely. This might not always happen.
>>>> > >
>>>> > > Turn on trace logging on the routers to see which route the
>>>> messages are
>>>> > > taking. Try the scenario over and over again.
>>>> > >
>>>> > > To turn on trace logging add the following to the router config
>>>> file -
>>>> > >
>>>> > > log {
>>>> > > module: DEFAULT
>>>> > > enable: trace+
>>>> > > output: path/to/qdrouterd.log
>>>> > > }
>>>> > >
>>>> > > If you want true load balancing between the consumers, put a router
>>>> R3 in
>>>> > > front of R1 and R2 and connect both consumers to R3. The producer
>>>> can
>>>> > > connect to any router.
>>>> > >
>>>> > > >
>>>> > > > You can find attached the new config files.
>>>> > > >
>>>> > > > Is this the correct way to resolve this kind of problem? Does it
>>>> sound
>>>> > > > reasonable to you?
>>>> > > >
>>>> > > > Best regards,
>>>> > > > Rabih and Ali
>>>> > > >
>>>> > > > On Thu, Jul 11, 2019 at 5:50 PM Ganesh Murthy <gmur...@redhat.com
>>>> >
>>>> > > wrote:
>>>> > > >
>>>> > > >>
>>>> > > >>
>>>> > > >> On Thu, Jul 11, 2019 at 10:37 AM Rabih M <
>>>> rabih.prom...@gmail.com>
>>>> > > wrote:
>>>> > > >>
>>>> > > >>> Hello,
>>>> > > >>>
>>>> > > >>> We are using Qpid dispatch router 1.7, Qpid broker 7.1.3 on
>>>> redhat
>>>> > > linux
>>>> > > >>> rhel 6.4.
>>>> > > >>>
>>>> > > >>> Use case description:
>>>> > > >>> We have a cluster of 2 dispatch routers and 2 brokers. A sharded
>>>> > queue
>>>> > > >>> "myQueue" on the 2 brokers.
>>>> > > >>> Here is an illustration:
>>>> > > >>>
>>>> > > >>> [image: Diagram.jpg]
>>>> > > >>>
>>>> > > >>> The producer produces 2 messages, the messages are load
>>>> balanced on
>>>> > the
>>>> > > >>> 2 brokers. Then, Consumer 1 and consumer 2 ask for a receive().
>>>> > > >>>
>>>> > > >> Does your receive() function issue one credit per call?
>>>> > > >>
>>>> > > >>>
>>>> > > >>> Our observation is that the consumer 1 consumes the first
>>>> message and
>>>> > > >>> the consumer 2 is never getting the second message.
>>>> > > >>> We are aware that the first dispatch router will do a prefetch
>>>> for
>>>> > the
>>>> > > 2
>>>> > > >>> messages but what is weird is that the prefetched message 2 is
>>>> never
>>>> > > routed
>>>> > > >>> to the second dispatch router and consumer 2.
>>>> > > >>> I attached the dispatch routers config.
>>>> > > >>>
>>>> > > >>> This is what is going on in my view -
>>>> > > >> 1. The producer sends two messages - Message 1 goes to Broker 1
>>>> and
>>>> > > >> Message 2 goes to Broker 2.
>>>> > > >> 2. Now Consumer 1 attaches to Router 1 and calls receive()
>>>> issuing one
>>>> > > >> credit. (The Consumer 2 has not yet attached to Router 2). The
>>>> > autolinks
>>>> > > >> from Broker 1 to Router 1 and Broker 2 to Router 2 each have a
>>>> > prefetch
>>>> > > of
>>>> > > >> 250 credits each. So,Message 1 and Message 2
>>>> > > >> both come down to Router 1. Message 1 is sent to Consumer 1.
>>>> Message 2
>>>> > > is
>>>> > > >> waiting on the outbound queue of Consumer 1 waiting to receive a
>>>> > credit
>>>> > > >> from Consumer 1.
>>>> > > >> 3. Now Consumer 2 shows up and wants to receive Message 2 but
>>>> that
>>>> > > >> message is already queued up to go to Consumer 1 who is still
>>>> keeping
>>>> > > his
>>>> > > >> connection open
>>>> > > >> 4. If now Consumer 1 drops out, the Message 2 will be RELEASED
>>>> back to
>>>> > > >> its broker and that message will be sent to Consumer 2.
>>>> > > >>
>>>> > > >> Have you tried attaching Consumer 1 and Consumer 2 first and then
>>>> > > >> subsequently bringing up the Producer 1 and send two messages ?
>>>> In
>>>> > that
>>>> > > >> case, I think, both consumers will receive one message.
>>>> > > >>
>>>> > > >> Thanks.
>>>> > > >>
>>>> > > >>> Do you have any idea how we can make consumer 2 receive the
>>>> message?
>>>> > > >>>
>>>> > > >>> Thanks for your help,
>>>> > > >>> Rabih
>>>> > > >>>
>>>> > > >>>
>>>> ---------------------------------------------------------------------
>>>> > > >>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>>>> > > >>> For additional commands, e-mail: users-h...@qpid.apache.org
>>>> > > >>
>>>> > > >>
>>>> > > >
>>>> ---------------------------------------------------------------------
>>>> > > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>>>> > > > For additional commands, e-mail: users-h...@qpid.apache.org
>>>> > >
>>>> >
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>>> For additional commands, e-mail: users-h...@qpid.apache.org
>>
>>
