Ekta created DISPATCH-2353:
------------------------------
Summary: More than configured consumers count
Key: DISPATCH-2353
URL: https://issues.apache.org/jira/browse/DISPATCH-2353
Project: Qpid Dispatch
Issue Type: Bug
Components: Router Node, Routing Engine
Affects Versions: 1.16.0
Environment: PROD
Reporter: Ekta
Hi There,
We have few issues with the qpid-dispatch router where we are seeing multiple
consumers are getting created randomly by our qpid dispatch routers. We are
unable to trace nor able to replicate this issue in our lower env's. Below is
the architecture diagram. Please provide any suggestions or feedback to help us
understand this issue better. Thanks
{*}Flow{*}:
Microservice (Kubernetes) ---> NLB (Load balancer) ---> *2* Qpid-Dispatch
Routers acting up as consumers in front of brokers ({*}1.16 version{*}) --->
*4* Broker pairs of activemq{*}(2.18 version){*} masters and slaves
({*}independent pairs{*})
*Problem statement # 1*
We are seeing more than the configured number of consumers count in our
activemq hawtio console causing messages to sit in delivering count which are
Un browsable since those messages are currently being delivered to its
consumers. Having only two qpid dispatch routers(acting as consumers) infront
of our activemq brokers, the count should always remain 2 but at times it is
going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s
to release the stuck/bad consumer so that messages can be processed/consumed.
Restarting of the client(microservice) and broker does NOT help in this case.
Only component that helps is QPID.
*Problem statement # 2*
At times we see messages going to delivering count even when there are only two
configured consumers(2qpids) showing in activemq hawtio console. We don't know
the cause to why the messages get stuck in delivering count. To resolve this,
we tried restarting the consumer service but that did not help. Next up we
tried restarting the brokers, that did not help and noticed that all the stuck
delivering messages, the broker slowly replayed back and Eventually the stuck
messages came back to delivering count post broker restart. To resolve this
issue, we are having to bounce the qpid’s to release the messages and that
fixes the issue.
*Version of qpid-dispatch router*
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
