[
https://issues.apache.org/jira/browse/AMQCPP-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oren updated AMQCPP-760:
------------------------
Environment:
* {*}ActiveMQ-CPP Version{*}: 3.9.5
* {*}ActiveMQ Broker Version{*}: 5.17.3
* {*}Broker Hosts{*}: cca-prdappqua01.icc.corp:20801,
cca-prdappqua02.icc.corp:20801
* {*}Client Host{*}: cca-prdappbta05.icc.corp
* {*}OS{*}: RHEL 7.9
* {*}AMQ_URL{*}:
failover:([tcp://cca-prdappqua01.icc.corp:20801,tcp://cca-prdappqua02.icc.corp:20801])?maxReconnectAttempts=10&initialReconnectDelay=1000&timeout=30000&randomize=false
* {*}Demo Application{*}: C++ program (demo_sys_amq_modules.exe) subscribing
to ActiveMQ.Advisory.Consumer.Queue.101 with a 30-second receive timeout.
(attached)
was:
* {*}ActiveMQ-CPP Version{*}: 3.9.5
* {*}ActiveMQ Broker Version{*}: 5.17.3
* {*}Broker Hosts{*}: cca-prdappqua01.icc.corp:20801,
cca-prdappqua02.icc.corp:20801
* {*}Client Host{*}: cca-prdappbta05.icc.corp
* {*}OS{*}: RHEL 7.9
* {*}AMQ_URL{*}:
failover:([tcp://cca-prdappqua01.icc.corp:20801,tcp://cca-prdappqua02.icc.corp:20801])?maxReconnectAttempts=10&initialReconnectDelay=1000&timeout=30000&randomize=false
* {*}Demo Application{*}: C++ program (demo_sys_amq_modules.exe) subscribing
to ActiveMQ.Advisory.Consumer.Queue.101 with a 30-second receive timeout.
(attached)
> ActiveMQ-CPP 3.9.5 Failover Timeout Issue with Advisory Topic in PROD
> ---------------------------------------------------------------------
>
> Key: AMQCPP-760
> URL: https://issues.apache.org/jira/browse/AMQCPP-760
> Project: ActiveMQ C++ Client
> Issue Type: Bug
> Components: Transports
> Environment: * {*}ActiveMQ-CPP Version{*}: 3.9.5
> * {*}ActiveMQ Broker Version{*}: 5.17.3
> * {*}Broker Hosts{*}: cca-prdappqua01.icc.corp:20801,
> cca-prdappqua02.icc.corp:20801
> * {*}Client Host{*}: cca-prdappbta05.icc.corp
> * {*}OS{*}: RHEL 7.9
> * {*}AMQ_URL{*}:
> failover:([tcp://cca-prdappqua01.icc.corp:20801,tcp://cca-prdappqua02.icc.corp:20801])?maxReconnectAttempts=10&initialReconnectDelay=1000&timeout=30000&randomize=false
> * {*}Demo Application{*}: C++ program (demo_sys_amq_modules.exe) subscribing
> to ActiveMQ.Advisory.Consumer.Queue.101 with a 30-second receive timeout.
> (attached)
> Reporter: Oren
> Priority: Major
> Attachments: activemq.log, activemq.xml, demo_sys_amq_modules.cpp
>
>
> h2. Summary
> We are experiencing a timeout issue in our PROD environment when using the
> ActiveMQ-CPP 3.9.5 client with a {{failover:}} URI, specifically when
> subscribing to the advisory topic
> {{{}ActiveMQ.Advisory.Consumer.Queue.101{}}}. The issue persists despite
> upgrading from 3.9.3 (where AMQCPP-610 was suspected) and correcting a typo
> in the {{{}AMQ_URL{}}}. The same code works in non-PROD environments
> (TEST/BETA) and in PROD with a direct {{tcp://}} URI.
> h2. Issue Description
> The demo times out in PROD when using the failover: protocol to receive
> advisory messages for Queue.101. Key observations:
> * Works in TEST/BETA with identical broker settings and failover: URI
> (maxReconnectAttempts=10).
> ^*[BETA l_onissa@cca-betappbta01 cca_domain]$
> AMQ_URL="failover:([tcp://cca-betappqua01.icc.corp:20801])"*^
> ^*[BETA l_onissa@cca-betappbta01 cca_domain]$ demo_sys_amq_modules.exe 101*^
> ^ActiveMQ-CPP initialized. Connecting to
> failover:([tcp://cca-betappqua01.icc.corp:20801])^
> ^Waiting for advisory messages on: ActiveMQ.Advisory.Consumer.Queue.101 ...^
> ^Received advisory message of type: Advisory^
> * Works in PROD with direct [tcp://cca-prdappqua01.icc.corp:20801].
> ^*[PROD l_onissa@cca-prdappbta05 cca_domain]$
> AMQ_URL="[tcp://cca-prdappqua01.icc.corp:20801]"*^
> ^*[PROD l_onissa@cca-prdappbta05 cca_domain]$ demo_sys_amq_modules.exe 101*^
> ^ActiveMQ-CPP initialized. Connecting to
> [tcp://cca-prdappqua01.icc.corp:20801]^
> ^Waiting for advisory messages on: ActiveMQ.Advisory.Consumer.Queue.101 ...^
> ^Received advisory message of type: Advisory^
> * Fails in PROD with failover: even after upgrading to 3.9.5 (fixing
> [AMQCPP-610]) and correcting a typo in the original AMQ_URL
> (maxReconnectAttemps=0, which defaulted to maxReconnectAttempts=-1).
> ^*[PROD l_onissa@cca-prdappbta05 cca_domain]$
> AMQ_URL="failover:([tcp://cca-prdappqua01.icc.corp:20801])"*^
> ^*[PROD l_onissa@cca-prdappbta05 cca_domain]$ demo_sys_amq_modules.exe 101*^
> ^ActiveMQ-CPP initialized. Connecting to
> failover:([tcp://cca-prdappqua01.icc.corp:20801])^
> ^Waiting for advisory messages on: ActiveMQ.Advisory.Consumer.Queue.101 ...^
> ^No advisory message received (timeout).^
> * Java-based clients work reliably in PROD with failover:.
> h2. Relevant Broker Log Entries
> From activemq.log on cca-prdappqua01 (ActiveMQ 5.17.3) activemq.log file
> attached:
> ^2025-05-19 21:21:23,839 | WARN | TopicSubscription:
> consumer=ID:cca-prdappbta07.icc.corp-14259-1747678799256-0:0:-1:1, ...
> dispatched=1000, delivered=0, matched=1001, ... has twice its prefetch limit
> pending, without an ack; it appears to be slow: [tcp://10.222.12.83:31247]^
> ^2025-05-19 17:31:22,146 | WARN | Transport Connection to:
> [tcp://10.222.12.74:25787] failed: Broken pipe (Write failed)^
> ^2025-05-24 20:25:15,567 | WARN | Transport Connection to:
> [tcp://10.222.12.84:33681] failed: Cannot send, channel has already failed^
> * Slow topic consumers on cca-prdappbta07, cca-prdappbta02, cca-prdappbta08
> suggest broker resource contention.
> * Failed connections indicate potential network instability affecting
> failover clients.
> h2. Suspected Causes
> # {*}Broker Overload{*}: Slow consumers may delay advisory message delivery,
> impacting failover clients.
> # {*}Network Issues{*}: Failed connections suggest network instability,
> disrupting failover retries or subscriptions.
> # {*}Advisory Message Absence{*}: Possible lack of consumer activity on
> Queue.101 in PROD.
> # {*}Residual Effects{*}: Earlier 3.9.3 clients with maxReconnectAttemps=0
> (defaulting to -1) may have stressed the broker, affecting current 3.9.5
> clients.
> # {*}Failover Transport{*}: Possible edge case in 3.9.5 failover handling
> for advisory topics.
> h2. Steps Taken
> * Upgraded from ActiveMQ-CPP 3.9.3 to 3.9.5 to address AMQCPP-610.
> * Corrected AMQ_URL typo (maxReconnectAttemps to maxReconnectAttempts=10).
> * Tested with direct tcp:// URI (works) and failover: (times out).
> * Verified broker connectivity via telnet (successful).
> * Added consumer for Queue.101 to trigger advisory messages (no success yet).
> * Enabled debug logging in the demo (logs pending).
> h2. Request for Assistance
> * Is there a known issue in ActiveMQ-CPP 3.9.5 with failover and advisory
> topic subscriptions under high broker load or network instability?
> * Could slow consumers (as seen in the log) affect failover client
> subscriptions? Any recommended configurations to mitigate this?
> * Are there additional failover parameters or patches in 3.9.5 to improve
> stability for advisory topics?
> * Suggestions for diagnosing broker-side issues (e.g., clearing stale
> connections without restart)?
> h2. Additional Information
> * {*}Broker Config{*}: advisorySupport is true (default for 5.17.3 see
> attached activemq.xml).
> * {*}Compile Command{*}:
> ^g++ -O2 -finline-functions -D_XOPEN_SOURCE=600 -m64 -fPIC -Wall
> \-L/usr/lib64 -lactivemq-cpp \-isystem /usr/include/activemq-cpp-3.9.5
> \-std=c++11 \src/demo_sys_amq_modules.cpp -o
> target/bin/demo_sys_amq_modules.exe^
> * {*}Next Steps Planned{*}: Test with non-advisory topic, increase receive
> timeout to 60s, check activemq.xml for advisory settings, and consider
> upgrading to 3.10.0.
> Please advise on next steps or known issues. Thank you for your support!
> Best regards,
> Oren
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
For further information, visit: https://activemq.apache.org/contact
