Hi Virgilio, What versions of Qpid and Broker were you using when you applied the patches to the broker (As we are currently using QPID 1.19.0 and Broker 2.27.1)? Additionally, could you please provide guidance on confirming whether the memory issue is associated with the usage of vectors within these long-running objects? If possible, please share any steps to follow. Thank you
@Ted Ross<mailto:[email protected]> Please find the below output of qdstat -m command for the qpid1 server which is consuming high memory currently. Memory Pools type size batch thread-max total in-threads rebal-in rebal-out ================================================================================================== qd_bitmask_t 24 64 128 1,920 1,344 382,869 382,878 qd_buffer_t 536 64 128 8,192 384 12,902,136 12,902,258 qd_composed_field_t 64 64 128 320 320 0 0 qd_composite_t 112 64 128 320 320 0 0 qd_connection_t 2,472 16 32 1,920 1,744 12,192 12,203 qd_connector_t 512 64 128 64 64 0 0 qd_deferred_call_t 32 64 128 384 256 312 314 qd_hash_handle_t 16 64 128 9,536 9,536 0 0 qd_hash_item_t 40 64 128 9,536 9,536 0 0 qd_iterator_t 160 64 128 3,648 384 275,400 275,451 qd_link_ref_t 24 64 128 2,368 192 60,144 60,178 qd_link_t 144 64 128 7,872 7,488 1,276 1,282 qd_listener_t 456 64 128 64 64 0 0 qd_log_entry_t 2,112 16 32 1,088 1,088 0 0 qd_management_context_t 48 64 128 64 64 0 0 qd_message_content_t 1,040 64 128 1,472 384 37,587 37,604 qd_message_t 128 64 128 2,688 384 1,716,133 1,716,169 qd_node_t 56 64 128 64 64 0 0 qd_parse_node_t 112 64 128 1,984 1,984 0 0 qd_parse_tree_t 32 64 128 64 64 0 0 qd_parsed_field_t 144 64 128 2,560 512 126,057 126,089 qd_pn_free_link_session_t 32 64 128 256 256 0 0 qd_session_t 56 64 128 6,400 5,888 1,391 1,399 qd_timer_t 72 64 128 64 64 0 0 qdr_action_t 136 64 128 3,328 192 65,499,099 65,499,148 qdr_address_config_t 80 64 128 576 576 0 0 qdr_address_t 416 64 128 1,152 1,152 0 0 qdr_auto_link_t 144 64 128 6,464 6,464 0 0 qdr_conn_identifier_t 112 64 128 64 64 0 0 qdr_connection_info_t 88 64 128 2,112 1,920 51,340 51,343 qdr_connection_ref_t 24 64 128 64 64 0 0 qdr_connection_t 632 64 128 2,112 1,920 51,340 51,343 qdr_connection_work_t 56 64 128 2,304 448 104,372 104,401 qdr_core_timer_t 40 64 128 64 64 0 0 qdr_delivery_cleanup_t 32 64 128 704 448 3,413,056 3,413,060 qdr_delivery_ref_t 24 64 128 512 384 1,671,475 1,671,477 qdr_delivery_t 312 64 128 2,496 192 1,709,088 1,709,124 qdr_field_t 40 64 128 3,136 256 79,489 79,534 qdr_forward_deliver_info_t 32 64 128 64 64 0 0 qdr_general_work_t 136 64 128 576 384 3,075,379 3,075,382 qdr_link_ref_t 24 64 128 17,088 13,888 32,905,297 32,905,347 qdr_link_t 552 64 128 7,936 7,488 104,286 104,293 qdr_link_work_t 48 64 128 3,456 512 32,470,813 32,470,859 qdr_node_t 88 64 128 64 64 0 0 qdr_query_t 344 64 128 192 192 107 107 qdr_terminus_t 64 64 128 6,016 320 1,553 1,642 qdtm_router_t 16 64 128 128 128 0 0 Memory Summary VmSize Pooled ==================== 12.1 GiB 25.5 MiB Below is the current statistics for qpid1 server - Router Statistics attr value =================================================== Version 1.19.0 Mode interior Router Id od-router-1-prod Worker Threads 4 Uptime 006:21:31:41 VmSize 12.1 GiB Area 0 Link Routes 0 Auto Links 6418 Links 7246 Nodes 1 Addresses 1088 Connections 1690 Presettled Count 0 Dropped Presettled Count 0 Accepted Count 106722455 Rejected Count 0 Released Count 273624 Modified Count 962 Deliveries Delayed > 1sec 4901525 Deliveries Delayed > 10sec 775262 Deliveries Stuck > 10sec 0 Deliveries to Fallback 0 Links Blocked 1002 Ingress Count 84870446 Egress Count 106011425 Transit Count 712114 Deliveries from Route Container 52221048 Deliveries to Route Container 32649333 Thanks, Nilesh Khokale From: Virgilio Fornazin <[email protected]> Sent: Friday, December 15, 2023 4:23 PM To: [email protected] Cc: Ajit Tathawade (Contractor) <[email protected]>; Nilesh Khokale (Contractor) <[email protected]>; Ted Ross <[email protected]> Subject: Re: High Memory consumption with Qpid Dispatch 1.19.0 [CAUTION: EXTERNAL SENDER] We had this issue with qpid ++ broker and was related to vectors used in long running objects (queues etc) c++11 introduced shrink_to_fit() and we patched the broker at that time to avoid those memory issues. On Fri, 15 Dec 2023 at 18:21 Ekta Awasthi <[email protected]<mailto:[email protected]>> wrote: Hello Tod, while running the qdstat -m command, I get below error. -sh-4.2$ qdstat -a ConnectionException: Connection amqp://0.0.0.0:amqp disconnected: Condition('proton.pythonio', 'Connection refused to all addresses') Ekta Awasthi, Engineer, EAI Operations & Support | Office Depot, Inc. 6600 North Military Trail | Boca Raton, FL 33496-2434<https://www.google.com/maps/search/6600+North+Military+Trail+%7C+Boca+Raton,+FL+33496-2434+%0D%0AOffice:+561?entry=gmail&source=g> Office: 561<https://www.google.com/maps/search/6600+North+Military+Trail+%7C+Boca+Raton,+FL+33496-2434+%0D%0AOffice:+561?entry=gmail&source=g>-438-3552 | Mobile: 206-966-5577 | [email protected]<mailto:[email protected]> -- Tips for EAI Support Engagement -- -EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS> -EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support -As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to [email protected]<mailto:[email protected]> or by initiating a MIRT ________________________________ From: Ted Ross <[email protected]<mailto:[email protected]>> Sent: Friday, December 15, 2023 2:48 PM To: Ekta Awasthi <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>>; Ajit Tathawade (Contractor) <[email protected]<mailto:[email protected]>>; Nilesh Khokale (Contractor) <[email protected]<mailto:[email protected]>> Subject: Re: High Memory consumption with Qpid Dispatch 1.19.0 [CAUTION: EXTERNAL SENDER] Ekta, You can get more granular memory-use data by using the "qdstat -m" command against the router when its memory footprint is larger than you think it should be. I assume you've been using this version for some time. It might be helpful to look into what other things changed right before the memory consumption problem started. -Ted On Fri, Dec 15, 2023 at 12:08 PM Ekta Awasthi <[email protected]<mailto:[email protected]>> wrote: Hi All & Tod, We are currently encountering elevated memory consumption with qpid dispatch version 1.19.0. Although the memory is released upon restarting qpid, it gradually accumulates again, surpassing 80% memory usage. As QPID in our case servers as a routing mechanism, handling traffic from NLB to QPID and then to the broker. While investigating the cause of this behavior and examining memory usage from New Relic (NR) graph indicates that the qdrouterd process is responsible for the memory consumption. We are seeking insights into the root cause of this issue and whether it may be related to the version (1.19.0). Please find additional information below. Architecture: NLB --> QPID(2 qpids acting as consumers) --> BROKER (Total of 3 pairs. Master/Slave configuration for HA) Qpids were restarted on 12-10-23 as you can see below the gradual increase has been happening ever since. Ekta Awasthi CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.
