dhairav opened a new issue, #9625: URL: https://github.com/apache/trafficserver/issues/9625
Hello, We use trafficserver as a reverse proxy for one of our caching systems. It is a 128G RAM system with about 56 TB of storage, wherein we have configured a (safe) RAM cache size of 92G. We've also configured an Average Object Size of 128K. According to our calculations, we should see at least 10+ GB of free memory on the system, but the RAM utilization of the system simply doesn't seem to come down, even at low-traffic periods. We typically hit 3-4 Gbps traffic and the RAM cache is almost always full, with a simple LRU configured. We have observed that the RAM utilization of the system has kept on increasing over time - and it has been OOM killed multiple times, causing traffic disruptions. What I would like to understand is why the RAM consumption is exceeding the RAM cache size explicitly by such a huge margin. The system is currently running at less than 2G of free RAM, with no other major processes running on the same. Here is the output from the `free` command - total: 125Gi used: 122Gi free: 961Mi shared: 2.0Mi buffers: 884Mi cache: 800Mi available: 1.5Gi I can see via `systemctl status trafficserver` that trafficserver is using 122G RAM `trafficserver.service - Apache Traffic Server is a fast, scalable and extensible caching proxy server. Loaded: loaded (/lib/systemd/system/trafficserver.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2023-04-11 00:29:36 UTC; 1 weeks 1 days ago Docs: man:traffic_server(8) Main PID: 54560 (traffic_manager) Tasks: 49 (limit: 9830) Memory: 122.0G CPU: 6d 20h 1min 19.626s CGroup: /system.slice/trafficserver.service ├─54560 /usr/bin/traffic_manager ├─54569 /usr/bin/traffic_server -M --httpport 80:fd=8,443:fd=9:ssl:proto=http,443:fd=10:ipv6 └─54571 traffic_crashlog --syslog --wait --host x86_64-pc-linux-gnu --user trafficserver Apr 11 00:29:36 dhairav systemd[1]: Started Apache Traffic Server is a fast, scalable and extensible caching proxy server.. Apr 11 00:29:36 dhairav traffic_manager[54560]: [E. Mgmt] log ==> [TrafficManager] using root directory '/usr' Apr 11 00:29:36 dhairav traffic_manager[54560]: NOTE: --- Manager Starting --- Apr 11 00:29:36 dhairav traffic_manager[54560]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 8.1.5 - (build # 081207 on Aug 12 2022 at 07:16:08) Apr 11 00:29:36 dhairav traffic_manager[54560]: NOTE: RLIMIT_NOFILE(7):cur(58981),max(58981) Apr 11 00:29:39 dhairav traffic_server[54569]: NOTE: --- traffic_server Starting --- Apr 11 00:29:39 dhairav traffic_server[54569]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 8.1.5 - (build # 081207 on Aug 12 2022 at 07:16:08) Apr 11 00:29:39 dhairav traffic_server[54569]: NOTE: RLIMIT_NOFILE(7):cur(58981),max(58981) Apr 11 00:29:39 dhairav traffic_manager[54569]: Traffic Server 8.1.5 Aug 12 2022 07:16:08 localhost Apr 11 00:29:39 dhairav traffic_manager[54569]: traffic_server: using root directory '/usr' ` Now as a short-term solution, I could create a boot script to prevent an OOM kill by adding an exception for trafficserver processes, but I believe it will not solve the inherent memory leak or any misconfiguration we might have done. I suspect something to do with the cache index overhead, but we've taken that into account in our calculations as well while setting the safer RAM cache 92G value. I'll be happy to attach my `records.config` file or anything else at my disposal to solve this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@trafficserver.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org