Re: [squid-users] Squid use all memory ram
We are still chasing this one down but made a major breakthrough. The leak is related to squid in intercept mode + SSL decryption + origin with invalid certs. In our case, the majority of the cases were related to Windows Update and Windows Defender domains, so a stopgap solution is to bypass decryption for these sites (eg, .update.microsoft.com). If you do, don't use dstdomain ACL, as the domain is not available at the time of the checking. Use something like ssl::server_name[_regex]. Hope this helps! On Fri, Jan 27, 2023 at 2:28 PM Gustavo Carvalho wrote: > Hi Hamilton, thanks for helping! > > I wish I could provide this log while squid is crashing, but there > have been no incidents since wednesday. From what I've heard, the RAM > on that server's VM has been increased to 32GB. > > Anyway, here is the squidclient mgr:mem log output. I hope it can be > helpful. > > On Thu, Jan 26, 2023 at 5:43 PM Hamilton Coutinho > wrote: > > > > Hi Gustavo, > > > > I'm seeing the same thing. I could narrow down (but can't say with 100% > confidence) to the code that does certificate verification when configured > for SSL decryption. What is the output of squidclient mgr:mem for you? Do > you see unexplainably high counts for in-use objects like HttpRequest, > PeekingPeerConnector, Comm::Connection, Security::ErrorDetail? > > > > > > On Thu, Jan 26, 2023 at 12:31 PM Gustavo Carvalho < > gustavocarv4...@gmail.com> wrote: > >> > >> Hi, > >> > >> I have Squid 5.6 on a FreeBSD 13.1 server with 16GB RAM > >> > >> I noticed that squid starts to consume a lot of ram until it starts to > >> consume swap space. When this happens, browsing becomes extremely > >> slow. > >> > >> This is happening at least once a week when I have to restart squid to > >> get it back to normal. > >> > >> Any ideas? > >> > >> # Wed Jan 25 08:30:00 -03 2023 # > >> > >> HTTP/1.1 200 OK > >> Server: squid > >> Mime-Version: 1.0 > >> Date: Wed, 25 Jan 2023 11:30:00 GMT > >> Content-Type: text/plain;charset=utf-8 > >> Expires: Wed, 25 Jan 2023 11:30:00 GMT > >> Last-Modified: Wed, 25 Jan 2023 11:30:00 GMT > >> X-Cache: MISS from .. > >> X-Cache-Lookup: MISS from ..:3128 > >> Via: 1.1 .. (squid) > >> Connection: close > >> > >> Squid Object Cache: Version 5.6 > >> Build Info: > >> Service Name: squid > >> Start Time: Thu, 19 Jan 2023 20:25:17 GMT > >> Current Time: Wed, 25 Jan 2023 11:30:00 GMT > >> Connection information for squid: > >> Number of clients accessing cache: 224 > >> Number of HTTP requests received: 7541590 > >> Number of ICP messages received: 0 > >> Number of ICP messages sent: 0 > >> Number of queued ICP replies: 0 > >> Number of HTCP messages received: 0 > >> Number of HTCP messages sent: 0 > >> Request failure ratio: 0.00 > >> Average HTTP requests per minute since start: 930.5 > >> Average ICP messages per minute since start: 0.0 > >> Select loop called: 78733524 times, 6.176 ms avg > >> Cache information for squid: > >> Hits as % of all requests: 5min: 8.4%, 60min: 12.1% > >> Hits as % of bytes sent: 5min: 21.6%, 60min: 14.1% > >> Memory hits as % of hit requests: 5min: 90.8%, 60min: 75.9% > >> Disk hits as % of hit requests: 5min: 4.0%, 60min: 19.7% > >> Storage Swap size: 2829956 KB > >> Storage Swap capacity: 90.0% used, 10.0% free > >> Storage Mem size: 16172 KB > >> Storage Mem capacity: 98.7% used, 1.3% free > >> Mean Object Size: 28.95 KB > >> Requests given to unlinkd: 186982 > >> Median Service Times (seconds) 5 min60 min: > >> HTTP Requests (All): 0.00562 0.01847 > >> Cache Misses: 0.15048 0.23230 > >> Cache Hits:0.0 0.0 > >> Near Hits: 0.14252 0.13498 > >> Not-Modified Replies: 0.00865 0.03066 > >> DNS Lookups: 0.0 0.00372 > >> ICP Queries: 0.0 0.0 > >> Resource usage for squid: > >> UP Time: 486282.612 seconds > >> CPU Time: 6.712 seconds > >> CPU Usage: 13.48% > >> CPU Usage, 5 minute avg: 26.89% > >> CPU Usage, 60 minute avg: 68.00% > >> Maximum Resident Size: 37896960 KB > >> Page faults with physical i/o: 10843 > >> Memory accounted for: > >> Total accounted: -1459461 KB > >> memPoolAlloc calls: 11408 > >> memPoolFree calls: 1888969689 > >> File descriptor usage for squid: > >> Maximum number of file descriptors: 4096 > >> Largest file desc currently in use: 2149 > >> Number of file desc currently in use: 679 > >> Files queued for open: 0 > >> Available number of file descriptors: 3417 > >> Reserved number of file descriptors: 100 > >> Store Disk files open: 0 > >> Internal Data Structures: > >> 97906 StoreEntries > >> 3002 StoreEntries with MemObjects > >> 2838 Hot Object Cache
Re: [squid-users] Squid use all memory ram
Hi Hamilton, thanks for helping! I wish I could provide this log while squid is crashing, but there have been no incidents since wednesday. From what I've heard, the RAM on that server's VM has been increased to 32GB. Anyway, here is the squidclient mgr:mem log output. I hope it can be helpful. On Thu, Jan 26, 2023 at 5:43 PM Hamilton Coutinho wrote: > > Hi Gustavo, > > I'm seeing the same thing. I could narrow down (but can't say with 100% > confidence) to the code that does certificate verification when configured > for SSL decryption. What is the output of squidclient mgr:mem for you? Do you > see unexplainably high counts for in-use objects like HttpRequest, > PeekingPeerConnector, Comm::Connection, Security::ErrorDetail? > > > On Thu, Jan 26, 2023 at 12:31 PM Gustavo Carvalho > wrote: >> >> Hi, >> >> I have Squid 5.6 on a FreeBSD 13.1 server with 16GB RAM >> >> I noticed that squid starts to consume a lot of ram until it starts to >> consume swap space. When this happens, browsing becomes extremely >> slow. >> >> This is happening at least once a week when I have to restart squid to >> get it back to normal. >> >> Any ideas? >> >> # Wed Jan 25 08:30:00 -03 2023 # >> >> HTTP/1.1 200 OK >> Server: squid >> Mime-Version: 1.0 >> Date: Wed, 25 Jan 2023 11:30:00 GMT >> Content-Type: text/plain;charset=utf-8 >> Expires: Wed, 25 Jan 2023 11:30:00 GMT >> Last-Modified: Wed, 25 Jan 2023 11:30:00 GMT >> X-Cache: MISS from .. >> X-Cache-Lookup: MISS from ..:3128 >> Via: 1.1 .. (squid) >> Connection: close >> >> Squid Object Cache: Version 5.6 >> Build Info: >> Service Name: squid >> Start Time: Thu, 19 Jan 2023 20:25:17 GMT >> Current Time: Wed, 25 Jan 2023 11:30:00 GMT >> Connection information for squid: >> Number of clients accessing cache: 224 >> Number of HTTP requests received: 7541590 >> Number of ICP messages received: 0 >> Number of ICP messages sent: 0 >> Number of queued ICP replies: 0 >> Number of HTCP messages received: 0 >> Number of HTCP messages sent: 0 >> Request failure ratio: 0.00 >> Average HTTP requests per minute since start: 930.5 >> Average ICP messages per minute since start: 0.0 >> Select loop called: 78733524 times, 6.176 ms avg >> Cache information for squid: >> Hits as % of all requests: 5min: 8.4%, 60min: 12.1% >> Hits as % of bytes sent: 5min: 21.6%, 60min: 14.1% >> Memory hits as % of hit requests: 5min: 90.8%, 60min: 75.9% >> Disk hits as % of hit requests: 5min: 4.0%, 60min: 19.7% >> Storage Swap size: 2829956 KB >> Storage Swap capacity: 90.0% used, 10.0% free >> Storage Mem size: 16172 KB >> Storage Mem capacity: 98.7% used, 1.3% free >> Mean Object Size: 28.95 KB >> Requests given to unlinkd: 186982 >> Median Service Times (seconds) 5 min60 min: >> HTTP Requests (All): 0.00562 0.01847 >> Cache Misses: 0.15048 0.23230 >> Cache Hits:0.0 0.0 >> Near Hits: 0.14252 0.13498 >> Not-Modified Replies: 0.00865 0.03066 >> DNS Lookups: 0.0 0.00372 >> ICP Queries: 0.0 0.0 >> Resource usage for squid: >> UP Time: 486282.612 seconds >> CPU Time: 6.712 seconds >> CPU Usage: 13.48% >> CPU Usage, 5 minute avg: 26.89% >> CPU Usage, 60 minute avg: 68.00% >> Maximum Resident Size: 37896960 KB >> Page faults with physical i/o: 10843 >> Memory accounted for: >> Total accounted: -1459461 KB >> memPoolAlloc calls: 11408 >> memPoolFree calls: 1888969689 >> File descriptor usage for squid: >> Maximum number of file descriptors: 4096 >> Largest file desc currently in use: 2149 >> Number of file desc currently in use: 679 >> Files queued for open: 0 >> Available number of file descriptors: 3417 >> Reserved number of file descriptors: 100 >> Store Disk files open: 0 >> Internal Data Structures: >> 97906 StoreEntries >> 3002 StoreEntries with MemObjects >> 2838 Hot Object Cache Items >> 97742 on-disk objects >> >> -- pfctl -si -- >> >> Status: Enabled for 25 days 22:58:24 Debug: Urgent >> >> State Table Total Rate >> current entries 8085 >> searches 6650475717 2965.4/s >> inserts133521957 59.5/s >> removals 133552376 59.5/s >> Counters >> match 605960865 270.2/s >> bad-offset 00.0/s >> fragment 10.0/s >> short 540.0/s >> normalize6590.0/s >> memory
Re: [squid-users] Squid use all memory ram
Hi Gustavo, I'm seeing the same thing. I could narrow down (but can't say with 100% confidence) to the code that does certificate verification when configured for SSL decryption. What is the output of squidclient mgr:mem for you? Do you see unexplainably high counts for in-use objects like HttpRequest, PeekingPeerConnector, Comm::Connection, Security::ErrorDetail? On Thu, Jan 26, 2023 at 12:31 PM Gustavo Carvalho wrote: > Hi, > > I have Squid 5.6 on a FreeBSD 13.1 server with 16GB RAM > > I noticed that squid starts to consume a lot of ram until it starts to > consume swap space. When this happens, browsing becomes extremely > slow. > > This is happening at least once a week when I have to restart squid to > get it back to normal. > > Any ideas? > > # Wed Jan 25 08:30:00 -03 2023 # > > HTTP/1.1 200 OK > Server: squid > Mime-Version: 1.0 > Date: Wed, 25 Jan 2023 11:30:00 GMT > Content-Type: text/plain;charset=utf-8 > Expires: Wed, 25 Jan 2023 11:30:00 GMT > Last-Modified: Wed, 25 Jan 2023 11:30:00 GMT > X-Cache: MISS from .. > X-Cache-Lookup: MISS from ..:3128 > Via: 1.1 .. (squid) > Connection: close > > Squid Object Cache: Version 5.6 > Build Info: > Service Name: squid > Start Time: Thu, 19 Jan 2023 20:25:17 GMT > Current Time: Wed, 25 Jan 2023 11:30:00 GMT > Connection information for squid: > Number of clients accessing cache: 224 > Number of HTTP requests received: 7541590 > Number of ICP messages received: 0 > Number of ICP messages sent: 0 > Number of queued ICP replies: 0 > Number of HTCP messages received: 0 > Number of HTCP messages sent: 0 > Request failure ratio: 0.00 > Average HTTP requests per minute since start: 930.5 > Average ICP messages per minute since start: 0.0 > Select loop called: 78733524 times, 6.176 ms avg > Cache information for squid: > Hits as % of all requests: 5min: 8.4%, 60min: 12.1% > Hits as % of bytes sent: 5min: 21.6%, 60min: 14.1% > Memory hits as % of hit requests: 5min: 90.8%, 60min: 75.9% > Disk hits as % of hit requests: 5min: 4.0%, 60min: 19.7% > Storage Swap size: 2829956 KB > Storage Swap capacity: 90.0% used, 10.0% free > Storage Mem size: 16172 KB > Storage Mem capacity: 98.7% used, 1.3% free > Mean Object Size: 28.95 KB > Requests given to unlinkd: 186982 > Median Service Times (seconds) 5 min60 min: > HTTP Requests (All): 0.00562 0.01847 > Cache Misses: 0.15048 0.23230 > Cache Hits:0.0 0.0 > Near Hits: 0.14252 0.13498 > Not-Modified Replies: 0.00865 0.03066 > DNS Lookups: 0.0 0.00372 > ICP Queries: 0.0 0.0 > Resource usage for squid: > UP Time: 486282.612 seconds > CPU Time: 6.712 seconds > CPU Usage: 13.48% > CPU Usage, 5 minute avg: 26.89% > CPU Usage, 60 minute avg: 68.00% > Maximum Resident Size: 37896960 KB > Page faults with physical i/o: 10843 > Memory accounted for: > Total accounted: -1459461 KB > memPoolAlloc calls: 11408 > memPoolFree calls: 1888969689 > File descriptor usage for squid: > Maximum number of file descriptors: 4096 > Largest file desc currently in use: 2149 > Number of file desc currently in use: 679 > Files queued for open: 0 > Available number of file descriptors: 3417 > Reserved number of file descriptors: 100 > Store Disk files open: 0 > Internal Data Structures: > 97906 StoreEntries > 3002 StoreEntries with MemObjects > 2838 Hot Object Cache Items > 97742 on-disk objects > > -- pfctl -si -- > > Status: Enabled for 25 days 22:58:24 Debug: Urgent > > State Table Total Rate > current entries 8085 > searches 6650475717 2965.4/s > inserts133521957 59.5/s > removals 133552376 59.5/s > Counters > match 605960865 270.2/s > bad-offset 00.0/s > fragment 10.0/s > short 540.0/s > normalize6590.0/s > memory 00.0/s > bad-timestamp 00.0/s > congestion 00.0/s > ip-option 00.0/s > proto-cksum00.0/s > state-mismatch1046740.0/s > state-insert 385010.0/s > state-limit00.0/s > src-limit
