Re: [squid-users] squid 3.5.23 memory usage
On 20/01/2017 1:23 p.m., Ivan Larionov wrote: > Hello. > > I'm pretty sure this question has been asked multiple times already, but > after reading everything I found I still can't figure out squid memory > usage patterns. > > We're currently trying to upgrade from squid 2.7 to squid 3.5 and memory > usage on squid 3 is much much higher compared to squid 2 with the same > configuration. One thing to be aware of with this big step in versions is that 3.x has a lot more things 64-bit enabled where 2.x was more 32-bit oriented. It is minor in any one place, but does add up when dealing with large numbers of objects. > > What do I see: > > squid running for several days with low traffic: > > # top > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 7367 squid 20 0 4780m 4.4g 5224 S 6.0 60.6 105:01.76 squid -N > > So it uses 4.4GB resident memory. Ok, let's see important config options: > > cache_mem 2298756 KB > maximum_object_size_in_memory 8 KB > memory_replacement_policy lru > cache_replacement_policy lru > > cache_dir aufs /mnt/services/squid/cache 445644 16 256 > > minimum_object_size 64 bytes # none-zero so we dont cache mistakes > maximum_object_size 102400 KB > > So we configured 2.2GB memory cache and 500GB disk cache. Disk cache is > quite big but current usage is only 3GB: > > # du -sh /mnt/services/squid/cache # cache_dir > 3.0G /mnt/services/squid/cache > > Now I'm looking into this page > http://wiki.squid-cache.org/SquidFaq/SquidMemory and see: > > 14 MB of memory per 1 GB on disk for 64-bit Squid > These wiki numbers are based on an average object size of 32KB. By setting "maximum_object_size_in_memory 8 KB" you reduce that by 3x so need to multiply the overhead per object by (3x more objects in same space) for the cache_mem value to get a better estimate. So, ... up to 100 MB of index for cache_mem ... up to 6 GB of index for cache_dir Another difference is the buffers in Squid-3 are a bit bigger than those used for Squid-2. Up to 256 KB per FD (Squid-2 stopped at 64KB). BUT, your pool details below show only the 16KB buffer being used much. So I doubt it is client connections related. > Which means disk cache should use ~50MB of RAM. > > All these means we have ~2.2GB ram used for everything else except > cache_mem and disk cache index. No, the index is also in that 2.2 GB which is not being used by the cache_mem. > > Let's see top pools from mgr:mem: > > Pool (KB) %Tot > mem_node 2298833 55.082 > Short Strings 622365 14.913 > HttpHeaderEntry 404531 9.693 > Long Strings 284520 6.817 > MemObject 182288 4.368 > HttpReply 155612 3.729 > StoreEntry739651.772 > Medium Strings711521.705 > cbdata MemBuf (12)355730.852 > LRU policy node 304030.728 > MD5 digest113800.273 > 16K Buffer1056 0.025 > > These pools consume ~35% of total squid memory usage: Short Strings, > HttpHeaderEntry, Long Strings, HttpReply. Looks suspicious. On squid 2 same > pools use 10 times less memory. The mem_node is the cache_mem space itself, plus active transactions data. The StoreEntry is the index entry for each object (cache_dir, cache_mem and in-transit). The MemObject is the index entry for each in-memory object (cache_mem and in-transit). The HttpReply are those cached objects in parsed format. The HttpHeaderEntry are all the headers in those reply objects. The various Strings are the individual words/lines etc in those headers. So we are under 1% values by the time we are done eliminating data stored in cache_mem objects and active transaction data. > > I found a bug which looks similar to our experience: > http://bugs.squid-cache.org/show_bug.cgi?id=4084. > Since you have configured your cache_mem to be 2.2 GB and total memory usage is 4.4 GB the report saying 55% of memory is used for mem_node looks fine to me. 50% of 4.4 GB is your 2.2 GB cache_mem setting, and the extra 5% is probably active transactions and maybe some nodes for the cache_dir data. So I dont think it is the issue I mentioned in comments 8. That said we have not fully identified what the bug problem was. > I'm attaching our config, mgr:info, mgr:mem and some system info I > collected. > > Could someone say if this is normal and why it's so much different from > squid 2? > Well, tentatively yes. The Squid-3 numbers all looks reasonably accurate. So there is no obvious sign of any problem from this one point in time. But if it worries you keep an eye on it for a week or so and see if anything starts to skew. Graphs like Martin had in comment 5 on that bug report would be a good indicator of whether there is a problem or just a new "normal" level. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-user
[squid-users] squid 3.5.23 memory usage
Hello. I'm pretty sure this question has been asked multiple times already, but after reading everything I found I still can't figure out squid memory usage patterns. We're currently trying to upgrade from squid 2.7 to squid 3.5 and memory usage on squid 3 is much much higher compared to squid 2 with the same configuration. What do I see: squid running for several days with low traffic: # top PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 7367 squid 20 0 4780m 4.4g 5224 S 6.0 60.6 105:01.76 squid -N So it uses 4.4GB resident memory. Ok, let's see important config options: cache_mem 2298756 KB maximum_object_size_in_memory 8 KB memory_replacement_policy lru cache_replacement_policy lru cache_dir aufs /mnt/services/squid/cache 445644 16 256 minimum_object_size 64 bytes # none-zero so we dont cache mistakes maximum_object_size 102400 KB So we configured 2.2GB memory cache and 500GB disk cache. Disk cache is quite big but current usage is only 3GB: # du -sh /mnt/services/squid/cache # cache_dir 3.0G /mnt/services/squid/cache Now I'm looking into this page http://wiki.squid-cache.org/SquidFaq/SquidMemory and see: 14 MB of memory per 1 GB on disk for 64-bit Squid Which means disk cache should use ~50MB of RAM. All these means we have ~2.2GB ram used for everything else except cache_mem and disk cache index. Let's see top pools from mgr:mem: Pool (KB) %Tot mem_node 2298833 55.082 Short Strings 622365 14.913 HttpHeaderEntry 404531 9.693 Long Strings 284520 6.817 MemObject 182288 4.368 HttpReply 155612 3.729 StoreEntry739651.772 Medium Strings711521.705 cbdata MemBuf (12)355730.852 LRU policy node 304030.728 MD5 digest113800.273 16K Buffer1056 0.025 These pools consume ~35% of total squid memory usage: Short Strings, HttpHeaderEntry, Long Strings, HttpReply. Looks suspicious. On squid 2 same pools use 10 times less memory. I found a bug which looks similar to our experience: http://bugs.squid-cache.org/show_bug.cgi?id=4084. I'm attaching our config, mgr:info, mgr:mem and some system info I collected. Could someone say if this is normal and why it's so much different from squid 2? -- With best regards, Ivan Larionov. HTTP/1.1 200 OK Server: squid/3.5.23 Mime-Version: 1.0 Date: Thu, 19 Jan 2017 23:39:50 GMT Content-Type: text/plain;charset=utf-8 Expires: Thu, 19 Jan 2017 23:39:50 GMT Last-Modified: Thu, 19 Jan 2017 23:39:50 GMT X-Cache: MISS from ip-172-22-10-120 X-Cache-Lookup: MISS from ip-172-22-10-120:3128 Connection: close Squid Object Cache: Version 3.5.23 Build Info: Service Name: squid Start Time: Fri, 13 Jan 2017 23:35:32 GMT Current Time: Thu, 19 Jan 2017 23:39:50 GMT Connection information for squid: Number of clients accessing cache: (client_db off) Number of HTTP requests received: 8195690 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 948.1 Average ICP messages per minute since start:0.0 Select loop called: 73529108 times, 7.054 ms avg Cache information for squid: Hits as % of all requests: 5min: 29.2%, 60min: 28.9% Hits as % of bytes sent:5min: 89.0%, 60min: 89.1% Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Disk hits as % of hit requests: 5min: 100.0%, 60min: 100.0% Storage Swap size: 2915344 KB Storage Swap capacity: 0.6% used, 99.4% free Storage Mem size: 2276524 KB Storage Mem capacity: 99.0% used, 1.0% free Mean Object Size: 4.00 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.01745 0.01745 Cache Misses: 0.02899 0.02451 Cache Hits:0.00091 0.00091 Near Hits: 0.0 0.0 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.0 0.00094 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:518657.265 seconds CPU Time: 6265.444 seconds CPU Usage: 1.21% CPU Usage, 5 minute avg:6.43% CPU Usage, 60 minute avg: 5.11% Maximum Resident Size: 18579360 KB Page faults with physical i/o: 0 Memory accounted for: Total accounted: -20826 KB memPoolAlloc calls: 2192400061 memPoolFree calls: 2194290230 File descriptor usage for squid: Maximum number of file descriptors: 524288 Largest file desc currently in use: 70
