> On Mon, 11 Apr 2011 07:52:41 -0400 (EDT), rpere...@lavabit.com wrote:
>> Hi
>>
>> How I can limit the ram memory use in my squid/tproxy box ?
>>
>> I have a fast server with 16Gb ram. The average bandwidth is about
>> 60-70
>> Mb/s.
>>
>> The bridge works well but when the cache and memory becomes full its
>> goes
>> slow and becomes unusable.
>>
>> The cache is 10G size.
>>
>> I see that a few hours to be working and have used the 16 GB of RAM
>> starts to run slow.
>>
>> Any help ?. I have configured some memory optimization options but
>> looks
>> don't help for me.
>>
>> Thanks in advance
>>
>> roberto
>>
>> This is my config:
>>
>> -------------------------------------
>>
>> cache_mem 10 MB
>> memory_pools off
>> cache_swap_low 94
>> cache_swap_high 95
>>
> <snip>
>> # Example rule allowing access from your local networks.
>> # Adapt to list your (internal) IP networks from where browsing
>> # should be allowed
>> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
>> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
>> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
>> acl localnet src fc00::/7 # RFC 4193 local private network range
>> acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
>> machines
>>
>> acl net-g1 src 200.117.xxx.xxx/24
>> acl net-g2 src 200.xxx.xxx.xxx/24
>> acl net-g3 src 190.xxx.xxx.xxx/24
>>
> <snip>
>>
>> http_access allow net-g1
>> http_access allow net-g2
>> http_access allow net-g3
>
> Being "allow" you can grab some extra speed by combining these all
> under one ACL name. "localnet" being the standard one for local network
> ranges.
>
> <snip>
>>
>> # Uncomment and adjust the following to add a disk cache directory.
>> cache_dir ufs /var/spool/squid 10000 64 256
>>
> <snip>
>
>
> Given those cache sizes, your Squid box should be using around 110 MB
> of RAM for index plus a little. Even assuming a worst-case of a minutes
> traffic accumulated in transit buffers comes nowhere close to filling 16
> GB up.
>
> Some questions that may help narrow down where the slow is coming from:
> What version of Squid is this?
> What are your avg object size?
> How many concurrent client connections?
> "slow" and "normal" response speeds?
> Do you notice any change in the Squid->Internet request types during
> the slowdown? (ie a move to extra MISS/HIT/IMS/REFRESH)
>
> And like Michael said the disk IO stats are important to look at. When
> the cache_dir gets to 94% full it will start spending CPU and disk
> cycles on erasing objects. If it reaches 95% a larger portion of cycles
> get used until it drops down below 94% again.
>
>
> Amos
>
Hi all !
These are two test samples when the proxy becomes slow.
Any clue ? Maybe an I/O disk issue.
roberto
First sample:
[jorge@proxy ~]$ squidclient mgr:info
HTTP/1.0 200 OK
Server: squid/3.1.1
Mime-Version: 1.0
Date: Wed, 20 Apr 2011 17:38:35 GMT
Content-Type: text/plain
Expires: Wed, 20 Apr 2011 17:38:35 GMT
Last-Modified: Wed, 20 Apr 2011 17:38:35 GMT
X-Cache: MISS from proxy.net-g.com.ar
X-Cache-Lookup: MISS from proxy.net-g.com.ar:3128
Via: 1.0 proxy.net-g.com.ar (squid/3.1.1)
Proxy-Connection: close
Squid Object Cache: Version 3.1.1
Start Time: Wed, 20 Apr 2011 17:34:35 GMT
Current Time: Wed, 20 Apr 2011 17:38:35 GMT
Connection information for squid:
Number of clients accessing cache: 171
Number of HTTP requests received: 99367
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 24781.1
Average ICP messages per minute since start: 0.0
Select loop called: 588320 times, 0.409 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 27.8%, 60min: 27.8%
Hits as % of bytes sent: 5min: 16.9%, 60min: 16.9%
Memory hits as % of hit requests: 5min: 6.3%, 60min: 6.3%
Disk hits as % of hit requests: 5min: 45.5%, 60min: 45.5%
Storage Swap size: 6029640 KB
Storage Swap capacity: 98.1% used, 1.9% free
Storage Mem size: 319540 KB
Storage Mem capacity: 15.4% used, 84.6% free
Mean Object Size: 14.94 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.30459 0.30459
Cache Misses: 0.42149 0.42149
Cache Hits: 0.00179 0.00179
Near Hits: 0.18699 0.18699
Not-Modified Replies: 0.00091 0.00091
DNS Lookups: 0.03374 0.03374
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 240.588 seconds
CPU Time: 165.222 seconds
CPU Usage: 68.67%
CPU Usage, 5 minute avg: 67.92%
CPU Usage, 60 minute avg: 67.92%
Process Data Segment Size via sbrk(): 259836 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
Total space in arena: 260112 KB
Ordinary blocks: 259914 KB 946 blks
Small blocks: 0 KB 0 blks
Holding blocks: 313188 KB 2048 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 197 KB
Total in use: 573102 KB 100%
Total free: 197 KB 0%
Total size: 573300 KB
Memory accounted for:
Total accounted: 517250 KB 90%
memPool accounted: 517250 KB 90%
memPool unaccounted: 56049 KB 10%
memPoolAlloc calls: 42349480
memPoolFree calls: 40063448
File descriptor usage for squid:
Maximum number of file descriptors: 16384
Largest file desc currently in use: 13418
Number of file desc currently in use: 13197
Files queued for open: 6
Available number of file descriptors: 3181
Reserved number of file descriptors: 100
Store Disk files open: 171
Internal Data Structures:
404884 StoreEntries
24853 StoreEntries with MemObjects
23380 Hot Object Cache Items
403505 on-disk objects
--------------------------
Second sample:
[jorge@proxy ~]$ squidclient mgr:info
HTTP/1.0 200 OK
Server: squid/3.1.1
Mime-Version: 1.0
Date: Thu, 21 Apr 2011 17:53:35 GMT
Content-Type: text/plain
Expires: Thu, 21 Apr 2011 17:53:35 GMT
Last-Modified: Thu, 21 Apr 2011 17:53:35 GMT
X-Cache: MISS from proxy.net-g.com.ar
X-Cache-Lookup: MISS from proxy.net-g.com.ar:3128
Via: 1.0 proxy.net-g.com.ar (squid/3.1.1)
Proxy-Connection: close
Squid Object Cache: Version 3.1.1
Start Time: Thu, 21 Apr 2011 17:44:32 GMT
Current Time: Thu, 21 Apr 2011 17:53:35 GMT
Connection information for squid:
Number of clients accessing cache: 201
Number of HTTP requests received: 237647
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 26240.8
Average ICP messages per minute since start: 0.0
Select loop called: 1117825 times, 0.486 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 26.5%, 60min: 26.0%
Hits as % of bytes sent: 5min: 13.4%, 60min: 13.8%
Memory hits as % of hit requests: 5min: 18.9%, 60min: 15.6%
Disk hits as % of hit requests: 5min: 33.5%, 60min: 36.9%
Storage Swap size: 6028184 KB
Storage Swap capacity: 98.1% used, 1.9% free
Storage Mem size: 784264 KB
Storage Mem capacity: 37.8% used, 62.2% free
Mean Object Size: 15.61 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.33943 0.33943
Cache Misses: 0.46965 0.44492
Cache Hits: 0.00286 0.00286
Near Hits: 0.19742 0.20843
Not-Modified Replies: 0.00179 0.00179
DNS Lookups: 0.01331 0.03079
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 543.383 seconds
CPU Time: 348.531 seconds
CPU Usage: 64.14%
CPU Usage, 5 minute avg: 65.91%
CPU Usage, 60 minute avg: 64.14%
Process Data Segment Size via sbrk(): 838644 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
Total space in arena: 838920 KB
Ordinary blocks: 838698 KB 1764 blks
Small blocks: 0 KB 0 blks
Holding blocks: 311892 KB 2048 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 221 KB
Total in use: 1150590 KB 100%
Total free: 221 KB 0%
Total size: 1150812 KB
Memory accounted for:
Total accounted: 1059722 KB 92%
memPool accounted: 1059722 KB 92%
memPool unaccounted: 91089 KB 8%
memPoolAlloc calls: 76008750
memPoolFree calls: 72446323
File descriptor usage for squid:
Maximum number of file descriptors: 16384
Largest file desc currently in use: 13386
Number of file desc currently in use: 13078
Files queued for open: 0
Available number of file descriptors: 3306
Reserved number of file descriptors: 100
Store Disk files open: 143
Internal Data Structures:
387582 StoreEntries
58603 StoreEntries with MemObjects
57148 Hot Object Cache Items
386166 on-disk objects
[jorge@proxy ~]$
