Re: [squid-users] low squid performance?
fre 2006-02-24 klockan 21:49 +0100 skrev Tomasz Kolaj: I found in archive: http://www.squid-cache.org/mail-archive/squid-dev/200212/0119.html How is it possible to get 2833 requests/second on 2xP3 1,4GHz box? Is it true? In a reverse proxy with epoll and a high hit ratio near 100% it is not entirely unrealistic. Doing the same in a forward Internet proxy is a quite different business. My result is poor in compare to his result;) (my max 135 requests/second with 95% usage of processor with logging turned off). Note: A propely configured Squid-2.5 will in most cases reach 100% CPU usage at about 60% of it's peak performance. Regards Henrik signature.asc Description: Detta är en digitalt signerad meddelandedel
Re: [squid-users] low squid performance?
Dnia wtorek, 28 lutego 2006 10:11, Henrik Nordstrom napisał: fre 2006-02-24 klockan 21:49 +0100 skrev Tomasz Kolaj: I found in archive: http://www.squid-cache.org/mail-archive/squid-dev/200212/0119.html How is it possible to get 2833 requests/second on 2xP3 1,4GHz box? Is it true? In a reverse proxy with epoll and a high hit ratio near 100% it is not entirely unrealistic. Doing the same in a forward Internet proxy is a quite different business. Ok, so what performance I should expect from my hardware/configuration? (in some kind of approximation). And ... what Can I do. I can't change processors. So: - I can add more discs (next wd raptors) - add more memory (ddr2) - change configuration My result is poor in compare to his result;) (my max 135 requests/second with 95% usage of processor with logging turned off). Note: A propely configured Squid-2.5 will in most cases reach 100% CPU usage at about 60% of it's peak performance. Hm, so when my squid reach 100% cpu usage on 130 reqs/sec I should expect that squid can do ~ 200reqs/sec ? Regards -- Tomasz
Re: [squid-users] low squid performance?
Dnia czwartek, 23 lutego 2006 16:17, Matus UHLAR - fantomas napisał:
On 23.02 14:25, Tomasz Kolaj wrote:
Dnia czwartek, 23 lutego 2006 11:32, napisałeś:
On 22.02 23:13, Tomasz Kolaj wrote:
I observed have too low performance. On 2x 64bit Xeon 2,8GHz 2GB
DDR2, 2x WD RAPTOR Squid 2.5.STABLE12 can answer max for 120
requests/s. 115 r/s - 97-98% usage of first processor. Second is
unusable for squid :/. I have two cache_dirs (aufs). One pre disk.
Maybe you have too many ACL's?
I pasted my squid.conf in one of last posts. I have much of addresses
bloacked in file spywaredomains.txt
sorry - the thread was broken and I didn't see it. (b)lame mailers who
break threads by not using References: or at least In-Reply-To: headers...
On 24.02 14:33, Tomasz Kolaj wrote:
refresh_pattern -i (.*jpg$|.*gif$|.*png$) 0 50% 28800
refresh_pattern -i (.*html$|.*htm|.*shtml|.*php) 0 20% 1440
acl mGG dstdomain .adserver.gadugadu.pl .adserver.gadu-gadu.pl
redirector_access deny !mGG
I'd use
redirector_access allow mGG
redirector_access deny all
but it shouldn't cause big diff.
squid compiled with options:
aragorn ~ # squid -v
Squid Cache: Version 2.5.STABLE12
configure options: --prefix=/usr --bindir=/usr/bin --exec-prefix=/usr
--sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man
--sysconfdir=/etc/squid --libexecdir=/usr/lib/squid
--enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap
--enable-linux-netfilter --enable-truncate --with-pthreads --enable-epoll
--enable-time-hack --disable-follow-x-forwarded-for
--host=x86_64-pc-linux-gnu --disable-snmp --enable-ssl --enable-underscores
--enable-storeio='diskd,coss,aufs,null' --enable-async-io
why truncate? did you find it effective?
with flags:
CFLAGS=-march=nocona -O3 -pipe -fomit-frame-pointer -ffast-math
-funroll-all-loops
CXXFLAGS=${CFLAGS} -fno-enforce-eh-specs
LDFLAGS=-Wl,-O1 -Wl,-Bdirect -Wl,-hashvals -Wl,-zdynsort
I would not use that heavy optimalization...
--
Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Enter any 12-digit prime number to continue.
Re: [squid-users] low squid performance?
Dnia poniedziałek, 27 lutego 2006 10:28, Matus UHLAR - fantomas napisał:
refresh_pattern -i (.*jpg$|.*gif$|.*png$) 0 50% 28800
refresh_pattern -i (.*html$|.*htm|.*shtml|.*php) 0 20% 1440
acl mGG dstdomain .adserver.gadugadu.pl .adserver.gadu-gadu.pl
redirector_access deny !mGG
I'd use
redirector_access allow mGG
redirector_access deny all
but it shouldn't cause big diff.
without redirector i have that same processor usage.
squid compiled with options:
aragorn ~ # squid -v
Squid Cache: Version 2.5.STABLE12
configure options: --prefix=/usr --bindir=/usr/bin --exec-prefix=/usr
--sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man
--sysconfdir=/etc/squid --libexecdir=/usr/lib/squid
--enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap
--enable-linux-netfilter --enable-truncate --with-pthreads --enable-epoll
--enable-time-hack --disable-follow-x-forwarded-for
--host=x86_64-pc-linux-gnu --disable-snmp --enable-ssl
--enable-underscores --enable-storeio='diskd,coss,aufs,null'
--enable-async-io
why truncate? did you find it effective?
configure script tells that truncate gives better performance.
but i tested without, and still that same usage.
with flags:
CFLAGS=-march=nocona -O3 -pipe -fomit-frame-pointer -ffast-math
-funroll-all-loops
CXXFLAGS=${CFLAGS} -fno-enforce-eh-specs
LDFLAGS=-Wl,-O1 -Wl,-Bdirect -Wl,-hashvals -Wl,-zdynsort
I would not use that heavy optimalization...
I tested -march=nocona -O2 -pipe only, and that same result.
-- cut --
sample_start_time = 1141056028.712087 (Mon, 27 Feb 2006 16:00:28 GMT)
sample_end_time = 1141056328.723774 (Mon, 27 Feb 2006 16:05:28 GMT)
client_http.requests = 124.701809/sec
client_http.hits = 49.184751/sec
client_http.errors = 0.00/sec
client_http.kbytes_in = 128.511660/sec
client_http.kbytes_out = 1141.532196/sec
client_http.all_median_svc_time = 0.036222 seconds
client_http.miss_median_svc_time = 0.092188 seconds
client_http.nm_median_svc_time = 0.003785 seconds
client_http.nh_median_svc_time = 0.150482 seconds
client_http.hit_median_svc_time = 0.004626 seconds
server.all.requests = 77.620310/sec
server.all.errors = 0.00/sec
server.all.kbytes_in = 968.955586/sec
server.all.kbytes_out = 103.229312/sec
server.http.requests = 77.620310/sec
server.http.errors = 0.00/sec
server.http.kbytes_in = 968.955586/sec
server.http.kbytes_out = 103.229312/sec
server.ftp.requests = 0.00/sec
server.ftp.errors = 0.00/sec
server.ftp.kbytes_in = 0.00/sec
server.ftp.kbytes_out = 0.00/sec
server.other.requests = 0.00/sec
server.other.errors = 0.00/sec
server.other.kbytes_in = 0.00/sec
server.other.kbytes_out = 0.00/sec
icp.pkts_sent = 0.00/sec
icp.pkts_recv = 0.00/sec
icp.queries_sent = 0.00/sec
icp.replies_sent = 0.00/sec
icp.queries_recv = 0.00/sec
icp.replies_recv = 0.00/sec
icp.replies_queued = 0.00/sec
icp.query_timeouts = 0.00/sec
icp.kbytes_sent = 0.00/sec
icp.kbytes_recv = 0.00/sec
icp.q_kbytes_sent = 0.00/sec
icp.r_kbytes_sent = 0.00/sec
icp.q_kbytes_recv = 0.00/sec
icp.r_kbytes_recv = 0.00/sec
icp.query_median_svc_time = 0.00 seconds
icp.reply_median_svc_time = 0.00 seconds
dns.median_svc_time = 0.003722 seconds
unlink.requests = 0.00/sec
page_faults = 0.00/sec
select_loops = 378.011941/sec
select_fds = 976.061976/sec
average_select_fd_period = 0.000944/fd
median_select_fds = 0.996094
swap.outs = 21.409166/sec
swap.ins = 53.114597/sec
swap.files_cleaned = 0.00/sec
aborted_requests = 3.576527/sec
syscalls.selects = 420.976933/sec
syscalls.disk.opens = 54.204555/sec
syscalls.disk.closes = 108.129121/sec
syscalls.disk.reads = 64.560818/sec
syscalls.disk.writes = 3565.071117/sec
syscalls.disk.seeks = 0.00/sec
syscalls.disk.unlinks = 11.406222/sec
syscalls.sock.accepts = 74.167111/sec
syscalls.sock.sockets = 52.031306/sec
syscalls.sock.connects = 52.031306/sec
syscalls.sock.binds = 52.031306/sec
syscalls.sock.closes = 92.786385/sec
syscalls.sock.reads = 469.155057/sec
syscalls.sock.writes = 520.233067/sec
syscalls.sock.recvfroms = 7.919691/sec
syscalls.sock.sendtos = 4.109840/sec
cpu_time = 277.429338 seconds
wall_time = 300.011687 seconds
cpu_usage = 92.472844%
-- cut --
Regards,
--
Tomasz
Re: [squid-users] low squid performance?
Dnia czwartek, 23 lutego 2006 16:17, Matus UHLAR - fantomas napisał:
On 23.02 14:25, Tomasz Kolaj wrote:
Dnia czwartek, 23 lutego 2006 11:32, napisałeś:
On 22.02 23:13, Tomasz Kolaj wrote:
I observed have too low performance. On 2x 64bit Xeon 2,8GHz 2GB
DDR2, 2x WD RAPTOR Squid 2.5.STABLE12 can answer max for 120
requests/s. 115 r/s - 97-98% usage of first processor. Second is
unusable for squid :/. I have two cache_dirs (aufs). One pre disk.
Maybe you have too many ACL's?
I pasted my squid.conf in one of last posts. I have much of addresses
bloacked in file spywaredomains.txt
sorry - the thread was broken and I didn't see it. (b)lame mailers who
break threads by not using References: or at least In-Reply-To: headers...
Ok, my mistake possible:
-- cut --
aragorn ~ # cat /etc/squid/squid.conf | grep -v ^# | tr -s '\n'
http_port 82.160.43.14:3128
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY
cache_mem 612 MB
maximum_object_size 8192 KB
maximum_object_size_in_memory 8 KB
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_dir aufs /var/cache/squid/dysk1 3 32 256
cache_dir aufs /var/cache/squid/dysk2 3 32 256
cache_access_log none
cache_store_log none
mime_table /etc/squid/mime.conf
redirect_children 15
request_header_max_size 20 KB
refresh_pattern -i (.*jpg$|.*gif$|.*png$) 0 50% 28800
refresh_pattern -i (.*html$|.*htm|.*shtml|.*php) 0 20% 1440
refresh_pattern . 0 20% 4320
half_closed_clients off
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl our_networks src 82.160.43.0/24 82.160.129.0/24
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 901 # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
cache_mgr admin
http_access allow manager localhost
http_access allow manager our_networks
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl mGG dstdomain .adserver.gadugadu.pl .adserver.gadu-gadu.pl
redirector_access deny !mGG
redirector_bypass on
redirect_program /home/gg_rewrite
acl spywaredomains dstdomain src /etc/squid/spywaredomains.txt
http_access deny spywaredomains
http_access allow our_networks
http_access allow localhost
http_access deny all
http_reply_access allow all
icp_access allow all
cache_mgr [EMAIL PROTECTED]
visible_hostname w3cache.abp.pl
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
dns_testnames onet.pl wp.pl microsoft.com abp.pl
logfile_rotate 10
append_domain .abp.pl
forwarded_for off
log_icp_queries off
cachemgr_passwd [cut] all
buffered_logs on
coredump_dir /var/cache/squid
store_dir_select_algorithm least-load
-- cut --
acl spywaredomains dstdomain src /etc/squid/spywaredomains.txt
http_access deny spywaredomains
but when I remove it from config squid still generate much processor
time.
What about epool? I aplied patch for squid_2.5 for tests.
I don't think that would help you much. Maybe using external redirector
(SquidGuard?) instead of squid itself would help - it may reside on another
CPU, while squid it one-CPU-process.
External redirector? But im redirecting only few requests, (to gadu-gadu
addserver).
squid compiled with options:
aragorn ~ # squid -v
Squid Cache: Version 2.5.STABLE12
configure options: --prefix=/usr --bindir=/usr/bin --exec-prefix=/usr
--sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man
--sysconfdir=/etc/squid --libexecdir=/usr/lib/squid
--enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap
--enable-linux-netfilter --enable-truncate --with-pthreads --enable-epoll
--enable-time-hack --disable-follow-x-forwarded-for
--host=x86_64-pc-linux-gnu --disable-snmp --enable-ssl --enable-underscores
--enable-storeio='diskd,coss,aufs,null' --enable-async-io
with flags:
CFLAGS=-march=nocona -O3 -pipe -fomit-frame-pointer -ffast-math
-funroll-all-loops
CXXFLAGS=${CFLAGS} -fno-enforce-eh-specs
LDFLAGS=-Wl,-O1 -Wl,-Bdirect -Wl,-hashvals -Wl,-zdynsort
Regards,
--
Tomasz
Re: [squid-users] low squid performance?
Hello, I found in archive: http://www.squid-cache.org/mail-archive/squid-dev/200212/0119.html How is it possible to get 2833 requests/second on 2xP3 1,4GHz box? Is it true? My result is poor in compare to his result;) (my max 135 requests/second with 95% usage of processor with logging turned off). Regards, -- Tomasz
Re: [squid-users] low squid performance?
Dnia czwartek, 23 lutego 2006 01:11, napisałeś:
* High latency clients
What do you mean high latecy clients?
The majority of my customers have a network path like:
client-squid-satellite-squid-internet
many of my clients: client-[radio line {12,34,54}mbps]-squid-internet
100 requests/second put my CPU usage in the high 80s (on a 32 bit Intel
Xeon 3.00GHz).
So my result isn't so bad. But I must tune squid to maximum possible
performance.
aragorn squid # squid -v
Squid Cache: Version 2.5.STABLE12
configure options: --prefix=/usr --bindir=/usr/bin
--exec-prefix=/usr
--sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man
--sysconfdir=/etc/squid --libexecdir=/usr/lib/squid
--enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap
--enable-linux-netfilter --enable-truncate --with-pthreads
--enable-epool
Hopefully that's just a misspelling. ;o)
Why?;) I did some wrong?
I'm testing epool patch like you said;
--disable-follow-x-forwarded-for --host=x86_64-pc-linux-gnu
--disable-snmp
--disable-ssl --enable-underscores
--enable-storeio='diskd,coss,aufs,null'
--enable-async-io
ah.. async-io, mayby better will be to specify number of async-io threads?
I don't see any other likely problems (not saying there aren't any).
Is chance to do something morre with hardware? I can add more memory banks or
hard discs (for example +2 wd raptors)
Regards,
--
Tomasz Kolaj
Re: [squid-users] low squid performance?
On 22.02 23:13, Tomasz Kolaj wrote: I observed have too low performance. On 2x 64bit Xeon 2,8GHz 2GB DDR2, 2x WD RAPTOR Squid 2.5.STABLE12 can answer max for 120 requests/s. 115 r/s - 97-98% usage of first processor. Second is unusable for squid :/. I have two cache_dirs (aufs). One pre disk. Maybe you have too many ACL's? -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Eagles may soar, but weasels don't get sucked into jet engines.
Re: [squid-users] low squid performance?
Dnia czwartek, 23 lutego 2006 11:32, napisałeś: On 22.02 23:13, Tomasz Kolaj wrote: I observed have too low performance. On 2x 64bit Xeon 2,8GHz 2GB DDR2, 2x WD RAPTOR Squid 2.5.STABLE12 can answer max for 120 requests/s. 115 r/s - 97-98% usage of first processor. Second is unusable for squid :/. I have two cache_dirs (aufs). One pre disk. Maybe you have too many ACL's? I pasted my squid.conf in one of last posts. I have much of addresses bloacked in file spywaredomains.txt acl spywaredomains dstdomain src /etc/squid/spywaredomains.txt http_access deny spywaredomains but when I remove it from config squid still generate much processor time. What about epool? I aplied patch for squid_2.5 for tests. Regards, -- Tomasz Kolaj
Re: [squid-users] low squid performance?
On 23.02 14:25, Tomasz Kolaj wrote: Dnia czwartek, 23 lutego 2006 11:32, napisałeś: On 22.02 23:13, Tomasz Kolaj wrote: I observed have too low performance. On 2x 64bit Xeon 2,8GHz 2GB DDR2, 2x WD RAPTOR Squid 2.5.STABLE12 can answer max for 120 requests/s. 115 r/s - 97-98% usage of first processor. Second is unusable for squid :/. I have two cache_dirs (aufs). One pre disk. Maybe you have too many ACL's? I pasted my squid.conf in one of last posts. I have much of addresses bloacked in file spywaredomains.txt sorry - the thread was broken and I didn't see it. (b)lame mailers who break threads by not using References: or at least In-Reply-To: headers... acl spywaredomains dstdomain src /etc/squid/spywaredomains.txt http_access deny spywaredomains but when I remove it from config squid still generate much processor time. What about epool? I aplied patch for squid_2.5 for tests. I don't think that would help you much. Maybe using external redirector (SquidGuard?) instead of squid itself would help - it may reside on another CPU, while squid it one-CPU-process. -- Matus UHLAR - fantomas, [EMAIL PROTECTED] ; http://www.fantomas.sk/ Warning: I wish NOT to receive e-mail advertising to this address. Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu. Spam = (S)tupid (P)eople's (A)dvertising (M)ethod
Re: [squid-users] low squid performance?
Hopefully that's just a misspelling. ;o) Why?;) I did some wrong? I'm testing epool patch like you said; What he meant, I think, was that it's poll, not pool... therefore, --enable-epool won't do a thing.
RE: [squid-users] low squid performance?
-Original Message-
From: Tomasz Kolaj [mailto:[EMAIL PROTECTED]
Sent: Saturday, January 28, 2006 10:01 AM
To: Chris Robertson
Subject: Re: [squid-users] low squid performance?
Dnia czwartek, 23 lutego 2006 01:11, napisałeś:
* High latency clients
What do you mean high latecy clients?
The majority of my customers have a network path like:
client-squid-satellite-squid-internet
many of my clients: client-[radio line
{12,34,54}mbps]-squid-internet
100 requests/second put my CPU usage in the high 80s (on a
32 bit Intel
Xeon 3.00GHz).
So my result isn't so bad. But I must tune squid to maximum possible
performance.
With epoll, 100 Req/sec puts my CPU at 23%. It made a huge difference.
aragorn squid # squid -v
Squid Cache: Version 2.5.STABLE12
configure options: --prefix=/usr --bindir=/usr/bin
--exec-prefix=/usr
--sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man
--sysconfdir=/etc/squid --libexecdir=/usr/lib/squid
--enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap
--enable-linux-netfilter --enable-truncate --with-pthreads
--enable-epool
Hopefully that's just a misspelling. ;o)
Why?;) I did some wrong?
I'm testing epool patch like you said;
--disable-follow-x-forwarded-for --host=x86_64-pc-linux-gnu
--disable-snmp
--disable-ssl --enable-underscores
--enable-storeio='diskd,coss,aufs,null'
--enable-async-io
ah.. async-io, mayby better will be to specify number of
async-io threads?
Try anything once. :o)
I don't see any other likely problems (not saying there aren't any).
Is chance to do something morre with hardware? I can add more
memory banks or
hard discs (for example +2 wd raptors)
More memory and more spindles (drives) certainly won't hurt, but you seem to be
CPU limited. Taking care of that problem will likely net you the best
improvement.
Regards,
--
Tomasz Kolaj
Chris
Re: [squid-users] low squid performance?
Dnia czwartek, 23 lutego 2006 18:32, napisałeś: With epoll, 100 Req/sec puts my CPU at 23%. It made a huge difference. I have still that same CPU usage.. mayby I aplied not this patch? More memory and more spindles (drives) certainly won't hurt, but you seem to be CPU limited. Taking care of that problem will likely net you the best improvement. So what is your proposition? Regards, -- Tomasz Kolaj
RE: [squid-users] low squid performance?
-Original Message- From: Tomasz Kolaj [mailto:[EMAIL PROTECTED] Sent: Thursday, February 23, 2006 11:17 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] low squid performance? Dnia czwartek, 23 lutego 2006 18:32, napisałeś: With epoll, 100 Req/sec puts my CPU at 23%. It made a huge difference. I have still that same CPU usage.. mayby I aplied not this patch? More memory and more spindles (drives) certainly won't hurt, but you seem to be CPU limited. Taking care of that problem will likely net you the best improvement. So what is your proposition? Regards, -- Tomasz Kolaj Likely I wasn't clear with the steps involved with applying the epoll patch. My apologies. Some of the following steps may be redundant in your case. 1) Grab most recent Squid source, untar, ungzip 2) Grab epoll patch 3) Grab autoconf 2.13 (http://mirrors.kernel.org/gnu/autoconf/autoconf-2.13.tar.gz) 4) Ungzip, untar, configure, make, and install. 5) Grab automake 1.5 (http://mirrors.kernel.org/gnu/automake/automake-1.5.tar.gz) 6) Ungzip, untar, etc. again. 7) Patch Squid source 8) Run bootstrap.sh in Squid source dir 9) Run ./configure --help to verify --enable-epoll is an option 10) Configure, make, make install Chris
Re: [squid-users] low squid performance?
Hello, I observed have too low performance. On 2x 64bit Xeon 2,8GHz 2GB DDR2, 2x WD RAPTOR Squid 2.5.STABLE12 can answer max for 120 requests/s. 115 r/s - 97-98% usage of first processor. Second is unusable for squid :/. I have two cache_dirs (aufs). One pre disk. aragorn ~ # squid -v Squid Cache: Version 2.5.STABLE12 configure options: --prefix=/usr --bindir=/usr/bin --exec-prefix=/usr --sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man --sysconfdir=/etc/squid --libexecdir=/usr/lib/squid --enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap --enable-digest-auth-helpers=password --enable-basic-auth-helpers=SASL,PAM,getpwnam,YP,NCSA,SMB,MSNT,multi-domain-NTLM,winbind --enable-external-acl-helpers=ip_user,unix_group,wbinfo_group,winbind_group --enable-ntlm-auth-helpers=SMB,fakeauth,no_check,winbind --enable-linux-netfilter --enable-ident-lookups --enable-useragent-log --enable-cache-digests --enable-delay-pools --enable-referer-log --enable-truncate --enable-arp-acl --with-pthreads --with-large-files --enable-htcp --enable-carp --enable-poll --disable-follow-x-forwarded-for --host=x86_64-pc-linux-gnu --disable-snmp --enable-ssl --enable-underscores --enable-storeio='ufs,diskd,coss,aufs,null' --enable-async-io from config: cache_mem 512MB aragorn ~ # uname -a Linux aragorn 2.6.15-gentoo-r5 #1 SMP Thu Feb 16 02:03:43 CET 2006 x86_64 Intel(R) Xeon(TM) CPU 2.80GHz GenuineIntel GNU/Linux glibc-2.3.90.20060207 with NPTL Can I do something to improve preformance? - Make your own build and installation of SQUID; configure only those options which you need. This may help for performance too. M.
Re: [squid-users] low squid performance?
Dnia środa, 22 lutego 2006 23:18, Mark Elsen napisał: - Make your own build and installation of SQUID; configure only those options which you need. This may help for performance too. Ok, I'll rebuild squid without not-needed options. I have top squid's usage at 18:00-21:00 so I'll check changes tommorow. What performance I should expect from this hardware? -- Tomasz Kolaj
RE: [squid-users] low squid performance?
-Original Message- From: Tomasz Kolaj [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 22, 2006 1:30 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] low squid performance? Dnia środa, 22 lutego 2006 23:18, Mark Elsen napisał: - Make your own build and installation of SQUID; configure only those options which you need. This may help for performance too. Ok, I'll rebuild squid without not-needed options. I have top squid's usage at 18:00-21:00 so I'll check changes tommorow. What performance I should expect from this hardware? -- Tomasz Kolaj The answer to that question is dependant on a whole host of variables, such as ACLs used, whether it's a proxy or an accelerator, the types of clients accessing it (client latency has a dramatic effect on CPU usage), types of content retrieved, how your cache_dirs are defined, etc. Various things that can reduce Squid performance: * regex based ACLs * High latency clients * blocking cache_dir configuration (e.g. using ufs instead of aufs or diskd) * Anti-virus scanning * Slow authentication back ends If none of these issues covers your problem, you might look into experimental solutions such as the epoll patch (http://devel.squid-cache.org/projects.html#epoll). Chris
Re: [squid-users] low squid performance?
Dnia środa, 22 lutego 2006 23:57, Chris Robertson napisał: [cut] The answer to that question is dependant on a whole host of variables, such as ACLs used, whether it's a proxy or an accelerator, the types of clients accessing it (client latency has a dramatic effect on CPU usage), types of content retrieved, how your cache_dirs are defined, etc. Various things that can reduce Squid performance: * regex based ACLs acl badURL url_regex -i .wmf$ #^ remove wmf after security leaks on ms wmf file format acl mGG url_regex ^http://adserver.gadu[\-]?gadu.pl/.*$ redirector_access deny !mGG redirector_bypass on redirect_program /home/gg_rewrite #^redirector ro replece banner in popular polish comunicator acl QUERY urlpath_regex cgi-bin \? #typical patterns refresh_pattern -i (.*jpg$|.*gif$|.*png$) 0 50% 28800 refresh_pattern -i (.*html$|.*htm|.*shtml|.*php) 0 20% 1440 refresh_pattern . 0 20% 4320 * High latency clients What do you mean high latecy clients? * blocking cache_dir configuration (e.g. using ufs instead of aufs or diskd) cache_dir aufs /var/cache/squid/dysk1 3 32 256 cache_dir aufs /var/cache/squid/dysk2 3 32 256 2x wd raptor 36GB * Anti-virus scanning second processor have lot of free time, but first i must tune up squid to ~130-140 req/s * Slow authentication back ends I don't have authentication backends, ACL from IP (acces filtered by netfilter too) If none of these issues covers your problem, you might look into experimental solutions such as the epoll patch (http://devel.squid-cache.org/projects.html#epoll). I recompiled withoud several options and with patch http://devel.squid-cache.org/cgi-bin/diff2/epoll-2_5.patch?s2_5 aragorn squid # squid -v Squid Cache: Version 2.5.STABLE12 configure options: --prefix=/usr --bindir=/usr/bin --exec-prefix=/usr --sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man --sysconfdir=/etc/squid --libexecdir=/usr/lib/squid --enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap --enable-linux-netfilter --enable-truncate --with-pthreads --enable-epool --disable-follow-x-forwarded-for --host=x86_64-pc-linux-gnu --disable-snmp --disable-ssl --enable-underscores --enable-storeio='diskd,coss,aufs,null' --enable-async-io fragmenst of squid.conf: -- cut -- http_port [ip:port] hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY cache_mem 512 MB maximum_object_size 16384 KB maximum_object_size_in_memory 16 KB cache_replacement_policy heap GDSF memory_replacement_policy heap GDSF cache_dir aufs /var/cache/squid/dysk1 3 32 256 cache_dir aufs /var/cache/squid/dysk2 3 32 256 cache_access_log /var/log/squid/access.log cache_store_log none mime_table /etc/squid/mime.conf redirect_children 15 auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off request_header_max_size 20 KB refresh_pattern -i (.*jpg$|.*gif$|.*png$) 0 50% 28800 refresh_pattern -i (.*html$|.*htm|.*shtml|.*php) 0 20% 1440 refresh_pattern . 0 20% 4320 half_closed_clients off acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl administracja src 82.160.43.0/24 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 563 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT cache_mgr admin http_access allow manager localhost http_access allow manager administracja http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports acl badURL url_regex -i .wmf$ acl mGG url_regex ^http://adserver.gadu[\-]?gadu.pl/.*$ redirector_access deny !mGG redirector_bypass on redirect_program /home/gg_rewrite acl spywaredomains dstdomain src /etc/squid/spywaredomains.txt acl our_networks src 82.160.43.0/24 82.160.129.0/24 http_access deny badURL http_access deny spywaredomains http_access allow our_networks http_access allow localhost http_access deny all http_reply_access allow all icp_access allow all cache_mgr [EMAIL PROTECTED] visible_hostname w3cache.abp.pl httpd_accel_host virtual httpd_accel_port 80 httpd_accel_with_proxy on httpd_accel_uses_host_header on dns_testnames onet.pl wp.pl microsoft.com abp.pl logfile_rotate 10 append_domain .abp.pl forwarded_for off log_icp_queries off cachemgr_passwd [cut] all buffered_logs on coredump_dir /var/cache/squid
RE: [squid-users] low squid performance?
-Original Message- From: Tomasz Kolaj [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 22, 2006 2:24 PM To: squid-users@squid-cache.org Subject: Re: [squid-users] low squid performance? Dnia środa, 22 lutego 2006 23:57, Chris Robertson napisał: [cut] The answer to that question is dependant on a whole host of variables, such as ACLs used, whether it's a proxy or an accelerator, the types of clients accessing it (client latency has a dramatic effect on CPU usage), types of content retrieved, how your cache_dirs are defined, etc. Various things that can reduce Squid performance: #^ remove wmf after security leaks on ms wmf file format acl mGG url_regex ^http://adserver.gadu[\-]?gadu.pl/.*$ If I'm reading the regex right, you could change this to... acl mGG dstdomain .adserver.gadugadu.pl .adserver.gadu-gadu.pl ...and you might see a reduction in CPU usage. I'm not sure how much of one though... redirector_access deny !mGG redirector_bypass on redirect_program /home/gg_rewrite #^redirector ro replece banner in popular polish comunicator [cut] * High latency clients What do you mean high latecy clients? The majority of my customers have a network path like: client-squid-satellite-squid-internet 100 requests/second put my CPU usage in the high 80s (on a 32 bit Intel Xeon 3.00GHz). [cut] aragorn squid # squid -v Squid Cache: Version 2.5.STABLE12 configure options: --prefix=/usr --bindir=/usr/bin --exec-prefix=/usr --sbindir=/usr/sbin --localstatedir=/var --mandir=/usr/share/man --sysconfdir=/etc/squid --libexecdir=/usr/lib/squid --enable-auth=basic,digest,ntlm --enable-removal-policies=lru,heap --enable-linux-netfilter --enable-truncate --with-pthreads --enable-epool Hopefully that's just a misspelling. ;o) --disable-follow-x-forwarded-for --host=x86_64-pc-linux-gnu --disable-snmp --disable-ssl --enable-underscores --enable-storeio='diskd,coss,aufs,null' --enable-async-io fragmenst of squid.conf: -- cut -- http_port [ip:port] hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? no_cache deny QUERY cache_mem 512 MB maximum_object_size 16384 KB maximum_object_size_in_memory 16 KB cache_replacement_policy heap GDSF memory_replacement_policy heap GDSF cache_dir aufs /var/cache/squid/dysk1 3 32 256 cache_dir aufs /var/cache/squid/dysk2 3 32 256 cache_access_log /var/log/squid/access.log cache_store_log none mime_table /etc/squid/mime.conf redirect_children 15 auth_param basic children 5 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours auth_param basic casesensitive off request_header_max_size 20 KB refresh_pattern -i (.*jpg$|.*gif$|.*png$) 0 50% 28800 refresh_pattern -i (.*html$|.*htm|.*shtml|.*php) 0 20% 1440 refresh_pattern . 0 20% 4320 half_closed_clients off acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl administracja src 82.160.43.0/24 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 563 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT cache_mgr admin http_access allow manager localhost http_access allow manager administracja http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports acl badURL url_regex -i .wmf$ acl mGG url_regex ^http://adserver.gadu[\-]?gadu.pl/.*$ redirector_access deny !mGG redirector_bypass on redirect_program /home/gg_rewrite acl spywaredomains dstdomain src /etc/squid/spywaredomains.txt acl our_networks src 82.160.43.0/24 82.160.129.0/24 http_access deny badURL http_access deny spywaredomains http_access allow our_networks http_access allow localhost http_access deny all http_reply_access allow all icp_access allow all cache_mgr [EMAIL PROTECTED] visible_hostname w3cache.abp.pl httpd_accel_host virtual httpd_accel_port 80 httpd_accel_with_proxy on httpd_accel_uses_host_header on dns_testnames onet.pl wp.pl microsoft.com abp.pl logfile_rotate 10 append_domain .abp.pl forwarded_for off log_icp_queries off cachemgr_passwd [cut] all buffered_logs on coredump_dir /var/cache/squid store_dir_select_algorithm least-load -- cut -- Thanks for advice. -- Tomasz Kolaj I don't see any other likely problems (not saying there aren't any). Chris
