Re: [squid-users] Squid slows under load
Hi again, Went into work the next day and carried on only to find that one of the SSDs was spewing CRC failures into the log. I don;t recall seeing any of these in the previous days and the logs didn;t contain any entries prior to my email so I assume that was a new problem. I've pulled the SSDs and replaced then with 160GB SATA II disks. I then ran curl-loader with 150 client load and the resulting 5 minute stats dump is posted below. sample_start_time = 1299342747.28318 (Sat, 05 Mar 2011 16:32:27 GMT) sample_end_time = 1299343047.45842 (Sat, 05 Mar 2011 16:37:27 GMT) client_http.requests = 115.309931/sec client_http.hits = 0.00/sec client_http.errors = 0.00/sec client_http.kbytes_in = 25.411849/sec client_http.kbytes_out = 195.575242/sec client_http.all_median_svc_time = 0.012346 seconds client_http.miss_median_svc_time = 0.013086 seconds client_http.nm_median_svc_time = 0.00 seconds client_http.nh_median_svc_time = 0.00 seconds client_http.hit_median_svc_time = 0.00 seconds server.all.requests = 0.299982/sec server.all.errors = 0.00/sec server.all.kbytes_in = 1.573241/sec server.all.kbytes_out = 0.14/sec server.http.requests = 0.266651/sec server.http.errors = 0.00/sec server.http.kbytes_in = 1.103269/sec server.http.kbytes_out = 0.253319/sec server.ftp.requests = 0.00/sec server.ftp.errors = 0.00/sec server.ftp.kbytes_in = 0.00/sec server.ftp.kbytes_out = 0.00/sec server.other.requests = 0.01/sec server.other.errors = 0.00/sec server.other.kbytes_in = 0.469973/sec server.other.kbytes_out = 0.076662/sec icp.pkts_sent = 0.00/sec icp.pkts_recv = 0.00/sec icp.queries_sent = 0.00/sec icp.replies_sent = 0.00/sec icp.queries_recv = 0.00/sec icp.replies_recv = 0.00/sec icp.replies_queued = 0.00/sec icp.query_timeouts = 0.00/sec icp.kbytes_sent = 0.00/sec icp.kbytes_recv = 0.00/sec icp.q_kbytes_sent = 0.00/sec icp.r_kbytes_sent = 0.00/sec icp.q_kbytes_recv = 0.00/sec icp.r_kbytes_recv = 0.00/sec icp.query_median_svc_time = 0.00 seconds icp.reply_median_svc_time = 0.00 seconds dns.median_svc_time = 4.177065 seconds unlink.requests = 0.00/sec page_faults = 0.00/sec select_loops = 202.268185/sec select_fds = 498.770865/sec average_select_fd_period = 0.002005/fd median_select_fds = 0.00 swap.outs = 0.063330/sec swap.ins = 0.00/sec swap.files_cleaned = 0.00/sec aborted_requests = 0.049997/sec syscalls.polls = 202.268185/sec syscalls.disk.opens = 0.063330/sec syscalls.disk.closes = 0.063330/sec syscalls.disk.reads = 0.00/sec syscalls.disk.writes = 0.406643/sec syscalls.disk.seeks = 0.00/sec syscalls.disk.unlinks = 0.00/sec syscalls.sock.accepts = 220.630446/sec syscalls.sock.sockets = 99.554185/sec syscalls.sock.connects = 0.216654/sec syscalls.sock.binds = 99.554185/sec syscalls.sock.closes = 209.474431/sec syscalls.sock.reads = 147.911360/sec syscalls.sock.writes = 242.719155/sec syscalls.sock.recvfroms = 2.516520/sec syscalls.sock.sendtos = 0.079995/sec cpu_time = 8.35 seconds wall_time = 300.017524 seconds cpu_usage = 2.783171% Whether the SSD replacement will give a real world cure will be seen Monday. Thanks again, Julian On 03/03/11 17:59, Pieter De Wit wrote: Hi Julian, The one stat that I can't see here is disk access. I know you said that you have SSD's, but what is the disk stats for your logging volume and the squid volume ? If you totally bypass the proxy, does it improve ? (could be that the squid server is getting shaped ?) Cheers, Pieter On 4/03/2011 06:46, Julian Pilfold-Bagwell wrote: Hi All, I've been having some problems with Squid and Dansguardian for a while now and despite lots of time on Google, haven't found a solution. The problem started a week or so back when I noticed that squid was slowing. A quick look through the logs showed it was running out of file descriptors so I upped the level to take account. The server was ancient so I bought in an HP Proliant DL120 (dual Pentium 2.80Ghz G6950 CPU 4GB of RAM). At the same time, I bought in 2 x 60GB SSD drives to use as cache space with the system on a RAID 1 array with 160GB SATA II disks. On this, I installed Ubuntu server 10.04.2 LTS with Squid 2.7 (from apt) and Dansguardian 2.10.1.1. The kernel version is 2.6.32-24-server and the server authenticates via a Samba PDC (v 3.5.6) using OpenLDAP/Winbind. The Samba version on the proxy machine is v 3.4.7 as supplied from the Ubuntu repo. This however also seems to run out of steam. My first thought was that it may have been running out of RAM so I ran htop. Both CPUs were topping out at 20% and out of the 4GB of RAM, 1.3GB was used. Next I checked the load on the NIC and found that it was running on average 400kB/s, with the odd burst at 5MB/s. As the load increased, web pages were taking up to 30-45 seconds to load. I bypassed Dansguardian and went in on 3128 with no change in
[squid-users] Squid slows under load
Hi All, I've been having some problems with Squid and Dansguardian for a while now and despite lots of time on Google, haven't found a solution. The problem started a week or so back when I noticed that squid was slowing. A quick look through the logs showed it was running out of file descriptors so I upped the level to take account. The server was ancient so I bought in an HP Proliant DL120 (dual Pentium 2.80Ghz G6950 CPU 4GB of RAM). At the same time, I bought in 2 x 60GB SSD drives to use as cache space with the system on a RAID 1 array with 160GB SATA II disks. On this, I installed Ubuntu server 10.04.2 LTS with Squid 2.7 (from apt) and Dansguardian 2.10.1.1. The kernel version is 2.6.32-24-server and the server authenticates via a Samba PDC (v 3.5.6) using OpenLDAP/Winbind. The Samba version on the proxy machine is v 3.4.7 as supplied from the Ubuntu repo. This however also seems to run out of steam. My first thought was that it may have been running out of RAM so I ran htop. Both CPUs were topping out at 20% and out of the 4GB of RAM, 1.3GB was used. Next I checked the load on the NIC and found that it was running on average 400kB/s, with the odd burst at 5MB/s. As the load increased, web pages were taking up to 30-45 seconds to load. I bypassed Dansguardian and went in on 3128 with no change in performance. Following the recommendations on other sites discovered via Google, I tuned and tweaked settings with no real benefit and I can't see that I changed anything to cause it to happen. The log files look fine, I have 1 file descriptors available and cachemgr shows plenty of spares. There are 50% more NTLM authenticators than are in use at any given time. The config file for Squid is shown below. I have had the number of authenticators set to 400 as I have 350 users but the number in use still peaked at around 50. If I've been a numpty and done something glaringly obvious, I'd be grateful if someone could point it out. If not, ask for info and I'll provide it. Thanks, Jools ## Squid.conf ## Start with authentication for clients auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp auth_param ntlm_param children 100 auth_param ntlm keep_alive on auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic auth_param basic children 100 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours ## Access Control Lists for filter bypass ## acl realtek dstdomain .realtek.com.tw acl tes dstdomain .tes.co.uk acl glogster dstdomain .glogster.com acl adobe-installer dstdomain .adobe.com # allow installs from adobe download manager acl actihealth dstdomain .actihealth.com .actihealth.net # Allow direct access for PE dept activity monitors acl spybotupdates dstdomain .safer-networking.org .spybotupdates.com # Allow updates for Spybot SD acl sims-update dstdomain .kcn.org.uk .capitaes.co.uk .capitasolus.co.uk .sims.co.uk # Allow SIMS to update itself directly acl kcc dstdomain .kenttrustweb.org.uk # Fix problem with county acl frenchconference dstdomain flashmeeting.e2bn.net acl emsonline dstdomain .emsonline.kent.gov.uk acl clamavdstdomain .db.gb.clamav.net acl ubuntudstdomain .ubuntu.com .warwick.ac.uk acl windowsupdate dstdomain windowsupdate.microsoft.com acl windowsupdate dstdomain .update.microsoft.com acl windowsupdate dstdomain download.windowsupdate.com acl windowsupdate dstdomain redir.metaservices.microsoft.com acl windowsupdate dstdomain images.metaservices.microsoft.com acl windowsupdate dstdomain c.microsoft.com acl windowsupdate dstdomain www.download.windowsupdate.com acl windowsupdate dstdomain wustat.windows.com acl windowsupdate dstdomain crl.microsoft.com acl windowsupdate dstdomain sls.microsoft.com acl windowsupdate dstdomain productactivation.one.microsoft.com acl windowsupdate dstdomain ntservicepack.microsoft.com acl windowsupdate dstdomain download.adobe.com acl comodo dstdomain download.comodo.com acl simsb2b dstdomain emsonline.kent.gov.uk acl powerman dstdomain pmstats.org acl ability dstdomain ability.com acl fulston dstdomain fulstonmanor.kent.sch.uk acl httpsproxy dstdomain .retiredsanta.com .atunnel.com .btunnel.com .ctunnel.com .dtunnel.com .ztunnel.com .partyaccount.com ## Access Control for filtered users ## acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 acl ntlm_users proxy_auth REQUIRED acl SSL_ports port 443 # https acl SSL_ports port 563 # snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl
Re: [squid-users] Squid slows under load
Hi Julian, The one stat that I can't see here is disk access. I know you said that you have SSD's, but what is the disk stats for your logging volume and the squid volume ? If you totally bypass the proxy, does it improve ? (could be that the squid server is getting shaped ?) Cheers, Pieter On 4/03/2011 06:46, Julian Pilfold-Bagwell wrote: Hi All, I've been having some problems with Squid and Dansguardian for a while now and despite lots of time on Google, haven't found a solution. The problem started a week or so back when I noticed that squid was slowing. A quick look through the logs showed it was running out of file descriptors so I upped the level to take account. The server was ancient so I bought in an HP Proliant DL120 (dual Pentium 2.80Ghz G6950 CPU 4GB of RAM). At the same time, I bought in 2 x 60GB SSD drives to use as cache space with the system on a RAID 1 array with 160GB SATA II disks. On this, I installed Ubuntu server 10.04.2 LTS with Squid 2.7 (from apt) and Dansguardian 2.10.1.1. The kernel version is 2.6.32-24-server and the server authenticates via a Samba PDC (v 3.5.6) using OpenLDAP/Winbind. The Samba version on the proxy machine is v 3.4.7 as supplied from the Ubuntu repo. This however also seems to run out of steam. My first thought was that it may have been running out of RAM so I ran htop. Both CPUs were topping out at 20% and out of the 4GB of RAM, 1.3GB was used. Next I checked the load on the NIC and found that it was running on average 400kB/s, with the odd burst at 5MB/s. As the load increased, web pages were taking up to 30-45 seconds to load. I bypassed Dansguardian and went in on 3128 with no change in performance. Following the recommendations on other sites discovered via Google, I tuned and tweaked settings with no real benefit and I can't see that I changed anything to cause it to happen. The log files look fine, I have 1 file descriptors available and cachemgr shows plenty of spares. There are 50% more NTLM authenticators than are in use at any given time. The config file for Squid is shown below. I have had the number of authenticators set to 400 as I have 350 users but the number in use still peaked at around 50. If I've been a numpty and done something glaringly obvious, I'd be grateful if someone could point it out. If not, ask for info and I'll provide it. Thanks, Jools ## Squid.conf ## Start with authentication for clients auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp auth_param ntlm_param children 100 auth_param ntlm keep_alive on auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic auth_param basic children 100 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours ## Access Control Lists for filter bypass ## acl realtek dstdomain .realtek.com.tw acl tes dstdomain .tes.co.uk acl glogster dstdomain .glogster.com acl adobe-installer dstdomain .adobe.com # allow installs from adobe download manager acl actihealth dstdomain .actihealth.com .actihealth.net # Allow direct access for PE dept activity monitors acl spybotupdates dstdomain .safer-networking.org .spybotupdates.com # Allow updates for Spybot SD acl sims-update dstdomain .kcn.org.uk .capitaes.co.uk .capitasolus.co.uk .sims.co.uk # Allow SIMS to update itself directly acl kcc dstdomain .kenttrustweb.org.uk # Fix problem with county acl frenchconference dstdomain flashmeeting.e2bn.net acl emsonline dstdomain .emsonline.kent.gov.uk acl clamavdstdomain .db.gb.clamav.net acl ubuntudstdomain .ubuntu.com .warwick.ac.uk acl windowsupdate dstdomain windowsupdate.microsoft.com acl windowsupdate dstdomain .update.microsoft.com acl windowsupdate dstdomain download.windowsupdate.com acl windowsupdate dstdomain redir.metaservices.microsoft.com acl windowsupdate dstdomain images.metaservices.microsoft.com acl windowsupdate dstdomain c.microsoft.com acl windowsupdate dstdomain www.download.windowsupdate.com acl windowsupdate dstdomain wustat.windows.com acl windowsupdate dstdomain crl.microsoft.com acl windowsupdate dstdomain sls.microsoft.com acl windowsupdate dstdomain productactivation.one.microsoft.com acl windowsupdate dstdomain ntservicepack.microsoft.com acl windowsupdate dstdomain download.adobe.com acl comodo dstdomain download.comodo.com acl simsb2b dstdomain emsonline.kent.gov.uk acl powerman dstdomain pmstats.org acl ability dstdomain ability.com acl fulston dstdomain fulstonmanor.kent.sch.uk acl httpsproxy dstdomain .retiredsanta.com .atunnel.com .btunnel.com .ctunnel.com .dtunnel.com .ztunnel.com .partyaccount.com ## Access Control for filtered users ## acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 acl ntlm_users proxy_auth REQUIRED acl SSL_ports port 443 # https acl SSL_ports port 563
