[squid-users] Squid slows under load
Hi All, I've been having some problems with Squid and Dansguardian for a while now and despite lots of time on Google, haven't found a solution. The problem started a week or so back when I noticed that squid was slowing. A quick look through the logs showed it was running out of file descriptors so I upped the level to take account. The server was ancient so I bought in an HP Proliant DL120 (dual Pentium 2.80Ghz G6950 CPU 4GB of RAM). At the same time, I bought in 2 x 60GB SSD drives to use as cache space with the system on a RAID 1 array with 160GB SATA II disks. On this, I installed Ubuntu server 10.04.2 LTS with Squid 2.7 (from apt) and Dansguardian 2.10.1.1. The kernel version is 2.6.32-24-server and the server authenticates via a Samba PDC (v 3.5.6) using OpenLDAP/Winbind. The Samba version on the proxy machine is v 3.4.7 as supplied from the Ubuntu repo. This however also seems to run out of steam. My first thought was that it may have been running out of RAM so I ran htop. Both CPUs were topping out at 20% and out of the 4GB of RAM, 1.3GB was used. Next I checked the load on the NIC and found that it was running on average 400kB/s, with the odd burst at 5MB/s. As the load increased, web pages were taking up to 30-45 seconds to load. I bypassed Dansguardian and went in on 3128 with no change in performance. Following the recommendations on other sites discovered via Google, I tuned and tweaked settings with no real benefit and I can't see that I changed anything to cause it to happen. The log files look fine, I have 1 file descriptors available and cachemgr shows plenty of spares. There are 50% more NTLM authenticators than are in use at any given time. The config file for Squid is shown below. I have had the number of authenticators set to 400 as I have 350 users but the number in use still peaked at around 50. If I've been a numpty and done something glaringly obvious, I'd be grateful if someone could point it out. If not, ask for info and I'll provide it. Thanks, Jools ## Squid.conf ## Start with authentication for clients auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp auth_param ntlm_param children 100 auth_param ntlm keep_alive on auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic auth_param basic children 100 auth_param basic realm Squid proxy-caching web server auth_param basic credentialsttl 2 hours ## Access Control Lists for filter bypass ## acl realtek dstdomain .realtek.com.tw acl tes dstdomain .tes.co.uk acl glogster dstdomain .glogster.com acl adobe-installer dstdomain .adobe.com # allow installs from adobe download manager acl actihealth dstdomain .actihealth.com .actihealth.net # Allow direct access for PE dept activity monitors acl spybotupdates dstdomain .safer-networking.org .spybotupdates.com # Allow updates for Spybot SD acl sims-update dstdomain .kcn.org.uk .capitaes.co.uk .capitasolus.co.uk .sims.co.uk # Allow SIMS to update itself directly acl kcc dstdomain .kenttrustweb.org.uk # Fix problem with county acl frenchconference dstdomain flashmeeting.e2bn.net acl emsonline dstdomain .emsonline.kent.gov.uk acl clamavdstdomain .db.gb.clamav.net acl ubuntudstdomain .ubuntu.com .warwick.ac.uk acl windowsupdate dstdomain windowsupdate.microsoft.com acl windowsupdate dstdomain .update.microsoft.com acl windowsupdate dstdomain download.windowsupdate.com acl windowsupdate dstdomain redir.metaservices.microsoft.com acl windowsupdate dstdomain images.metaservices.microsoft.com acl windowsupdate dstdomain c.microsoft.com acl windowsupdate dstdomain www.download.windowsupdate.com acl windowsupdate dstdomain wustat.windows.com acl windowsupdate dstdomain crl.microsoft.com acl windowsupdate dstdomain sls.microsoft.com acl windowsupdate dstdomain productactivation.one.microsoft.com acl windowsupdate dstdomain ntservicepack.microsoft.com acl windowsupdate dstdomain download.adobe.com acl comodo dstdomain download.comodo.com acl simsb2b dstdomain emsonline.kent.gov.uk acl powerman dstdomain pmstats.org acl ability dstdomain ability.com acl fulston dstdomain fulstonmanor.kent.sch.uk acl httpsproxy dstdomain .retiredsanta.com .atunnel.com .btunnel.com .ctunnel.com .dtunnel.com .ztunnel.com .partyaccount.com ## Access Control for filtered users ## acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 acl ntlm_users proxy_auth REQUIRED acl SSL_ports port 443 # https acl SSL_ports port 563 # snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl
[squid-users] NTLM Auth problem
Hi All, I have a problem with NTLM authentication on squid-2.6.STABLE21-6.el5 on CentOS 5.5. If I run /usr/bin/ntml_auth --username=jpb --domain=BGS, it returns success. Samba (v3.5.6) file sharing works as does winbind's wbinfo -, wbinfo -g, wbinfo -t so I'm fairly sure that both Samba and winbind are functioning OK. If I go to a client and try to visit a website, I get the pop up credentials box but entering the same credentials as on the ntlm_auth line above generates the following with the virtual XP being a VM and the jpb-workstation being a Linux box: [2011/02/23 22:49:05.671790, 3] libsmb/ntlmssp.c:65(debug_ntlmssp_flags) Got NTLMSSP neg_flags=0xa2088207 [2011/02/23 22:49:05.674159, 3] libsmb/ntlmssp.c:747(ntlmssp_server_auth) Got user=[bgs0001] domain=[BGS] workstation=[VIRTUAL-XP] len1=24 len2=24 [2011/02/23 22:49:05.675008, 3] utils/ntlm_auth.c:598(winbind_pw_check) Login for user [BGS]\[bgs0001]@[VIRTUAL-XP] failed due to [Invalid handle] [2011/02/23 23:03:24.838232, 3] libsmb/ntlmssp.c:65(debug_ntlmssp_flags) Got NTLMSSP neg_flags=0x00088207 [2011/02/23 23:03:24.845152, 3] libsmb/ntlmssp.c:747(ntlmssp_server_auth) Got user=[jpb] domain=[] workstation=[jpb-desktop] len1=24 len2=24 [2011/02/23 23:03:24.845972, 3] utils/ntlm_auth.c:598(winbind_pw_check) Login for user []\[jpb]@[jpb-desktop] failed due to [Invalid handle] [2011/02/23 23:03:40.780692, 3] libsmb/ntlmssp.c:65(debug_ntlmssp_flags) Got NTLMSSP neg_flags=0x00088207 [2011/02/23 23:03:40.782125, 3] libsmb/ntlmssp.c:747(ntlmssp_server_auth) Got user=[jpb] domain=[bgs] workstation=[jpb-desktop] len1=24 len2=24 [2011/02/23 23:03:40.782938, 3] utils/ntlm_auth.c:598(winbind_pw_check) Login for user [bgs]\[jpb]@[jpb-desktop] failed due to [Invalid handle] [2011/02/23 23:05:13.260874, 3] libsmb/ntlmssp.c:65(debug_ntlmssp_flags) Got NTLMSSP neg_flags=0x00088207 [2011/02/23 23:05:13.262425, 3] libsmb/ntlmssp.c:747(ntlmssp_server_auth) Got user=[jpb] domain=[] workstation=[jpb-desktop] len1=24 len2=24 [2011/02/23 23:05:13.263254, 3] utils/ntlm_auth.c:598(winbind_pw_check) Login for user []\[jpb]@[jpb-desktop] failed due to [Invalid handle] Given that using the ntlm_auth command directly succeeds, I'm unsure as to whether this a problem with Samba, Squid or the interaction between the two. I've set the permissions on the winbind privileged pipe to 750, created a group called winbindd_priv and added the squid user to that group. There are no messages relating to being unable to read from the pipe. There are other people that have had the same problem but nothing I've looked at has solved it yet. Has anyone else been here? Thanks. Julian
[squid-users] Bypassing proxy authentication
Hi all, i have a squid proxy server (v2.6.STABLE21-3.el5) running on CentOS 5.4. It's set up for NTLM authentication for use with Windows XP and it works perfectly.However, I have a piece of software that needs to contact a stats site and I've tried running proxycfg - p 172.20.0.5:8002 and have also added an acl called stats and set an http_access allow rule for the acl but it still doesn't contact the site. Ohter sites that we've had problems with have identical ACLs and rules and these work so I know the syntax is correct. The software manufacturers only know Windows proxy servers and don't seem to be able to help much. Is there any way I can bypass the authentication or is this implied by the access rule. Thanks, Julian PB
[squid-users] [SOLVED] [squid-users] Bypassing proxy authentication
Julian Pilfold-Bagwell wrote: Amos Jeffries wrote: Julian Pilfold-Bagwell wrote: Hi all, i have a squid proxy server (v2.6.STABLE21-3.el5) running on CentOS 5.4. It's set up for NTLM authentication for use with Windows XP and it works perfectly.However, I have a piece of software that needs to contact a stats site and I've tried running proxycfg - p 172.20.0.5:8002 and have also added an acl called stats and set an http_access allow rule for the acl but it still doesn't contact the site. Ohter sites that we've had problems with have identical ACLs and rules and these work so I know the syntax is correct. The software manufacturers only know Windows proxy servers and don't seem to be able to help much. Is there any way I can bypass the authentication or is this implied by the access rule. Sounds like you are almost there. Any http_access lines above the first line which tests for auth will permit/deny access without needing auth themselves. Amos Stunning service :) Thanks very much Amos, much appreciated, all up and running as required. All the best, Julian
[squid-users] Slow connection through proxy
Hi All, I have a problem with my proxy and Windows clients on certain ip ranges on my network. I've just upgraded my network from a single LDAP/Samba server running on Mandriva 2007 to a dual redundant setup with DNS, NTP and LDAP master/slave on two servers with a seperate PDC and BDC pair authenticating and providing file shares. Authentication on the network for users is fast as lightning. On the old network I had a Mandriva 2007 box with Squid proxying and NTLM auth and this machine has been moved to the new setup. Clients are spread across three IP ranges 172.20.0., 172.20.1. and 172.20.2. with the 0 range being assigned static IPs and the one and two ranges collecting an IP from DHCPD. If I connect a client to the network, it obtains an address from the DHCP server along with DNS, gateway and WINS server settings but the connection via Squid is slow e.g. 30-120 seconds to obtain a page. If I take the settings from ipconfig and enter them manually but with an IP in the 172.20.0 range, it works perfectly with pages appearing withing 1-2 seconds. nslookup returns IP's within a second on the proxy and clients and su'ing to a user account on the proxy takes a split second, suggesting that nss and pam_smb are authenticating OK. On the old network, the proxy worked fine across al three IP ranges, on the new it behaves as above. Is there anywhere I should be looking in particular for clues to this one. I'll be out of the office until Monday but I'll check the mail as soon as I can for a reply. Many thanks, Julian PB
Re: [squid-users] Squid and Windows Update
Henrik Nordstrom wrote: tor 2007-06-21 klockan 14:22 +0100 skrev Julian Pilfold-Bagwell: If I am to guess you might need to allow access to the windows update servers without using authentication. Is it possible to do that while retaining authentication for users? Yes. Just allow access to the windows update servers before where you normally require authentication. Regards Henrik Hi again, Does the first acl line: acl winupdate dstdomain .microsoft.com .windowsupdate.com not do this? I put the always_direct rule in before the mynetwork rule but it doesn't seem to do the trick. Thanks, Jools
Re: [squid-users] Squid and Windows Update - SOLVED!!
Hi Henrik, It's cured. You were right about allowing access to winupdate. The confusing aspect is that some time back, we had to wrestle for a day to get it working after Windows updated itself. It turned out that you had to use the always_direct directive to get it work as it would crash out otherwise. Don't know what Microsoft have done to Windows Update but it now has to go back to http_allow. Thanks again, much appreciated, All the best, Julian Pilfold-Bagwell
[squid-users] Squid and Windows Update
Hi All, I have an NTLM authenticated squid proxy and an trying to get to Windows Update. Up until about 3 weeks ago it worked OK but then stopped and I haven't been able to get it going since. I have microsoft.com and windowsupdate.com in an always_direct acl and have used proxycfg to set the proxy up on the windows boxes. I've also ticked http 1.1 connection on proxy in IE6's options. I've spent hours on Google without finding any solution. Could someone have a look through the acls below to see if I've missed something please. Cheers, Jools PS: Below is a snap from the proxy log showing what's happening when I try to connect. Thanks. # Log Output 1182427844.513 RELEASE -1 62992ED631E0F39DDA8C8DC2F898F266 407 1182427844 0 1182427844 text/html 1325/1325 GET http://go.microsoft.com/fwlink/? 1182427844.520 RELEASE -1 2E6A5C7F93EEE6901CCCEE0DEB5A2229 407 1182427844 0 1182427844 text/html 1325/1325 GET http://go.microsoft.com/fwlink/? 1182427844.533 RELEASE -1 DEE0F5C0483083C6578A92A5A262DBA8 407 1182427844 0 1182427844 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427844.868 RELEASE -1 A8ABED5E2C14C5B1E9D0C071634A6A5F 407 1182427844 0 1182427844 text/html 1325/1325 GET http://go.microsoft.com/fwlink/? 1182427844.898 RELEASE -1 8A2AF11EB29DC53BECCE375C51ED2564 407 1182427844 0 1182427844 text/html 1325/1325 GET http://go.microsoft.com/fwlink/? 1182427845.371 RELEASE -1 E376783F93B586292C10EB17CEED8C0D 302 1182427844-1 1182427784 text/html 135/135 GET http://go.microsoft.com/fwlink/? 1182427845.395 RELEASE -1 DB56627F467C065BB2717F8C4807EE04 302 1182427844-1 1182427784 text/html 135/135 GET http://go.microsoft.com/fwlink/? 1182427845.959 RELEASE -1 FC48317C07A19CD1D257DF7931B8CF91 407 1182427845 0 1182427845 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427845.965 RELEASE -1 9FDB6B061BB1A01FD5774EDCF57BFE72 407 1182427845 0 1182427845 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427845.968 RELEASE -1 24E1583A4D3FE04F9CC5D92791D8234F 407 1182427845 0 1182427845 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427846.017 RELEASE -1 307158AE09CFED627438DB4C97BB6DE7 407 1182427846 0 1182427846 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427848.314 RELEASE -1 B54B1B79B60C0A9EE18BCC5F376CCCF0 407 1182427848 0 1182427848 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427848.335 RELEASE -1 106150D23930001055AB50F33462E587 407 1182427848 0 1182427848 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427848.385 RELEASE -1 8F2EB8EA5C13E1999AA8BBA44C8DE2CC 407 1182427848 0 1182427848 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427848.608 RELEASE -1 9AAF6E2DA487093383A0DD59ADB264B4 407 1182427848 0 1182427848 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427848.628 RELEASE -1 552B7EA2E74614B8A4E9E82E193FC296 407 1182427848 0 1182427848 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427848.631 RELEASE -1 B2701012D1DE2296A7678125A6841581 407 1182427848 0 1182427848 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427848.681 RELEASE -1 6194E73C33414591F76E8645DD78AF71 407 1182427848 0 1182427848 text/html 1301/1301 CONNECT update.microsoft.com:443 1182427848.928 RELEASE -1 2B64CB519E1123FE9772D9D2FD6B9D23 407 1182427848 0 1182427848 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427848.959 RELEASE -1 BAB09BA63C9B037455216ED743BDE755 407 1182427848 0 1182427848 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427849.014 RELEASE -1 964028CC20022B536F59877D37745174 407 1182427849 0 1182427849 text/html 1463/1463 POST http://stats.update.microsoft.com/ReportingWebService/ReportingWebService.asmx 1182427850.033 RELEASE -1 36FDA330BD08904D927FB76ABD56B1D1 407 1182427850 0 1182427850 text/html 1292/1292 CONNECT urs.microsoft.com:443 1182427850.075 RELEASE -1 B5335E465AA32ED4259749CBB2AC4236 407 1182427850 0 1182427850 text/html 1292/1292 CONNECT urs.microsoft.com:443 1182427850.127 RELEASE -1 0D4261BD99331073CAE9F2FA94E0EE61 407 1182427850 0 1182427850 text/html 1292/1292 CONNECT urs.microsoft.com:443 1182427850.130 RELEASE -1 32CCE2EA2FB00E6CA57DF5D5F2CC6799 407 1182427850 0 1182427850
