[squid-users] proxy become very slow during peak time
Hi there, I am running squid 2.5 on freebsd 7, and my squid box respond very slow during peak hours. my squid machine have twin dual core processors, 4 ram and following hdds. Filesystem SizeUsed Avail Capacity Mounted on /dev/da0s1a9.7G241M8.7G 3%/ devfs 1.0K1.0K 0B 100%/dev /dev/da0s1f 73G 35G 32G52%/cache1 /dev/da0s1g 73G2.0G 65G 3%/cache2 /dev/da0s1e 39G2.5G 33G 7%/usr /dev/da0s1d 58G6.4G 47G12%/var below are the status and settings i have done. i need further guidance to improve the box. last pid: 50046; load averages: 1.02, 1.07, 1.02 up 7+20:35:29 15:21:42 26 processes: 2 running, 24 sleeping CPU states: 25.4% user, 0.0% nice, 1.3% system, 0.8% interrupt, 72.5% idle Mem: 378M Active, 1327M Inact, 192M Wired, 98M Cache, 112M Buf, 3708K Free Swap: 4096M Total, 20K Used, 4096M Free PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND 49819 sbt1 1050 360M 351M CPU3 3 92:43 98.14% squid 487 root1 960 4372K 2052K select 0 57:00 3.47% natd 646 root1 960 16032K 12192K select 3 54:28 0.00% snmpd 49821 sbt1 -40 3652K 1048K msgrcv 0 0:13 0.00% diskd 49822 sbt1 -40 3652K 1048K msgrcv 0 0:10 0.00% diskd 49864 root1 960 3488K 1536K CPU2 1 0:04 0.00% top 562 root1 960 3156K 1008K select 0 0:04 0.00% syslogd 717 root1 80 3184K 1048K nanslp 0 0:02 0.00% cron 49631 x-man 1 960 8384K 2792K select 0 0:01 0.00% sshd 49635 root1 200 5476K 2360K pause 0 0:00 0.00% csh 49628 root1 40 8384K 2776K sbwait 1 0:00 0.00% sshd 710 root1 960 5616K 2172K select 1 0:00 0.00% sshd 49634 x-man 1 80 3592K 1300K wait 1 0:00 0.00% su 49820 sbt1 -80 1352K 496K piperd 3 0:00 0.00% unlinkd 49633 x-man 1 80 3456K 1280K wait 3 0:00 0.00% sh 765 root1 50 3156K 872K ttyin 1 0:00 0.00% getty 766 root1 50 3156K 872K ttyin 2 0:00 0.00% getty 767 root1 50 3156K 872K ttyin 2 0:00 0.00% getty 769 root1 50 3156K 872K ttyin 3 0:00 0.00% getty 771 root1 50 3156K 872K ttyin 1 0:00 0.00% getty 770 root1 50 3156K 872K ttyin 0 0:00 0.00% getty 768 root1 50 3156K 872K ttyin 3 0:00 0.00% getty 772 root1 50 3156K 872K ttyin 1 0:00 0.00% getty 47303 root1 80 8080K 3560K wait 1 0:00 0.00% squid 426 root1 960 1888K 420K select 0 0:00 0.00% devd 146 root1 200 1356K 668K pause 0 0:00 0.00% adjkerntz pxy# iostat tty da0pass0 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 126 12.79 5 0.06 0.00 0 0.00 4 0 1 0 95 pxy# vmstat procs memory pagedisks faults cpu r b w avmfre flt re pi pofr sr da0 pa0 in sy cs us sy id 1 3 0 458044 10326812 0 0 030 5 0 0 273 1721 2553 4 1 95 pxy# netstat -am 1376/1414/2790 mbufs in use (current/cache/total) 1214/1372/2586/25600 mbuf clusters in use (current/cache/total/max) 1214/577 mbuf+clusters out of packet secondary zone in use (current/cache) 147/715/862/12800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 3360K/5957K/9317K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/7/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines "netstat -an | grep "TIME_WAIT" | more " command 17 scroll pages of crt. some lines from squid.conf cache_mem 256 MB cache_replacement_policy heap LFUDA memory_replacement_policy heap GDSF cache_swap_low 80 cache_swap_high 90 cache_dir diskd /cache2 6 16 256 Q1=72 Q2=64 cache_dir diskd /cache1 6 16 256 Q1=72 Q2=64 cache_log /var/log/squid25/cache.log cache_access_log /var/log/squid25/access.log cache_store_log none half_closed_clients off maximum_object_size 1024 KB pxy# sysctl -a | grep maxproc kern.maxproc: 6164 kern.maxprocperuid: 5547 kern.ipc.somaxconn: 1024 kern.maxfiles: 12328 kern.maxfilesperproc: 11095 net.inet.ip.portrange.randomtime: 45 net.inet.ip.portrange.randomcps: 10 net.inet.ip.portrange.randomized: 1 net.inet.ip.portrange.reservedlow: 0 net.i
Re: [squid-users] How to setup squid proxy to run in fail-over mode
Thanks to all for replies. Sorry i didn't mentioned the plateform I am using to run squid on which is freebsd 7. I have visited the linux-ha site, where it says the software is supported for freebsd too but their is no distribution for freebsd, so can u people tell me which distribution i can use for feebsd 7? Thanks & Regards, A Sami On Mon, Jun 15, 2009 at 4:07 PM, Muhammad Sharfuddin wrote: > just a question > >>2. Use an HA solution such as Ultramonkey3. Here you could do >>Active-Active. > Why Ultramonkey3.. why not HA from http://www.linux-ha.org/ > > -Sharfuddin > > A PC is like a aircondition. If you open Windows it just don't funktion > properly anymore > > On Mon, 2009-06-15 at 12:12 +0200, Luis Daniel Lucio Quiroz wrote: >> There are 2 ways as far as I know to do this possible: >> >> 1. Use de WPAD protocol: lets say PROXY squid1; PROXY squid2 (this is fail >> over) >> 2. Use an HA solution such as Ultramonkey3. Here you could do Active-Active. >> >> Kind regards, >> >> LD >> Le lundi 15 juin 2009 11:09:28, Sagar Navalkar a écrit : >> > Hey Remy, >> > >> > The DNS server does not determine which server is down, however If It is >> > unable to resolve the 1st entry, it will automatically go down to the 2nd >> > entry. >> > >> > Regards, >> > >> > Sagar Navalkar >> > Team Leader >> > >> > >> > -Original Message- >> > From: Mario Remy Almeida [mailto:malme...@isaaviation.ae] >> > Sent: Monday, June 15, 2009 1:36 PM >> > To: Sagar Navalkar >> > Cc: squid-users@squid-cache.org; 'abdul sami' >> > Subject: RE: [squid-users] How to setup squid proxy to run in fail-over >> > mode >> > >> > Hi Sagar, >> > >> > Just a Question? >> > >> > How can a DNS server determine that the primary server is down and it >> > should resolve the secondary server IP? >> > >> > //Remy >> > >> > On Mon, 2009-06-15 at 11:21 +0530, Sagar Navalkar wrote: >> > > Hi Abdul, >> > > >> > > Please try to enter 2 different IPs in the DNS >> > > >> > > 10.xxx.yyy.zz1 (proxyA) as primary (proxyA-Name should be same on both >> > > the servers.) >> > > 10.xxx.yyy.zz2 (proxyA) as secondary. >> > > >> > > Start squid services on both the servers (Primary & Secondary) >> > > >> > > If Primary server fails, the DNS will resolve secondary IP for proxyA & >> > >> > the >> > >> > > squid on second server will kick in automatically.. >> > > >> > > Hope am able to explain it properly. >> > > >> > > Regards, >> > > >> > > Sagar Navalkar >> > > >> > > >> > > -Original Message- >> > > From: abdul sami [mailto:sami.me...@gmail.com] >> > > Sent: Monday, June 15, 2009 11:17 AM >> > > To: squid-users@squid-cache.org >> > > Subject: [squid-users] How to setup squid proxy to run in fail-over mode >> > > >> > > Dear all, >> > > >> > > Now that i have setup a proxy server, as a next step i want to run it >> > > in fail-over high availability mode, so that if one proxy is down due >> > > to any reason, second proxy should automatically be up and start >> > > serving requests. >> > > >> > > any help in shape of articles/steps would be highly appreciated. >> > > >> > > Thanks and regards, >> > > >> > > A Sami >> > >> > --- >> >- -- >> > Disclaimer and Confidentiality >> > >> > >> > This material has been checked for computer viruses and although none has >> > been found, we cannot guarantee that it is completely free from such >> > problems >> > and do not accept any liability for loss or damage which may be caused. >> > Please therefore check any attachments for viruses before using them on >> > your >> > own equipment. If you do find a computer virus please inform us >> > immediately so that we may take appropriate action. This communication is >> > intended solely >> > for the addressee and is confidential. If you are not the intended >> > recipient, >> > any disclosure, copying, distribution or any action taken or omitted to be >> > taken in reliance on it, is prohibited and may be unlawful. The views >> > expressed in this message are those of the individual sender, and may not >> > necessarily be that of ISA. >> > >
[squid-users] How to setup squid proxy to run in fail-over mode
Dear all, Now that i have setup a proxy server, as a next step i want to run it in fail-over high availability mode, so that if one proxy is down due to any reason, second proxy should automatically be up and start serving requests. any help in shape of articles/steps would be highly appreciated. Thanks and regards, A Sami
Re: [squid-users] Transparent proxy with HTTPS on freebsd
Deal All, So champs now the interesting part starts. ok A few days ago we had proxy configured in the following way. DR Site \ \ int: bge0 intt: bge1 internal net -> lan switch -> \Squid on BSD -> firewall -> public net IP=X \ IP=Y \ Branches 1. Above diagram shows that our internal net & and DR site is connected to squid on interface bge0 and uses transparent proxy whereas branches come to bge1 and uses manual proxy to get access to internet. 2. in above configuration http and https was working perfectly fine. after that in our company major changes were made in network and in result our proxy working scenario also changed as below. DR site | int: bge0 | int: bge1 internal net -> lan switch -> Squid on BSD| -> firewall -> public net IP=X \ IP=Y \ Branches 1. By network guys DR site traffic forcibly shifted to bge1, and resultantly internet access at DR site stopped functioning. 2. my colleague who was previously looking proxy changed following rule in ipfw file as below (as per his statement), and after that internet access for http started working but https traffic stopped working at both sides where transparent proxy was working i-e at DR site and internal net, however https still work at branches. RULE: ipfw add divert natd all from any to any via bge1 CHANGED TO: RULE: ipfw add divert natd all from internal net/24 to any via bge1 3. my network colleague told me that proxy is adding it's address as source address to http packets but not to https, and passes https packets with source address of internal net, which is ultimately blocked at perimeter firewall. now pls note that i have freshly started working on squid couple of months has only passed. so when https didn't run, i gone through documentation, forums etc (one example is of your previous answers) and found that https would not work on squid on transprent configuration & Got SURPRISED that how it was working previously then. anyways now when i say this to my head that squid on transparent proxy mode wont work for https he is not ready to accept. I argued with network colleagues that there must be some other setttings had been done for https but the do not agree and say that we had checked every thing and no such settings was there proxy was doing all functionality, Repeating Problem: Currently proxy adds it address as source to http traffic but not https, in https case it simple forwards packets with soruce address of internal net. and perimeter firewall allows proxy ip traffic and drops internal net addresses, resultantly https does not work. So this is the whole story and i have got really stuck, what should i do. .SUGGESTIONS DESPARITLY NEEDED. With Regards, > > > > > On Thu, Apr 30, 2009 at 8:24 AM, Amos Jeffries wrote: >>> First of all let me Thank you v much to all for replies. >>> >>> i am searching/reading for PAC / port forwarding for squid on FreeBSD, >>> but it would be grateful to me if you provide me an example/source. >> >> http://wiki.squid-cache.org/Technology/WPAD >> >>> >>> again i repeat i only want to allow https site like (gmail, yahoo) >>> behind my transparent proxy to work. >>> >> >> Once the requests are going to Squid properly this is a simple matter of >> ACLs. >> >> Amos >> >> >
Re: [squid-users] Transparent proxy with HTTPS on freebsd
Deal All, So champs now the interesting part starts. ok A few days ago we had proxy configured in the following way. DR Site \ int: bge0 int: bge1 internal netlan switch\Squid on BSD-external firewallpublic net IP=X \ IP=Y \ Branches 1. Above diagram shows that our internal net & and DR site is connected to squid on interface bge0 and uses transparent proxy whereas branches come to bge1 and uses manual proxy to get access to internet. 2. in above configuration http and https was working perfectly fine. after that in our company major changes were made in network and in result our proxy working scenario also changed as below. DR site | int: bge0 | int: bge1 internal netlan switchSquid on BSD|-external firewallpublic net IP=X \ IP=Y \ Branches 1. By network guys DR site traffic forcibly shifted to bge1, and resultantly internet access at DR site stopped functioning. 2. my colleague who was previously looking proxy changed following rule in ipfw file as below (as per his statement), and after that internet access for http started working but https traffic stopped working at both sides where transparent proxy was working i-e at DR site and internal net, however https still work at branches. RULE: ipfw add divert natd all from any to any via bge1 CHANGED TO: RULE: ipfw add divert natd all from internal net/24 to any via bge1 3. my network colleague told me that proxy is adding it's address as source address to http packets but not to https, and passes https packets with source address of internal net, which is ultimately blocked at perimeter firewall. now pls note that i have freshly started working on squid couple of months has only passed. so when https didn't run, i gone through documentation, forums etc (one example is of your previous answers) and found that https would not work on squid on transprent configuration & Got SURPRISED that how it was working previously then. anyways now when i say this to my head that squid on transparent proxy mode wont work for https he is not ready to accept. I argued with network colleagues that there must be some other setttings had been done for https but the do not agree and say that we had checked every thing and no such settings was there proxy was doing all functionality, Repeating Problem: Currently proxy adds it address as source to http traffic but not https, in https case it simple forwards packets with soruce address of internal net. and perimeter firewall allows proxy ip traffic and drops internal net addresses, resultantly https does not work. So this is the whole story and i have got really stuck, what should i do. .SUGGESTIONS DESPARITLY NEEDED. With Regards, On Thu, Apr 30, 2009 at 8:24 AM, Amos Jeffries wrote: >> First of all let me Thank you v much to all for replies. >> >> i am searching/reading for PAC / port forwarding for squid on FreeBSD, >> but it would be grateful to me if you provide me an example/source. > > http://wiki.squid-cache.org/Technology/WPAD > >> >> again i repeat i only want to allow https site like (gmail, yahoo) >> behind my transparent proxy to work. >> > > Once the requests are going to Squid properly this is a simple matter of > ACLs. > > Amos > >
Re: [squid-users] Transparent proxy with HTTPS on freebsd
First of all let me Thank you v much to all for replies. i am searching/reading for PAC / port forwarding for squid on FreeBSD, but it would be grateful to me if you provide me an example/source. again i repeat i only want to allow https site like (gmail, yahoo) behind my transparent proxy to work. With Regards, .Goody. On Wed, Apr 29, 2009 at 7:03 PM, Stefan Hartmann wrote: > Goody, > > if you simply want to have http and https go through the same unix box, > you can use squid for http and a port forwarding (for example using > iptables) for https. > > Regards, > Stefan > > > nyoman karna wrote: >> nope, >> you can NOT use transparent proxy for HTTPS. >> >> since using transparent proxy for HTTPS >> will be considered as man-in-the-middle attack. >> >> you probably may use PAC (as Amos suggested) >> but IMO it ruin the basic idea of using transparent proxy >> (which is user does not need to put any setting in their browser) >> >> >> Nyoman Bogi Aditya Karna >> IM Telkom >> http://www.imtelkom.ac.id >> >> >> >> >> --- On Wed, 4/29/09, goody goody wrote: >> >>> From: goody goody >>> Subject: Re: [squid-users] Transparent proxy with HTTPS on freebsd >>> To: squid-users@squid-cache.org >>> Cc: "Amos Jeffries" >>> Date: Wednesday, April 29, 2009, 7:30 AM >>> >>> Dear Amos, >>> >>> i say http works but https doesn't behind transparent proxy >>> (no proxy details specified in browser) and this is simply I >>> just want to achieve as some sites such as yahoo, gmail use >>> https to connect to. >>> >>> so if you guide my how can i configure squid to allow https >>> sites to connect behind transparent proxy. >>> >>> Further info regarding squid and bsd os is as follows. >>> >>> squid version info >>> >>> Squid Cache: Version 2.5.STABLE10 >>> configure options: --enable-storeio=diskd,ufs >>> --enable-snmp --with-openssl=/opt/ssl '--enable-auth=basic >>> ntlm' --enable-wccp '--enable-removal-policies=heap lru' >>> >>> BSD OS Info >>> >>> FreeBSD XXX 5.4-RELEASE FreeBSD 5.4-RELEASE #0: Fri Mar 30 >>> 18:16:33 PKT 2007 r...@xxx.abc.com.:/usr/src/sys/i386/compile/BSD-ROUTER >>> i386 >>> >>> an early response would be very much appreciated. >>> >>> Regards, >>> >>> >>> --- On Wed, 4/29/09, Amos Jeffries >>> wrote: >>> >>>> From: Amos Jeffries >>>> Subject: Re: [squid-users] Transparent proxy with >>> HTTPS on freebsd >>>> To: "abdul sami" >>>> Cc: squid-users@squid-cache.org >>>> Date: Wednesday, April 29, 2009, 1:49 PM >>>> abdul sami wrote: >>>>> Dear all, >>>>> >>>>> subject settings doesn't work when i set the >>>> transparent proxy though >>>>> http traffic works. on analysis of traffic i have >>> come >>>> to know that >>>>> proxy doesn't add it's source address to https >>> traffic >>>> rather simply >>>>> forwards it with local net address to >>> gateway/firewall >>>> device which >>>>> ultimately drops the packets. >>>>> >>>>> any suggestion in shape of steps/article would >>> be >>>> highly appreciated. >>>>> Regards, >>>> Pardon? >>>> HTTPS being transparently intercepted (miracle >>> #1) and the >>>> users not phoning you about being attacked? (miracle >>> #2). >>>> HTTPS == HTTP via _secure_ SSL. >>>> transparent proxy == man-in-middle network attack on >>>> traffic. >>>> >>>> HTTPS was created to prevent transparent interception >>>> amongst other things. So yes I'm not surprised it >>> won't >>>> work. >>>> >>>> What are you trying to achieve with this? >>>> >>>> Amos >>>> -- Please be using >>>> Current Stable Squid 2.7.STABLE6 or 3.0.STABLE14 >>>> Current Beta Squid 3.1.0.7 >>>> >>> >>> >>> >> >> >> >> >> > > -- > 09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c0 > --- > OnlineDienst Nordbayern | http://www.odn.de/ | Internet-Systemhaus > GmbH & Co.KG | E-Mail: ha...@odn.de | Hosting, Housing > Steinstr. 19 | Tel: 0911 / 933877-0 | Consulting, VoIP > 90419 Nuernberg - Germany | Fax: 0911 / 933877-55 | Programmierung > GF Christiane Teichgräber | AG Nürnberg HRA 13304 | > >
[squid-users] Transparent proxy with HTTPS on freebsd
Dear all, subject settings doesn't work when i set the transparent proxy though http traffic works. on analysis of traffic i have come to know that proxy doesn't add it's source address to https traffic rather simply forwards it with local net address to gateway/firewall device which ultimately drops the packets. any suggestion in shape of steps/article would be highly appreciated. Regards,
