Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
On 15/02/2013 10:12 p.m., dahanhsi wrote: Hi all, I use squid as a reverse proxy, and make thousands of connection to the it. Which version of Squid? What do you mean by thousands of connections. 1's of tousands? 10's of thousands? 100's of thousands? There are one ten of all connections can not establish in TCP layer, because squid does not respond SYN-ACK to client's SYN packet. How can I solve it? Thanks Check ulimit settings for Squid? Check your cache.log for messages about running out of filedescriptors? Once you find out what the cause is you an find out what to do about it. Amos
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
Thanks for your reply, provide more information below: 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:12 p.m., dahanhsi wrote: Hi all, I use squid as a reverse proxy, and make thousands of connection to the it. Which version of Squid? I use Squid 2.7 What do you mean by thousands of connections. 1's of tousands? 10's of thousands? 100's of thousands? # netstat -nat|grep -i 80|wc -l the result vary from 4651 to 9404 There are one ten of all connections can not establish in TCP layer, because squid does not respond SYN-ACK to client's SYN packet. How can I solve it? Thanks Check ulimit settings for Squid? # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655360 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Check your cache.log for messages about running out of filedescriptors? I set my limit.conf to: rootsoftnofile 655360 roothardnofile 655360 Once you find out what the cause is you an find out what to do about it. Amos
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
On 15/02/2013 10:43 p.m., dahanhsi wrote: Thanks for your reply, provide more information below: 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:12 p.m., dahanhsi wrote: Hi all, I use squid as a reverse proxy, and make thousands of connection to the it. Which version of Squid? I use Squid 2.7 Output of squid -v please. What do you mean by thousands of connections. 1's of tousands? 10's of thousands? 100's of thousands? # netstat -nat|grep -i 80|wc -l the result vary from 4651 to 9404 There are one ten of all connections can not establish in TCP layer, because squid does not respond SYN-ACK to client's SYN packet. How can I solve it? Thanks Check ulimit settings for Squid? # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655360 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Check your cache.log for messages about running out of filedescriptors? I set my limit.conf to: rootsoftnofile 655360 roothardnofile 655360 That does not answer the question. Squid may have been built or configured with a limit of less than 655360 filedescriptors. cache.log should tell you if Squid is reaching some limit like this. Amos
Re: [squid-users] Netflix+squid
Hi Amos, I still haven't configured/deployed anything yet. My approach is to have a server in the U.S. But I thought maybe there is a better solution/approach to this deployment. Maybe a proxy server local to them and configure it to use my proxy server in the U.S as its upstream proxy. Thanks Monah On 15/02/2013 1:24 p.m., mb...@whywire.com wrote: Hi all, A friend of mine has a company outside the U.S, and wants to provide Netflix to his customers. Since I can setup a proxy here for him and have his clients use my proxy to access netflix, is there any other solution that can optimize it even better. Better than what? you have not provided any information on what configuration settings you are using, we cannot tell whether you configured it for good performance or not. Can you cache the videos by the way? Unknown. You will want to look into the cached object size limits (default maximum_object_size directive is probably too small for large videos). then look into whether the videos are actually cacheable. Paste one of their URLs into redbot.org for info on that. Amos
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
Hi Amos, 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:43 p.m., dahanhsi wrote: Thanks for your reply, provide more information below: 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:12 p.m., dahanhsi wrote: Hi all, I use squid as a reverse proxy, and make thousands of connection to the it. Which version of Squid? I use Squid 2.7 Output of squid -v please. # squid -v Squid Cache: Version 2.7.STABLE9 configure options: '--prefix=/usr' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--sysconfdir=/etc/squid' '--localstatedir=/var/spool/squid' '--datadir=/usr/share/squid' '--enable-async-io' '--with-pthreads' '--enable-storeio=ufs,aufs,coss,diskd,null' '--enable-linux-netfilter' '--enable-arp-acl' '--enable-epoll' '--enable-removal-policies=lru,heap' '--enable-snmp' '--enable-delay-pools' '--enable-htcp' '--enable-cache-digests' '--enable-underscores' '--enable-referer-log' '--enable-useragent-log' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-large-files' '--with-maxfd=65536' 'amd64-debian-linux' 'build_alias=amd64-debian-linux' 'host_alias=amd64-debian-linux' 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS=' What do you mean by thousands of connections. 1's of tousands? 10's of thousands? 100's of thousands? # netstat -nat|grep -i 80|wc -l the result vary from 4651 to 9404 There are one ten of all connections can not establish in TCP layer, because squid does not respond SYN-ACK to client's SYN packet. How can I solve it? Thanks Check ulimit settings for Squid? # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655360 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Check your cache.log for messages about running out of filedescriptors? I set my limit.conf to: rootsoftnofile 655360 roothardnofile 655360 That does not answer the question. Squid may have been built or configured with a limit of less than 655360 filedescriptors. cache.log should tell you if Squid is reaching some limit like this. my cache.log: 2013/02/15 8:30:10| Starting Squid Cache version 2.7.STABLE9 for x86_64-debian-linux-gnu... 2013/02/15 8:30:10| Process ID 8136 2013/02/15 8:30:10| With 2048 file descriptors available 2013/02/15 8:30:10| Using epoll for the IO loop 2013/02/15 8:30:10| DNS Socket created at 0.0.0.0, port 6450, FD 6 2013/02/15 8:30:10| Adding nameserver 8.8.8.8 from /etc/resolv.conf 2013/02/15 8:30:10| User-Agent logging is disabled. 2013/02/15 8:30:10| Referer logging is disabled. 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/access.log 2013/02/15 8:30:10| Unlinkd pipe opened on FD 12 2013/02/15 8:30:10| Swap maxSize 8192 + 8388608 KB, estimated 645907 objects 2013/02/15 8:30:10| Target number of buckets: 32295 2013/02/15 8:30:10| Using 32768 Store buckets 2013/02/15 8:30:10| Max Mem size: 8388608 KB 2013/02/15 8:30:10| Max Swap size: 8192 KB 2013/02/15 8:30:10| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/store.log 2013/02/15 8:30:10| Rebuilding storage in /var/spool/squid (CLEAN) 2013/02/15 8:30:10| Using Least Load store dir selection 2013/02/15 8:30:10| Set Current Directory to /var/spool/squid 2013/02/15 8:30:10| Loaded Icons. 2013/02/15 8:30:10| Accepting accelerated HTTP connections at 0.0.0.0, port 80, FD 14. 2013/02/15 8:30:10| Accepting ICP messages at 0.0.0.0, port 3130, FD 15. 2013/02/15 8:30:10| HTCP Disabled. 2013/02/15 8:30:10| WCCP Disabled. 2013/02/15 8:30:10| Configuring localhost Parent localhost/12080/0 2013/02/15 8:30:10| Ready to serve requests. 2013/02/15 8:30:10| Done reading /var/spool/squid swaplog (11 entries) 2013/02/15 8:30:10| Finished rebuilding storage from disk. 2013/02/15 8:30:10|11 Entries scanned 2013/02/15 8:30:10| 0 Invalid entries. 2013/02/15 8:30:10| 0 With invalid flags. 2013/02/15 8:30:10|11 Objects loaded. 2013/02/15 8:30:10| 0 Objects expired. 2013/02/15 8:30:10| 0 Objects cancelled. 2013/02/15 8:30:10| 0 Duplicate URLs purged. 2013/02/15 8:30:10| 0 Swapfile clashes avoided. 2013/02/15 8:30:10| Took 0.3
Re: [squid-users] Securing squid3
Oh, this was a lot of information! :D So here goes. Im only using squeeze on the production server. On the testing server Im running wheezy, but not squid. Only havp. And yeah, I seems a bit poor but I was only testing this as proof of concept. Or to satisfy my inner nerd. Im not going to use this solution in the long run, and like you say there is more options as well. So now Im going to check them out since Im done with this. Could a managed switch help me out here? Instead of the crazy iptables/forwarding/redirecting on the server? Right now Im researching a small HP procurve to manage these connections for me, is this the normal route (no pun intended) to do this? Was thinking about setting up the switch directly in front of 192.168.0.1 and redirect the traffic to 192.168.0.24 before it hits the server. Thanks for the other links. Im under the weather here, got the flu and a toot ache signed by satan him self. So Im going to check it out when Im fit for fight again. And about the cve patches Im like hoping the debian team is on top of this. If you have other information - please keep it to your self :D -Andreas On Feb 15, 2013, at 06:11 , Amos Jeffries squ...@treenet.co.nz wrote: On 15/02/2013 10:18 a.m., Andreas Westvik wrote: So i actually got it working! Client - gateway - havp - squid - internets I actually had blocked my self totally from squid3, so that was quite the head scratch. It turned out that http access deny all has to be at the bottom of the config file. ;) :-) You started this thread with a question on how to make Squid secure. If you are using the Squeeze or Wheezy package you are not secure, the Squeeze package is missing patches for 3 CVE vulnerabilities, the Wheezy package is currently missing 1. Also, since you have a good handle on where the traffic is coming from you can lock down the proxy listening port. I wouls suggest s small vriant of teh mangle table rule which can be found here: http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxDnat By adding a -s !192.168.* stanza to exclude your internal clients from the port block you can give them service while halting all external access. So then I pasted this into squid.conf cache_peer 192.168.0.24 parent 3127 0 no-query no-digest And then I reloaded and everything just worked. Now my second server running debian wheezy is a first gen macbook. So that is not a beast. But it workes just fine. The log folder is mounted in the ram to use most of the speed. I made a little screencast of the thing working Have a look https://vimeo.com/59687536 Thanks for the help everyone! :) On Feb 14, 2013, at 17:24 , Andreas Westvik andr...@spbk.no wrote: havp supports parent setup, and as far as I have seen, it should be setup before squid. Now, I can always switch this around, and move the squid3 setup to 192.168.0.24 and setup havp on 192.168.0.1 of course. But 192.168.0.1 is running debian production and Debian does not support havp on a squeeze. So Im using a debian wheezy for havp in the mean while. And its not installed via apt. HAVP appears to be a neglected project. You may want to update the scanner to another AV (clamav with c-icap perhapse). NP: With ICAP you can plug in almost any AV scanner system into Squid and only have the MISS traffic being scanned, pre-scanned HITS still served out of cache at full speed. ICAP also supports streamed scanning from the latest AV systems, where the client gets delivery far faster. * serving from cache without re-scanning is a controverisial topic though. It is fast on the HITs, but permits any infections in cache to be delivered even after scanner signatures are updated. If squid caches infected files, the local clamav should take care of that anyways? Since havp on the other server are using clamav as well. Try plugging clamav directly into Squid. c-icap works for most people (unless you are one of the lucky ones with trouble). I really don't think the iptables rules should be that difficult to setup up, since I intercept the web traffic with this: iptables -t nat -A PREROUTING -i eth3 -p tcp --dport 80 -j REDIRECT --to-port 3128 So it's basically the same thing, but kinda like -j REDIRECT -to-destination 192.168.0.24:3127 But it's not working! grr! REDIRECT is a special case of DNAT target which redirects to the hosts main IP address. You cannot specify a destination IP on REDIRECT target, you can on DNAT. The LInuxDnat wiki page I linked to above has all the details you need for this - the iptables rules are the same for any proxy which accepts NAT'd traffic. So... * When your box IP is dynamically assigned and not known in advance use REDIRECT. * When your box is statically assigned use DNAT to the IP Squid is listening on. Squid-3.2+ provide protection against the CVE-2009-0801 security vulnerability in NAT and
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
On 15/02/2013 11:53 p.m., dahanhsi wrote: Hi Amos, 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:43 p.m., dahanhsi wrote: Thanks for your reply, provide more information below: 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:12 p.m., dahanhsi wrote: Hi all, I use squid as a reverse proxy, and make thousands of connection to the it. Which version of Squid? I use Squid 2.7 Output of squid -v please. # squid -v Squid Cache: Version 2.7.STABLE9 configure options: '--prefix=/usr' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--sysconfdir=/etc/squid' '--localstatedir=/var/spool/squid' '--datadir=/usr/share/squid' '--enable-async-io' '--with-pthreads' '--enable-storeio=ufs,aufs,coss,diskd,null' '--enable-linux-netfilter' '--enable-arp-acl' '--enable-epoll' '--enable-removal-policies=lru,heap' '--enable-snmp' '--enable-delay-pools' '--enable-htcp' '--enable-cache-digests' '--enable-underscores' '--enable-referer-log' '--enable-useragent-log' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-large-files' '--with-maxfd=65536' 'amd64-debian-linux' 'build_alias=amd64-debian-linux' 'host_alias=amd64-debian-linux' 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS=' What do you mean by thousands of connections. 1's of tousands? 10's of thousands? 100's of thousands? # netstat -nat|grep -i 80|wc -l the result vary from 4651 to 9404 There are one ten of all connections can not establish in TCP layer, because squid does not respond SYN-ACK to client's SYN packet. How can I solve it? Thanks Check ulimit settings for Squid? # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655360 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Check your cache.log for messages about running out of filedescriptors? I set my limit.conf to: rootsoftnofile 655360 roothardnofile 655360 That does not answer the question. Squid may have been built or configured with a limit of less than 655360 filedescriptors. cache.log should tell you if Squid is reaching some limit like this. my cache.log: 2013/02/15 8:30:10| Starting Squid Cache version 2.7.STABLE9 for x86_64-debian-linux-gnu... 2013/02/15 8:30:10| Process ID 8136 2013/02/15 8:30:10| With 2048 file descriptors available 2013/02/15 8:30:10| Using epoll for the IO loop 2013/02/15 8:30:10| DNS Socket created at 0.0.0.0, port 6450, FD 6 2013/02/15 8:30:10| Adding nameserver 8.8.8.8 from /etc/resolv.conf 2013/02/15 8:30:10| User-Agent logging is disabled. 2013/02/15 8:30:10| Referer logging is disabled. 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/access.log 2013/02/15 8:30:10| Unlinkd pipe opened on FD 12 2013/02/15 8:30:10| Swap maxSize 8192 + 8388608 KB, estimated 645907 objects 2013/02/15 8:30:10| Target number of buckets: 32295 2013/02/15 8:30:10| Using 32768 Store buckets 2013/02/15 8:30:10| Max Mem size: 8388608 KB 2013/02/15 8:30:10| Max Swap size: 8192 KB 2013/02/15 8:30:10| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/store.log 2013/02/15 8:30:10| Rebuilding storage in /var/spool/squid (CLEAN) 2013/02/15 8:30:10| Using Least Load store dir selection 2013/02/15 8:30:10| Set Current Directory to /var/spool/squid 2013/02/15 8:30:10| Loaded Icons. 2013/02/15 8:30:10| Accepting accelerated HTTP connections at 0.0.0.0, port 80, FD 14. 2013/02/15 8:30:10| Accepting ICP messages at 0.0.0.0, port 3130, FD 15. 2013/02/15 8:30:10| HTCP Disabled. 2013/02/15 8:30:10| WCCP Disabled. 2013/02/15 8:30:10| Configuring localhost Parent localhost/12080/0 2013/02/15 8:30:10| Ready to serve requests. 2013/02/15 8:30:10| Done reading /var/spool/squid swaplog (11 entries) 2013/02/15 8:30:10| Finished rebuilding storage from disk. 2013/02/15 8:30:10|11 Entries scanned 2013/02/15 8:30:10| 0 Invalid entries. 2013/02/15 8:30:10| 0 With invalid flags. 2013/02/15 8:30:10|11 Objects loaded. 2013/02/15 8:30:10| 0 Objects expired. 2013/02/15 8:30:10| 0 Objects cancelled. 2013/02/15 8:30:10| 0 Duplicate URLs purged. 2013/02/15 8:30:10| 0 Swapfile clashes avoided. 2013/02/15 8:30:10| Took
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 11:53 p.m., dahanhsi wrote: Hi Amos, 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:43 p.m., dahanhsi wrote: Thanks for your reply, provide more information below: 2013/2/15 Amos Jeffries squ...@treenet.co.nz: On 15/02/2013 10:12 p.m., dahanhsi wrote: Hi all, I use squid as a reverse proxy, and make thousands of connection to the it. Which version of Squid? I use Squid 2.7 Output of squid -v please. # squid -v Squid Cache: Version 2.7.STABLE9 configure options: '--prefix=/usr' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--sysconfdir=/etc/squid' '--localstatedir=/var/spool/squid' '--datadir=/usr/share/squid' '--enable-async-io' '--with-pthreads' '--enable-storeio=ufs,aufs,coss,diskd,null' '--enable-linux-netfilter' '--enable-arp-acl' '--enable-epoll' '--enable-removal-policies=lru,heap' '--enable-snmp' '--enable-delay-pools' '--enable-htcp' '--enable-cache-digests' '--enable-underscores' '--enable-referer-log' '--enable-useragent-log' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-carp' '--enable-follow-x-forwarded-for' '--with-large-files' '--with-maxfd=65536' 'amd64-debian-linux' 'build_alias=amd64-debian-linux' 'host_alias=amd64-debian-linux' 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS=' What do you mean by thousands of connections. 1's of tousands? 10's of thousands? 100's of thousands? # netstat -nat|grep -i 80|wc -l the result vary from 4651 to 9404 There are one ten of all connections can not establish in TCP layer, because squid does not respond SYN-ACK to client's SYN packet. How can I solve it? Thanks Check ulimit settings for Squid? # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655360 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Check your cache.log for messages about running out of filedescriptors? I set my limit.conf to: rootsoftnofile 655360 roothardnofile 655360 That does not answer the question. Squid may have been built or configured with a limit of less than 655360 filedescriptors. cache.log should tell you if Squid is reaching some limit like this. my cache.log: 2013/02/15 8:30:10| Starting Squid Cache version 2.7.STABLE9 for x86_64-debian-linux-gnu... 2013/02/15 8:30:10| Process ID 8136 2013/02/15 8:30:10| With 2048 file descriptors available 2013/02/15 8:30:10| Using epoll for the IO loop 2013/02/15 8:30:10| DNS Socket created at 0.0.0.0, port 6450, FD 6 2013/02/15 8:30:10| Adding nameserver 8.8.8.8 from /etc/resolv.conf 2013/02/15 8:30:10| User-Agent logging is disabled. 2013/02/15 8:30:10| Referer logging is disabled. 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/access.log 2013/02/15 8:30:10| Unlinkd pipe opened on FD 12 2013/02/15 8:30:10| Swap maxSize 8192 + 8388608 KB, estimated 645907 objects 2013/02/15 8:30:10| Target number of buckets: 32295 2013/02/15 8:30:10| Using 32768 Store buckets 2013/02/15 8:30:10| Max Mem size: 8388608 KB 2013/02/15 8:30:10| Max Swap size: 8192 KB 2013/02/15 8:30:10| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/store.log 2013/02/15 8:30:10| Rebuilding storage in /var/spool/squid (CLEAN) 2013/02/15 8:30:10| Using Least Load store dir selection 2013/02/15 8:30:10| Set Current Directory to /var/spool/squid 2013/02/15 8:30:10| Loaded Icons. 2013/02/15 8:30:10| Accepting accelerated HTTP connections at 0.0.0.0, port 80, FD 14. 2013/02/15 8:30:10| Accepting ICP messages at 0.0.0.0, port 3130, FD 15. 2013/02/15 8:30:10| HTCP Disabled. 2013/02/15 8:30:10| WCCP Disabled. 2013/02/15 8:30:10| Configuring localhost Parent localhost/12080/0 2013/02/15 8:30:10| Ready to serve requests. 2013/02/15 8:30:10| Done reading /var/spool/squid swaplog (11 entries) 2013/02/15 8:30:10| Finished rebuilding storage from disk. 2013/02/15 8:30:10|11 Entries scanned 2013/02/15 8:30:10| 0 Invalid entries. 2013/02/15 8:30:10| 0 With invalid flags. 2013/02/15 8:30:10|11 Objects loaded. 2013/02/15 8:30:10| 0 Objects expired. 2013/02/15 8:30:10| 0 Objects
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
On 02/15/2013 11:11 AM, dahanhsi wrote: There you go then. Squid is not permitted to _use_ more than 1651 FD. Every client TCP connection uses at least 1, sometimes 2 FD. When all the FD are used up Squid waits until some are free'd before accepting more client connections. With from 4651 to 9404 I would set your max_filedescriptors to at least 18000. It can be anything up to the ulimit max. I set max_filedescriptors to 655360, and confirm that ulimit -n is also 655360. After restart Squid, I observe that rate of connection timeout in client is still about 10%, and no additional error such as filedescriptors error found in cache.log or dmesg. any ideas? thanks ulimit -n must be run as the same user that the proxy is running. In debian/ubuntu that user is proxy, and if you type ulimit as root you will get a different answer that if you type ulimit logged in as proxy user. Be sure to check the ulimit for the right user.
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
ulimit -n must be run as the same user that the proxy is running. In debian/ubuntu that user is proxy, and if you type ulimit as root you will get a different answer that if you type ulimit logged in as proxy user. Be sure to check the ulimit for the right user Or you can check current limits using: /proc/SQUIDPID/limits
[squid-users] uploading attachments in hotmail failing
A Hotmail account converted to hotmail's new outlook style email which I believe uses sliverlight, consistently fails to allow file attachments when going through proxy. Any ideas? Squid Cache: Version 3.1.19
RE: [squid-users] uploading attachments in hotmail failing
Resolved The request was never even making it to the proxy server. The workstation was using a pac file that had some return DIRECT for hotmail.com live.com login.live.com Fmail.live.com Not sure why they were interfering, but removing them from the pac file made the problem go away.
[squid-users] Redirect Youtube out second ISP
I'm wondering if it's possible to use squid to redirect youtube out a second ISP line. We have two connections and I'd like to push all youtube out the second connection. I was thinking I could put a second squid proxy on that line and then redirect all youtube traffic to it, but I'm not sure how to start this. Thanks Ryan
Re: [squid-users] Redirect Youtube out second ISP
On 16/02/2013 11:43, Stinn, Ryan wrote: I'm wondering if it's possible to use squid to redirect youtube out a second ISP line. We have two connections and I'd like to push all youtube out the second connection. I was thinking I could put a second squid proxy on that line and then redirect all youtube traffic to it, but I'm not sure how to start this. Thanks Ryan Hi, Look at the cache_peer_access option if you have the second server. You could also use a dual gateway option, but this needs some work on iptables/iproute. Cheers, Pieter
Re: [squid-users] Squid does not respond to TCP SYN when there are thousands of connection
2013/2/15 Amm ammdispose-sq...@yahoo.com: ulimit -n must be run as the same user that the proxy is running. In debian/ubuntu that user is proxy, and if you type ulimit as root you will get a different answer that if you type ulimit logged in as proxy user. Be sure to check the ulimit for the right user Or you can check current limits using: /proc/SQUIDPID/limits Hi, I set my /etc/security/limit.conf: * softnofile 655360 * hardnofile 655360 so user squid also has limit in 655360: # cat /proc/SQUIDPID/limits Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimitedseconds Max file size unlimitedunlimitedbytes Max data size unlimitedunlimitedbytes Max stack size10485760 unlimitedbytes Max core file sizeunlimitedunlimitedbytes Max resident set unlimitedunlimitedbytes Max processes 257742 257742 processes Max open files655360 655360 files Max locked memory 6553665536bytes Max address space unlimitedunlimitedbytes Max file locksunlimitedunlimitedlocks Max pending signals 257742 257742 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Max realtime timeout unlimitedunlimitedus and my fd.file-max is 655360 too: # cat /proc/sys/fs/file-max 655360
Re: [squid-users] Redirect Youtube out second ISP
On 16/02/2013 12:01 p.m., Pieter De Wit wrote: On 16/02/2013 11:43, Stinn, Ryan wrote: I'm wondering if it's possible to use squid to redirect youtube out a second ISP line. We have two connections and I'd like to push all youtube out the second connection. I was thinking I could put a second squid proxy on that line and then redirect all youtube traffic to it, but I'm not sure how to start this. Thanks Ryan Hi, Look at the cache_peer_access option if you have the second server. You could also use a dual gateway option, but this needs some work on iptables/iproute. Cheers, Pieter It is also worth looking at tcp_outgoing_tos and tcp_outgoing_mark in Squid-3.2 or later. PS. you may need a version with http://bugs.squid-cache.org/show_bug.cgi?id=3767 fixed (will be in 3.2.8 next month, daily 3.2 bug-fix packages have it now). Amos
Re: [squid-users] Redirect Youtube out second ISP
On 16/02/2013 3:11 p.m., Amos Jeffries wrote: On 16/02/2013 12:01 p.m., Pieter De Wit wrote: On 16/02/2013 11:43, Stinn, Ryan wrote: I'm wondering if it's possible to use squid to redirect youtube out a second ISP line. We have two connections and I'd like to push all youtube out the second connection. I was thinking I could put a second squid proxy on that line and then redirect all youtube traffic to it, but I'm not sure how to start this. Thanks Ryan Hi, Look at the cache_peer_access option if you have the second server. You could also use a dual gateway option, but this needs some work on iptables/iproute. Cheers, Pieter It is also worth looking at tcp_outgoing_tos and tcp_outgoing_mark in Squid-3.2 or later. PS. you may need a version with http://bugs.squid-cache.org/show_bug.cgi?id=3767 fixed (will be in 3.2.8 next month, daily 3.2 bug-fix packages have it now). Oops. I meant http://bugs.squid-cache.org/show_bug.cgi?id=3723, but both are worth it. Amos Amos
Re: [squid-users] Redirect Youtube out second ISP
- Original Message - From: Stinn, Ryan ryan.st...@htcsd.ca To: squid-users@squid-cache.org squid-users@squid-cache.org Cc: Sent: Saturday, 16 February 2013 4:13 AM Subject: [squid-users] Redirect Youtube out second ISP I'm wondering if it's possible to use squid to redirect youtube out a second ISP line. We have two connections and I'd like to push all youtube out the second connection. Try this: acl dstdom_regex yt -i youtube tcp_outgoing_address yt 1.2.3.4 1.2.3.4 is IP address of 2nd line (should be on same machine as squid). Amm.
[squid-users] auth for system services
On a Windows desktop there are often a bunch of system services that make http connections, either running as a system account or running as a user but that don't know how to authenticate. The list of these exceptions is tedious to maintain so it would be good to be able to authorise the users IP address once they have successfully authenticated to squid, sort of like the old style 'pop before smtp' auth used to work. If such a solution was scriptable I could also use something like netfilter ipsets to allow access on non-http ports using squid authentication. What hooks exist to allow this sort of thing? Thanks James
Re: [squid-users] auth for system services
On 16/02/2013 3:23 p.m., James Harper wrote: On a Windows desktop there are often a bunch of system services that make http connections, either running as a system account or running as a user but that don't know how to authenticate. The list of these exceptions is tedious to maintain so it would be good to be able to authorise the users IP address once they have successfully authenticated to squid, sort of like the old style 'pop before smtp' auth used to work. Tedious to maintain? what exactly are you listing? I list services by approving and whitelisting destination domains. That is no much work, as the listing only needs adapting when the first time you encounter a service. A second, third... thousandth client system using that service does not make any difference. If such a solution was scriptable I could also use something like netfilter ipsets to allow access on non-http ports using squid authentication. What hooks exist to allow this sort of thing? Some Warnings first: * Be careful with this. It is not very safe to trust an IP just because you saw credentials from it earlier on a completely different connection. * Clients are able to run proxy software and share their internet connection with the world very easily these days. * You loose all tracking of any infections or malicious software they may be infected with. * You loose the ability for users to share machines. User A can shut down the machine, user B restart it and if they are fast enough the proxy shares the session started by user A. * the above means you loose the ability to identify which user is doing what actions (once the IP-based session is active the credentials are not used or logged). Anyways What you want to look at is the session helper, with its active mode (-a command line parameter). http://www.squid-cache.org/Versions/v3/3.2/manuals/ext_session_acl.html For example: external_acl_type session ttl=300 %SRC */usr/local/squid/libexec/ext_session_acl -t 300 -a* # allow client IPs which have already logged in earlier acl sessionActive external session http_access allow sessionActive # deny anyone not logged in (triggers the login process) acl auth proxy_auth REQUIRED http_access deny !auth # allow clients with login and create a session for them acl sessionStart external session LOGIN http_access allow auth sessionLogin If you want to be fancy you can add the following snippet *above* the sessionActive ACL test and setup a script which when the user logs off their machine makes a web request (without credentials) to http://example.com/logout . That will help avoid the session-sharing problem provided people logout properly. # magic logout. Visit the URL http://example.com/logout from a script on the box to log this client out of the session when they logout or shut down. acl sessionLogout external session LOGOUT acl logoutMagic url_regex ^http://example.com/logout$ http_access deny magicLogout sessionLogout Amos
RE: [squid-users] auth for system services
On 16/02/2013 3:23 p.m., James Harper wrote: On a Windows desktop there are often a bunch of system services that make http connections, either running as a system account or running as a user but that don't know how to authenticate. The list of these exceptions is tedious to maintain so it would be good to be able to authorise the users IP address once they have successfully authenticated to squid, sort of like the old style 'pop before smtp' auth used to work. Tedious to maintain? what exactly are you listing? I list services by approving and whitelisting destination domains. That is no much work, as the listing only needs adapting when the first time you encounter a service. A second, third... thousandth client system using that service does not make any difference. That's what I thought originally, but things like CRL's (every CA seems to use a new one - I've 'fixed' java 5 times in the past week) and skydrive (breaks every month or so as Microsoft change things) require continual maintenance and doesn't fail nicely. Some Warnings first: * Be careful with this. It is not very safe to trust an IP just because you saw credentials from it earlier on a completely different connection. Credentials will time out * Clients are able to run proxy software and share their internet connection with the world very easily these days. That's true of the existing username/password authentication anyway. But the site is small enough that we'd notice. * You loose all tracking of any infections or malicious software they may be infected with. How so? Username is first logged against IP address, then IP address is logged. Tracking is easy. * You loose the ability for users to share machines. User A can shut down the machine, user B restart it and if they are fast enough the proxy shares the session started by user A. I thought about that. Firstly, the above scenario doesn't happen, and if it did the login records are present on the PC anyway. * the above means you loose the ability to identify which user is doing what actions (once the IP-based session is active the credentials are not used or logged). But the IP address is, so the problem becomes a reporting problem. Anyways What you want to look at is the session helper, with its active mode (-a command line parameter). http://www.squid- cache.org/Versions/v3/3.2/manuals/ext_session_acl.html Thanks for taking the time to write all of this. Now I know that what I want to do is possible I can consider whether it is the best road forward. Have you ever considered integrating a SOCKS style proxy into squid? It requires a smart client of course but I can do that much under Windows. James
