Re: [squid-users] Squid TProxy Problem
Amos, Hi Thanks for your reply. Ragarding the documentation, I have inserted the following routing rules: ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 Now, access.log is populated with proper logs, but clients can not surf the web, I mean the proxy server is unable to forward http responses to clients' browsers. When the client enters for example www.google.com, the connection to the http server is established but the process halts at Waiting for www.google.com and after a while Squid reports the unablility to retreive the requested URL. By the way, we have disabled selinux. Any ideas? Warm Regards, Ali Majdzadeh Kohbanani 2011/6/6 Amos Jeffries : > On 06/06/11 06:32, Ali Majdzadeh wrote: >> >> Hello All, >> I have setup the following configuration: >> Squid (3.1.12) (--enable-linux-netfilter passed as the one and only >> configure option) >> Kernel (2.6.38.3) >> iptables (1.4.11) >> >> I have added the following two directives in squid.conf: >> http_port 3128 >> http_port 3129 tproxy >> >> Also, I have configured iptables with the following rules: >> iptables -t mangle -N DIVERT >> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT >> iptables -t mangle -A DIVERT -j MARK --set-mark 1 >> iptables -t mangle -A DIVERT -j ACCEPT >> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY >> --tproxy-mark 0x1/0x1 --on-port 3129 >> >> Everything work as expected, I mean, the users can surf the web and >> the proxy server is transparent. The problem is that actually there is >> no caching. I mean, both cache.log and access.log files are empty. On > > That would be transparency to the point of not going through the proxy. > access.log should have entries for each request. > >> the other hand, if I manually set the proxy configuration in clients' >> browsers (the IP address of the squid server and port number 3128) >> everything is OK; the log files are incremented and objects are >> cached. >> >> Have anyone faced the same issue? > > Some. Its usually boiled down to missing out some details omitted. building > against libcap2 or routing packets to the squid box for example. > > Are the packet counters on that -j TPROXY rule showing captures? > > Did you follow the rest of the feature config? > ie the special sub-routing table? OS packet filtering toggles? selinux > updated to allow tproxy? > > Is this box even routing or bridging port 80 traffic for the network? > > Amos > -- > Please be using > Current Stable Squid 2.7.STABLE9 or 3.1.12 > Beta testers wanted for 3.2.0.8 and 3.1.12.2 >
Re: [squid-users] need a simple transparent caching conf
On 06/06/11 11:55, MrNicholsB wrote: Squid is caching content, but it is NOT serving cache to my clients and frankly its driving me nuts, I dont need a 101 on squid, I just need a basic conf. I wish the devs would include a basic transparent cache proxy conf with squid to save noobs like me the trouble. My clients are (rant warning) We can't bundle it. * This TCP hijacking is no topic for "noobs" as you put it. * "transparent" rides a fine line of legality in most of the world. Just like downloading MP3s and AVIs, everybody "noob" tries it anyway. We do distribute the 19 configs via the wiki. * http://wiki.squid-cache.org/ConfigExamples/#Interception as you can see; a different config for every device, firewall software, and firewall feature on the market. That list is also only for the common ones we get told about. /rant MANUALLY aimed at the proxy at port 3128, they can surf just fine, so good. Problem worked around then. Time to relax before looking at alternatives calmly. NAT is NOT required on the box, I just need a conf that actually WORKS. Good. Lets keep it completely out of the picture until the caching bit is figured out. This is getting absurd, I dont understand why its not serving up cached content, I download ANYTHING you know 13mb exe files from a site, then go download the same file on another pc and BAM!! fresh content NOT Ah, there is a sign that (a) the PC are each asking for different content (one URL has multiple "variants" in HTTP), or (b) the server is producing different content for each unique client. Once you have a recent enough version of Squid we can give you debug settings to log the headers and see what is going on. served from the cache, wtf am I doing wrong here!?!?!?! Still doing all this with 3.0.STABLE1 ? yes/no ? Caching behaviour and HTTP compliance has undergone a LOT of good changes since then. Ive tried several confs and they all FAIL to actually serve up cache, below is my latest attempt. http_port 3128 transparent hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? cache deny QUERY acl apache rep_header Server ^Apache access_log /var/log/squid3/access.log squid hosts_file /etc/hosts refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320 acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 563 # https, snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 631 # cups acl Safe_ports port 873 # rsync acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost acl lan src 10.10.1.0/24 http_access allow localhost http_access allow lan http_access deny all http_reply_access allow all icp_access allow all visible_hostname katmai.local always_direct allow all coredump_dir /var/spool/squid3 cache_dir ufs /var/spool/squid3 2 32 256 maximum_object_size 200 MB maximum_object_size_in_memory 50 MB -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
Re: [squid-users] Squid TProxy Problem
On 06/06/11 06:32, Ali Majdzadeh wrote: Hello All, I have setup the following configuration: Squid (3.1.12) (--enable-linux-netfilter passed as the one and only configure option) Kernel (2.6.38.3) iptables (1.4.11) I have added the following two directives in squid.conf: http_port 3128 http_port 3129 tproxy Also, I have configured iptables with the following rules: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129 Everything work as expected, I mean, the users can surf the web and the proxy server is transparent. The problem is that actually there is no caching. I mean, both cache.log and access.log files are empty. On That would be transparency to the point of not going through the proxy. access.log should have entries for each request. the other hand, if I manually set the proxy configuration in clients' browsers (the IP address of the squid server and port number 3128) everything is OK; the log files are incremented and objects are cached. Have anyone faced the same issue? Some. Its usually boiled down to missing out some details omitted. building against libcap2 or routing packets to the squid box for example. Are the packet counters on that -j TPROXY rule showing captures? Did you follow the rest of the feature config? ie the special sub-routing table? OS packet filtering toggles? selinux updated to allow tproxy? Is this box even routing or bridging port 80 traffic for the network? Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
Re: [squid-users] About reply_body_max_size
On 06/06/11 06:24, Odhiambo WASHINGTON wrote: Reading squid.conf.documented for 3.1.9, I see this portion: TAG: reply_body_max_size size [acl acl...] # This option specifies the maximum size of a reply body. It can be # used to prevent users from downloading very large files, such as # MP3's and movies. When the reply headers are received, the # reply_body_max_size lines are processed, and the first line where # all (if any) listed ACLs are true is used as the maximum body size # for this reply. # # This size is checked twice. First when we get the reply headers, # we check the content-length value. If the content length value exists # and is larger than the allowed size, the request is denied and the # user receives an error message that says "the request or reply # is too large." If there is no content-length, and the reply # size exceeds this limit, the client's connection is just closed # and they will receive a partial reply. # # WARNING: downstream caches probably can not detect a partial reply # if there is no content-length header, so they will cache # partial responses and give them out as hits. You should NOT # use this option if you have downstream caches. # # WARNING: A maximum size smaller than the size of squid's error messages # will cause an infinite loop and crash squid. Ensure that the smallest # non-zero value you use is greater that the maximum header size plus # the size of your largest error page. # # If you set this parameter none (the default), there will be # no limit imposed. # # Configuration Format is: # reply_body_max_size SIZE UNITS [acl ...] # ie. # reply_body_max_size 10 MB Now, having the following line causes squid (3.1.9) to grok: reply_body_max_size 0 KB deny all squid -k parse gives this: 2011/06/05 21:03:05| aclParseAclList: ACL name 'KB' not found. FATAL: Bungled squid.conf line 60: reply_body_max_size 0 KB deny all Squid Cache (Version 3.1.9): Terminated abnormally. CPU Usage: 0.007 seconds = 0.007 user + 0.000 sys Maximum Resident Size: 4376 KB Page faults with physical i/o: 0 Where is the problem? There are two special cases for traffic size: "0". Meaning no body is permitted. "none". Meaning no limit applied or "unlimited" size. Units are not relevant on these and Squid does not currently accept any. You can still add ACLs after these special values to indicate _when_ they apply. NP: the default is not to limit any replies. An implicit: "reply_body_max_size none all". The word "deny" is also not relevant in reply_body_max_size. Squid ACL lines have a general syntax "$directive $value $conditions". The $value applies only when the $conditions are all matching. In the case of http_access the $value is permission or rejection (allow/deny). In reply_body_max_size the $value is the limit being set. So what you would be used to as allow/deny elsewhere is written "n KB" here. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
[squid-users] need a simple transparent caching conf
Squid is caching content, but it is NOT serving cache to my clients and frankly its driving me nuts, I dont need a 101 on squid, I just need a basic conf. I wish the devs would include a basic transparent cache proxy conf with squid to save noobs like me the trouble. My clients are MANUALLY aimed at the proxy at port 3128, they can surf just fine, so NAT is NOT required on the box, I just need a conf that actually WORKS. This is getting absurd, I dont understand why its not serving up cached content, I download ANYTHING you know 13mb exe files from a site, then go download the same file on another pc and BAM!! fresh content NOT served from the cache, wtf am I doing wrong here!?!?!?! Ive tried several confs and they all FAIL to actually serve up cache, below is my latest attempt. http_port 3128 transparent hierarchy_stoplist cgi-bin ? acl QUERY urlpath_regex cgi-bin \? cache deny QUERY acl apache rep_header Server ^Apache access_log /var/log/squid3/access.log squid hosts_file /etc/hosts refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320 acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 563 # https, snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 563 # https, snews acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 631 # cups acl Safe_ports port 873 # rsync acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost acl lan src 10.10.1.0/24 http_access allow localhost http_access allow lan http_access deny all http_reply_access allow all icp_access allow all visible_hostname katmai.local always_direct allow all coredump_dir /var/spool/squid3 cache_dir ufs /var/spool/squid3 2 32 256 maximum_object_size 200 MB maximum_object_size_in_memory 50 MB
[squid-users] Squid TProxy Problem
Hello All, I have setup the following configuration: Squid (3.1.12) (--enable-linux-netfilter passed as the one and only configure option) Kernel (2.6.38.3) iptables (1.4.11) I have added the following two directives in squid.conf: http_port 3128 http_port 3129 tproxy Also, I have configured iptables with the following rules: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129 Everything work as expected, I mean, the users can surf the web and the proxy server is transparent. The problem is that actually there is no caching. I mean, both cache.log and access.log files are empty. On the other hand, if I manually set the proxy configuration in clients' browsers (the IP address of the squid server and port number 3128) everything is OK; the log files are incremented and objects are cached. Have anyone faced the same issue? Warm Regards, Ali Majdzadeh Kohbanani
[squid-users] About reply_body_max_size
Reading squid.conf.documented for 3.1.9, I see this portion: TAG: reply_body_max_size size [acl acl...] # This option specifies the maximum size of a reply body. It can be # used to prevent users from downloading very large files, such as # MP3's and movies. When the reply headers are received, the # reply_body_max_size lines are processed, and the first line where # all (if any) listed ACLs are true is used as the maximum body size # for this reply. # # This size is checked twice. First when we get the reply headers, # we check the content-length value. If the content length value exists # and is larger than the allowed size, the request is denied and the # user receives an error message that says "the request or reply # is too large." If there is no content-length, and the reply # size exceeds this limit, the client's connection is just closed # and they will receive a partial reply. # # WARNING: downstream caches probably can not detect a partial reply # if there is no content-length header, so they will cache # partial responses and give them out as hits. You should NOT # use this option if you have downstream caches. # # WARNING: A maximum size smaller than the size of squid's error messages # will cause an infinite loop and crash squid. Ensure that the smallest # non-zero value you use is greater that the maximum header size plus # the size of your largest error page. # # If you set this parameter none (the default), there will be # no limit imposed. # # Configuration Format is: # reply_body_max_size SIZE UNITS [acl ...] # ie. # reply_body_max_size 10 MB Now, having the following line causes squid (3.1.9) to grok: reply_body_max_size 0 KB deny all squid -k parse gives this: 2011/06/05 21:03:05| aclParseAclList: ACL name 'KB' not found. FATAL: Bungled squid.conf line 60: reply_body_max_size 0 KB deny all Squid Cache (Version 3.1.9): Terminated abnormally. CPU Usage: 0.007 seconds = 0.007 user + 0.000 sys Maximum Resident Size: 4376 KB Page faults with physical i/o: 0 Where is the problem? -- Best regards, Odhiambo WASHINGTON, Nairobi,KE
Re: [squid-users] dst vs dstdomain speed
On 05/06/11 22:56, E.S. Rosenberg wrote: Hi, Is dst easier/faster for squid then dstdomain to handle? I'm asking this because I see a lot of the pre-made black/white lists seem to be of the dst type while it seems to me that dstdomain is more effective and easier to manage since you don't need to add an entry for every single host on a domain you want to block/allow you just add .domain.tld to the list. Also as far as I understand when a user tries to use an IP instead of a domain name if the IP is known to be matched to a domain in a list whatever rule was applied to said list will be applied to the IP even though it is not mentioned specifically in the list? dstdomain is bit dynamic. It is fast for domains and "slow" for raw-IP. It does a plain text match on the value the client gave (whether domain FQDN or textual IP representation). If there was a raw-IP AND it is working in a "slow" access list it will lookup and try to match on the rDNS. dst must always lookup the IP. So is always "slow" category. On raw-IP requests it can be the faster one. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
[squid-users] dst vs dstdomain speed
Hi, Is dst easier/faster for squid then dstdomain to handle? I'm asking this because I see a lot of the pre-made black/white lists seem to be of the dst type while it seems to me that dstdomain is more effective and easier to manage since you don't need to add an entry for every single host on a domain you want to block/allow you just add .domain.tld to the list. Also as far as I understand when a user tries to use an IP instead of a domain name if the IP is known to be matched to a domain in a list whatever rule was applied to said list will be applied to the IP even though it is not mentioned specifically in the list? Thanks and regards, Eli
Re: [squid-users] lots of UDP connections
On 05/06/11 16:55, Bal Krishna Adhikari wrote: On 06/04/2011 12:59 PM, Amos Jeffries wrote: Bal Krishna Adhikari 6/3/2011 6:13 AM Hello, I found a lot of UDP connections that is coming to my proxy servers. I don't find the cause of such one-way traffics to my servers. The sample UDP traffic is as :- 14:00:07.506612 IP 41.209.69.146.10027> x.x.x.x.65453: UDP, length 30 14:00:07.518118 IP 121.218.37.254.41597> x.x.x.x.64338: UDP, length 30 14:00:07.572559 IP 85.224.143.193.29978> x.x.x.x.62782: UDP, length 30 14:00:07.596554 IP 183.87.200.42.36895> x.x.x.x.15786: UDP, length 30 14:00:07.642820 IP 180.215.37.96.49977> x.x.x.x.49458: UDP, length 30 14:00:07.653055 IP 117.195.138.64.24314> x.x.x.x.44985: UDP, length 33 14:00:07.739963 IP 82.31.238.101.50534> x.x.x.x.52750: UDP, length 30 14:00:07.783452 IP 86.83.107.196.41870> x.x.x.x.62782: UDP, length 30 14:00:07.809677 IP 94.246.23.15.59003> x.x.x.x.27462: UDP, length 30 14:00:07.837415 IP 75.156.164.147.49398> x.x.x.x.34847: UDP, length 30 14:00:07.841668 IP 82.8.212.242.25931> x.x.x.x.24869: UDP, length 30 14:00:07.841697 IP 89.136.112.99.42182> x.x.x.x.52750: UDP, length 30 14:00:07.854215 IP 99.191.156.208.18162> x.x.x.x.64338: UDP, length 30 14:00:07.885386 IP 88.147.72.252.60224> x.x.x.x.19151: UDP, length 30 14:00:07.960841 IP 68.169.185.192.63480> x.x.x.x.58638: UDP, length 30 14:00:08.071763 IP 79.113.242.42.31998> x.x.x.x.33995: UDP, length 30 14:00:08.078260 IP 94.202.49.109.61957> x.x.x.x.26071: UDP, length 67 14:00:08.101495 IP 82.169.68.179.19605> x.x.x.x.45682: UDP, length 30 14:00:08.113238 IP 86.99.42.7.15086> x.x.x.x.11706: UDP, length 67 14:00:08.127979 IP 62.195.70.253.45266> x.x.x.x.37050: UDP, length 30 14:00:08.163992 IP 2.82.207.195.38343> x.x.x.x.26680: UDP, length 30 14:00:08.183453 IP 68.81.206.57.25923> x.x.x.x.18378: UDP, length 30 14:00:08.237689 IP 108.120.241.254.47249> x.x.x.x.39433: UDP, length 30 14:00:08.256906 IP 99.161.157.254.41719> x.x.x.x.26680: UDP, length 30 14:00:08.291885 IP 121.136.175.247.12577> x.x.x.x.16485: UDP, length 67 14:00:08.315427 IP 121.144.158.120.30845> x.x.x.x.61415: UDP, length 30 14:00:08.317404 IP 115.117.219.18.25817> x.x.x.x.59936: UDP, length 30 Anyone has any idea if the traffic is genuine or some kind of attack ? x.x.x.x is my proxy server. --- Bal Krishna On 04/06/11 01:16, Chad Naugle wrote: > Check the hostname of these IP addresses. They could be DNS replies, > using random ports for source/destinations. Squid can generate tons of > DNS traffic. I don't think its genuine Squid traffic. DNS, ICP and HTCP all use a fixed well-known port at one end and a rarely changing port at the other. It could be anything else on the box though. There are a few CVE attacks this could be, two using DNS and one HTCP. If you have a Squid 2.7.STABLE8+, 3.0.STABLE23+ or 3.1.1+ you are safe from those. They are just annoying. If you have a Squid-3.1+ with an IPv6 address publicly advertised this could be a sign of v6 connection attempts. Several IP tunnel protocols involve UDP handshakes. Amos I'm currently using 2.7 STABLE9. And the connection seems increased then earlier. Blocking the UDP other then DNS and SNMP from outside can solve the problem ? We can't answer that. It may not be a problem. You need to find out what it actually is. Blocking it will stop it doing anything, but until you know what it is that may just be creating a different problem. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
