Re: [squid-users] transparent proxy myself?
El 04/07/12 02:07, Amos Jeffries escribió: > On 04.07.2012 07:05, Linos wrote: >> Hi, >> i have configured transparent proxy sometimes for the local network LAN, >> but >> now i want to actually control the output traffic from the machine >> running the >> squid itself without have to configure manually browsers and other network >> programs, i can't get it to work using iptables, what rule should i be using? >> >> Regards, >> Miguel Angel. > > > Sorry your message is not clear. What do you have setup? and what exactly are > you trying to reach? > > From what I gather; > you have a somewhat proper proxy configuration already in use and want to > downgrade it to a full-blown MITM security attack on your clients instead? > > This might help: > http://wiki.squid-cache.org/SquidFaq/ConfiguringBrowsers#Recommended_network_configuration > > > Amos > Yes, my message it's not clear, sorry, i will explain through examples, suppose my Squid computer it's 192.168.1.254 and my lan 192.168.1.0/24, to get computers in my lan use transparent proxy i do: 1) configure squid.conf http_port (in 192.168.1.254 Squid computer) in "transparent" mode 2) i use 192.168.1.254 as gateway for computers in my lan 3) in Squid computer i use "iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 3128" After that if, for example, the computer 192.168.1.2 browse the net the url visited from there appear in access.log at Squid computer as usual without the need to configure the browser of this computer to use the proxy, perfect. What i don't know how to do it's intercept web traffic from a browser used in 192.168.1.254, i have one office where the Squid computer it's a desktop as well and i would like to transparent proxy the browsers of the Squid machine itself. Regards, Miguel Angel.
Re: [squid-users] slow reconfigure on squid3
Hi all, thanks for your responses... versions - I use the standard ones with Debian squeeze (2.7.stable9 and 3.1.6) Yes there are lots of helpers - 25 NTLM helpers and 10 squiduguard helpers, so this could account for slow reconfig. Upgrading to 3.2 seems like a good bet - are there ready-rooled squid 3.2 debs available for Squeeze or do I have to make my own? We currently run squid in 3 different flavours of authentication - NTLM for PCs, ident for macs and digest for guest network, so have 3 distinct squid setups running on our proxy server. Would it be worth setting these all up as non-caching, then set up a parent caching server, or will setting them up as cache peers make them share their caches at all? cheers Jim UK On 2 July 2012 14:44, Marcus Kool wrote: > Squid reconfigure can indeed take a long time. Especially when Squid > uses lots of memory and starts helpers. Starting helpers takes a > large amount of kernel resources when Squid is large, e.g. 2+ GB > since it forks itself and replaces its copy by a new process. The > fork can take a long time. If you use a URL rewritor you can > easily have 24 or more of them and this makes 24 copies of a large > process. > > How large is squid ? > Can you post the output of >ps -o pid,stime,sz,vsz,rss,args -C squid > > I wrote a test program to test the performance of forking X times > a large process. I can post it if you are interested. > > Marcus > > > > On 07/02/2012 05:08 AM, Mr J Potter wrote: >> >> Hi all, >> >> Does anyone have any tips on how to fix this issue: >> >> We've just moved to squid3 from squid2, and now when we do squid3 -k >> reconfigure we get about 30 seconds of squid refusing/failing to >> forward requests while it rejigs itself. I don't know if this is >> squid3 rescanning the cache or doing something with squidguard (we >> have a fairly complex+large squidguard setup)? I don't think this >> happened with squid2. >> >> What can we do to make this less noticeable? >> >> - make it reconfigure faster? >> - multiple squid servers - can we do failover somehow (either proxy >> DNS record points to them both, or they automatically redirect (is >> this what cache peers are for?))? >> - go back to squid 2 - I didn't see any end user benefits of squid3 >> over squid2... >> >> any help would be greatly appreciated. >> >> thanks >> >> Jim Potter >> UK >> >> >
[squid-users] Bug with direct proxying in transparent mode?
Hi all,
I am running squid 3.1.12 and I have noticed that if I connect to squid
and make a request to itself while the box has "transparent" enabled on
the port it causes squid to run out of file descriptors.
So for example:
$ telnet localhost 6045
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET http://localhost:6045 HTTP/1.0
I then see:
$ sudo ls /proc//fd | wc -l
932
And:
2012/07/03 17:03:21| client_side.cc(3032) okToAccept: WARNING! Your
cache is running out of filedescriptors
2012/07/03 17:03:21| WARNING: Forwarding loop detected for:
GET / HTTP/1.1
Host: localhost.alba.local:6045
I noticed that the code has this comment in src/client_side_reply.cc:
/// Deny loops for accelerator and interceptor. TODO: deny in all modes?
if (r->flags.loopdetect &&
(http->flags.accel || http->flags.intercepted)) {
Seems like either the answer to the TODO is yes or for this request we
should have flags.intercepted set (not the first one request but the
next one).
Cheers,
--
Michael Graham
[squid-users] DSTDOMAIN Wildcards and Multiple http_port
Hello Team, Please bare with me if this was resolved before, but I am asking out of curiosity and need to test it soon What is the difference and implications of the following as dstdomain entries? # Assuming namasenda.com is a registered domain, and the outcome is controlling any domain with the word "namasenda" like hatenamasenda.com or .net a) .\namasenda\. b) .namasenda. c) .namasenda.com d) .\namasenda\.com Is a) & b) correct, anyway? I want one squid instance to listen on ports 80, 8080, & 3128 Do I just do "http_port 3128 8080 80 transparent" work? Thank you, # Edmonds
Re: [squid-users] slow reconfigure on squid3
On 4/07/2012 7:49 p.m., Mr J Potter wrote: Hi all, thanks for your responses... versions - I use the standard ones with Debian squeeze (2.7.stable9 and 3.1.6) Yes there are lots of helpers - 25 NTLM helpers and 10 squiduguard helpers, so this could account for slow reconfig. Upgrading to 3.2 seems like a good bet - are there ready-rooled squid 3.2 debs available for Squeeze or do I have to make my own? Right now you have to roll your own. There is a 3.1.20 in Wheezy/Testing repositories that should work a lot better than 3.1.6 from Squeeze. You may want to try that and see if it resolves this issue, there were quite a few resource consumption issues resolved across the 3.1 series. We currently run squid in 3 different flavours of authentication - NTLM for PCs, ident for macs and digest for guest network, so have 3 distinct squid setups running on our proxy server. Would it be worth setting these all up as non-caching, then set up a parent caching server, or will setting them up as cache peers make them share their caches at all? Either way works. * sibling peers has extra traffic coordinating the caches and almost (but not always) getting it right. * one parent caching proxy creates a potential bottleneck at the parent. Choice is yours. Amos
Re: [squid-users] yahoo mail problem with tproxy (squid 3.1.19, kernel 3.2.21)
--- On Mon, 7/2/12, Ming-Ching Tiew wrote: > > No your symptom and mine are totally different. With the > limited testing, I don't see any problem with any OSes, any > sites. I only see problem when visiting yahoo mail. Meaning > when I use Windows XP, firefox, IE, Linux with firefox to > visit any sites, the response is decent and acceptable. I > just can't go to yahoo mail (classic). But when I switch to > nat REDIRECT mode, it works perfectedly including yahoo > mail. > > > I tried with kernel 2.6.37.6 and had the same problem. Further to it, also encounter another website which will hit the same problem. So it must be either the squid version or my setup is wrong somewhere. But the strange thing is those web sites which work, continue to work while those websites (such as yahoo mail) continue to give the same problem. Appreciate any comments about the next best thing to try. Thanks.
Re: [squid-users] DSTDOMAIN Wildcards and Multiple http_port
On 4/07/2012 8:37 p.m., Edmonds Namasenda wrote: Hello Team, Please bare with me if this was resolved before, but I am asking out of curiosity and need to test it soon What is the difference and implications of the following as dstdomain entries? # Assuming namasenda.com is a registered domain, and the outcome is controlling any domain with the word "namasenda" like hatenamasenda.com or .net a) .\namasenda\. b) .namasenda. c) .namasenda.com d) .\namasenda\.com Is a) & b) correct, anyway? No they are not. Neither is (d). (c) is correct dstdomain syntax but the "wildcard" is label-based, not character-based. .namasenda.com will match namasenda.com, www.namasenda.com, www2.namasenda.com, other.namasenda.com but not: anamasenda.com or hatenamasenda.com or anything else with a different 2nd-tier label than "namasenda". I want one squid instance to listen on ports 80, 8080, & 3128 Do I just do "http_port 3128 8080 80 transparent" work? No, one line per http_port entry. Amos
Re: [squid-users] DSTDOMAIN Wildcards and Multiple http_port
>> # Assuming namasenda.com is a registered domain, and the outcome is >> controlling any domain with the word "namasenda" like >> hatenamasenda.com or .net >> a) .\namasenda\. >> b) .namasenda. >> c) .namasenda.com >> d) .\namasenda\.com >> >> Is a) & b) correct, anyway? > > > No they are not. Neither is (d). > > (c) is correct dstdomain syntax but the "wildcard" is label-based, not > character-based. > Is it possible to have / write a label-based dstdomain syntax? How? > .namasenda.com will match namasenda.com, www.namasenda.com, > www2.namasenda.com, other.namasenda.com > but not: anamasenda.com or hatenamasenda.com or anything else with a > different 2nd-tier label than "namasenda". > > >> >> I want one squid instance to listen on ports 80, 8080, & 3128 >> Do I just do "http_port 3128 8080 80 transparent" work? > > > No, one line per http_port entry. > > Amos > Thank you, # Edmonds.
Re: [squid-users] slow reconfigure on squid3
On 07/04/2012 04:49 AM, Mr J Potter wrote: Hi all, thanks for your responses... versions - I use the standard ones with Debian squeeze (2.7.stable9 and 3.1.6) Yes there are lots of helpers - 25 NTLM helpers and 10 squiduguard helpers, so this could account for slow reconfig. I have seen a performance as low as 1 second per helper for a Squid process of approximately 4 GB. How large is your Squid process and how long does it take to start the helpers? Upgrading to 3.2 seems like a good bet - are there ready-rooled squid 3.2 debs available for Squeeze or do I have to make my own? We currently run squid in 3 different flavours of authentication - NTLM for PCs, ident for macs and digest for guest network, so have 3 distinct squid setups running on our proxy server. Would it be worth setting these all up as non-caching, then set up a parent caching server, or will setting them up as cache peers make them share their caches at all? This would work well, IF the "non-caching" Squid has a small memory footprint and needs all the helpers and the parent Squid has a large memory footprint and does not need helpers. Maybe the child can have a small memory cache instead of no cache. Squidguard also needs more resources than ufdbGuard since it uses 10 database caches and a database on disk (which is cached in the file system buffers) where ufdbGuard uses one copy of the URL database in its own memory. The database format of squidguard uses 2-4 times more bytes than the format of ufdbGuard reducing further the need for memory resources. Marcus cheers Jim UK On 2 July 2012 14:44, Marcus Kool wrote: Squid reconfigure can indeed take a long time. Especially when Squid uses lots of memory and starts helpers. Starting helpers takes a large amount of kernel resources when Squid is large, e.g. 2+ GB since it forks itself and replaces its copy by a new process. The fork can take a long time. If you use a URL rewritor you can easily have 24 or more of them and this makes 24 copies of a large process. How large is squid ? Can you post the output of ps -o pid,stime,sz,vsz,rss,args -C squid I wrote a test program to test the performance of forking X times a large process. I can post it if you are interested. Marcus On 07/02/2012 05:08 AM, Mr J Potter wrote: Hi all, Does anyone have any tips on how to fix this issue: We've just moved to squid3 from squid2, and now when we do squid3 -k reconfigure we get about 30 seconds of squid refusing/failing to forward requests while it rejigs itself. I don't know if this is squid3 rescanning the cache or doing something with squidguard (we have a fairly complex+large squidguard setup)? I don't think this happened with squid2. What can we do to make this less noticeable? - make it reconfigure faster? - multiple squid servers - can we do failover somehow (either proxy DNS record points to them both, or they automatically redirect (is this what cache peers are for?))? - go back to squid 2 - I didn't see any end user benefits of squid3 over squid2... any help would be greatly appreciated. thanks Jim Potter UK
[squid-users] acl to allow sites on SQL or LDAP
Hi all, I am administering 3 squid 3.0.STABLE19-1ubuntu0.2 proxies on 3 different sites, and managed to read group membership on LDAP using external_acl_type and squid_ldap_group without a problem. The last bit I need to make this a dream proxy cluster is also store the allowed sites on LDAP (preferably). I searched the net for something like this, but all I get is about user auth, nothing regarding allowed sites list. Can someone help me find the way for that, if any? Thanks in advance and best regards. -- Marcio Merlone
[squid-users] IP Address Change
I am running squid 3.19 on debian lenny. I have changed the ip address in /etc/network/interfaces. Majority of sites work. However, there are instances where squid responds to a workstation request with the old ip as return address. Where could squid still be getting the old ip from? What is there to do? Thanks in advance
[squid-users] Re: IP Address Change
sorry for the false alarm. As it turns out, it was calling java application which had old ip for proxy. Thanks On Wed, Jul 4, 2012 at 1:25 PM, Roman Gelfand wrote: > I am running squid 3.19 on debian lenny. I have changed the ip > address in /etc/network/interfaces. Majority of sites work. However, > there are instances where squid responds to a workstation request with > the old ip as return address. > > Where could squid still be getting the old ip from? What is there to do? > > Thanks in advance
Re: [squid-users] acl to allow sites on SQL or LDAP
On 7/4/2012 5:37 PM, Marcio Merlone wrote: Hi all, I am administering 3 squid 3.0.STABLE19-1ubuntu0.2 proxies on 3 different sites, and managed to read group membership on LDAP using external_acl_type and squid_ldap_group without a problem. The last bit I need to make this a dream proxy cluster is also store the allowed sites on LDAP (preferably). I searched the net for something like this, but all I get is about user auth, nothing regarding allowed sites list. Can someone help me find the way for that, if any? Thanks in advance and best regards. Hey there Marcio, squid is loading the acls\rules at startup or reconfiguring. there for using regular squid rules you can't use DB such as LDAP, mysql or any other DB.(there are other open options) i wouldn't recommend you to use LDAP as a DB for this kind of operation because it's pretty slow for it. the other options are: URL_REWRITE,ICAP,EXTERNAL_ACL. i wrote a nice ICAP server that was meant to do url manipulation but seems that it can do much more. it uses MYSQL as temp DB to store and retrieve specific data on urls for cache so it's MYSQL\PG\SQLITE\LDAP ready. i am working now on effective way to add filtering mechanism into it. i have basic model that works. this model should be the same for filtering or as ACLS, you will just need to change the destination page to any page you want like "porn is not available right now please try this later at home" or other nice pages you like. if you are willing to do the testings with me and built some skeleton for it to fit sysadmins i will be more then happy to work on it. the basic "domain" match is pretty simple to implement and it's kind of done already. the next thing to be done is the dstdomain ".example.dom" joker. about regex acls i will might use some other technique to load it from DB into memory and only when the DB changed to update the regex into memory. regex is a very slow acl and basically should be used wisely. talk with me Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
[squid-users] Re: transparent (intercepting?) without wccp, options?
> "Amos" == Amos Jeffries writes: > On 04.07.2012 15:54, Ezequiel Birman wrote: >>> "Eliezer" == Eliezer Croitoru writes: >> > >> http://wiki.squid-cache.org/ConfigExamples/Intercept/DebianWithRedirectorAndReporting >> >> > this is a good way to start but it wont be a transparent >> proxy but > a "nat" proxy but it can be good for your needs as >> anyway you have > nat in the RV042. >> >> Are you sure? The only mention to nat in is in order to redirect >> port 80 to 3128 on squid box. This is the intro: >> ... > Yes. There are 4 protocol layers involved. ebtables - rules stops > it being a bridge "transparent relay/proxy" and makes it routed > traffic. iptables - rules use NAT (interception proxy) instead of > TPROXY (transparent proxy). squid - config file uses > URL-rewriters to prevent Squid being a HTTP protocol "transparent > proxy" (HTTP definition of "transparent proxy" is the Squid > default behaviour). > There is a lot of people confused by the meaning of the word > "transparent". With good reason, it has been used out of context > so much. Where should I start then? Could yo point me to some doc, tutorial or config example to implement what Eliezer suggested? I mean beside the books which I didn't buy yet. Regards -- Ezequiel Birman
Re: [squid-users] Re: transparent (intercepting?) without wccp, options?
On 7/5/2012 4:48 AM, Ezequiel Birman wrote: "Amos" == Amos Jeffries writes: > On 04.07.2012 15:54, Ezequiel Birman wrote: >>> "Eliezer" == Eliezer Croitoru writes: >> > >> http://wiki.squid-cache.org/ConfigExamples/Intercept/DebianWithRedirectorAndReporting >> >> > this is a good way to start but it wont be a transparent >> proxy but > a "nat" proxy but it can be good for your needs as >> anyway you have > nat in the RV042. >> >> Are you sure? The only mention to nat in is in order to redirect >> port 80 to 3128 on squid box. This is the intro: >> ... > Yes. There are 4 protocol layers involved. ebtables - rules stops > it being a bridge "transparent relay/proxy" and makes it routed > traffic. iptables - rules use NAT (interception proxy) instead of > TPROXY (transparent proxy). squid - config file uses > URL-rewriters to prevent Squid being a HTTP protocol "transparent > proxy" (HTTP definition of "transparent proxy" is the Squid > default behaviour). > There is a lot of people confused by the meaning of the word > "transparent". With good reason, it has been used out of context > so much. Where should I start then? Could yo point me to some doc, tutorial or config example to implement what Eliezer suggested? I mean beside the books which I didn't buy yet. Regards dont worry! i dont know anyone that masters linux and got it all from books he didnt bout :) it's pretty simple to implement as long you do understand the concepts. you will just need to practice and see how all of it actually fits together as a puzzle. start with a bridge interface and bridge tools. it depends on what linux distro you are using. debian is a nice and simple one. you need to install the bridge tools + ebtables and configure the bridge interface for two Ethernet interfaces. the next step is to add the bridge interface ip address and default route. all the above can be done in the /etc/...somewhere this link: http://wiki.debian.org/BridgeNetworkConnections#Libvirt_and_bridging can help you a bit. on debian it will work just like that.. config.. apply settings.. connect one cable .. connect second cable ...done. after that you can install\compile squid3.1 will be here to help if you need something. Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il
[squid-users] Windows build machine
Hi, some time ago, I donated a windows vm for squid dev team to use as a windows build machine, just want to confirm it is still in use, otherwise I plan to decommission it. please let me know! Cheers.
[squid-users] Re: Windows build machine
On 5/07/2012 3:55 p.m., Yucong Sun (叶雨飞) wrote: Hi, some time ago, I donated a windows vm for squid dev team to use as a windows build machine, just want to confirm it is still in use, otherwise I plan to decommission it. please let me know! Cheers. Thank you for that machine. It was helpful. I'm not sure what the current status on need is, other than all the windows machines needing a bit of update before we can continue tests. I have cc'd Kinkie (Francesco) who has been managing the testing machines to see if he can clarify if we still need it. Amos Jeffries Squid Project
