Re: [squid-users] high memory usage (squid 3.2.0)
Il 10/04/2013 17:22, Mr Dash Four ha scritto: Marcello Romani wrote: Il 10/04/2013 13:59, Mr Dash Four ha scritto: Marcello Romani wrote: Il 09/04/2013 19:33, Mr Dash Four ha scritto: > [snip] if the maximum_object_size_in_memory is reduced, then I suppose squid's memory footprint will have to go down too, which makes the cache_mem option a bit useless. I think will just store more objects in RAM. I am sorry, but I don't understand that logic. If I set cache_mem (which is supposed to be the limit of ram squid is going to use for caching), then the maximum_object_size_in_memory should be irrelevant. The *number* of objects to be placed in memory should depend on cache_mem, not the other way around. You're wrong. Each object that squid puts into cache_mem can have a different size. Thus the number of objects stored in cache_mem will vary over time depending on the traffic and selection algorithms. >> I don't see how I am wrong in what I've posted above. You wrote: "if the maximum_object_size_in_memory is reduced, then I suppose squid's memory footprint will have to go down too, which makes the cache_mem option a bit useless." (Perhaps you should've written: which *would make* the cache_mem option a bit useless.) I haven't made real-life measurements to test how maximum_object_size_in_memory affects squid memory footprint, but my feeling is that lowering it woud *not* decrease memory usage. I would expect instead an *increase* in total memory consumption because more objects in cache_mem would mean more memory used for the indexes needed to manage them. > I am not saying that the number of objects placed in ram will be constant, all I am saying is that the total memory used of all objects placed in ram should not be 6 times the cache_mem value I've specified in my configuration file - that is simply wrong, no matter how you twist it. What currently seems to happen is that cache_mem is completely ignored and squid is trying to shove up as many objects into my ram as possible, to the point where nothing else on that machine is able to function nominally. This is like putting cart in front of the horse - ridiculous! As stated elsewhere, previous versions of squid had memory leaks. That doesn't mean squid is _designed_ to put as many objects in ram as possible. Well, as I indicated previously, my cache_mem is 200MB. Current memory usage of squid was 1.3GB - more than 6 times what I have indicated. That is not a simple memory "leak" - that is one hell of a raging torrent if you ask me! Also, the cache_mem value must not be confused with a hard limit on total squid memory usage (which AFAIK cannot be set). For example there's also the memory used to manage the on-disk cache (10MB per GB IIRC - google it for a reliable answer). Even if we account for that, I don't see why squid should be occupying 6 times more memory from what I restricted it to use. This is what the official squid wiki has to say about this ratio: "rule of thumb: cache_mem is usually one third of the total memory consumption." But you see... it's just a "rule of thumb". Squid uses additional memory to manage on-disk cache. Again, from the squid memory page: "10 MB of memory per 1 GB on disk for 32-bit Squid 14 MB of memory per 1 GB on disk for 64-bit Squid" So if you have a very large on-disk cache but specify a low cache_mem parameter, the 6:1 ratio can be easily exceeded. Suppose you specify cache_mem 32MB and have a 40GB cache_dir. That would give (at least) 32MB + 40GB / 1GB * 10MB = 432MB. 432 / 32 = 13.5 I'm not saying this would be a sensible configuration, nor denying there's an actual problem in your case. Plus, I'm not claiming I would be able to predict a squid instance memory usage (I prefer to graph that over time with munin). It's just that IMVHO you're barking at the wrong tree :-) -- Marcello Romani
Re: [squid-users] HTML Realtime Report SqStat in SQUID
> When I do I access http://192.168.0.1/ Apache page appears, then I think that > is missing in some additional configuration may be due SQUID or modify any > part of the SqStat PHP code. > > First described above ask for your kind help to correct my problem. Hi Daniel, On the SqStat page, I see: " Copy file config.inc.php.defaults to config.inc.php, edit config.inc.php to specify your squid proxy server IP and port. " If that doesn't work, you probably want to contact SqStat's author. -- /kinkie
Re: [squid-users] squid-internal-mgr not found - cannot login to cachemgr
On Thu, Apr 11, 2013 at 2:28 AM, brendan kearney wrote: > resending because i got a mailer-daemon failure for HTML formatting... > > all, > > i am running squid 3.2.5 on fedora 16 64 bit on two separate boxes, > load balanced with HA Proxy. i am trying to access cachemgr on either > one of the squid instances, and both exhibit the behaviour where the > squid-internal-mgr URI is not found. attempts to login via the HA > Proxy VIP as well as with no proxy configured (direct access) have > been tried. both ways produce the same error. below is some header > info: > > http://192.168.25.1/squid-internal-mgr/ [...] > # > - > # TAG: http_port > http_port 192.168.25.1:3128 [...] > can anyone tell my why i am not able to get logged into the cachemgr? > the page presents, but the login fails. cachemgr.conf has the IP of > both proxies listed, and /etc/httpd/conf.d/squid.conf has the right > access allowed by network. /usr/lib64/squid/cachemgr.cgi is chmod'd > 755 (rwxr-xr-x) and is chown'd root:root. Hi Brendan, the reason is that Squid is listening on port 3128, you're connecting to the Apache server listening on port 80. -- /kinkie
Re: [squid-users] high memory usage (squid 3.2.0)
On 11/04/2013 7:11 p.m., Marcello Romani wrote: Il 10/04/2013 17:22, Mr Dash Four ha scritto: Marcello Romani wrote: Il 10/04/2013 13:59, Mr Dash Four ha scritto: Marcello Romani wrote: Il 09/04/2013 19:33, Mr Dash Four ha scritto: > [snip] if the maximum_object_size_in_memory is reduced, then I suppose squid's memory footprint will have to go down too, which makes the cache_mem option a bit useless. I think will just store more objects in RAM. I am sorry, but I don't understand that logic. If I set cache_mem (which is supposed to be the limit of ram squid is going to use for caching), then the maximum_object_size_in_memory should be irrelevant. The *number* of objects to be placed in memory should depend on cache_mem, not the other way around. You're wrong. Each object that squid puts into cache_mem can have a different size. Thus the number of objects stored in cache_mem will vary over time depending on the traffic and selection algorithms. >> I don't see how I am wrong in what I've posted above. You wrote: "if the maximum_object_size_in_memory is reduced, then I suppose squid's memory footprint will have to go down too, which makes the cache_mem option a bit useless." (Perhaps you should've written: which *would make* the cache_mem option a bit useless.) I haven't made real-life measurements to test how maximum_object_size_in_memory affects squid memory footprint, but my feeling is that lowering it woud *not* decrease memory usage. I would expect instead an *increase* in total memory consumption because more objects in cache_mem would mean more memory used for the indexes needed to manage them. Smaller maximum_object_size_in_memory *raises* the memory usage. As the cache_mem is the fixed baseline amount of memory used for RAM cache, and each object uses a indexing memory structure as well. Limiting memory to storing smaller objects means more of them fit into the same fixed-size cache_mem and thus require more index entries ... with more overheads. > I am not saying that the number of objects placed in ram will be constant, all I am saying is that the total memory used of all objects placed in ram should not be 6 times the cache_mem value I've specified in my configuration file - that is simply wrong, no matter how you twist it. What currently seems to happen is that cache_mem is completely ignored and squid is trying to shove up as many objects into my ram as possible, to the point where nothing else on that machine is able to function nominally. This is like putting cart in front of the horse - ridiculous! As stated elsewhere, previous versions of squid had memory leaks. That doesn't mean squid is _designed_ to put as many objects in ram as possible. Well, as I indicated previously, my cache_mem is 200MB. Current memory usage of squid was 1.3GB - more than 6 times what I have indicated. That is not a simple memory "leak" - that is one hell of a raging torrent if you ask me! I agree. Unfortunately there were a few of those introduced in the 3.2 development code. They are now polished out by the QA process. Mr Dash Four, you said 3.2.0 version? We never released a three-numeric version ending in '0', because the '0' set are the development beta releases, there is always a fourth numeric indicatign what beta release it is. QA process is only completed and thing like memory leaks expected to be removed when we reached 3.2.1 release. Also, the cache_mem value must not be confused with a hard limit on total squid memory usage (which AFAIK cannot be set). For example there's also the memory used to manage the on-disk cache (10MB per GB IIRC - google it for a reliable answer). Even if we account for that, I don't see why squid should be occupying 6 times more memory from what I restricted it to use. This is what the official squid wiki has to say about this ratio: "rule of thumb: cache_mem is usually one third of the total memory consumption." But you see... it's just a "rule of thumb". Squid uses additional memory to manage on-disk cache. Again, from the squid memory page: "10 MB of memory per 1 GB on disk for 32-bit Squid 14 MB of memory per 1 GB on disk for 64-bit Squid" So if you have a very large on-disk cache but specify a low cache_mem parameter, the 6:1 ratio can be easily exceeded. Also Squid requires up to 256 KB per client transaction (averages around 16KB per FD). So at 200MB with 4000 clients 1.3 GB can also easily be reached despite everything else memory-related being disabled. I think this is what Alex was referring to earlier when he mentioned the problem may still exist (and not be a leak) even if you elimiated caching as a source of the problem. No need to be a developer to test this, just add "cache deny all" and "cache_mem 0" to the config file and see if the memor usage remains. Suppose you specify cache_mem 32MB and have a 40GB cache_dir. That would give (at least) 32MB + 40GB / 1GB * 10MB = 432M
Re: [squid-users] redirecting
On 10/04/2013 11:59 p.m., folkert wrote: Hi, I'm trying to get the following to work: - all requests must go to the redirect_program.z after the request went through the redirect program: - all requests must be forwarder to proxy A - UNLESS they are to 172.19.0.0/16 or listed in /usr/local/etc/squid3-directurls.txt - if they are in /usr/local/etc/squid3-directurls.txt they must be forwarded to proxy B - if they are to 172.19.0.0/16 they must go to directly to the destination (172.19.0.1:8008 mostly) I tried the following: acl ir_ah dst 172.19.0.0/16 redirector_access deny ir_ah BUT your policy statement said ... "all requests must go to the redirect_program." redirect_program /usr/local/bin/ir always_direct allow ir_ah always_direct deny all "if they are to 172.19.0.0/16 they must go to directly to the destination " Okay. That is *kind* of working. If any one of teh URL domains IP addresses is in the 172.19.0.0/16 range the request will go direct even if the IP picked is not in that range. So be Careful. cache_peer A parent 8123 0 default cache_peer B parent 8080 0 acl allowed_servers dstdomain "/usr/local/etc/squid3-directurls.txt" cache_peer_access A deny allowed_servers cache_peer_access A deny ir_ah cache_peer_access A allow all "all requests must be forwarder to proxy A - UNLESS they are to 172.19.0.0/16 or listed in /usr/local/etc/squid3-directurls.txt " OKAY. That is configured. cache_peer_access B deny ir_ah cache_peer_access B allow all "if they are in /usr/local/etc/squid3-directurls.txt they must be forwarded to proxy B " BROKEN. All the config says is any request which does not match ir_ah will be directed to proxy B. You need that to be: cache_peer_access B allow allowed_servers cache_peer_access B deny all What works: - all request initially go to the redirect_program, it then allows them to be processed - BUT: after the redirect_program acks them, it sends the user back to the redirect_program! What I see is that Squid sends requests to 172.19.0.1 via its redirector while it should go straight through to 172.19.0.1. How can I fix this? Um, this matches your policy statements about what you wanted to happen. But does not match what the config says should be happening. In order for this to happen the "dst" ACL when applied to the client original URL domain needs to be producing something other than a 172.19.0.0/16 IP address. Note that this ACL does not match against the output of the redirector. I'm not sure what needs to be done to make it more predictable. Avoiding "dst" type when possible is a good idea. The other thing to look at is *how* the redirector is "ACKing" the URL. A proper redirect is just a referral to another location, the client will followup with another request. This could be what you are seeing. Otherwise the only way I can think of you could get double helper lookups like this is forwarding loops somewhere causing the transaction to re-enter Squid. Amos
Re: [squid-users] Order of authentication schemes in Proxy-Authenticate
On 10/04/2013 4:23 p.m., Alan wrote: Is there any way to influence the order in which Squid sends the Proxy-Authenticate headers to the client? I already tried changing the order in the config file to no avail. That was the way to do it. Please test carefully and IF you have solid evidence of Squid disobeying the config order please open a bug report about it. Background: I have a squid 3.3.3 proxy using both kerberos and radius. A capture shows it offers both Basic and Negotiate authentication schemes, in separate headers and in that order. IE seems to try Negotiate first and Basic later, disregarding the order in which the headers appear. Firefox seems to be trying in the same order as the headers appear (I haven't confirmed that changing the order would fix this). The RFC doesn't seem to mention anything about which one should be tried first, so both approaches seem reasonable. I haven't been able to find any configuration option in Firefox to change the order either. This should clarify for you what is going on: http://wiki.squid-cache.org/action/show/Features/Authentication#Can_I_use_different_authentication_mechanisms_together.3F The order Squid sends the headers is *supposed* to be irrelevant. Any browser following that order is buggy. Amos
Re: [squid-users] squid 3.x expected max throughput
On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: I was aware of that page. As you said, it's often RPS so it's not relevant for me. It is more relevant than you seem to think. Squid processing overheads are tied tightly to the request parsing and ACL testing processes. These are relatively fixed overheads for each request regardless of the request size. IIRC the 50Mbps traffic was on average HTTP request traffic objects in a medium sized ISP - we do not have any good detaisl on what that average was though. With very little extra overheads the same proxy on same hardware might also reach 100Mbps. All that is required is the avg object size to go from, for example, 16KB to 32KB which are both within the bounds of "average HTTP traffic" sizes (11-63KB). Amos Youssef On Apr 10, 2013, at 2:02 PM, Kinkie wrote: You probably want to check http://wiki.squid-cache.org/KnowledgeBase/Benchmarks Unfortunately the benchmarks reported are often expressed as RPS and not bandwidth. On Wed, Apr 10, 2013 at 1:21 PM, Youssef Ghorbal wrote: On Apr 10, 2013, at 10:37 AM, Kinkie wrote: On Tue, Apr 9, 2013 at 11:35 PM, Youssef Ghorbal wrote: Hello, Is there any recent figures about max throughput to be expected from a squid 3.x install (on recent hardware) in the scenario of a single stream downloading a large file (> 1GB) (read not cacheable) I'm aware that's not a performance metric per se, but it's one of the scenarios we have to deal with. Few weeks ago, Amos talked about 50Mb/s (client + server) for a squid 3.1 Hi, 50mb/s seems a very conservative estimate to me; in that scenario Squid is essentially acting as a network pipe. Assuming this is a lab (and we can thus ignore bandwidth and latency on the internet link), the expectation is that this kind of scenario for squid will be CPU and network I/O bound, so in order to give any sensible answer we'd need to know what kind of network interface you would use (fast-ethernet? Giga-ethernet copper? Giga-ethernet fiber? Even faster?), what kind of CPU and what kind of system architecture (server-class? pc-class? virtual?) I'm agree, it's a LAB and we can ignore bandwidth and latency of Internet link. And it's exactly what you said it's a scenario where Squid is essentially acting as a network pipe. Here is a summary of the hardware : - 1Gb ethernet NIC (copper) : the same of client and server traffic. - 16GB RAM - 2 CPUs Quad Core (Xeon E5420 2.5Ghz per core) : I think that the many cores are not relevent since a single stream will eventually be handled by a single core. - FreeBSD 8.3 amd64 What I'm seeing right now is ~50Mb/s on 3.1 release (as Amos said earlier) which seems very conservative estimate to me too, and I was seeking infos on what can be expected in a perfect world. If it's the current best figures I can get, that's fine and I'll not be looking to optmise anything else. If it can do a lot better (let's say 10 times better) I'll try to investgate time to reach this (upgrade to the last release, tune the configuration, tune the system etc) Youssef
Re: [squid-users] not working tproxy in squid 3.2
On Tue, Apr 02, 2013 at 12:52:58AM +1300, Amos Jeffries wrote: > On 1/04/2013 7:40 p.m., Oleg wrote: > In your case with kernel limits of 800MB per-process this config > will guarantee it gets killed quickly. No memory leak required: > > cache_mem 900 MB > > From your config I see Squid is using its default 256 MB of > cache_mem. So you should expect to see at least 300MB of Squid RAM > usage normally. I'm ready for 300MB, but i'm not ready for 800MB. > The difference between 6 and 7 is the kernel version. Some Kernels > are known to have TPROXY bugs. > Also, the Debian kernels had non-working TPROXY for many releases > after the apparently identical upstream kernel version was working > very well. This affects Debian 6 for certain I'm not sure about 7. This is not an issue for us - we use a custom 3.2.38 kernel. > >>modprobe xt_MARK > >FATAL: Module xt_MARK not found. > > I would guess this is related to the problem. > > Theory: without MARK support in the kernel the TPROXY connections > are looping through Squid until the active connection state consumes > 800MB and gets killed. > Can you verify that at all? What kernel config option is responsible for this? # grep MARK /boot/config-3.2.38-my CONFIG_NETWORK_SECMARK=y CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_CONNTRACK_SECMARK=y CONFIG_NETFILTER_XT_MARK=m CONFIG_NETFILTER_XT_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m CONFIG_NETFILTER_XT_TARGET_MARK=m CONFIG_NETFILTER_XT_TARGET_SECMARK=m CONFIG_NETFILTER_XT_MATCH_CONNMARK=m CONFIG_NETFILTER_XT_MATCH_MARK=m CONFIG_NET_SCH_DSMARK=m CONFIG_CLS_U32_MARK=y Now, we stay at squid 3.1.20 from Debian 7. And, as before, we see tcp packets, but client browser doesn't open any site (may be packets are broken?) tcpdump of one http request (10.232.194.5 - client): 16:12:57.971120 IP 10.232.194.5.3733 > 87.251.132.181.80: Flags [S], seq 1252681145, win 65535, options [mss 1348,nop,nop,sackOK], length 0 16:12:57.971165 IP 87.251.132.181.80 > 10.232.194.5.3733: Flags [S.], seq 2610504694, ack 1252681146, win 14600, options [mss 1460,nop,nop,sackOK], length 0 16:12:57.971569 IP 10.232.194.5.3734 > 87.251.132.181.80: Flags [S], seq 3247035523, win 65535, options [mss 1348,nop,nop,sackOK], length 0 16:12:57.971608 IP 87.251.132.181.80 > 10.232.194.5.3734: Flags [S.], seq 901187601, ack 3247035524, win 14600, options [mss 1460,nop,nop,sackOK], length 0 16:12:57.973064 IP 10.232.194.5.3733 > 87.251.132.181.80: Flags [.], ack 1, win 65535, length 0 16:12:57.973195 IP 87.251.132.181.80 > 10.232.194.5.3733: Flags [F.], seq 1, ack 1, win 14600, length 0 16:12:57.973379 IP 10.232.194.5.3734 > 87.251.132.181.80: Flags [.], ack 1, win 65535, length 0 16:12:57.973458 IP 87.251.132.181.80 > 10.232.194.5.3734: Flags [F.], seq 1, ack 1, win 14600, length 0 16:12:57.975361 IP 10.232.194.5.3733 > 87.251.132.181.80: Flags [P.], seq 1:301, ack 1, win 65535, length 300 16:12:57.975388 IP 87.251.132.181.80 > 10.232.194.5.3733: Flags [R], seq 2610504695, win 0, length 0 16:12:57.975396 IP 10.232.194.5.3733 > 87.251.132.181.80: Flags [.], ack 2, win 65535, length 0 16:12:57.975409 IP 87.251.132.181.80 > 10.232.194.5.3733: Flags [R], seq 2610504696, win 0, length 0 16:12:57.975612 IP 10.232.194.5.3734 > 87.251.132.181.80: Flags [.], ack 2, win 65535, length 0 16:12:57.977060 IP 10.232.194.5.3734 > 87.251.132.181.80: Flags [F.], seq 1, ack 2, win 65535, length 0 16:12:57.977085 IP 87.251.132.181.80 > 10.232.194.5.3734: Flags [.], ack 2, win 14600, length 0 16:12:58.004864 IP 10.232.194.5.3735 > 87.251.132.181.80: Flags [S], seq 641201190, win 65535, options [mss 1348,nop,nop,sackOK], length 0 16:12:58.004897 IP 87.251.132.181.80 > 10.232.194.5.3735: Flags [S.], seq 2722793776, ack 641201191, win 14600, options [mss 1460,nop,nop,sackOK], length 0 16:12:58.014947 IP 10.232.194.5.3735 > 87.251.132.181.80: Flags [.], ack 1, win 65535, length 0 16:12:58.015059 IP 87.251.132.181.80 > 10.232.194.5.3735: Flags [F.], seq 1, ack 1, win 14600, length 0 16:12:58.016445 IP 10.232.194.5.3735 > 87.251.132.181.80: Flags [.], ack 2, win 65535, length 0 16:12:58.196105 IP 10.232.194.5.3735 > 87.251.132.181.80: Flags [P.], seq 1:301, ack 2, win 65535, length 300 16:12:58.196133 IP 87.251.132.181.80 > 10.232.194.5.3735: Flags [R], seq 2722793778, win 0, length 0
Re: [squid-users] squid 3.x expected max throughput
On Apr 11, 2013, at 12:13 PM, Amos Jeffries wrote: > On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: >> I was aware of that page. >> As you said, it's often RPS so it's not relevant for me. > > It is more relevant than you seem to think. Squid processing overheads are > tied tightly to the request parsing and ACL testing processes. These are > relatively fixed overheads for each request regardless of the request size. Thank you for the explanation. It confirms what I was expecting myself. What I meant, is that in my case I have ONE big object (the infamous >1G file) and 1 client and I want to know what to expect on the troughput point of view. As kinkie said, in my scenario, Squid is basically a network pipe. the overhead (ACL parsing etc is done one time) Maybe, the question here is how does Squid handle data buffers, or at which rate does it perform server reads on the one side and to client writes on the other. > IIRC the 50Mbps traffic was on average HTTP request traffic objects in a > medium sized ISP - we do not have any good detaisl on what that average was > though. With very little extra overheads the same proxy on same hardware > might also reach 100Mbps. All that is required is the avg object size to go > from, for example, 16KB to 32KB which are both within the bounds of "average > HTTP traffic" sizes (11-63KB). Yousef
Re: [squid-users] Local Squid to Reverse Squid to keyserver.ubuntu.com
On 8/04/2013 4:24 a.m., Christopher H. Laco wrote: Ok, I've solved this as much as I need too without digging into the squid source itself. I fired up tcpdump and took a capture of the failed attempt from the proxy to the keyserver using 3.1.19, then a capture of the successful attempt from the proxy to the keyserver using 3.3.3 (sorry for the dots, pulled directly from tcpdump -XX) Failed: GET./.HTTP/1.1.. User-Agent:.curl/7.22.0.(86_64-pc-linux-gnu).libcurl/7.22.0.OpenSSL/1.0.1.zlib/1.2.3.4.libidn/1.23.librtmp/2.3.. Host:.keyserver.ubuntu.com.. Accept:.*/*.. Via:.1.1.localhost.(squid/3.1.19).. X-Forwarded-For:.10.10.10.20. Cache-Control:.max-age=259200.. Connection:.keep-alive Success: GET./.HTTP/1.1.. User-Agent:.curl/7.22.0.(x86_64-pc-linux-gnu).libcurl/7.22.0.OpenSSL/1.0.1.zlib/1.2.3.4.libidn/1.23.librtmp/2.3.. Host:.keyserver.ubuntu.com.. Accept:.*/*.. Via:.1.1.opencenter-proxy.(squid/3.3.3).. X-Forwarded-For:.10.10.10.20.. Cache-Control:.max-age=259200.. Connection:.keep-alive The only difference in the request is the hostname/version in Via The difference in the response is that for the Failure, the remote keyserver is actually the one returning the Squid Access Denied page. That wasn't apparent since my local proxy default is also "localhost". 14:59:10.690245 IP cassava.canonical.com.http > 10.0.2.15.52015: Flags [P.], seq 1:1389, ack 293, win 65535, length 1388 0x: 0800 2788 0ca6 5254 0012 3502 0800 4500 ..'...RT..5...E. 0x0010: 0594 6563 4006 4dfe 5bbd 5a37 0a00 ..ec..@.M.[.Z7.. 0x0020: 020f 0050 cb2f 1205 4802 3a61 994a 5018 ...P./..H.:a.JP. 0x0030: 0054 4854 5450 2f31 2e30 2034 ...T..HTTP/1.0.4 0x0040: 3033 2046 6f72 6269 6464 656e 0d0a 5365 03.Forbidden..Se 0x0050: 7276 6572 3a20 7371 7569 642f 332e 312e rver:.squid/3.1. 0x0060: 3139 0d0a 4d69 6d65 2d56 6572 7369 6f6e 19..Mime-Version 0x0070: 3a20 312e 300d 0a44 6174 653a 2053 756e :.1.0..Date:.Sun 0x0080: 2c20 3037 2041 7072 2032 3031 3320 3134 ,.07.Apr.2013.14 0x0090: 3a35 393a 3130 2047 4d54 0d0a 436f 6e74 :59:10.GMT..Cont 0x00a0: 656e 742d 5479 7065 3a20 7465 7874 2f68 ent-Type:.text/h 0x00b0: 746d 6c0d 0a43 6f6e 7465 6e74 2d4c 656e tml..Content-Len 0x00c0: 6774 683a 2033 3430 380d 0a58 2d53 7175 gth:.3408..X-Squ 0x00d0: 6964 2d45 7272 6f72 3a20 4552 525f 4143 id-Error:.ERR_AC 0x00e0: 4345 5353 5f44 454e 4945 4420 300d 0a56 CESS_DENIED.0..V At this point, I changed visible_hostname to something other than localhost, and now the requests through 3.1.19 work as expected. So, with that said, it seems that this problem is some form of logic issue within the 3.1.19 codebase where if the source and upstream proxies are named the same (and the same version*), shenanigans ensue with the upstream proxy response. Maybe this is a setting somewhere I don't know about or some form of circular processing protection gone wrong. What you are looking at is the feature we call "forwarding loop detection". The Via: header is scanned by each proxy for entries of *its own* (supposedly unique) combination of HTTP protocol support level, machine hostname (unique all by itself when one follows the RFCs), proxy software version, and some extra fluff header octets to prevent false matches. As you found when all three of these details are exactly lining up ... including the setting of your proxies hostname to that of the remote upstream proxy (uhm, "localhost"). The upstream *will* reject your traffic to protect itself against abuse. NP: For anyone reading this. If you are an admin needing to share visible_hostname values between multiple proxies in a hierarchy look at the unique_hostname directive. *Now, even more interesting, if I change visible_hostname to localhost in the 3.3.3 version and make the request to the upstream keyserver, things also work. Maybe this is just an issue with the 3.1.x codebase as it was. I might test this local with a few test proxy servers and various visible_hostname incantations. To recap: - 3.1.19 visible_hostname defaults to localhost. always change it. :-) Well, no and maybe... it defaults to localhost only if gethostname() fails along with an rDNS lookup on the resulting name. This is true for all Squid right back to Squid-1.1 AFAIK. What differs is the amount of bugginess in the gethostname() and rDNS lookup logics - 3.1 had a few intolerances which made localhost appear more often than later releases. - 3.3.3 visible_hostname defaults to gethosename() - 3.1.19 connecting to an upstream 3.1.19, both with the same name "localhost" returns access denied. - keyserver.ubuntu.com should probably fix it's visible_hostname setting. Sadly they are *far* from alone in doing so ... Amos
Re: [squid-users] squid 3.x expected max throughput
On 11/04/2013 10:48 p.m., Youssef Ghorbal wrote: On Apr 11, 2013, at 12:13 PM, Amos Jeffries wrote: On 11/04/2013 12:23 a.m., Youssef Ghorbal wrote: I was aware of that page. As you said, it's often RPS so it's not relevant for me. It is more relevant than you seem to think. Squid processing overheads are tied tightly to the request parsing and ACL testing processes. These are relatively fixed overheads for each request regardless of the request size. Thank you for the explanation. It confirms what I was expecting myself. What I meant, is that in my case I have ONE big object (the infamous >1G file) and 1 client and I want to know what to expect on the troughput point of view. As kinkie said, in my scenario, Squid is basically a network pipe. the overhead (ACL parsing etc is done one time) Maybe, the question here is how does Squid handle data buffers, or at which rate does it perform server reads on the one side and to client writes on the other. Well, in absence of any other loading it would be one read per packet just like you (or was it someone earlier) are seeing. With an internal event call passing the buffered data through Squid to write the same amount on the output. The "select" loop speed varies with a maximum loop time held around 10ms and decreasing if packets arrive faster. You will have to try to follow the code to understand the buffers unfortunately. I'm still learning thet part myself. For CONNECT requests its easy, the input buffer is the output buffer - one fread() causes fwrite() on the same buffer. For other requests is varies as Store systems are also involved which may (or may not) cause a file I/O to occur in the middle or a bufffer copy (I'm not sure here). Amos IIRC the 50Mbps traffic was on average HTTP request traffic objects in a medium sized ISP - we do not have any good detaisl on what that average was though. With very little extra overheads the same proxy on same hardware might also reach 100Mbps. All that is required is the avg object size to go from, for example, 16KB to 32KB which are both within the bounds of "average HTTP traffic" sizes (11-63KB). Yousef
Re: [squid-users] Issue related to using Squid 3.1 or 3.29 and accessing a site that uses a recursive DNS record. (30 seconds to bring up site)
On 11/04/2013 1:52 a.m., Duncan, Brian M. wrote: I opened a bug report on this today. Thanks One last thing to check is what happens when you query for records on that domain. That is the major change between 3.1 and 3.2 DNS. What I get is: ## time host -t A webapps.kattenlaw.com webapps.kattenlaw.com has address 38.98.128.19 0.000u 0.004s 0:00.06 0.0% 0+0k 0+0io 0pf+0w ## time host -t webapps.kattenlaw.com ;; connection timed out; no servers could be reached 0.000u 0.004s 0:12.03 0.0% 0+0k 0+0io 0pf+0w Which indicates that dns_timeout directive in Squid can be reduced to workaround that server brokenness on their end to some degree. Amos
[squid-users] Re: peer-cache question
Fine, at least some progress. Now you should upgrade your squid and verify, that http forwarding still works. http://wiki.squid-cache.org/Features/BumpSslServerFirst is based on squid 3.3 But you have 3.1 You should not use any other wiki/info as directly mentioned here in the forum. Because the developers are here from time to time, too. So you will get up-to-date info. And from most knowledgable source. As a step-by-step procedure, I would first to try to set up explicit (non-transparent) SSL-proxy with squid, to be accessed because of definition in the browsers proxy settings. So it would mean, the google-proxy must be SSL-capable first and only. This should be the simplest possible https-proxy config. In case, then you setup a proxy.pac for the browser, this would be a simple, alternate solution regarding splitting up the traffic to different squids. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/peer-cache-question-tp4659419p4659499.html Sent from the Squid - Users mailing list archive at Nabble.com.
[squid-users] Re: Order of authentication schemes in Proxy-Authenticate
There is a bug in new FF regarding kerberos auth. May be, that matters: https://bugzilla.mozilla.org/show_bug.cgi?id=857291 Regarding: http://wiki.squid-cache.org/action/show/Features/Authentication#Can_I_use_different_authentication_mechanisms_together.3F It states: >Due to a bug in common User-Agents (most notably Microsoft Internet Explorer) the order the auth-schemes are configured is relevant. RFC 2617, chapter 4.6, states: A user agent MUST choose to use the strongest auth-scheme it understands. Microsoft Internet Explorer instead chooses the first authe-scheme (in the order they are offered) it understands < However, AFAIK, this NOT correct (any more) regarding Internet Explorer. Starter of this thread confirmes this. Pls, update. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Order-of-authentication-schemes-in-Proxy-Authenticate-tp4659436p4659501.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] high memory usage (squid 3.2.0)
On 04/10/2013 03:37 PM, Mr Dash Four wrote: > Alex Rousskov wrote: >> The total memory used by Squid is, roughly: >> >>1) initial memory used by Squid shortly after start + >>2) memory used to store memory-cached objects + >>3a) memory used to index memory-cached objects + >>3b) memory used to index disk-cached objects + >>4) memory used for current transactions + >>5) memory leaked since Squid start >> >> cache_mem determines the maximum value of (2) >> >> cache_mem and mean object size in memory cache determine the maximum >> value of (3a) because they determine the maximum number of objects in >> the memory cache. >> >> cache_mem has no significant direct effect on the other four components. >> Depending on the local circumstances, every other component alone may >> exceed cache_mem 5 times or more, especially when cache_mem is >> relatively small, e.g., 200MB. And taken together, they can exceed >> 6*cache_mem level even easier. >> >> For example, a 10KB/s memory leak will consume 1GB of RAM in less than >> two days. Similarly, an overloaded Squid may grow (4) at rates exceeding >> 1MB/s. > Well, in that case, whoever did the testing of squid before releasing > such a buggy software should be shot on site or be locked in a dark > garage 24/7 somewhere and taught how to use valgrind. FWIW, I recommend avoiding non-constructive and negative comments like this if your primary objective is to get help solving your problem. Such comments reduce the pool of people willing to help you. Find other ways or other forums to vent your anger, frustration, or dissatisfaction. My examples simply demonstrate how Squid memory management works. I am trying to give you correct information about Squid so that you can correct your assumptions and it becomes easier for you to troubleshoot problems. Nothing else. >> Now back to your actual problem. An idle Squid with 200 MB memory cache >> and 200 MB disk cache should not use 1.3GB of RAM unless component (1) >> is very high due to some huge ACL lists or other unusual complications. >> If your (1) is just a few MBs and your idle Squid uses 1.3GB, then I >> would suspect a memory leak. > At start, squid memory consumption is about 25-30MB - give or take a few > megabytes, keeping in mind that according to the logs I have in excess > of 14500 objects (verified by squid at startup) in the disk cache. This sound fine. How much does your _idle_ Squid consume after handling traffic for a while (e.g., filling memory cache and then running under load for a little bit more)? Why am I asking about idle Squid? Because idle Squid does not have component (4), which often dominates (especially if Squid is overloaded) and which is difficult to measure otherwise. >> If you want to reduce Squid memory usage, I recommend eliminating memory >> leaks as suspects first because they are bugs and because they may >> consume all available RAM on your machine. > I am not a squid developer - I am a user. Sure, but you do not have to be a developer to detect and eliminate memory leaks. In most cases, you just need to follow instructions, preferably without attacking software and people who are trying to help you. The best first step is to upgrade to a supported Squid version. The second step could be to measure memory consumption of an idle Squid after it handles traffic. Producing a "memory consumption and load versus time" graph often helps. More steps may be required, depending on the outcomes of the first ones. > I'll try to see if the latest version has the above behaviour and if > that turns out to be the case and I have found no solution the the > problem I described in my initial post, then I have to just forget that > squid exists and use something else which is more reliable, like polipo > for example. While this is a reasonable plan if you do not need any Squid-specific features, there is no need to pummel Squid while you are trying to get Squid help. If you must vent about Squid, you can do that on the polipo mailing list when you run into problems with that software (but I would not recommend that either). Cheers, Alex.
[squid-users] high traffic with google
Hi, we are handling a rather large network ( ~140Kusers ) and we use one unique public IP address for internet traffic. This lead google to get suspicious with us ( captcha with each search ) Do you know if google can whitelist us in some way? where to contact them? any way to smartly bypass this behavior? Thanks Alex
Re: [squid-users] Need help on SSL bump and certificate chain
Hi Guy, We want to be a man-in-the middle but we want to get the approval from clients/end-users out of band by accepting the terms and conditions. The self signed certificates is sort of ok with browsers. But many other applications like dropbox sync, AV dat update, vpn , etc fail because of the untrusted certificate. On top of it we have some headless devices in our network as well. Since we anyway have this information in our terms and conditions we would like to move to a trusted chain so that all the applications work as expected.. Gentlemen, I see some users have already asked help/reported bug about the same thing like, http://www.squid-cache.org/mail-archive/squid-users/201112/0197.html. I also see that changes have been done in squid to support this behavior as well. http://www.squid-cache.org/mail-archive/squid-dev/201110/0207.html I followed the steps from this thread for configuration and I still dont see the chain information sent to the clients. http://www.squid-cache.org/mail-archive/squid-users/201109/0037.html So has the behavior of squid changed in recent times? Or am I missing something in my configuration. How to make squid send the entire certificate chain to clients? Please help. Regards, Prasanna
RE: [squid-users] Issue related to using Squid 3.1 or 3.29 and accessing a site that uses a recursive DNS record. (30 seconds to bring up site)
>One last thing to check is what happens when you query for records >on that domain. That is the major change between 3.1 and 3.2 DNS. >What I get is: >## time host -t A webapps.kattenlaw.com >webapps.kattenlaw.com has address 38.98.128.19 >0.000u 0.004s 0:00.06 0.0% 0+0k 0+0io 0pf+0w >## time host -t webapps.kattenlaw.com >;; connection timed out; no servers could be reached >0.000u 0.004s 0:12.03 0.0% 0+0k 0+0io 0pf+0w >Which indicates that dns_timeout directive in Squid can be reduced to >workaround that server brokenness on their end to some degree. >Amos Thanks! I missed that directive. And yes when I lowered it I was able to decrease the initial lookup time to 5 seconds if I set the directive to 5 seconds. I am wondering what tradeoff I will have, if anything else will break by lowering this too low. Is the timeout on lookup I experience due to the internal resolver trying to do an ipv6 host name lookup? I received similar #'s to the above when it tried for the record it took 15 seconds till it timed out. Thanks again === CIRCULAR 230 DISCLOSURE: Pursuant to Regulations Governing Practice Before the Internal Revenue Service, any tax advice contained herein is not intended or written to be used and cannot be used by a taxpayer for the purpose of avoiding tax penalties that may be imposed on the taxpayer. === CONFIDENTIALITY NOTICE: This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies. === NOTIFICATION: Katten Muchin Rosenman LLP is an Illinois limited liability partnership that has elected to be governed by the Illinois Uniform Partnership Act (1997). ===
[squid-users] Question about encryption of data
Hello everyone, I have a question about squid. Forgive me if it is a basic one. Is it possible to encrypt data between squid server and other users? I am planning to use squid as a tool to bypass internet filtering. And it is important for me to have the data encrypted, I know that they are using DPI systems so they can easily recognize unencrypted traffic. Keshavarzian
[squid-users] Re: Question about encryption of data
You might use (secure) SPDY. Although still under development, and a new, ongoing google project. For a beginning: https://groups.google.com/forum/?fromgroups=#!topic/spdy-dev/pON010b6oiM -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/Question-about-encryption-of-data-tp4659506p4659507.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] squid-internal-mgr not found - cannot login to cachemgr
according to "man cachemgr.cgi", the default proxy port is assumed if no port is specified. if i am mistaken, where would be the appropriate place to configure the port for the cachemgr? CONFIGURATION Configuration examples for many common web servers can be found in the Squid FAQ wiki. http://wiki.squid-cache.org/SquidFaq ./cachemgr.conf /etc/squid/cachemgr.conf The access configuration file defining which Squid servers may be managed via this cachemgr.cgi program. Each line specifies a server:port followed by an optional description The server name may contain shell wildcard characters such as *, [] etc. A quick selection dropdown menu is automatically constructed from the simple server names. Specifying :port is optional. If not specified then the default proxy port is assumed. :* or :any matches any port on the target server. On Thu, Apr 11, 2013 at 4:41 AM, Kinkie wrote: > On Thu, Apr 11, 2013 at 2:28 AM, brendan kearney wrote: >> resending because i got a mailer-daemon failure for HTML formatting... >> >> all, >> >> i am running squid 3.2.5 on fedora 16 64 bit on two separate boxes, >> load balanced with HA Proxy. i am trying to access cachemgr on either >> one of the squid instances, and both exhibit the behaviour where the >> squid-internal-mgr URI is not found. attempts to login via the HA >> Proxy VIP as well as with no proxy configured (direct access) have >> been tried. both ways produce the same error. below is some header >> info: >> >> http://192.168.25.1/squid-internal-mgr/ > > [...] >> # >> - >> # TAG: http_port >> http_port 192.168.25.1:3128 > [...] > >> can anyone tell my why i am not able to get logged into the cachemgr? >> the page presents, but the login fails. cachemgr.conf has the IP of >> both proxies listed, and /etc/httpd/conf.d/squid.conf has the right >> access allowed by network. /usr/lib64/squid/cachemgr.cgi is chmod'd >> 755 (rwxr-xr-x) and is chown'd root:root. > > Hi Brendan, > the reason is that Squid is listening on port 3128, you're connecting > to the Apache server listening on port 80. > > -- > /kinkie
Re: [squid-users] Question about encryption of data
The most common response to your question would be to suggest that you use a vpn server to connect your clients to your server then to squid in a secured fashion, However, there are many other ways to achieve your goal as well, you could do it via a socks5 proxy, your clients to your server, and forward all traffic internally to your squid, and I am not certain about this next one, but I suppose in theory it could be possible to force all http traffic via https. if you did some magik on your squid proxy with ssl certs, essentially forcing an https connection from your clients to your squid. - Signed, Fix Nichols http://www.squidblacklist.org
