Re: [squid-users] TProxy and client_dst_passthru
On 09/11/2016 10:23 AM, Amos Jeffries wrote: > The only visible problem is why that 2% exists. > > ==> ORIGINAL_DST is should *only* ever be used on MISS or > REFRESH/revalidate traffic. Never on a HIT. Thus zero (0%) hit-ratio is > the expected behaviour. It is possible that a terminology clash affects Squid code and Squid output interpretation. There are two popular approaches: A. "Hit" means "served from the cache" or, more precisely, "the response is based on the cache entry present in the cache at the time of request interpretation". "Pure hits" (or similar) are served without an attempt to contact the origin server. "Revalidation hits" (or similar) are served after an attempt to check the cached entry freshness with the origin server (that did not result in replacing or removing the previously cached entry). B. "Hit" means served without an attempt to contact the origin server. Revalidation transactions are not hits (by definition). The above definitions are not precise and do not cover all use cases, but that is not important for highlighting the key difference between them. Many people, including Amos in his email quoted above AFAICT, are using approach "B". Many people, including myself, are using approach "A". Squid code, output, and documentation mix the two approaches, unfortunately. Be careful when you interpret what you see. HTH, Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] TProxy and client_dst_passthru
On 12/09/2016 3:04 a.m., Omid Kosari wrote: > > I refer to following messages .i have same problem > The "problem" is misunderstanding of the log entry meaning. > > FredT wrote >> Hi Amos, >> >> We have done additional tests in production with ISPs and the ORIGINAL_DST >> in tproxy cannot be cached. >> In normal mode (not tproxy), ORIGINAL_DST can be cached, no problem. >> But once in tproxy (http_port 3128 tproxy), no way, it's impossible to get >> TCP_HIT. >> >> We have played with the client_dst_passthru and the host_verify_strict, >> many combinaisons on/off. >> By settings client_dst_passthru ON and host_verify_strict OFF, we can >> reduce the number of ORIGINAL_DST (generating DNS "alerts" in the >> cache.log) but it makes issues with HTTPS websites (facebook, hotmail, >> gmail, etc...). Nod. That is the purpose of those controls. So they are working. >> We have also tried many DNS servers (internals and/or externals), same >> issue. >> >> I read what you explain in your previous email but it seems there is >> something weird. >> The problem is that the ORIGINAL_DST could be up to 25% of the traffic >> with some installations meaning this part is "out-of-control" in term of >> cache potential. Any server type could be up to 100%. The type of server used implies nothing about caching potential. The reverse is true: caching potential implies server type, with adjustments for traffic mode type and squid.conf settings. For example: HIT implies HEIR_NONE. MISS with intercept/tproxy implies ORIGINAL_DST or a peer. REFRESH with intercept/tproxy implies ORIGINAL_DST or a peer. MISS in forward-proxy implies DIRECT or a peer. REFRESH in forward-proxy implies DIRECT or a peer. > > FredT wrote >> Hi Eliezer, >> >> Well, we have done many tests with Squid (3.1 to 3.5.x), disabling >> "client_dst_passthru" (off) will stop the DNS entry as explained in the >> wiki, the option directly acts on the flag "ORIGINAL_DST". >> As you know, ORIGINAL_DST switches the optimization off (ex: StoreID) then >> it's not possible to cache the URL (ex: >> http://cdn2.example.com/mypic.png). ORIGINAL_DST does nothing. It is simply a label indicating which type of server supplied the HTTP response message for the transaction. >> >> In no tproxy/NAT mode, the client_dst_passthru works perfectly by >> disabling the DNS entry control, so optimization is done correctly. >> But in tproxy/NAT, the client_dst_passthru has no effect, we see >> ORIGINAL_DST in logs. >> >> So, maybe I'm totaly wrong here the client_dst_passthru is not related to >> the ORIGINAL_DST, "client_dst_passthru on" makes Squid use ORIGINAL_DST (client provided) server instead of DIRECT (DNS lookup) server(s) for *all* intercepted traffic. Even requests where DIRECT is possible. > or there is an explaination why the client_dst_passthru >> does not act in tproxy/NAT... >> There is. The "transparent" part of "transparent interception proxy" means that MISS should use the same server the client was originally sending its request to (the ORIGINAL_DST server). >> Bye Fred > > please look at following results ... > > 60% vs 2% hit ratio(bytes) . The problem is ORIGINAL_DST > The only visible problem is why that 2% exists. ==> ORIGINAL_DST is should *only* ever be used on MISS or REFRESH/revalidate traffic. Never on a HIT. Thus zero (0%) hit-ratio is the expected behaviour. For the same reason that a report of the log traffic using "grep -v HIT" will show zero cache ratio. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] [squid-announce] Squid 4.0.14 beta is available
The Squid HTTP Proxy team is very pleased to announce the availability of the Squid-4.0.14 release! This release is a bug fix release resolving several issues found in the prior Squid releases. The major changes to be aware of: * Bugs #4404 and #4503: access.log responses marked _ABORTED The past few Squid-4 releases have incorrectly logged several transaction types as ABORTED. This hopefully resolves the remaininginstances of that behaviour. * Make Squid death due to overloaded helpers optional Previous Squid versions would halt the entire Squid process if a helper became too non-responsive and its lookup queue became overloaded. This release allows the Squid handling behaviour to be configured to simulate an ERR helper response instead of always halting. * Crashes on shutdown The Squid shutdown process when set to a short timeout was crashing while cleaning up idle ICAP connections. This resolves the ICAP issues, however some other sources of shutdown crash still remain to be fixed. * Various HTTP/1.1 compliance updates Previous Squid releases have a number of compliance issues with RFC 7230 updated HTTP specifications. This release fixes several issues involved with detecting invalid message framing and required error reponse generation. * General portability and stability changes This release also includes a large number of code cleanup fixes too small to mention individually, but which resolve a lot of portability and build issues. The release size appears to be very large, however the majority of alterations are in documentation and translation updates for Squid-4. All users of Squid-4.0.x are encouraged to upgrade to this release as soon as possible. All users of Squid-3 are encouraged to test this release out and plan for upgrades where possible. See the ChangeLog for the full list of changes in this and earlier releases. Please refer to the release notes at http://www.squid-cache.org/Versions/v4/RELEASENOTES.html when you are ready to make the switch to Squid-4 This new release can be downloaded from our HTTP or FTP servers http://www.squid-cache.org/Versions/v4/ ftp://ftp.squid-cache.org/pub/squid/ ftp://ftp.squid-cache.org/pub/archive/4/ or the mirrors. For a list of mirror sites see http://www.squid-cache.org/Download/http-mirrors.html http://www.squid-cache.org/Download/mirrors.html If you encounter any issues with this release please file a bug report. http://bugs.squid-cache.org/ Amos Jeffries ___ squid-announce mailing list squid-annou...@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-announce ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] [squid-announce] Squid 3.5.21 is available
The Squid HTTP Proxy team is very pleased to announce the availability of the Squid-3.5.21 release! This release is a bug fix release resolving several issues found in the prior Squid releases. The major changes to be aware of: * Bug #4534: assertion failure in xcalloc when using many cache_dir Squid is documented as supporting up to 64 cache directories, but would crash with a memory allocation error if more than a few were actually configured. * Bug #4542: authentication credentials IP TTL updated incorrectly This bug caused error in max_user_ip ACL accounting to allow clients to shift IP address more times than configured. This bug fix may have an effect on IPv6 clients using "proviacy adressing" to rotate IPs. * Bug #4428: mal-formed Cache-Control:stale-if-error header This bug shows up as incorrect stale-if-error values being relayed by Squid breaking the use of this feature in the recipients. Squid now relays the header values correctly. * Bug #3025: Proxy-Authenticate problem using ICAP server With this change Squid now treats the ICAP REQMOD adaptation point as a part of itself with regards to proxy authentication. The Proxy-Authentication header received from the client is delivered as part of the HTTP request headers in expectation that the ICAP service may authenticate and/or produce 407 response itself. Note that use of stateful or connection-oriented authentication schemes is not possible. HTTP is designed to operate in a stateless way and any deviation from that design requires Squid to perform special message processing. * HTTP: MUST always revalidate Cache-Control:no-cache responses. This bug shows up as Squid not revalidating some responses until they became stale according to refresh_pattern heuristic rules (specifically the minimum caching age). Squid now revalidates these objects on every request. * HTTP: do not allow Proxy-Connection to override Connection header The Proxy-Connection: header is a long-deprecated experimental header. For the past decade Squid has been actively stripping it out of relayed traffic. This release continues the removal process by also preventing it from having any effect on Squid client connection persistence when a Connection: header is present. * SSL CN wildcard must only match a single domain component [fragment]. This bug shows up as incorrect matching (or non-matching) of the ss::server_name ACL against TLS certificate values. Squid now treats the certificate CN fields according to X.509 domain matching requirements instead of HTTP domain matching requirements. All users of Squid-3 are encouraged to upgrade to this release as soon as possible. See the ChangeLog for the full list of changes in this and earlier releases. Please refer to the release notes at http://www.squid-cache.org/Versions/v3/3.5/RELEASENOTES.html when you are ready to make the switch to Squid-3.5 Upgrade tip: "squid -k parse" is starting to display even more useful hints about squid.conf changes. This new release can be downloaded from our HTTP or FTP servers http://www.squid-cache.org/Versions/v3/3.5/ ftp://ftp.squid-cache.org/pub/squid/ ftp://ftp.squid-cache.org/pub/archive/3.5/ or the mirrors. For a list of mirror sites see http://www.squid-cache.org/Download/http-mirrors.html http://www.squid-cache.org/Download/mirrors.html If you encounter any issues with this release please file a bug report. http://bugs.squid-cache.org/ Amos Jeffries ___ squid-announce mailing list squid-annou...@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-announce ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] TProxy and client_dst_passthru
Antony Stone wrote > On Thursday 08 September 2016 at 12:27:42, Omid Kosari wrote: > >> Hi Fred, >> >> Same problem here . Do you found any solution or workaround ? > > Please clarify which message you are reply / referring to. > > Thanks, > > > Antony. > > -- > Archaeologists have found a previously-unknown dinosaur which seems to > have > had a very large vocabulary. They've named it Thesaurus. > >Please reply to the > list; > please *don't* CC > me. > ___ > squid-users mailing list > squid-users@.squid-cache > http://lists.squid-cache.org/listinfo/squid-users I refer to following messages .i have same problem FredT wrote > Hi Amos, > > We have done additional tests in production with ISPs and the ORIGINAL_DST > in tproxy cannot be cached. > In normal mode (not tproxy), ORIGINAL_DST can be cached, no problem. > But once in tproxy (http_port 3128 tproxy), no way, it's impossible to get > TCP_HIT. > > We have played with the client_dst_passthru and the host_verify_strict, > many combinaisons on/off. > By settings client_dst_passthru ON and host_verify_strict OFF, we can > reduce the number of ORIGINAL_DST (generating DNS "alerts" in the > cache.log) but it makes issues with HTTPS websites (facebook, hotmail, > gmail, etc...). > We have also tried many DNS servers (internals and/or externals), same > issue. > > I read what you explain in your previous email but it seems there is > something weird. > The problem is that the ORIGINAL_DST could be up to 25% of the traffic > with some installations meaning this part is "out-of-control" in term of > cache potential. > > All help is welcome here > Thanks in advance. > > Bye Fred FredT wrote > Hi Eliezer, > > Well, we have done many tests with Squid (3.1 to 3.5.x), disabling > "client_dst_passthru" (off) will stop the DNS entry as explained in the > wiki, the option directly acts on the flag "ORIGINAL_DST". > As you know, ORIGINAL_DST switches the optimization off (ex: StoreID) then > it's not possible to cache the URL (ex: > http://cdn2.example.com/mypic.png). > > In no tproxy/NAT mode, the client_dst_passthru works perfectly by > disabling the DNS entry control, so optimization is done correctly. > But in tproxy/NAT, the client_dst_passthru has no effect, we see > ORIGINAL_DST in logs. > > So, maybe I'm totaly wrong here the client_dst_passthru is not related to > the ORIGINAL_DST, or there is an explaination why the client_dst_passthru > does not act in tproxy/NAT... > > Bye Fred please look at following results As you know the following command shows statistics of line which only have ORIGINAL_DST tail -n 100 /var/log/squid/access.log | grep -a ORIGINAL_DST | calamaris --config-file /etc/calamaris/calamaris.conf --all-useful-reports | more - -- -- Proxy statistics - -- -- Total amount: requests 378310 unique hosts/users:hosts 1859 Total Bandwidth:Byte 16453M Proxy efficiency (HIT [kB/sec] / DIRECT [kB/sec]):factor 1.22 Average speed increase:% 0.39 TCP response time of 100% requests: msec 0M - -- -- Cache statistics - -- -- Total amount cached:requests 11945 Request hit rate: % 3.16 Bandwidth savings: Byte 355M Bandwidth savings in Percent (Byte hit rate): % 2.16 Average cached object size: Byte 0M Average direct object size: Byte 0M Average object size:Byte 0M - -- -- # Incoming TCP-requests by status status request % sec/req Byte % kB/sec -- - -- --- -- --- HIT11945 3.161.94 355M 2.16 15.66 TCP_REFRESH_UNMODIFIED_ABORTED 104 0.03 44.89 158M 0.96 34.55 TCP_REFRESH_UNMODIFIED11795 3.120.77 119M 0.72 13.47 TCP_REFRESH_UNMODIFIED_TIMEDOUT 8 0.00 1108.82 79M 0.48 9.09 TCP_HIT_ABORTED
