[squid-users] Bug 2973 - Memory leak when handling pathless http requests
I just filed a new bug and wondered if anyone here had seen a similar
problem or had any suggestions about how to track down the possible
memory leak.
* http://bugs.squid-cache.org/show_bug.cgi?id=2973
There seems to be quite a bad memory leak in the way Squid handles HTTP
requests which do not contain a path. For example, one of our customers Squid
servers, deployed in transparent mode, is receiving many thousands of such
requests, presumably some sort of DOS attack on the named web server.
{{{
GET HTTP/1.1
Host: aferist.su
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.2)
Gecko/20100115 Firefox/3.6b1 (de) (TL-FF) (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: Keep-Alive
}}}
Squid logs these as TCP_DENIED/400
{{{
1278006100.745 0 1.2.3.4 TCP_DENIED/400 870 GET NONE:// - NONE/- text/html
}}}
When the attack starts, we observe a rapid increase in the Squid resident
memory size until eventually Squid crashes.
-RichardW.
Re: [squid-users] how to cache youtube
On Fri, Apr 9, 2010 at 6:35 AM, Kinkie gkin...@gmail.com wrote: On Fri, Apr 9, 2010 at 12:32 AM, sameer khan khanza...@hotmail.com wrote: Please see http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube thanks kinkie, but does it work ? any one experience with below link ? will be much appreciated for any help. To the best of my knowledge, it does. The only way to be sure is by trying. Sameer, Kinkie, There is one thing to beware of. I have recently encountered Squid segfaults with the Youtube loop detection patch on latest Squid 2.7.STABLE8/9. Previous versions of Squid worked reliably with this loop detection patch. The problem has already been recorded on the Lusca bug tracker. See: * http://code.google.com/p/lusca-cache/issues/detail?id=86 You will also find an updated loop detection patch contributed by chudy.fernandez - I haven't yet tested it but will report back when I do. It's also worth noting that the alternative minimum_object_size 512 bytes workaround no longer seems to work. I have only done a brief investigation, but it seems that many Youtube redirect responses are now larger than 512 bytes. -RichardW.
[squid-users] TProxy for Squid-2.7.STABLE8
Hi Henrik, Amos, etc
I've been trying to compile Squid-2.7.STABLE8 (squid-2.HEAD-20100222)
but am having difficulty applying the Visolve TProxy-4 patch
* http://www.visolve.com/squid/squid-tproxy.php
The patch no longer applies cleanly. I spent some time trying to
resolve the conflicts, and after successful compilation, Squid is
listening on its port, but also complains to cachelog as follows and
it's not spoofing the source IP:
{{{
Accepting proxy HTTP connections at 192.168.251.106, port 800, FD 27.
...
commBind: Cannot bind socket FD 31 to 192.168.251.106:800: (98)
Address already in use
}}}
I'm compiling on an Ubuntu 9.10 machine with Linux kernel
2.6.31-19-generic and Linux headers packages installed
{{{
aptitude search ~ilinux-headers
i linux-headers-2.6.31-19
- Header files related to Linux kernel version 2.6.31
i linux-headers-2.6.31-19-generic
- Linux kernel headers for version 2.6.31 on x86/x86_64
i A linux-headers-generic
- Generic Linux kernel headers
}}}
I'm deploying this on a Slackware based box with custom Linux Kernel
2.6.31.6 (TProxy module enabled)
{{{
cachebox# dmesg | grep -i tproxy
NF_TPROXY: Transparent proxy support initialized, version 4.1.0
NF_TPROXY: Copyright (c) 2006-2007 BalaBit IT Ltd.
}}}
I think the problem might be caused by this recent patch to the libcap
code, particularly - around tools.c:
* http://www.squid-cache.org/Versions/v2/HEAD/changesets/12640.patch
It looked like the changes to tools.c that had previously been applied
by the Tproxy patch are now part of the 2.7 tree, but re-factored
slightly. Then again I may be totally off the mark :)
I've attached my latest version of the patch in which I rejected all
the Tproxy changes to tools.c.
Has anyone already prepared a more up to date version of the Tproxy
patch? If not, I'd like to help fix the patch, but perhaps someone can
quickly summarise what might be the problem and what needs doing.
-RichardW.
Index: configure
===
--- configure (revision 9786)
+++ configure (working copy)
@@ -9554,7 +9554,6 @@
grp.h \
libc.h \
linux/netfilter_ipv4.h \
- linux/netfilter_ipv4/ip_tproxy.h \
malloc.h \
math.h \
memory.h \
@@ -29104,10 +29103,10 @@
fi
if test $LINUX_TPROXY; then
-{ $as_echo $as_me:$LINENO: checking if TPROXY header files are installed 5
-$as_echo_n checking if TPROXY header files are installed... 6; }
+{ echo $as_me:$LINENO: checking if sys/capability header files are installed 5
+echo $ECHO_N checking if sys/capability header files are installed... $ECHO_C 6; }
# hold on to your hats...
-if test $ac_cv_header_linux_netfilter_ipv4_ip_tproxy_h = yes test $LINUX_NETFILTER = yes; then
+if test $ac_cv_header_sys_capability_h = yes test $LINUX_NETFILTER = yes; then
LINUX_TPROXY=yes
cat confdefs.h \_ACEOF
@@ -29122,8 +29121,12 @@
_ACEOF
fi
-{ $as_echo $as_me:$LINENO: result: $LINUX_TPROXY 5
-$as_echo $LINUX_TPROXY 6; }
+{ echo $as_me:$LINENO: result: $LINUX_TPROXY 5
+echo ${ECHO_T}$LINUX_TPROXY 6; }
+
+if test $LINUX_TPROXY = no ; then
+echo WARNING: Cannot find necessary system capability headers files
+echo Linux TProxy-4 support WILL NOT be enabled
if test $use_libcap != yes; then
{ $as_echo $as_me:$LINENO: WARNING: Missing needed capabilities (libcap or libcap2) for TPROXY 5
$as_echo $as_me: WARNING: Missing needed capabilities (libcap or libcap2) for TPROXY 2;}
@@ -29131,11 +29134,6 @@
sleep 10
fi
fi
-if test $LINUX_TPROXY = no test $LINUX_NETFILTER = yes; then
-echo WARNING: Cannot find TPROXY headers, you need to patch your kernel with the
-echo tproxy package from:
-echo - lynx http://www.balabit.com/downloads/files/tproxy/;
-sleep 10
fi
if test -z $USE_GNUREGEX ; then
Index: configure.in
===
--- configure.in (revision 9786)
+++ configure.in (working copy)
@@ -1802,7 +1802,6 @@
grp.h \
libc.h \
linux/netfilter_ipv4.h \
- linux/netfilter_ipv4/ip_tproxy.h \
malloc.h \
math.h \
memory.h \
@@ -2946,9 +2945,9 @@
dnl Linux Netfilter/TPROXY support requires some specific header files and libcap
dnl Shamelessly copied from shamelessly copied from above
if test $LINUX_TPROXY; then
-AC_MSG_CHECKING(if TPROXY header files are installed)
+AC_MSG_CHECKING(if sys/capability header files are installed)
# hold on to your hats...
-if test $ac_cv_header_linux_netfilter_ipv4_ip_tproxy_h = yes test $LINUX_NETFILTER = yes; then
+if test $ac_cv_header_sys_capability_h = yes test $LINUX_NETFILTER = yes; then
LINUX_TPROXY=yes
AC_DEFINE(LINUX_TPROXY, 1, [Enable real Transparent Proxy support for Netfilter TPROXY.])
else
@@ -2961,13 +2960,12 @@
LINUX_TPROXY=no
sleep 10
fi
+
+if test $LINUX_TPROXY = no ; then
+echo WARNING: Cannot find necessary system capability
Re: [squid-users] Configure a transparent proxy to pass through non-http port 80 traffic [Was: How to handle the error: Unsupported method 'BitTorrent']
On Sat, Jan 9, 2010 at 1:10 PM, Amos Jeffries squ...@treenet.co.nz wrote: I would not worry about that. P2P apps which use port 80 usually have other methods of connecting. Particularly their own dedicated protocol ports. Leave those open and they work better. The apps which do not use port 80 for HTTP properly (ie doing correct HTTP tunneling) are in violation of web usage standards. Your contracts should not allow you to be penalized for providing a properly working web proxy to your clients. Thanks Amos, Sorry for not replying sooner. I agree and I think I was wrong about the proportion of non-http traffic. The problem lay elsewhere. If you must look at it, then the workaround hack of identifying packets data content has to be done in the iptables routing levels. This is a tricky problem since there is no guarantee that the needed data is in the first packet of a connection. Once packets enter Squid its too late to bypass. Yeah, we're using a Foundry ServerIron L7 switch which seems to have a facility to reconstruct the http headers and use those in routing policies. Sounds like magic to me, but if I manage to get that working, I'll report back. I'm also still interested in the wccp_return_method as a way of bypassing non-http traffic, but in a previous thread it seemed that Squid doesn't support this yet: * http://www.squid-cache.org/mail-archive/squid-users/200811/0130.html * http://www.mail-archive.com/squid-users@squid-cache.org/msg63741.html * http://old.nabble.com/WCCP-load-balancing-and-TPROXY-fully-transparent-interception-td20299256.html Thanks for your help. -RichardW.
[squid-users] Configure a transparent proxy to pass through non-http port 80 traffic [Was: How to handle the error: Unsupported method 'BitTorrent']
On Wed, Dec 3, 2008 at 4:44 AM, Amos Jeffries squ...@treenet.co.nz wrote: Khemara Lyn wrote: How can I handle this error in Cache log: parseHttpRequest: Unsupported method 'BitTorrent' Is it serious or does it affect Squid performance? It's only a waste of TCP connections, if you have available fd and socket capacity on the system you can safely ignore it. Sorry to re-open this ancient thread, but I'm interested in another aspect of this. I am working for a small ISP customer who have an obligation not to block *any* traffic. We have set up Squid in transparent mode and we are using a Brocade smart switch / router to redirect port 80 traffic to the Squid machine. It all works, but we notice an immediate and significant drop in outbound traffic which we are fairly sure is caused by Squid blocking non-http port 80 traffic. Can anyone suggest a way to only pass http traffic to Squid - and let other non-http traffic to pass through. Look forward to hearing your suggestions. -RichardW.
Re: [squid-users] Issue with multiple Squids / TPROXY / WCCP and DNS load balancing
On Fri, Aug 14, 2009 at 5:15 PM, Matus UHLAR - fantomasuh...@fantomas.sk wrote: snip Squid ignores the original destination of connection in order to provide correct content. Many servers provide the same (their own) content independently on what Host: you ask for, so the cache would get filled with incorrect content. That's one of downfalls when using inctercepting proxy. Hi Matus, Thanks for taking the time to reply. I don't understand the point you made above. In any case, I asked the same question on #squid and Henrik Nordstrom pointed out that we can work around this problem by using the source IP address for redirection, rather than the destination address. We've been using this successfully for the last couple of days. The only downside is that our two Squid caches now contain duplicate objects. We're going to see if we can modify the Squid source to use the requested destination IP address rather than looking it up again. I'll post here if we make any progress. snip avoid using the proxy or explain why do you see different host than squid does... It's caused by DNS host records with multple IP addresses (commonly used for load balancing eg on the akmai network). When the client looks up the host, it gets one IP address, and when Squid then does a DNS lookup shortly afterwards it receives a different IP address. rich...@largo:~$ dig assets.macys.com A +short assets.macys.com.edgesuite.net. a796.b.akamai.net. 80.157.169.145 80.157.169.195 rich...@largo:~$ dig assets.macys.com A +short assets.macys.com.edgesuite.net. a796.b.akamai.net. 80.157.169.195 80.157.169.145 This causes the Cisco router to redirect the response to the other Squid server which just drops it. -RichardW.
[squid-users] Issue with multiple Squids / TPROXY / WCCP and DNS load balancing
Hello, I'm seeing exactly the same problem as was described in this email (in 2004): * http://www.squid-cache.org/mail-archive/squid-dev/200407/0008.html The http client does a DNS lookup, then Squid repeats the DNS lookup and receives a different host IP - which means that Squid makes it's proxy request to a different IP than the original request. Our theory is that the WCCP router redirects the response to the wrong Squid. Anyone else seen the same problem? Anyone got a work around? In that mailing list thread, Henrik had suggested that someone needs to submit a patch to allow Squid to connect to the same IP address as the client connection, but I can't tell if anyone has done this yet. Look forward to hearing from anyone with answers. Squid 2.7 STABLE6 -RichardW.
Re: [squid-users] WCCP+Squid not working. Could use some of your experience.
2009/1/23 Anthony DeMatteis adematt...@commspeed.net: Greetings Group, I'm new to this group... We're an ISP trying to control some of our bandwidth issues. I've never set up squid before. I have a working squid server, working very well, including caching youtube vids. However, this is via setting up the proxy settings in the browser and pointing to the caching server's ip address:3128 or using acl's on the router and redirecting traffic to the caching server. I would like to set it up transparently using wccp. I would rather go the wccp route to allow traffic to continue to flow in the event the caching server(s) die. I understand wccpv2 provides this feature. My problem is getting the gre tunnel to work. I've been googling for two days. I've used info from pages 143-149 of Squid: The Definitive Guide. No luck getting wccp tunnel working. I've managed to get this: Hello Tony, The following commands are useful for debugging WCCP problems. * CISCO IOS debug ip wccp events debug ip wccp packets This two commands will make the router log useful WCCP debug info. * squid.conf debug_options 80,3 This will log detailed wccp info to the squid cachelog. See http://squid.cvs.sourceforge.net/viewvc/squid/squid/doc/debug-sections.txt?view=markup * Use tcpdump on the physical and gre interfaces to watch packets arriving from the Cisco router. * Configure your firewall to log dropped packets, and search for any dropped packets originating from the Cisco router. Perhaps this recent blog will be helpful: http://fakrul.wordpress.com/2008/12/11/transparent-squid-proxy-server-with-wccp-support/ You should be aware that if you are deploying a standard transparent Squid proxy, all your web traffic will appear to come from the IP address of the Squid box. For an ISP this can cause problems for users if they are accessing sites (eg download sites) that limit concurrent access based on client source IP. To get round this, there is a patch for Squid called TPROXY which allows it to spoof the source IP address of the original user. This is well supported on Linux, but I'm not sure about FreeBSD (see http://cacheboy.blogspot.com/2009/01/freebsd-tproxy-works.html) Hope that helps. -RichardW. -- Richard Wall ApplianSys Ltd http://www.appliansys.com
Re: [squid-users] WCCP load balancing and TPROXY fully transparent interception
2008/12/19 Bin Liu binliu.l...@gmail.com: I'm interested to know if you have managed to get this working reliably for your ISP environment? Not yet. We are still doing some test in our own environment. Bin, Thanks for your reply. We're working on a Squid configuration for ISP customers and it would be really interesting to hear about any potential problems with this configuration. How far have you gone to make Squid truly transparent eg * suppressing the Squid headers, error messages etc. * Is there any way to configure Squid / Cisco to give SYN_ACK, connection refused and ICMP host unreachable responses rather than Squid error messages? * Can you force Squid to make its request from the same source port as the client. You mean totally transparent and the clients don't even know the existence of squid by any means? It seems a little bit difficult... Yeah, I agree. It's going to be impossible to totally hide Squid, but we're going to do as much as we can. * If someone uses port 80 for a protocol other than http, can Squid reject the redirected traffic in such a way that it is passed through directly instead? WCCPv2 can support this feature by Packet Return Method. (See http://www.cisco.com/en/US/docs/ios/12_0t/12_0t3/feature/guide/wccp.html, search Web Cache Packet Return. Also mentioned in your url: http://bazaar.launchpad.net/~squid3/squid/3.1/annotate/9363?file_id=draftwilsonwccpv212o-20070417152110-s6qkuxj8uabe-1) But Henrik said squid hadn't implemented this feature yet. (See http://www.squid-cache.org/mail-archive/squid-users/200811/0130.html) Thanks for the links. -RichardW. -- Richard Wall Support Engineer ApplianSys Ltd http://www.appliansys.com (t) +44 (0)24 7643 0094 (f) +44 (0)87 0762 7063 (e) richard.w...@appliansys.com
[squid-users] WCCP v2 connection dropped intermittently: HERE_I_AM not received by router
: Called 2008/12/18 17:11:46| wccp2HereIam: sending to service id 0 2008/12/18 17:11:46| wccp2_update_md5_security: called 2008/12/18 17:11:46| Sending HereIam packet size 160 2008/12/18 17:11:56| wccp2HereIam: Called 2008/12/18 17:11:56| wccp2HereIam: sending to service id 0 2008/12/18 17:11:56| wccp2_update_md5_security: called 2008/12/18 17:11:56| Sending HereIam packet size 160 2008/12/18 17:12:06| wccp2HereIam: Called 2008/12/18 17:12:06| wccp2HereIam: sending to service id 0 2008/12/18 17:12:06| wccp2_update_md5_security: called 2008/12/18 17:12:06| Sending HereIam packet size 160 2008/12/18 17:12:16| wccp2HereIam: Called 2008/12/18 17:12:16| wccp2HereIam: sending to service id 0 2008/12/18 17:12:16| wccp2_update_md5_security: called 2008/12/18 17:12:16| Sending HereIam packet size 160 2008/12/18 17:12:26| wccp2HereIam: Called 2008/12/18 17:12:26| wccp2HereIam: sending to service id 0 2008/12/18 17:12:26| wccp2_update_md5_security: called 2008/12/18 17:12:26| Sending HereIam packet size 160 2008/12/18 17:12:36| wccp2HereIam: Called 2008/12/18 17:12:36| wccp2HereIam: sending to service id 0 2008/12/18 17:12:36| wccp2_update_md5_security: called 2008/12/18 17:12:36| Sending HereIam packet size 160 2008/12/18 17:12:36| wccp2HandleUdp: Called. 2008/12/18 17:12:36| Incoming WCCPv2 I_SEE_YOU length 128. 2008/12/18 17:12:36| Complete packet received 2008/12/18 17:12:36| Incoming WCCP2_I_SEE_YOU Received ID old=1502 new=1504. 2008/12/18 17:12:36| Cleaning out cache list 2008/12/18 17:12:36| Adding ourselves as the only cache 2008/12/18 17:12:36| Change detected - queueing up new assignment }}} -- Richard Wall ApplianSys Ltd http://www.appliansys.com
Re: [squid-users] WCCP load balancing and TPROXY fully transparent interception
2008/11/5 Bin Liu binliu.l...@gmail.com: snip I have 2 squid servers, squid A and squid B, both implented TPROXY and connected to the same Cisco router: Internet | | squid ARoutersquid B | | Customers Here squid A wants to send a HTTP request to original destination server, the routers just forwards this packet, it's OK; but when the response packet from the original server returns in, how does the router redirect that packet? Redirect it to squid A or squid B? As there's no connection table in router memory or any mark in the packet, how can the router determine that this response packet should be forwarded to squid A? squid A -- (request to original server) -- router -- original server -- (response) -- router -- squid A or B? Hi Bin, You may already have got the answer to this, but I have recently been setting this up and had the same question. Seems the key is in the Redirection with Hash Assignment: * http://bazaar.launchpad.net/~squid3/squid/3.1/annotate/9363?file_id=draftwilsonwccpv212o-20070417152110-s6qkuxj8uabe-1 (LINE 549) In the config example that Henrik linked to (above) the outbound requests are redirected to a particular Squid, based on a hash of their destination IP and the returning responses are redirected based on their source ip. This way the response is redirected to the Squid that made the spoofed request. Clever in theory; and in my minimal test setup it does seem to work. I'm interested to know if you have managed to get this working reliably for your ISP environment? Has it caused an particular problems for your customers? How far have you gone to make Squid truly transparent eg * suppressing the Squid headers, error messages etc. * Is there any way to configure Squid / Cisco to give SYN_ACK, connection refused and ICMP host unreachable responses rather than Squid error messages? * Can you force Squid to make its request from the same source port as the client. * If someone uses port 80 for a protocol other than http, can Squid reject the redirected traffic in such a way that it is passed through directly instead? Look forward to any information you can provide. -RichardW. -- Richard Wall ApplianSys Ltd http://www.appliansys.com
Re: [squid-users] Update Accelerator, Squid and Windows Update Caching
On Fri, Oct 10, 2008 at 12:30 PM, Amos Jeffries [EMAIL PROTECTED] wrote:
Richard Wall wrote:
Hi,
I've been reading through the archive looking for information about
squid 2.6 and windows update caching. The FAQ mentions problems with
range offsets but it's not really clear which versions of Squid this
applies to.
All versions. The FAQ was the result of my experiments mid last year. With
some tweaks made early his year since Vista came out.
We haven't done a intensive experiments with Vista yet.
Hi Amos,
I'm still investigating Windows Update caching (with 2.6.STABLE17/18)
First of all, I have been doing some tests to try and find out the
problem with Squid and Content-Range requests.
* I watch the squid logs as a vista box does its automatic updates
and I can see that *some* of its requests use ranges. (so far I have
only seen these when it requests .psf files...some of which seem to be
very large files...so the range request makes sense) See:
http://groups.google.hr/group/microsoft.public.windowsupdate/browse_thread/thread/af5db07dc2db9713
# zcat squid.log.192.168.1.119.2008-10-16.gz | grep
multipart/byteranges | awk '{print $7}' | uniq | while read URL; do
echo $URL; wget --spider $URL 21 | grep Length; done
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/10/windows6.0-kb956390-x86_2d03c4b14b5bad88510380c14acd2bffc26436a7.psf
Length: 91,225,471 (87M) [application/octet-stream]
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/05/windows6.0-kb950762-x86_0cc2989b92bc968e143e1eeae8817f08907fd715.psf
Length: 834,868 (815K) [application/octet-stream]
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/03/windows6.0-kb948590-x86_ed27763e42ee2e20e676d9f6aa13f18b84d7bc96.psf
Length: 755,232 (738K) [application/octet-stream]
http://www.download.windowsupdate.com/msdownload/update/software/crup/2008/09/windows6.0-kb955302-x86_1e40fd3ae8f95723dbd76f837ba096adb25f3829.psf
Length: 7,003,447 (6.7M) [application/octet-stream]
...
* I have found that curl can make range requests so I've been using
it to test how Squid behavesand it seems to do the right thing. eg
- First ask for a range : The correct range is returned X-Cache: MISS
- Repeat the range request : The correct range is returned X-Cache: MISS
- Request the entire file: The entire file is correctly returned X-Cache: MISS
- Repeat the request: X-Cache: HIT
- Repeat the previous range request: X-Cache: HIT
- Request a different range: X-Cache: HIT
curl --range 1000-1002 --header Pragma: -v -x http://127.0.0.1:3128
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/05/windows6.0-kb950762-x86_0cc2989b92bc968e143e1eeae8817f08907fd715.psf
/dev/null
Looking back through the archive I find this conversation from 2005:
http://www.squid-cache.org/mail-archive/squid-users/200504/0669.html
...but the behaviour there sounds like a result of setting:
range_offset_limit -1
Seems to me that Squid should do a good job of Windows Update caching.
There is another thread discussing how to override MS update cache
control headers:
http://www.squid-cache.org/mail-archive/squid-users/200508/0596.html
but I don't see anything evil in the server response headers
today. I guess the client may be sending no-cache headers...I'll
double check that later.
Is there some other case that I'm missing?
I'm going to experiment, but if anyone has any positive or
negative experience of Squid and windows update caching, I'd be really
interested to hear from you.
In case Squid cannot do windows update caching by its self, I'm also
looking at integrating Update Accelerator
(http://update-accelerator.advproxy.net/) script with standard squid
2.6 and wondered if anyone else had any experience of this.
The update accelerator script is just a perl wrapper around wget which
is configured as a Squid url_rewrite_program. It's not clear to me
what this script is doing that Squid wouldn't do by its self.
Strange indeed.
I got update accelerator working with Squid but I'm still not
convinced that it's necessary (see above).
-RichardW.
[squid-users] Update Accelerator, Squid and Windows Update Caching
Hi, I've been reading through the archive looking for information about squid 2.6 and windows update caching. The FAQ mentions problems with range offsets but it's not really clear which versions of Squid this applies to. I'm going to experiment, but if anyone has any positive or negative experience of Squid and windows update caching, I'd be really interested to hear from you. In case Squid cannot do windows update caching by its self, I'm also looking at integrating Update Accelerator (http://update-accelerator.advproxy.net/) script with standard squid 2.6 and wondered if anyone else had any experience of this. The update accelerator script is just a perl wrapper around wget which is configured as a Squid url_rewrite_program. It's not clear to me what this script is doing that Squid wouldn't do by its self. Thanks. -RichardW.
Re: [squid-users] Update Accelerator, Squid and Windows Update Caching
On Fri, Oct 10, 2008 at 12:30 PM, Amos Jeffries [EMAIL PROTECTED] wrote: Richard Wall wrote: I've been reading through the archive looking for information about squid 2.6 and windows update caching. The FAQ mentions problems with range offsets but it's not really clear which versions of Squid this applies to. All versions. The FAQ was the result of my experiments mid last year. With some tweaks made early his year since Vista came out. We haven't done a intensive experiments with Vista yet. Thanks Amos, Okay. Well I'm planning on testing with Vista updates so I'll try and report my findings here. snip In case Squid cannot do windows update caching by its self, I'm also looking at integrating Update Accelerator (http://update-accelerator.advproxy.net/) script with standard squid 2.6 and wondered if anyone else had any experience of this. The update accelerator script is just a perl wrapper around wget which is configured as a Squid url_rewrite_program. It's not clear to me what this script is doing that Squid wouldn't do by its self. Strange indeed. I'll let you know how it goes :) -RichardW.
[squid-users] Job Opportunity: Squid support engineer at ApplianSys, UK
Hello, I work for a company called ApplianSys, in Coventry, UK. We sell a range of server appliances, one of which, the CACHEBox, is based around Squid 2.6 http://www.appliansys.com/products/ We're currently recruiting for developers, support engineers and sales people. We are particularly looking for people familiar with installing and administering Squid and Linux There is a list of current vacancies on our website and details about how to apply: http://www.appliansys.com/company/employment.html If you're looking for a job and are able to re-locate to the Midlands (UK) we'd really like to hear from you. -RichardW. -- Richard Wall Support Engineer ApplianSys Ltd http://www.appliansys.com (t) +44 (0)24 7643 0094 (f) +44 (0)87 0762 7063 (e) [EMAIL PROTECTED]
Re: [squid-users] RAID is good
On Thu, Mar 27, 2008 at 1:59 AM, Marcus Kool [EMAIL PROTECTED] wrote: snip Only one cache directory per disk is recommended while you have 4 cache directories on one file system. Consider dropping 2 COSS cache directories so that you have 1 COSS and 1 AUFS. Yep, I understand. Unfortunately in that area I'm restricted by a 2GB file size limit, beyond which various system binaries don't recognise the COSS files. Don't ask. :) There are ways round it though and for a one off benchmark I can setup a 7.5 GB COSS and a 2.5GB AUFS store. I'll let you know if this improves the benchmark results. I also intend to run a benchmark on the same hardware but with JBOD, and hopefully, given time I'll be able to benchmark a cluster of boxes. Kinkie and I rewrote the RAID for Squid section of the FAQ and it includes more details about price, performance and reliability trade-offs. Yep, that's a really good write up. Thanks. -RichardW.
Re: [squid-users] RAID is good (was: Re: [squid-users] Hardware setup ?)
On Tue, Mar 25, 2008 at 1:23 PM, Marcus Kool [EMAIL PROTECTED] wrote: I wish that the wiki for RIAD is rewritten. Companies depend on internet access and a working Squid proxy and therefore the advocated no problem if a single disk fails is not from today's reality. One should also consider the difference between simple RAID and extremely advanced RAID disk systems Recently I've spent a fair bit of time benchmarking a Squid system whose COSS and AUFS storage (10GB total) + access logging are on a RAID0 array of two consumer grade SATA disks. For various reasons, I'm stuck with RAID0 for now, but I thought you might be interested to hear that the box performs pretty well. The box can handle a 600 - 700 Req/Sec Polygraph polymix-4 benchmark with a ~40% document hit ratio. usage Doubling the total storage to 20GB, increased the doc hit ratio to 55%, but hit response times began to increase noticably during the top phases. CPU was about 5% idle during the top phases. Logs were being rotated and compressed every five minutes. CPU usage never Some initial experiments suggest that removing RAID doesn't particularly improve performance, but I intend to do a more thorough set of benchmarks soon. I'm not sure how relevant this is to your discussion. I don't know how RAID0 performance is expected to compare to RAID5. I'll post here if and when I do more benchmarking without RAID. -RichardW. == Spec == CPU: Intel(R) Celeron(R) CPU 2.53GHz RAM: 3GB Disks: 2 x Seagate Barracuda 160GB Squid: 2.6.STABLE17 Linux Kernel: 2.6.23.8 FS: reiserfs == Squid Conf (extract) == # NETWORK OPTIONS http_port 800 transparent # MEMORY CACHE OPTIONS cache_mem 152 MB maximum_object_size_in_memory 50 KB # DISK CACHE OPTIONS cache_replacement_policy lru # TOTAL AVAILABLE STORAGE: 272445 MB # MEMORY STORAGE LIMIT: 46694 MB # CONFIGURED STORAGE LIMIT: 1 MB cache_dir coss /squid_data/squid/coss0 2000 max-size=16000 cache_swap_log /squid_data/squid/%s cache_dir coss /squid_data/squid/coss1 2000 max-size=16000 cache_swap_log /squid_data/squid/%s cache_dir coss /squid_data/squid/coss2 2000 max-size=16000 cache_swap_log /squid_data/squid/%s cache_dir aufs /squid_data/squid 4000 16 256 max_open_disk_fds 0 maximum_object_size 2 KB # LOGFILE OPTIONS debug_options ALL,1 buffered_logs on logfile_rotate 10 # MISCELLANEOUS memory_pools_limit 10 MB memory_pools off cachemgr_passwd none all client_db off
Re: [squid-users] NTLM authentication testing
On 2/18/08, Adrian Chadd [EMAIL PROTECTED] wrote:
Thats basically right - Squid doesn't handle the NTLM itself, it just
passes the blob right through. The helper framework can handle hundreds
of requests a second without too much thought; I'd like to spend some
time figuring out what Samba is doing thats so slow. I thought that winbind
was actually handling the NTLM challenge/response stuff itself and caching
data rather than passing it upstream to the DC for every request.
I haven't yet looked at it, so I can't say for certain that is correct.
I've done some pretty unscientific tests using curl against our Squid box.
* CPU: Intel(R) Celeron(R) CPU 2.53GHz
* MemTotal: 2075628 kB
* Squid2.6 STABLE17 (using epoll)
* NTLM auth_param ntlm children 100
I've been running multiple curl instances on four clients as follows:
{{{
for i in {1..100}; do
while true; do
curl -x 192.168.1.97:800 \
--proxy-ntlm \
--proxy-user DOMAINNAME\\username:password \
--include \
--silent \
--header Pragma: http://www.mydomain.com/index.html /dev/null
done
sleep 1
done
}}}
According to cachemgr this is generating a load of ~250req/sec.
client_http.requests = 252.175917/sec
client_http.hits = 126.159625/sec
client_http.errors = 0.00/sec
client_http.kbytes_in = 90.109732/sec
client_http.kbytes_out = 2735.581866/sec
client_http.all_median_svc_time = 0.851301 seconds
client_http.miss_median_svc_time = 0.000911 seconds
client_http.nm_median_svc_time = 0.00 seconds
client_http.nh_median_svc_time = 0.00 seconds
client_http.hit_median_svc_time = 0.806511 seconds
First problem is that you have to reinterpret the Squid reported hit
ratios when using NTLM auth. Only half of these are hits, the other
half being TCP_DENIED/407 that form part of the NTLM auth negotiation.
Second problem is that the majority of requests seem to result in auth
requests to the DC. There is an article describing Win2003 performance
counters showing Number of auth requests / sec, but those counters
don't seem to exist on my copy.
* http://support.microsoft.com/kb/928576
Instead I used the difference in a minute of the total number of
security events (as shown in the titel bar of the windows event
viewer.
* ~127 successful auth events per second
...which is about the same as the client_http.hits reported by squid.
I have the following setting defined in smb.conf:
* winbind cache time = 10
...which clearly isn't being respected.
* Does anyone else see this behaviour or have you managed to get auth
requests cached by winbindd?
* Can winbindd even do caching of auth reqests or is it only
concerned with caching other domain data?
If anyone has answers, I'd really appreciate to hear from you. I'll
continue to experiment and will post my findings.
-RichardW.
Re: [squid-users] NTLM authentication testing
On 2/19/08, Adrian Chadd [EMAIL PROTECTED] wrote: G'day, THanks for this stuff. Could you possibly try hitting it hard enough to cause Squid to back up on pending authentications? It'd be good to replicate a fail situation; we can then take that to the samba guys and ask wtf? Adrian, Yep I've seen that and it's easy to reproduce by lowering the number of authenticators. So when I start squid configured with: auth_param ntlm children 50 # /usr/local/squid/sbin/squid -d100 -X -N -D -f /RamDisk/squid.conf 2008/02/19 14:29:09| WARNING: All ntlmauthenticator processes are busy. 2008/02/19 14:29:09| WARNING: up to 50 pending requests queued 2008/02/19 14:29:11| storeDirWriteCleanLogs: Starting... 2008/02/19 14:29:11| WARNING: Closing open FD 64 2008/02/19 14:29:11| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed on fd=64: (1) Operation not permitted 2008/02/19 14:29:11| Finished. Wrote 93 entries. 2008/02/19 14:29:11| Took 0.0 seconds (140060.2 entries/sec). FATAL: Too many queued ntlmauthenticator requests (251 on 50) Aborted # echo $? 134 It exits immediatly with return code 134 -RichardW.
Re: [squid-users] NTLM authentication testing
On 2/19/08, Guido Serassio [EMAIL PROTECTED] wrote:
At 14:40 19/02/2008, Richard Wall wrote:
First problem is that you have to reinterpret the Squid reported hit
ratios when using NTLM auth. Only half of these are hits, the other
half being TCP_DENIED/407 that form part of the NTLM auth negotiation.
This is caused by the NTLM over HTTP authentication sequence, look
here for details:
http://davenport.sourceforge.net/ntlm.html
Guido,
Yep, I've looked at it, but have not completely absorbed it yet :)
Second problem is that the majority of requests seem to result in auth
requests to the DC. There is an article describing Win2003 performance
counters showing Number of auth requests / sec, but those counters
don't seem to exist on my copy.
* http://support.microsoft.com/kb/928576
Correct, you should request the hotfix to Microsoft.
Thanks will search it out.
What Samba version do you are using ?
I remember that in Samba 3.0.25 there was big changes into winbindd
regarding off-line logon support, but I don't know if this could help.
# /usr/upgrade/samba/sbin/winbindd --version
Version 3.0.24
So I guess I'll try compiling the latest version. Thanks for th tip.
Another question, what type of NTLM authentication is supported by curl ?
Lan manager/NTLMv1 or full NTLMv2 ? (See the previous link for details)
I'm not sure, but in full debug mode, curl will show the various
headers it exchanges with the server.
It seems to correspond to:
* http://devel.squid-cache.org/ntlm/client_proxy_protocol.html
...but of course we're starting at point 4 which means that in real
life, there'd be even more squid requests I guess.
Anyway, here's the output from curl. Does this give enough information
to work out which type is being used?
{{{
* About to connect() to proxy 10.0.0.12 port 800 (#0)
* Trying 10.0.0.12... connected
* Connected to 10.0.0.12 (10.0.0.12) port 800 (#0)
* Proxy auth using NTLM with user 'COVENTRYOFFICE\stafftest'
GET http://www.squid-cache.org/Images/img4.jpg HTTP/1.1
Proxy-Authorization: NTLM TlRMTVNTUAABBoIIAAA=
User-Agent: curl/7.16.4 (i486-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3.3 libidn/1.0
Host: www.squid-cache.org
Accept: */*
Proxy-Connection: Keep-Alive
* HTTP 1.0, assume close after body
HTTP/1.0 407 Proxy Authentication Required
Server: squid/2.6.STABLE17
Date: Tue, 19 Feb 2008 15:03:05 GMT
Content-Type: text/html
Content-Length: 1371
Expires: Tue, 19 Feb 2008 15:03:05 GMT
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
Proxy-Authenticate: NTLM
TlRMTVNTUAACDgAOADAGgokAN+ZK+JnmUOEAAIoAigA+Q09WRU5UUllPRkZJQ0UCABwAQwBPAFYARQBOAFQAUgBZAE8ARgBGAEkAQwBFAAEAEABBAFAALQBUAEUAUwBUADIABAAcAGMAYQBjAGgAZQAuAGUAMgBiAG4ALgBvAHIAZwADAC4AYQBwAC0AdABlAHMAdAAyAC4AYwBhAGMAaABlAC4AZQAyAGIAbgAuAG8AcgBnAAA=
X-Cache: MISS from ntlmsquidbox.test
X-Cache-Lookup: NONE from ntlmsquidbox.test:800
Via: 1.0 ntlmsquidbox.test:800 (squid/2.6.STABLE17)
* HTTP/1.0 proxy connection set to keep alive!
Proxy-Connection: keep-alive
* Ignoring the response-body
{ [data not shown]
* Connection #0 to host 10.0.0.12 left intact
* Issue another request to this URL:
'http://www.squid-cache.org/Images/img4.jpg'
* Re-using existing connection! (#0) with host 10.0.0.12
* Connected to 10.0.0.12 (10.0.0.12) port 800 (#0)
* Proxy auth using NTLM with user 'COVENTRYOFFICE\stafftest'
GET http://www.squid-cache.org/Images/img4.jpg HTTP/1.1
Proxy-Authorization: NTLM
TlRMTVNTUAADGAAYAEAYABgAWA4ADgBwCQAJAH4IAAgAhwAABoKJAFb2ATKsj8TWAA6YY1ymLs5AgU5/lxbNCYtJnhdC67O5c0NPVkVOVFJZT0ZGSUNFc3RhZmZ0ZXN0cG9seXNydjE=
User-Agent: curl/7.16.4 (i486-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e
zlib/1.2.3.3 libidn/1.0
Host: www.squid-cache.org
Accept: */*
Proxy-Connection: Keep-Alive
* HTTP 1.0, assume close after body
HTTP/1.0 200 OK
Date: Tue, 19 Feb 2008 15:00:26 GMT
Server: Apache/2.2.6 (FreeBSD) mod_ssl/2.2.6 OpenSSL/0.9.7e-p1 DAV/2
PHP/5.2.5 with Suhosin-Patch
Last-Modified: Mon, 22 Jan 2007 10:51:58 GMT
ETag: 6daaa8-7083-d9b9ef80
Accept-Ranges: bytes
Content-Length: 28803
Content-Type: image/jpeg
Age: 159
X-Cache: HIT from ntlmsquidbox.test
HTTP/1.0 407 Proxy Authentication Required
Server: squid/2.6.STABLE17
Date: Tue, 19 Feb 2008 15:03:05 GMT
Content-Type: text/html
Content-Length: 1371
Expires: Tue, 19 Feb 2008 15:03:05 GMT
X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
Proxy-Authenticate: NTLM
TlRMTVNTUAACDgAOADAGgokAN+ZK+JnmUOEAAIoAigA+Q09WRU5UUllPRkZJQ0UCABwAQwBPAFYARQBOAFQAUgBZAE8ARgBGAEkAQwBFAAEAEABBAFAALQBUAEUAUwBUADIABAAcAGMAYQBjAGgAZQAuAGUAMgBiAG4ALgBvAHIAZwADAC4AYQBwAC0AdABlAHMAdAAyAC4AYwBhAGMAaABlAC4AZQAyAGIAbgAuAG8AcgBnAAA=
X-Cache: MISS from ntlmsquidbox.test
X-Cache-Lookup: NONE from ntlmsquidbox.test:800
Via: 1.0 ntlmsquidbox.test:800 (squid/2.6.STABLE17)
Proxy-Connection: keep-alive
HTTP/1.0 200 OK
Date: Tue, 19 Feb 2008 15:00:26
[squid-users] Squid automatically builds the COSS storage file
Hi, My first post to squid-users, so first let me thank the developers for their work on Squid. I'm working with 2.6.STABLE17 and am experimenting with COSS storage. According to the COSS Faq page, you have to create the COSS storage file (using dd) before pointing Squid to it: * http://wiki.squid-cache.org/SquidFaq/CyclicObjectStorageSystem I have found that infact, you can simply configure Squid with some non-existent COSS file and if it is not there and Squid has permissions to write to the parent directory, Squid will build the file for you. I was going to update the wiki page, but thought I'd check here first, to see if there are any benefits to creating the COSS file manually or circumstances in which it is required? Before realising that I could let Squid manage the COSS file creation, I wrote a script to check for the existence of the file and check that it had the expected size. If not then it DDs the file as part of the squid start up. This seemed to work, but after running squid for a while, I find that the file size has increased and therefore my script thinks there's a problem. * I wondered whether this is expected behaviour? * If the size of the COSS file does change unexpectedly, does it signify a problem? * I find that I can reuse an existing COSS file, but tell Squid that it's capacity is greater than the size of the exisiting file. Should Squid complain about this, or will it dynamically resize the file once it reaches capacity? Thanks in advance. -RichardW.
Re: [squid-users] NTLM authentication testing
Hi Adrian, My comments are below. On 2/18/08, Adrian Chadd [EMAIL PROTECTED] wrote: I've got one customer who is asking for some testing of Squid in a large NTLM environment. The problem, as those who have tried it will have encountered, is that although Squid can keep up with it, the Samba/Winbind stuff plainly just can't. This is something that I'm currently very interested in. I had heard that NTLM auth could significantly reduce Squids throughput but haven't seen any figures. I couldn't tell from your message above whether you / your customer has already tried deploying Squid / NTLM auth in live environment. If so, I'm really interested to know what request rate Squid was able to maintain. I understand from the documentation, that the three stage NTLM authentication negotiation has to be repeated for every new connection and that this is the bottleneck. I'd assumed that winbindd was able to CACHE the NTLM user credentials, so that subsequent requests would not result in network calls to the NTLM authentication server. Is this your understanding? So I'm looking for some tools to let me craft and fire off NTLM type authentication stuff to a proxy. I don't really care if they're free or not, unix or windows. If anyone knows of anything that'll let me create -lots- of NTLM authentication requests and fire them through a proxy then please, please let me know. We were considering the possibility of using something like Selenium control the web browser and send requests that way, but some further googling suggests that curl may be able to send NTLM Proxy auth requests. Hopefully the result from all of this will be slightly better NTLM interoperability. -RichardW.
Re: [squid-users] NTLM authentication testing
On 2/18/08, Richard Wall [EMAIL PROTECTED] wrote: googling suggests that curl may be able to send NTLM Proxy auth requests. Sorry forgot to include the link: * http://curl.haxx.se/docs/manpage.html#--proxy-ntlm -RichardW.
Re: [squid-users] NTLM authentication testing
On 2/18/08, Adrian Chadd [EMAIL PROTECTED] wrote: Well, I'll be stuffed: violet:~ adrian$ curl --help | grep ntlm --ntlm Enable HTTP NTLM authentication (H) --proxy-ntlmEnable NTLM authentication on the proxy (H) I wonder how well it'll work. Oh well, time to have a play! Thanks! It does seem to work with the following options: curl -v -x 192.168.1.97:800 --proxy-ntlm --proxy-user DOMAINNAME\\username:password http://www.google.com/ The -v reveals the details of the NTLM authentication headers. I'll let you know if I get any further. -RichardW.
