Re: [squid-users] squid-3.1 client POST buffering
On Wed, Dec 01, 2010 at 11:40:52AM +, Graham Keeling wrote: > Hello, > > I am convinced that this is a serious bug, so I have entered a proper bug > report. > > It is bug 3113: > > http://bugs.squid-cache.org/show_bug.cgi?id=3113 I have created a simple patch that seems to fix the problem for me on squid-3.1.9, and I have attached it to the bug report.
Re: [squid-users] squid-3.1 client POST buffering
Hello, I am convinced that this is a serious bug, so I have entered a proper bug report. It is bug 3113: http://bugs.squid-cache.org/show_bug.cgi?id=3113
Re: [squid-users] squid-3.1 client POST buffering
On Tue, Nov 30, 2010 at 11:31:45AM +, Graham Keeling wrote: > On Tue, Nov 30, 2010 at 09:46:47PM +1300, Amos Jeffries wrote: > > On 30/11/10 21:23, Oguz Yilmaz wrote: > >> On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries > >> wrote: > >>> On 30/11/10 04:04, Oguz Yilmaz wrote: > >>>> > >>>> Graham, > >>>> > >>>> This is the best explanation I have seen about ongoing upload problem > >>>> in proxy chains where squid is one part of the chain. > >>>> > >>>> On our systems, we use Squid 3.0.STABLE25. Before squid a > >>>> dansguardian(DG) proxy exist to filter. Results of my tests: > >>>> > >>>> 1- > >>>> DG+Squid 2.6.STABLE12: No problem of uploading > >>>> DG+Squid 3.0.STABLE25: Problematic > >>>> DG+Squid 3.1.8: Problematic > >>>> DG+Squid 3.2.0.2: Problematic > >>>> > >>>> 2- We have mostly prıblems with the sites with web based upload status > >>>> viewers. Like rapidshare, youtube etc... > >>>> > >>>> 3- If Squid is the only proxy, no problem of uploading. > >>>> > >>>> 4- ead_ahead_gap 16 KB does not resolv the problem > >>>> > >>>> > >>>> Dear Developers, > >>>> > >>>> Can you propose some other workarounds for us to test? The problem is > >>>> encountered with most active sites of the net, unfortunately. > >>> > >>> This sounds like the same problem as > >>> http://bugs.squid-cache.org/show_bug.cgi?id=3017 > >> > > > > Sorry, crossing bug reports in my head. > > > > This one is closer to the suck-everything behaviour you have seen: > > http://bugs.squid-cache.org/show_bug.cgi?id=2910 > > > > both have an outside chance of working. > > I have tried both suggestions, and neither of them make a difference > (changes to BodyPipe.h and client_side_request.cc). > > I am keen to try any further suggestions, or provide you with debug output, > or whatever you like. > > This problem is extremely easy for me to reproduce. > It happens without any authentication, and with squid as the only proxy > between > my browser and the website. > > Shall I enter a proper bug report? To demonstrate the problem happening, I set on 'debug_options 33,2' and re-ran my test. This shows that ConnStateData::makeSpaceAvailable() in client_side.cc will eat memory forever. I can turn on more debug if needed, but others should be able to easily reproduce this. 2010/11/30 11:57:17.482| growing request buffer: notYetUsed=4095 size=8192 2010/11/30 11:57:17.483| growing request buffer: notYetUsed=8191 size=16384 2010/11/30 11:57:17.483| growing request buffer: notYetUsed=16383 size=32768 2010/11/30 11:57:17.484| growing request buffer: notYetUsed=32767 size=65536 2010/11/30 11:57:17.486| growing request buffer: notYetUsed=65535 size=131072 2010/11/30 11:57:17.488| growing request buffer: notYetUsed=131071 size=262144 2010/11/30 11:57:17.506| growing request buffer: notYetUsed=262143 size=524288 2010/11/30 11:57:17.533| growing request buffer: notYetUsed=524287 size=1048576 2010/11/30 11:57:17.586| growing request buffer: notYetUsed=1048575 size=2097152 2010/11/30 11:57:17.692| growing request buffer: notYetUsed=2097151 size=4194304 2010/11/30 11:57:17.884| growing request buffer: notYetUsed=4194303 size=8388608 2010/11/30 11:57:18.308| growing request buffer: notYetUsed=8388607 size=16777216 2010/11/30 11:57:19.136| growing request buffer: notYetUsed=16777215 size=33554432 2010/11/30 11:57:20.792| growing request buffer: notYetUsed=33554431 size=67108864 2010/11/30 11:57:23.957| growing request buffer: notYetUsed=67108863 size=134217728 2010/11/30 11:57:31.176| growing request buffer: notYetUsed=134217727 size=268435456 2010/11/30 11:57:58.433| growing request buffer: notYetUsed=268435455 size=536870912 ...
Re: [squid-users] squid-3.1 client POST buffering
On Tue, Nov 30, 2010 at 09:46:47PM +1300, Amos Jeffries wrote: > On 30/11/10 21:23, Oguz Yilmaz wrote: >> On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries wrote: >>> On 30/11/10 04:04, Oguz Yilmaz wrote: Graham, This is the best explanation I have seen about ongoing upload problem in proxy chains where squid is one part of the chain. On our systems, we use Squid 3.0.STABLE25. Before squid a dansguardian(DG) proxy exist to filter. Results of my tests: 1- DG+Squid 2.6.STABLE12: No problem of uploading DG+Squid 3.0.STABLE25: Problematic DG+Squid 3.1.8: Problematic DG+Squid 3.2.0.2: Problematic 2- We have mostly prıblems with the sites with web based upload status viewers. Like rapidshare, youtube etc... 3- If Squid is the only proxy, no problem of uploading. 4- ead_ahead_gap 16 KB does not resolv the problem Dear Developers, Can you propose some other workarounds for us to test? The problem is encountered with most active sites of the net, unfortunately. >>> >>> This sounds like the same problem as >>> http://bugs.squid-cache.org/show_bug.cgi?id=3017 >> > > Sorry, crossing bug reports in my head. > > This one is closer to the suck-everything behaviour you have seen: > http://bugs.squid-cache.org/show_bug.cgi?id=2910 > > both have an outside chance of working. I have tried both suggestions, and neither of them make a difference (changes to BodyPipe.h and client_side_request.cc). I am keen to try any further suggestions, or provide you with debug output, or whatever you like. This problem is extremely easy for me to reproduce. It happens without any authentication, and with squid as the only proxy between my browser and the website. Shall I enter a proper bug report?
Re: [squid-users] squid-3.1 client POST buffering
On Thu, Nov 25, 2010 at 04:36:49PM +, Graham Keeling wrote: > Hello, > > I have upgraded to squid-3.1 recently, and found a change of behaviour. > I have been using dansguardian in front of squid. > > It appears to be because squid now buffers uploaded POST data slightly > differently. > In versions < 3.1, it would take some data, send it through to the website, > and then ask for some more. > In 3.1 version, it appears to take as much from the client as it can without > waiting for what it has already got to be uploaded to the website. > > This means that dansguardian quickly uploads all the data into squid, and > then waits for a reply, which is a long time in coming because squid still > has to upload everything to the website. > And then dansguardian times out on squid after two minutes. > > > I noticed the following squid configuration option. Perhaps what I need is > a similar thing for buffering data sent from the client. > > # TAG: read_ahead_gap buffer-size > # The amount of data the cache will buffer ahead of what has been > # sent to the client when retrieving an object from another server. > #Default: > # read_ahead_gap 16 KB > > Comments welcome! > > Graham. Upon further experimentation, I have found that squid-3.1.x (specifically, I have tried squid-3.1.8 and squid-3.1.9) behaves very badly with POST uploads. It just increases the input buffer forever, until the upload is finished, or the machine runs out of memory. This problem exists when connecting directly to squid without dansguardian in the way. This problem doesn't exist on my old squid-2.5 installation.
[squid-users] squid-3.1 client POST buffering
Hello, I have upgraded to squid-3.1 recently, and found a change of behaviour. I have been using dansguardian in front of squid. It appears to be because squid now buffers uploaded POST data slightly differently. In versions < 3.1, it would take some data, send it through to the website, and then ask for some more. In 3.1 version, it appears to take as much from the client as it can without waiting for what it has already got to be uploaded to the website. This means that dansguardian quickly uploads all the data into squid, and then waits for a reply, which is a long time in coming because squid still has to upload everything to the website. And then dansguardian times out on squid after two minutes. I noticed the following squid configuration option. Perhaps what I need is a similar thing for buffering data sent from the client. # TAG: read_ahead_gap buffer-size # The amount of data the cache will buffer ahead of what has been # sent to the client when retrieving an object from another server. #Default: # read_ahead_gap 16 KB Comments welcome! Graham.
Re: [squid-users] Blocking or allowing specific arbitrary request headers in squid-3.1.
On Tue, Mar 16, 2010 at 09:46:44PM +, Amos Jeffries wrote: > On Tue, 16 Mar 2010 14:06:59 +0000, Graham Keeling > wrote: > > Hello, > > > > In older versions of squid, I was able to block or allow specific > arbitrary > > request headers. For example: > > > > header_access X-SomeRandomHeaderA allow all > > header_access X-SomeRandomHeaderB deny all > > > > In squid-3.1 (and 3.0, I think), the equivalent of header_access for > > request > > headers is now request_header_access. > > > > But if I try this, squid gets upset and doesn't start: > > request_header_access X-SomeRandomHeaderA allow all > > "allow all" is the default. You can ignore those settings. > > > request_header_access X-SomeRandomHeaderB deny all > > > > It says: > > 2010/03/16 13:55:19| parse_http_header_access: unknown header name > > 'X-SomeRandomHeaderA' > > > > So, it seems that you can only add headers that squid knows about > > internally. > > Which is what this page says: > > http://www.squid-cache.org/Doc/config/request_header_access/ > > > > You can only specify known headers for the header name. > > Other headers are reclassified as 'Other'. You can also > > refer to all the headers with 'All'. > > > > I could use 'Other', but it means that I have to treat all unknown > headers > > in the same way. Unless I'm missing something. > > > > > > So, can anybody tell me how to block or allow specific arbitrary request > > headers in squid-3.1? > > > > Not possible in Squid-3. Removing random headers is a violation of HTTP > protocol and can seriously break things when not understood. > > If you can present to us some information about these special headers that > shows they are in fact deserving of stripping, we can add them. Thanks for your reply. What I am doing is using a url/content filter in front of squid. I have the filter listening on two ports. One port is for filtering with authentication. One port is for filtering without authentication. The filter adds a header that says which port a request came in on. I then have a squid acl rule that matches the port in this header, which turns on authentication in squid. But, I don't want squid to then forward my new header out to the web server. And I don't want to use 'Other' to block it, as that will block application X and its proprietory headers. > There is always the eCAP/ICAP filtering add-on interface now available for > local control. > > > Rant: I for one got tired of handling complaints because application X > would not work through Squid when the admin stripped away all it's > proprietary headers. > > > Amos
[squid-users] Blocking or allowing specific arbitrary request headers in squid-3.1.
Hello, In older versions of squid, I was able to block or allow specific arbitrary request headers. For example: header_access X-SomeRandomHeaderA allow all header_access X-SomeRandomHeaderB deny all In squid-3.1 (and 3.0, I think), the equivalent of header_access for request headers is now request_header_access. But if I try this, squid gets upset and doesn't start: request_header_access X-SomeRandomHeaderA allow all request_header_access X-SomeRandomHeaderB deny all It says: 2010/03/16 13:55:19| parse_http_header_access: unknown header name 'X-SomeRandomHeaderA' So, it seems that you can only add headers that squid knows about internally. Which is what this page says: http://www.squid-cache.org/Doc/config/request_header_access/ You can only specify known headers for the header name. Other headers are reclassified as 'Other'. You can also refer to all the headers with 'All'. I could use 'Other', but it means that I have to treat all unknown headers in the same way. Unless I'm missing something. So, can anybody tell me how to block or allow specific arbitrary request headers in squid-3.1? Thank you.
Re: [squid-users] squid-3.1.0.15 and WCCPv1
On Fri, Jan 29, 2010 at 03:59:29PM +, Graham Keeling wrote: > On Fri, Jan 29, 2010 at 11:35:00AM +0000, Graham Keeling wrote: > > Hello, > > > > I am trying to upgrade my squid from 2.5 to 3.1. > > I have got all my old configuration working on 3.1, except... > > I am coming across a problem with WCCPv1. > > > > 172.16.13.56 is the address of the squid box. > > 172.16.13.2 is the address of the cisco router. > > I have wccp_router=172.16.13.2 in my squid.conf. > > > > squid-2.5 connects to UDP port 2048, I get replies, and everything else then > > works. Here is a tcpdump of the initial connection: > > > > 16:12:13.404466 IP 172.16.13.56.2048 > 172.16.13.2.2048: UDP, length 52 > > 16:12:13.406764 IP 172.16.13.2.2048 > 172.16.13.56.2048: UDP, length 64 > > > > > > But, squid-3.1 looks like it is trying to connect to UDP port 0 on the > > cisco. > > Here is the equivalent tcpdump with squid-3.1: > > > > 15:59:10.093415 IP 172.16.13.56.2048 > 172.16.13.2.0: UDP, length 52 > > 15:59:10.094423 IP 172.16.13.2 > 172.16.13.56: ICMP 172.16.13.2 udp port 0 > > unreachable, length > > > > > > I have looked at the src/wccp.c for squid-2.5, and it is clear that the > > port is > > being set to 2048 for the connection to the router. > > I have also looked at the source for 2.6, 2.7 and 3.0 (src/wccp.cc for this > > version). > > In all those, it appears to be setting the port on the outgoing connection. > > > > However, in the 3.1 source, it doesn't. > > > > Is this a bug? > > > > Has anybody got WCCPv1 working with squid-3.1? > > > Further information: > I've now tried squid-3.0.STABLE21, and WCCPv1 worked fine. > > Conclusion: > WCCPv1 is broken in squid-3.1. > > Is this the correct list to be reporting this to? My squid-3.1 WCCPv1 appears to work with the attached patch that I just made. diff -u -r1.1 wccp.cc --- src/wccp.cc 22 Dec 2009 13:54:53 - 1.1 +++ src/wccp.cc 29 Jan 2010 16:19:46 - @@ -146,6 +146,7 @@ } Config.Wccp.address.SetPort(WCCP_PORT); +Config.Wccp.router.SetPort(WCCP_PORT); theWccpConnection = comm_open_listener(SOCK_DGRAM, IPPROTO_UDP,
Re: [squid-users] squid-3.1.0.15 and WCCPv1
On Fri, Jan 29, 2010 at 11:35:00AM +, Graham Keeling wrote: > Hello, > > I am trying to upgrade my squid from 2.5 to 3.1. > I have got all my old configuration working on 3.1, except... > I am coming across a problem with WCCPv1. > > 172.16.13.56 is the address of the squid box. > 172.16.13.2 is the address of the cisco router. > I have wccp_router=172.16.13.2 in my squid.conf. > > squid-2.5 connects to UDP port 2048, I get replies, and everything else then > works. Here is a tcpdump of the initial connection: > > 16:12:13.404466 IP 172.16.13.56.2048 > 172.16.13.2.2048: UDP, length 52 > 16:12:13.406764 IP 172.16.13.2.2048 > 172.16.13.56.2048: UDP, length 64 > > > But, squid-3.1 looks like it is trying to connect to UDP port 0 on the cisco. > Here is the equivalent tcpdump with squid-3.1: > > 15:59:10.093415 IP 172.16.13.56.2048 > 172.16.13.2.0: UDP, length 52 > 15:59:10.094423 IP 172.16.13.2 > 172.16.13.56: ICMP 172.16.13.2 udp port 0 > unreachable, length > > > I have looked at the src/wccp.c for squid-2.5, and it is clear that the port > is > being set to 2048 for the connection to the router. > I have also looked at the source for 2.6, 2.7 and 3.0 (src/wccp.cc for this > version). > In all those, it appears to be setting the port on the outgoing connection. > > However, in the 3.1 source, it doesn't. > > Is this a bug? > > Has anybody got WCCPv1 working with squid-3.1? Further information: I've now tried squid-3.0.STABLE21, and WCCPv1 worked fine. Conclusion: WCCPv1 is broken in squid-3.1. Is this the correct list to be reporting this to?
[squid-users] squid-3.1.0.15 and WCCPv1
Hello, I am trying to upgrade my squid from 2.5 to 3.1. I have got all my old configuration working on 3.1, except... I am coming across a problem with WCCPv1. 172.16.13.56 is the address of the squid box. 172.16.13.2 is the address of the cisco router. I have wccp_router=172.16.13.2 in my squid.conf. squid-2.5 connects to UDP port 2048, I get replies, and everything else then works. Here is a tcpdump of the initial connection: 16:12:13.404466 IP 172.16.13.56.2048 > 172.16.13.2.2048: UDP, length 52 16:12:13.406764 IP 172.16.13.2.2048 > 172.16.13.56.2048: UDP, length 64 But, squid-3.1 looks like it is trying to connect to UDP port 0 on the cisco. Here is the equivalent tcpdump with squid-3.1: 15:59:10.093415 IP 172.16.13.56.2048 > 172.16.13.2.0: UDP, length 52 15:59:10.094423 IP 172.16.13.2 > 172.16.13.56: ICMP 172.16.13.2 udp port 0 unreachable, length I have looked at the src/wccp.c for squid-2.5, and it is clear that the port is being set to 2048 for the connection to the router. I have also looked at the source for 2.6, 2.7 and 3.0 (src/wccp.cc for this version). In all those, it appears to be setting the port on the outgoing connection. However, in the 3.1 source, it doesn't. Is this a bug? Has anybody got WCCPv1 working with squid-3.1?
