Re: [squid-users] squid-3.1 client POST buffering
On Wed, Dec 01, 2010 at 11:40:52AM +, Graham Keeling wrote: > Hello, > > I am convinced that this is a serious bug, so I have entered a proper bug > report. > > It is bug 3113: > > http://bugs.squid-cache.org/show_bug.cgi?id=3113 I have created a simple patch that seems to fix the problem for me on squid-3.1.9, and I have attached it to the bug report.
Re: [squid-users] squid-3.1 client POST buffering
Hello, I am convinced that this is a serious bug, so I have entered a proper bug report. It is bug 3113: http://bugs.squid-cache.org/show_bug.cgi?id=3113
Re: [squid-users] squid-3.1 client POST buffering
On Tue, Nov 30, 2010 at 11:31:45AM +, Graham Keeling wrote: > On Tue, Nov 30, 2010 at 09:46:47PM +1300, Amos Jeffries wrote: > > On 30/11/10 21:23, Oguz Yilmaz wrote: > >> On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries > >> wrote: > >>> On 30/11/10 04:04, Oguz Yilmaz wrote: > > Graham, > > This is the best explanation I have seen about ongoing upload problem > in proxy chains where squid is one part of the chain. > > On our systems, we use Squid 3.0.STABLE25. Before squid a > dansguardian(DG) proxy exist to filter. Results of my tests: > > 1- > DG+Squid 2.6.STABLE12: No problem of uploading > DG+Squid 3.0.STABLE25: Problematic > DG+Squid 3.1.8: Problematic > DG+Squid 3.2.0.2: Problematic > > 2- We have mostly prıblems with the sites with web based upload status > viewers. Like rapidshare, youtube etc... > > 3- If Squid is the only proxy, no problem of uploading. > > 4- ead_ahead_gap 16 KB does not resolv the problem > > > Dear Developers, > > Can you propose some other workarounds for us to test? The problem is > encountered with most active sites of the net, unfortunately. > >>> > >>> This sounds like the same problem as > >>> http://bugs.squid-cache.org/show_bug.cgi?id=3017 > >> > > > > Sorry, crossing bug reports in my head. > > > > This one is closer to the suck-everything behaviour you have seen: > > http://bugs.squid-cache.org/show_bug.cgi?id=2910 > > > > both have an outside chance of working. > > I have tried both suggestions, and neither of them make a difference > (changes to BodyPipe.h and client_side_request.cc). > > I am keen to try any further suggestions, or provide you with debug output, > or whatever you like. > > This problem is extremely easy for me to reproduce. > It happens without any authentication, and with squid as the only proxy > between > my browser and the website. > > Shall I enter a proper bug report? To demonstrate the problem happening, I set on 'debug_options 33,2' and re-ran my test. This shows that ConnStateData::makeSpaceAvailable() in client_side.cc will eat memory forever. I can turn on more debug if needed, but others should be able to easily reproduce this. 2010/11/30 11:57:17.482| growing request buffer: notYetUsed=4095 size=8192 2010/11/30 11:57:17.483| growing request buffer: notYetUsed=8191 size=16384 2010/11/30 11:57:17.483| growing request buffer: notYetUsed=16383 size=32768 2010/11/30 11:57:17.484| growing request buffer: notYetUsed=32767 size=65536 2010/11/30 11:57:17.486| growing request buffer: notYetUsed=65535 size=131072 2010/11/30 11:57:17.488| growing request buffer: notYetUsed=131071 size=262144 2010/11/30 11:57:17.506| growing request buffer: notYetUsed=262143 size=524288 2010/11/30 11:57:17.533| growing request buffer: notYetUsed=524287 size=1048576 2010/11/30 11:57:17.586| growing request buffer: notYetUsed=1048575 size=2097152 2010/11/30 11:57:17.692| growing request buffer: notYetUsed=2097151 size=4194304 2010/11/30 11:57:17.884| growing request buffer: notYetUsed=4194303 size=8388608 2010/11/30 11:57:18.308| growing request buffer: notYetUsed=8388607 size=16777216 2010/11/30 11:57:19.136| growing request buffer: notYetUsed=16777215 size=33554432 2010/11/30 11:57:20.792| growing request buffer: notYetUsed=33554431 size=67108864 2010/11/30 11:57:23.957| growing request buffer: notYetUsed=67108863 size=134217728 2010/11/30 11:57:31.176| growing request buffer: notYetUsed=134217727 size=268435456 2010/11/30 11:57:58.433| growing request buffer: notYetUsed=268435455 size=536870912 ...
Re: [squid-users] squid-3.1 client POST buffering
On Tue, Nov 30, 2010 at 09:46:47PM +1300, Amos Jeffries wrote: > On 30/11/10 21:23, Oguz Yilmaz wrote: >> On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries wrote: >>> On 30/11/10 04:04, Oguz Yilmaz wrote: Graham, This is the best explanation I have seen about ongoing upload problem in proxy chains where squid is one part of the chain. On our systems, we use Squid 3.0.STABLE25. Before squid a dansguardian(DG) proxy exist to filter. Results of my tests: 1- DG+Squid 2.6.STABLE12: No problem of uploading DG+Squid 3.0.STABLE25: Problematic DG+Squid 3.1.8: Problematic DG+Squid 3.2.0.2: Problematic 2- We have mostly prıblems with the sites with web based upload status viewers. Like rapidshare, youtube etc... 3- If Squid is the only proxy, no problem of uploading. 4- ead_ahead_gap 16 KB does not resolv the problem Dear Developers, Can you propose some other workarounds for us to test? The problem is encountered with most active sites of the net, unfortunately. >>> >>> This sounds like the same problem as >>> http://bugs.squid-cache.org/show_bug.cgi?id=3017 >> > > Sorry, crossing bug reports in my head. > > This one is closer to the suck-everything behaviour you have seen: > http://bugs.squid-cache.org/show_bug.cgi?id=2910 > > both have an outside chance of working. I have tried both suggestions, and neither of them make a difference (changes to BodyPipe.h and client_side_request.cc). I am keen to try any further suggestions, or provide you with debug output, or whatever you like. This problem is extremely easy for me to reproduce. It happens without any authentication, and with squid as the only proxy between my browser and the website. Shall I enter a proper bug report?
Re: [squid-users] squid-3.1 client POST buffering
-- Oguz YILMAZ On Tue, Nov 30, 2010 at 10:46 AM, Amos Jeffries wrote: > On 30/11/10 21:23, Oguz Yilmaz wrote: >> >> On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries >> wrote: >>> >>> On 30/11/10 04:04, Oguz Yilmaz wrote: Graham, This is the best explanation I have seen about ongoing upload problem in proxy chains where squid is one part of the chain. On our systems, we use Squid 3.0.STABLE25. Before squid a dansguardian(DG) proxy exist to filter. Results of my tests: 1- DG+Squid 2.6.STABLE12: No problem of uploading DG+Squid 3.0.STABLE25: Problematic DG+Squid 3.1.8: Problematic DG+Squid 3.2.0.2: Problematic 2- We have mostly prıblems with the sites with web based upload status viewers. Like rapidshare, youtube etc... 3- If Squid is the only proxy, no problem of uploading. 4- ead_ahead_gap 16 KB does not resolv the problem Dear Developers, Can you propose some other workarounds for us to test? The problem is encountered with most active sites of the net, unfortunately. >>> >>> This sounds like the same problem as >>> http://bugs.squid-cache.org/show_bug.cgi?id=3017 >> > > Sorry, crossing bug reports in my head. > > This one is closer to the suck-everything behaviour you have seen: > http://bugs.squid-cache.org/show_bug.cgi?id=2910 > > both have an outside chance of working. > I have tried the patch proposed (BodyPipe.h). However does not work. Note: My system is based on Linux os. >> >> In my tests, no NTLM auth was used. >> The browser has proxy confguration targeting DG and DG uses squid as >> provider proxy. If you think it will work, I can try the patch >> located in the bug case. >> Upload will stop at about 1MB, so is it about SQUID_TCP_SO_RCVBUF? > > AIUI, Squid is supposed to read SQUID_TCP_SO_RCVBUF + read_ahead_gap and > wait for some of that to pass on to the server before grabbing some more. > > Amos > -- > Please be using > Current Stable Squid 2.7.STABLE9 or 3.1.9 > Beta testers wanted for 3.2.0.3 >
Re: [squid-users] squid-3.1 client POST buffering
On 30/11/10 21:23, Oguz Yilmaz wrote: On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries wrote: On 30/11/10 04:04, Oguz Yilmaz wrote: Graham, This is the best explanation I have seen about ongoing upload problem in proxy chains where squid is one part of the chain. On our systems, we use Squid 3.0.STABLE25. Before squid a dansguardian(DG) proxy exist to filter. Results of my tests: 1- DG+Squid 2.6.STABLE12: No problem of uploading DG+Squid 3.0.STABLE25: Problematic DG+Squid 3.1.8: Problematic DG+Squid 3.2.0.2: Problematic 2- We have mostly prıblems with the sites with web based upload status viewers. Like rapidshare, youtube etc... 3- If Squid is the only proxy, no problem of uploading. 4- ead_ahead_gap 16 KB does not resolv the problem Dear Developers, Can you propose some other workarounds for us to test? The problem is encountered with most active sites of the net, unfortunately. This sounds like the same problem as http://bugs.squid-cache.org/show_bug.cgi?id=3017 Sorry, crossing bug reports in my head. This one is closer to the suck-everything behaviour you have seen: http://bugs.squid-cache.org/show_bug.cgi?id=2910 both have an outside chance of working. In my tests, no NTLM auth was used. The browser has proxy confguration targeting DG and DG uses squid as provider proxy. If you think it will work, I can try the patch located in the bug case. Upload will stop at about 1MB, so is it about SQUID_TCP_SO_RCVBUF? AIUI, Squid is supposed to read SQUID_TCP_SO_RCVBUF + read_ahead_gap and wait for some of that to pass on to the server before grabbing some more. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.9 Beta testers wanted for 3.2.0.3
Re: [squid-users] squid-3.1 client POST buffering
On Tue, Nov 30, 2010 at 10:05 AM, Amos Jeffries wrote: > On 30/11/10 04:04, Oguz Yilmaz wrote: >> >> Graham, >> >> This is the best explanation I have seen about ongoing upload problem >> in proxy chains where squid is one part of the chain. >> >> On our systems, we use Squid 3.0.STABLE25. Before squid a >> dansguardian(DG) proxy exist to filter. Results of my tests: >> >> 1- >> DG+Squid 2.6.STABLE12: No problem of uploading >> DG+Squid 3.0.STABLE25: Problematic >> DG+Squid 3.1.8: Problematic >> DG+Squid 3.2.0.2: Problematic >> >> 2- We have mostly prıblems with the sites with web based upload status >> viewers. Like rapidshare, youtube etc... >> >> 3- If Squid is the only proxy, no problem of uploading. >> >> 4- ead_ahead_gap 16 KB does not resolv the problem >> >> >> Dear Developers, >> >> Can you propose some other workarounds for us to test? The problem is >> encountered with most active sites of the net, unfortunately. > > This sounds like the same problem as > http://bugs.squid-cache.org/show_bug.cgi?id=3017 In my tests, no NTLM auth was used. The browser has proxy confguration targeting DG and DG uses squid as provider proxy. If you think it will work, I can try the patch located in the bug case. Upload will stop at about 1MB, so is it about SQUID_TCP_SO_RCVBUF? > > Amos > -- > Please be using > Current Stable Squid 2.7.STABLE9 or 3.1.9 > Beta testers wanted for 3.2.0.3 >
Re: [squid-users] squid-3.1 client POST buffering
On 30/11/10 04:04, Oguz Yilmaz wrote: Graham, This is the best explanation I have seen about ongoing upload problem in proxy chains where squid is one part of the chain. On our systems, we use Squid 3.0.STABLE25. Before squid a dansguardian(DG) proxy exist to filter. Results of my tests: 1- DG+Squid 2.6.STABLE12: No problem of uploading DG+Squid 3.0.STABLE25: Problematic DG+Squid 3.1.8: Problematic DG+Squid 3.2.0.2: Problematic 2- We have mostly prıblems with the sites with web based upload status viewers. Like rapidshare, youtube etc... 3- If Squid is the only proxy, no problem of uploading. 4- ead_ahead_gap 16 KB does not resolv the problem Dear Developers, Can you propose some other workarounds for us to test? The problem is encountered with most active sites of the net, unfortunately. This sounds like the same problem as http://bugs.squid-cache.org/show_bug.cgi?id=3017 Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.9 Beta testers wanted for 3.2.0.3
Re: [squid-users] squid-3.1 client POST buffering
Graham, This is the best explanation I have seen about ongoing upload problem in proxy chains where squid is one part of the chain. On our systems, we use Squid 3.0.STABLE25. Before squid a dansguardian(DG) proxy exist to filter. Results of my tests: 1- DG+Squid 2.6.STABLE12: No problem of uploading DG+Squid 3.0.STABLE25: Problematic DG+Squid 3.1.8: Problematic DG+Squid 3.2.0.2: Problematic 2- We have mostly prıblems with the sites with web based upload status viewers. Like rapidshare, youtube etc... 3- If Squid is the only proxy, no problem of uploading. 4- ead_ahead_gap 16 KB does not resolv the problem Dear Developers, Can you propose some other workarounds for us to test? The problem is encountered with most active sites of the net, unfortunately. Best Regards, -- Oguz YILMAZ On Thu, Nov 25, 2010 at 6:36 PM, Graham Keeling wrote: > > Hello, > > I have upgraded to squid-3.1 recently, and found a change of behaviour. > I have been using dansguardian in front of squid. > > It appears to be because squid now buffers uploaded POST data slightly > differently. > In versions < 3.1, it would take some data, send it through to the website, > and then ask for some more. > In 3.1 version, it appears to take as much from the client as it can without > waiting for what it has already got to be uploaded to the website. > > This means that dansguardian quickly uploads all the data into squid, and > then waits for a reply, which is a long time in coming because squid still > has to upload everything to the website. > And then dansguardian times out on squid after two minutes. > > > I noticed the following squid configuration option. Perhaps what I need is > a similar thing for buffering data sent from the client. > > # TAG: read_ahead_gap buffer-size > # The amount of data the cache will buffer ahead of what has been > # sent to the client when retrieving an object from another server. > #Default: > # read_ahead_gap 16 KB > > Comments welcome! > > Graham. >
Re: [squid-users] squid-3.1 client POST buffering
On 27/11/10 05:10, Graham Keeling wrote: On Thu, Nov 25, 2010 at 04:36:49PM +, Graham Keeling wrote: Hello, I have upgraded to squid-3.1 recently, and found a change of behaviour. I have been using dansguardian in front of squid. It appears to be because squid now buffers uploaded POST data slightly differently. In versions< 3.1, it would take some data, send it through to the website, and then ask for some more. In 3.1 version, it appears to take as much from the client as it can without waiting for what it has already got to be uploaded to the website. This means that dansguardian quickly uploads all the data into squid, and then waits for a reply, which is a long time in coming because squid still has to upload everything to the website. And then dansguardian times out on squid after two minutes. I noticed the following squid configuration option. Perhaps what I need is a similar thing for buffering data sent from the client. # TAG: read_ahead_gap buffer-size # The amount of data the cache will buffer ahead of what has been # sent to the client when retrieving an object from another server. #Default: # read_ahead_gap 16 KB Comments welcome! Graham. Upon further experimentation, I have found that squid-3.1.x (specifically, I have tried squid-3.1.8 and squid-3.1.9) behaves very badly with POST uploads. It just increases the input buffer forever, until the upload is finished, or the machine runs out of memory. This problem exists when connecting directly to squid without dansguardian in the way. This problem doesn't exist on my old squid-2.5 installation. Buffer max is 64KB. I'm thinking this is related to http://bugs.squid-cache.org/show_bug.cgi?id=2910 Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.9 Beta testers wanted for 3.2.0.3
Re: [squid-users] squid-3.1 client POST buffering
On Thu, Nov 25, 2010 at 04:36:49PM +, Graham Keeling wrote: > Hello, > > I have upgraded to squid-3.1 recently, and found a change of behaviour. > I have been using dansguardian in front of squid. > > It appears to be because squid now buffers uploaded POST data slightly > differently. > In versions < 3.1, it would take some data, send it through to the website, > and then ask for some more. > In 3.1 version, it appears to take as much from the client as it can without > waiting for what it has already got to be uploaded to the website. > > This means that dansguardian quickly uploads all the data into squid, and > then waits for a reply, which is a long time in coming because squid still > has to upload everything to the website. > And then dansguardian times out on squid after two minutes. > > > I noticed the following squid configuration option. Perhaps what I need is > a similar thing for buffering data sent from the client. > > # TAG: read_ahead_gap buffer-size > # The amount of data the cache will buffer ahead of what has been > # sent to the client when retrieving an object from another server. > #Default: > # read_ahead_gap 16 KB > > Comments welcome! > > Graham. Upon further experimentation, I have found that squid-3.1.x (specifically, I have tried squid-3.1.8 and squid-3.1.9) behaves very badly with POST uploads. It just increases the input buffer forever, until the upload is finished, or the machine runs out of memory. This problem exists when connecting directly to squid without dansguardian in the way. This problem doesn't exist on my old squid-2.5 installation.
