Re: [squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
On Wed, 08 Jun 2011 16:47:34 -0500, Harry Putnam wrote: Amos Jeffries writes: [...] This may help... http://www.extremetech.com/article2/0,2845,1854196,00.asp Thanks for the continuing good information. About that citation above: The author of the hacks on that page has suggested that readers might edit a `user.js' file. Probably something of a dumb question but, is that something I am supposed to create myself, like: `%APPDATA%\Mozilla\Firefox\Profiles\xxx.default\user.js Or should that file already exist? I don't see a file with that name on a windows or linux install of recent firefox. I'm not sure what the author was smoking when he mentioned that. The settings are all available for viewing and editing by entering about:config in the browser address bar. Amos
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
Amos Jeffries writes: [...] Harry wrote: >> What kind of things should I do to start tracking down what is >> hampering the connections so bad. >> > Amos replied: > This may help... > http://www.extremetech.com/article2/0,2845,1854196,00.asp After creating a user.js file in the mozilla profile default containing: --- 8< snip -- 8< snip -- 8http://www.ford-trucks.com/ Between 17-19 seconds ---- ---=--- - Then trying the connection with no proxy: kill firefox, start FF, clear cache, Type in http://www.ford-trucks.com/ Connection was fully completed in 8 seconds again. ---- ---=--- - Now trying the first experiment (privoxy alone as proxy) but this time with no `user.js' file: Connection is completed in 16 seconds plus. ---- ---=--- - Now with user.js in place but no proxy Connection completes in 8 seconds ---- ---=--- - ---- ---=--- - And since this is a squid group. Proxy set to squid alone (user.js in place) The connection to about 30 seconds for roughly 60% of that ford cite to load. I abandoned the completing the connection after 2 full minutes. ---- ---=--- - Then to be fair, I removed the user.js hacks. Restarted firefox, (with proxy set to squid alone and no user.js). First, it took a goodly while for my home page (goggle.com/ig) to load, cleared the cache and typed in: http://www.ford-trucks.com About 60% loaded in 12 seconds and completed in about 35 seconds. ---- ---=--- - Apparently the `user.js' hack isn't all that helpful. And in fact appears to be something of a hindrance. (Note: I ran these experiments repeatedly.. the results shown above are about the average results obtained) I don't know what to make of it all, I guess my configs are pretty weak being fairly default for both privoxy and squid, but so far it appears that both squid and privoxy cause serious slowdown of page loading. My full configs: , | Squid: |www.jtan.com/~reader/sqcfg/disp.cgi ( squid.conf with all default |comments). | |www.jtan.com/~reader/sqcfg/strp.cgi (squid.conf stripped). | | Privoxy: |www.jtan.com/~reader/prcfg/disp.cgi (privoxy config with all |default comments). | |www.jtan.com/~reader/prcfg/strp.cgi (privoxy config stripped). `
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
Amos Jeffries writes: [...] > This may help... > http://www.extremetech.com/article2/0,2845,1854196,00.asp Thanks for the continuing good information. About that citation above: The author of the hacks on that page has suggested that readers might edit a `user.js' file. Probably something of a dumb question but, is that something I am supposed to create myself, like: `%APPDATA%\Mozilla\Firefox\Profiles\xxx.default\user.js Or should that file already exist? I don't see a file with that name on a windows or linux install of recent firefox.
Re: [squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
On 05/06/11 11:55, Harry Putnam wrote: Amos Jeffries writes: [...] Harry wrote (summarized -ed hp): Adding squid and privoxy into a proxy setup seems to really really slow down my browsing as compared to browsing with a direct connection. (no proxy) And asks if this is normal. [...] Speed gain/loss/other depends on what you are moving from. MORE IMPORTANTLY: how you define "slow"! Keep in mind that you also now have around 2x the processing going on with 2 proxies. The difference added by Squid can be at least 10ms. Some people call that noticeable slowdown. Some dont care about anything less than a second. I'm guessing its more than seconds slower but not really sure how to gage the difference reliably, so as not to be giving flawed information here or have difference due to caching or something. Can you suggest a method to arrive at a fairly good comparison? The "developer tools" in any NetKit browser (Safari, Chrome, Gecko etc) or FireBug in Firefox or Iceweasel browsers shows a wealth of timing information on the network analysis panel. One forced load with and one without will tell you pretty clearly what the timing and speed difference is. And whether its page processing or transfer lag. * 3.1 is about 10-20% slower than the latest 2.7 on the same config. With the older versions of 3.1 being on the slower end of that scale as we work to optimize and fix things throughout the series. Wouldn't 20% be noticeable? So you're saying to back down a few versions for now? Possibly. It varies a lot on where and how you run Squid. The "slow" here is in CPU time cycles. So if your CPU is maxed out you see it as real seconds, if you have spare CPU it should not be noticeably different. * Moving to Squid from a non-proxy setup can be a major drop down depending on the browser age. The browsers themselves drop the parallel fetch rate from hundreds down to under 10. Browser tweaking is the only way to avoid this. I'm using firefox 4.X on all home lan machines (that have a gui). Can you recommend some documentation that might help with what you called `Browser tweaking', I've never done anything special to a browser other than add or subtract add-on tools. * Moving from browser->privoxy to a browser->squid->privoxy setup you should have seen only a small drop. Some possibilities are Squid using slow disks (maybe RAID), or Squid box is swapping, or the bandwidth is being routed down the same physical links to/from Squid. No raid, and the hardware of the server is P4 Intel(R) Celeron(R) CPU 3.06GHz and 2GBram running on oldish IDE discs ---- ---=--- - Probably should have included some questions about the squid.conf and privoxy/config in the first post. Maybe there are things in there that are not good left as default. I realize that both tools have several config files. I've left all but the two main ones in default state and have posted my working /etc/squid/squid.conf and /etc/privoxy/config There is a lot of debris in the comments but still I thought it might be useful to leave it all in for jogging memories...But also included a way to prune the comments by changing the name of the cgi script at the end of the URL from `disp.cgi' to `strp.cgi'. Any coaching would be well appreciated. Looks good. :) The audit results so far: * patch is reversed. Please diff the files the other way around on the next one :) * "_SQUID_INLINE_ static" is not as portable as we would like. So we have to avoid it for now. * instead of debugs() level 0 or 1 we have a macro DBG_CRITICAL or DBG_IMPORTANT to indicate how bad the problem is. Most of what you have at level 28,0 only needs 28,DBG_IMPORTANT. * rather than naming the function all over the place please use the macro HERE to start the debugs text in debugs level 2 and higher. * also, stuff important enough for level 0 and 1 is usually user visible and should be describing a problem+solution clearly enough not to need the internal function name. compileRE(): * the bad pattern error message seems to work better like this: debugs(28, DBG_IMPORTANT, "WARNING: Skipping invalid regex in " << cfg_filename << " line " << config_lineno << ": '" << config_input_line << "': " << errbuf); debugs(28, 7, HERE << "compiled regex:'" << RE << "': " << errbuf); compileOptimisedREs(): * the if (RElen > BUFSIZ-1) case is done by aclParseRegexList(). It is redundant here. * you follow "if (RElen + largeREindex + 3 < BUFSIZ-1)" with four ++ operations and assignments. Can the fourth overflow the buffer? * at the end of the if-elif-else sequence you state "do the loop again to add the RE to largeRE" then continue;. What is that comment meaning? the compiled pattern is used as prefix for further appended patterns? compileUnoptimisedREs(): * the if (RElen > BUFSIZ-1) case is done by aclParseRegexList(). It is redundant h
Re: [squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
On 05/06/11 13:54, Harry Putnam wrote: Amos Jeffries writes: [...] Speed gain/loss/other depends on what you are moving from. MORE IMPORTANTLY: how you define "slow"! OK, getting down to cases here. Here is one test.. First clear the cache just the simple way tools/options/advanced/network `clear now' Close firefox (4.0.1) Start firefox (It starts on google which is struggling to resolve) Clear the cache once more Type this URL in http://www.ford-trucks.com Start stopwatch, then start firefox on that address by hitting enter.. I get 1:32 (one min, 32 sec) with chain of browser => squid => privoxy in place Now with no proxy at all. CLear the cache, kill firefox. Start firefox, (it starts on google very quickly), clear the cache once more for good measure. Type in http://www.ford-trucks.com, Start the stopwatch, then start firefox on that address by hitting enter... I get 0:9 (9 seconds) with no proxy in place That is something on the order of 900% faster... I think. ---- ---=--- - What kind of things should I do to start tracking down what is hampering the connections so bad. This may help... http://www.extremetech.com/article2/0,2845,1854196,00.asp Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
Amos Jeffries writes: [...] > Speed gain/loss/other depends on what you are moving from. > > MORE IMPORTANTLY: how you define "slow"! > OK, getting down to cases here. Here is one test.. First clear the cache just the simple way tools/options/advanced/network `clear now' Close firefox (4.0.1) Start firefox (It starts on google which is struggling to resolve) Clear the cache once more Type this URL in http://www.ford-trucks.com Start stopwatch, then start firefox on that address by hitting enter.. I get 1:32 (one min, 32 sec) with chain of browser => squid => privoxy in place Now with no proxy at all. CLear the cache, kill firefox. Start firefox, (it starts on google very quickly), clear the cache once more for good measure. Type in http://www.ford-trucks.com, Start the stopwatch, then start firefox on that address by hitting enter... I get 0:9 (9 seconds) with no proxy in place That is something on the order of 900% faster... I think. ---- ---=--- - What kind of things should I do to start tracking down what is hampering the connections so bad.
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
Amos Jeffries writes: [...] Harry wrote (summarized -ed hp): Adding squid and privoxy into a proxy setup seems to really really slow down my browsing as compared to browsing with a direct connection. (no proxy) And asks if this is normal. [...] > Speed gain/loss/other depends on what you are moving from. > > MORE IMPORTANTLY: how you define "slow"! > > Keep in mind that you also now have around 2x the processing going on > with 2 proxies. The difference added by Squid can be at least > 10ms. Some people call that noticeable slowdown. Some dont care about > anything less than a second. I'm guessing its more than seconds slower but not really sure how to gage the difference reliably, so as not to be giving flawed information here or have difference due to caching or something. Can you suggest a method to arrive at a fairly good comparison? > * 3.1 is about 10-20% slower than the latest 2.7 on the same > config. With the older versions of 3.1 being on the slower end of that > scale as we work to optimize and fix things throughout the series. Wouldn't 20% be noticeable? So you're saying to back down a few versions for now? > * Moving to Squid from a non-proxy setup can be a major drop down > depending on the browser age. The browsers themselves drop the > parallel fetch rate from hundreds down to under 10. Browser tweaking > is the only way to avoid this. I'm using firefox 4.X on all home lan machines (that have a gui). Can you recommend some documentation that might help with what you called `Browser tweaking', I've never done anything special to a browser other than add or subtract add-on tools. > * Moving from browser->privoxy to a browser->squid->privoxy setup you > should have seen only a small drop. Some possibilities are Squid using > slow disks (maybe RAID), or Squid box is swapping, or the bandwidth is > being routed down the same physical links to/from Squid. No raid, and the hardware of the server is P4 Intel(R) Celeron(R) CPU 3.06GHz and 2GBram running on oldish IDE discs ---- ---=--- - Probably should have included some questions about the squid.conf and privoxy/config in the first post. Maybe there are things in there that are not good left as default. I realize that both tools have several config files. I've left all but the two main ones in default state and have posted my working /etc/squid/squid.conf and /etc/privoxy/config There is a lot of debris in the comments but still I thought it might be useful to leave it all in for jogging memories...But also included a way to prune the comments by changing the name of the cgi script at the end of the URL from `disp.cgi' to `strp.cgi'. Any coaching would be well appreciated. ---- ---=--- - squid.conf WITH comments: www.jtan.com/~reader/sqcfg/disp.cgi squid.conf WITH OUT comments www.jtan.com/~reader/sqcfg/strp.cgi = privoxy's config WITH comments: www.jtan.com/~reader/prcfg/disp.cgi privoxy's config WITH OUT comments: www.jtan.com/~reader/prcfg/strp.cgi
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
Eliezer Croitoru writes: > another question. > are you by any change made a speedtest on the current setup? > if you do get the right speed but wrong speed to get the page > processed it's a known side effect of privoxy. Is there some standard way to do a speed test? I'm not sure I'd know how to do one that was at least semi-scientific. Find a slow loading page and do `time firefox URL'... killing it soon as it loads? Then clean the cache and do it again with squid and privoxy in the loop?
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
sichent writes: >> Setup: Gentoo linux OS on squid and privoxy home lan server >> Squid-3.1.12 >> privoxy-3.0.17 >> >> I'm not running an html server, just trying to use squid and privoxy >> for my own browsing. >> > > Why not to use ICAP or URL rewriter functionality built into Squid to > achieve the same results as privoxy instead of having this "chaining" > setup? Well for starts, I'd be jumping off into stuff I know nothing about and more than likely end up spending way more time than I should on it. I don't know much about squid and privoxy either but at least have tinkered with them several times over the years.
Re: [squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
On 04/06/11 16:23, Eliezer Croitoru wrote: On 04/06/2011 02:05, sichent wrote: Setup: Gentoo linux OS on squid and privoxy home lan server Squid-3.1.12 privoxy-3.0.17 I'm not running an html server, just trying to use squid and privoxy for my own browsing. Why not to use ICAP or URL rewriter functionality built into Squid to achieve the same results as privoxy instead of having this "chaining" setup? Sorry for possible offtopic :) What offtopic? the only problem i can thing of is cpu and process management on this server. icap is a nice idea and i can recommend one http://greasyspoon.sourceforge.net/ you must know basic programming to make it work and they have rules samples. if your hardware capabilities are low you will get most likely poor results on both privoxy and ICAP. also i will send later the settings for working with greasyspoon on squid 3.2 another question. are you by any change made a speedtest on the current setup? if you do get the right speed but wrong speed to get the page processed it's a known side effect of privoxy. Regards Eliezer sich Speed gain/loss/other depends on what you are moving from. MORE IMPORTANTLY: how you define "slow"! Keep in mind that you also now have around 2x the processing going on with 2 proxies. The difference added by Squid can be at least 10ms. Some people call that noticeable slowdown. Some dont care about anything less than a second. * 3.1 is about 10-20% slower than the latest 2.7 on the same config. With the older versions of 3.1 being on the slower end of that scale as we work to optimize and fix things throughout the series. * Moving to Squid from a non-proxy setup can be a major drop down depending on the browser age. The browsers themselves drop the parallel fetch rate from hundreds down to under 10. Browser tweaking is the only way to avoid this. * Moving from browser->privoxy to a browser->squid->privoxy setup you should have seen only a small drop. Some possibilities are Squid using slow disks (maybe RAID), or Squid box is swapping, or the bandwidth is being routed down the same physical links to/from Squid. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.8 and 3.1.12.2
Re: [squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
On 04/06/2011 02:05, sichent wrote: Setup: Gentoo linux OS on squid and privoxy home lan server Squid-3.1.12 privoxy-3.0.17 I'm not running an html server, just trying to use squid and privoxy for my own browsing. Why not to use ICAP or URL rewriter functionality built into Squid to achieve the same results as privoxy instead of having this "chaining" setup? Sorry for possible offtopic :) What offtopic? the only problem i can thing of is cpu and process management on this server. icap is a nice idea and i can recommend one http://greasyspoon.sourceforge.net/ you must know basic programming to make it work and they have rules samples. if your hardware capabilities are low you will get most likely poor results on both privoxy and ICAP. also i will send later the settings for working with greasyspoon on squid 3.2 another question. are you by any change made a speedtest on the current setup? if you do get the right speed but wrong speed to get the page processed it's a known side effect of privoxy. Regards Eliezer sich
[squid-users] Re: Should I see a massive slowdown when chaining squid => privoxy
Setup: Gentoo linux OS on squid and privoxy home lan server Squid-3.1.12 privoxy-3.0.17 I'm not running an html server, just trying to use squid and privoxy for my own browsing. Why not to use ICAP or URL rewriter functionality built into Squid to achieve the same results as privoxy instead of having this "chaining" setup? Sorry for possible offtopic :) sich
