Re: [squid-users] performance drop moving from 3.0 to 3.1?
On Fri, 18 Mar 2011, Amos Jeffries wrote: On 18/03/11 10:05, da...@lang.hm wrote: ping, any comments on this? excluding acl's, cache_peer* and *direct config entries (~500 lines worth, all IP, servername, port# or url_regex based) Tested with or without all those ACLs? They do make a difference to speed, even the fast ACL tests. I would expect them to, but my issue isn't with the overall speed, but rather with the relative speed of the two versions when running the same ruleset. It appears that 3.1 is significantly slower under these conditions than 3.0. the remaining config file is http_port 8000 icp_port 0 visible_hostname gromit1 cache_effective_user proxy cache_effective_group proxy appaend_domain .invalid.server.name NP: append_domain ? typo on my end, due to oddities in how I access my personal mail from work I can't do a cut-n-paste so I retyped this. pid_filename /var/run/squid.pid cache_dir null /tmp client_db off cache_access_log syslog squid NP: Squid needs a syslog format spec. Same as you would use in the syslog config. syslog:daemon.1 or some such. And the directive name is now just access_log cache_log /var/log/squid/cache.log cache_store_log none coredump_dir none no_cache deny all NP: directive name is just cache. thanks, I'll correct it what would I need to do to track down the cause of this performance drop? That same question is the topic of some discussion(s) in squid-dev. http://www.squid-cache.org/mail-archive/squid-dev/201101/0106.html thanks, I'll go through this tomorrow. There is about 30% CPU load increase as well as the raw speed drop. That 30% is IMO what you are measuring. When topping out the CPU it obviously can't handle many more RPS. * adding IPv6 support - copying, checking version and text'ifying larger IPs a lot is SLOW. - looking up DNS twice ( and A) is relatively slower. - failover when connecting via a network with broken IPv6 connectivity results slower server connect times. any transit network blocking ICMPv6 breaks *your* IP failover. 3.1 was compiled without IPv6 support (I'll report all the config options in the morning) * adding async support - more overheads on every single async step / call. - some events being queued for immediate execution holding up others. - lots of legacy code calling handlers needlessly on errors. Under async this is a full event scheduling cycle/delay on each such call. possible * HTTP/1.1 (is not explicitly mentioned by Alex, but...) - lot more logics checking whether HTTP/1.1 features are to be used. - chunking feature is a slow encoding, performed on all unknown-length requests to servers. Which form a large % of POST. Gets a bit worse in 3.HEAD where its also performed on many GET replies. since I was blasting with ab, i believe that it was doing HTTP/1.0 about as simple as you can get. Some are offset by optimizations and fixes later, so its not cut-n-dry. Work is underway by Alex and Co. to identify the problems. We all work on ways to grab performance back when found. Most of these optimizations won't make it into 3.1, but 3.2 hopes to be better. any feel for how 3.2 is doing (how close is it to no longer being 'release candidate', which for some strange reason scares management types ;-) That thread is a bit outdated now. Contact Alex for some commit points that still need to be performance tested, and how to do that testing. will do. David Lang Amos David Lang On Sun, 13 Mar 2011, da...@lang.hm wrote: I'm using squid in a pure access control mode (all caching disabled) and am looking to move from 3.0 to 3.1, but when I'm doing lab tests with it I am seeing a significant performance drop. when doing a simple small request test (using ab to hammer the proxy retrieving 40 byte pages) 3.0 is reaching 4200 request/sec while under the exact same conditions 3.1.11 is barely topping 1400 requests/sec. if I request larger pages (100K), 3.0 does ~650 requests/sec while 3.1 only manages ~480 requests/sec. going up to 1M pagess, 3.0 does 90 requests/sec while 3.1 does 60 this is with identical configuration except that 3.0 has the null disk cache driver configured while 3.1 has that line commented out. In all of these cases, squid is maxing out at 100% of the single core it has available to it. David Lang
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On 18/03/11 21:54, da...@lang.hm wrote: On Fri, 18 Mar 2011, Amos Jeffries wrote: snip Some are offset by optimizations and fixes later, so its not cut-n-dry. Work is underway by Alex and Co. to identify the problems. We all work on ways to grab performance back when found. Most of these optimizations won't make it into 3.1, but 3.2 hopes to be better. any feel for how 3.2 is doing (how close is it to no longer being 'release candidate', which for some strange reason scares management types ;-) So far its looking like no earlier than end-April. FWIW we have some selective 3.2 builds running happily in production. They just need a bit of testing before use. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] performance drop moving from 3.0 to 3.1?
cache_log /var/log/squid/cache.log cache_store_log none coredump_dir none no_cache deny all NP: directive name is just cache. Hi, Which directive of these should be just cache in 3.1? Thanks Alex
Re: [squid-users] Fw: squid download file larger than 2GB
Hello, Thank you for your kind help. I have installed squid as per below configuration option on my 32-bit linux RHEL 4 OS Squid Cache: Version 3.1.11-20110316 configure options: '--prefix=/usr/local/squidnew2' '--enable-delay-pools' '--enable-arp-acl' '--enable-basic-auth-helpers=NCSA' '--with-filedescriptors=4096' '--with-large-files' '--disable-ipv6' --with-squid=/root/squid-3.1.11-20110316 --enable-ltdl-convenience Now, i am facing typical issue. I stopped my old running squid ans start newly compile squid. But it automatically restarts after few minutes. If i keep my old squid also running than the new squid works fine. I have observed following error message in new squid cache.log file 2011/03/18 13:13:03| comm_old_accept: FD 322: (22) Invalid argument 2011/03/18 13:13:03| FTP data connection from unexpected server ([::]), expecting 140.252.25.92 2011/03/18 13:13:03| assertion failed: comm.cc:1583: fd = 0 What could be the reason? Regards Jigar --- On Wed, 3/16/11, Amos Jeffries squ...@treenet.co.nz wrote: From: Amos Jeffries squ...@treenet.co.nz Subject: Re: [squid-users] Fw: squid download file larger than 2GB To: squid-users@squid-cache.org Date: Wednesday, March 16, 2011, 4:39 AM On 16/03/11 23:19, Jigar Raval wrote: Hello, We are facing issue of downloading file larger than 2GB using squid. Following is the status of squid. We have installed it with large-file support. Squid Cache: Version 3.0.STABLE25 configure options: '--prefix=/usr/local/squidnew' '--enable-delay-pools' '--enable-arp-acl' '--disable-internal-dns' That is not a great idea. It forces Squid to use the slow/blocking OS resolver (max capacity ~200 req/sec). '--enable-basic-auth-helpers=NCSA,LDAP,YP' '--enable-large-files' That should be: --with-large-files Following is the status of linux kernel. It is 32-bit 2.6.9-5.ELsmp #1 SMP Wed Jan 5 19:30:39 EST 2005 i686 i686 i386 GNU/Linux We are getting following error in store.log file while downloading iso using http or ftp. WARNING: preventing off_t overflow for http://ftp.jaist.ac.jp/pub/Linux /openSUSE/distribution/11.4/iso/openSUSE-11.4-DVD-x86_64.iso WARNING: preventing off_t overflow for http://mirrors.isu.net.sa/pub/opensuse/distribution/11.4/iso/openSUSE-11.4-DVD-x86_64.iso We have also used iptables to redirect all port 80 request to our defined squid port. What could be the reason ?. Should we upgrade to 64-bit linux ? The message indicates off_t is 32-bit. --with-large-files makes it 64-bit if your compiler supports the ILP32_OFFBIG environment. I suggest an upgrade to Squid-3.1.10 or later though. There were some cache size accounting problems with 2GB files discovered and fixed recently. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
[squid-users] Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
Hi, I'm trying squid 3.1.10 with ntlm and kerberos. The kinit, klist process works good even net join is working. The problem im facing is when trying to start winbind service and using wbinfo. Always the service is not starting giving the error message lib/util_sock.c:1771(create_pipe_sock) invalid permissions on socket directory /var/run/samba/winbindd_privileged winbindd/winbindd.c:1412(main) winbindd_setup_listeners() failed Right now the ownership of /var/run/samba/winbindd_privileged is set to proxy:winbindd_priv with permissions of 0777 (for testing only), still the service doesn't start. I made the change of permissions to reflect in the service script also, /etc/init.d/winbind. I'm using ubuntu 10.04 (lucid). On the side note, after editing the winbind service script, when I run this command sudo update-rc.d winbind start 21 2 3 4 5 . I get a warning saying update-rc.d: warning: winbind stop runlevel arguments (none) do not match LSB Default-Stop values (0 1 6) System start/stop links for /etc/init.d/winbind already exist. Is there a known solution for this issue? Regards
[squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards On 18 March 2011 14:45, Go Wow gow...@gmail.com wrote: Hi, I'm trying squid 3.1.10 with ntlm and kerberos. The kinit, klist process works good even net join is working. The problem im facing is when trying to start winbind service and using wbinfo. Always the service is not starting giving the error message lib/util_sock.c:1771(create_pipe_sock) invalid permissions on socket directory /var/run/samba/winbindd_privileged winbindd/winbindd.c:1412(main) winbindd_setup_listeners() failed Right now the ownership of /var/run/samba/winbindd_privileged is set to proxy:winbindd_priv with permissions of 0777 (for testing only), still the service doesn't start. I made the change of permissions to reflect in the service script also, /etc/init.d/winbind. I'm using ubuntu 10.04 (lucid). On the side note, after editing the winbind service script, when I run this command sudo update-rc.d winbind start 21 2 3 4 5 . I get a warning saying update-rc.d: warning: winbind stop runlevel arguments (none) do not match LSB Default-Stop values (0 1 6) System start/stop links for /etc/init.d/winbind already exist. Is there a known solution for this issue? Regards
Re: [squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
On 18/03/11 10:47, Go Wow wrote: Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards Check /etc/init.d/winbind (or /etc/init.d/samba if you don't have separate scripts for winbind) to make sure it does not set permissions on the directory. Some distributions seem to do this, I think it might even be in upstream Samba. Just comment it out if it's doing it - it seems a stupid think to put in an init script to me. Cheers Alex
Re: [squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
There is a script in /etc/init.d/winbind I tried editing it but still no luck. I check /etc/init.d/smbd but there is no mentioning about winbind. On 18 March 2011 15:02, Alex Crow a...@nanogherkin.com wrote: On 18/03/11 10:47, Go Wow wrote: Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards Check /etc/init.d/winbind (or /etc/init.d/samba if you don't have separate scripts for winbind) to make sure it does not set permissions on the directory. Some distributions seem to do this, I think it might even be in upstream Samba. Just comment it out if it's doing it - it seems a stupid think to put in an init script to me. Cheers Alex
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On 18/03/11 22:50, Alex Crow wrote: cache_log /var/log/squid/cache.log cache_store_log none coredump_dir none no_cache deny all NP: directive name is just cache. Hi, Which directive of these should be just cache in 3.1? The one which used to be called no_cache back in Squid-2.2. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] Fw: squid download file larger than 2GB
On 18/03/11 23:26, Jigar Raval wrote: Hello, Thank you for your kind help. I have installed squid as per below configuration option on my 32-bit linux RHEL 4 OS Squid Cache: Version 3.1.11-20110316 configure options: '--prefix=/usr/local/squidnew2' '--enable-delay-pools' '--enable-arp-acl' '--enable-basic-auth-helpers=NCSA' '--with-filedescriptors=4096' '--with-large-files' '--disable-ipv6' --with-squid=/root/squid-3.1.11-20110316 --enable-ltdl-convenience Now, i am facing typical issue. I stopped my old running squid ans start newly compile squid. But it automatically restarts after few minutes. If i keep my old squid also running than the new squid works fine. I have observed following error message in new squid cache.log file 2011/03/18 13:13:03| comm_old_accept: FD 322: (22) Invalid argument 2011/03/18 13:13:03| FTP data connection from unexpected server ([::]), expecting 140.252.25.92 2011/03/18 13:13:03| assertion failed: comm.cc:1583: fd= 0 What could be the reason? On b**r. This is http://bugs.squid-cache.org/show_bug.cgi?id=3177 The patch there prevents the crashing. But cannot fix the underlying FTP data connection being broken on arrival, as shown by the ([::]). Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
On 19/03/11 00:15, Go Wow wrote: There is a script in /etc/init.d/winbind I tried editing it but still no luck. I check /etc/init.d/smbd but there is no mentioning about winbind. On 18 March 2011 15:02, Alex Crow wrote: On 18/03/11 10:47, Go Wow wrote: Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards Check /etc/init.d/winbind (or /etc/init.d/samba if you don't have separate scripts for winbind) to make sure it does not set permissions on the directory. Some distributions seem to do this, I think it might even be in upstream Samba. Just comment it out if it's doing it - it seems a stupid think to put in an init script to me. Cheers Alex The correct configuration is detailed here: http://wiki.squid-cache.org/ConfigExamples/Authenticate/Ntlm#winbind_privileged_pipe_permissions One major gotcha: RHEL and a few other OS patch a hard-coded value for this directive. So that removing it from config still fails. In that case a full re-build without the distro patch is required. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
Thanks Amos. I was going to try with cache_effective_user setting in squid.conf but I will try this config first. Will update you guys. Regards On 18 March 2011 17:06, Amos Jeffries squ...@treenet.co.nz wrote: On 19/03/11 00:15, Go Wow wrote: There is a script in /etc/init.d/winbind I tried editing it but still no luck. I check /etc/init.d/smbd but there is no mentioning about winbind. On 18 March 2011 15:02, Alex Crow wrote: On 18/03/11 10:47, Go Wow wrote: Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards Check /etc/init.d/winbind (or /etc/init.d/samba if you don't have separate scripts for winbind) to make sure it does not set permissions on the directory. Some distributions seem to do this, I think it might even be in upstream Samba. Just comment it out if it's doing it - it seems a stupid think to put in an init script to me. Cheers Alex The correct configuration is detailed here: http://wiki.squid-cache.org/ConfigExamples/Authenticate/Ntlm#winbind_privileged_pipe_permissions One major gotcha: RHEL and a few other OS patch a hard-coded value for this directive. So that removing it from config still fails. In that case a full re-build without the distro patch is required. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
[squid-users] ipcCreate: fork: (12) Cannot allocate memory
Hi, Squid has started to NOT come back up after log rotate. Here is snippett from cache.log. Machine has 1G ram and cache_mem is set to 500MB, cache_replacement_policy heap GDSF memory_replacement_policy heap GDSF storeDirWriteCleanLogs: Starting... 2011/03/18 04:00:01| 65536 entries written so far. 2011/03/18 04:00:01|131072 entries written so far. 2011/03/18 04:00:01|196608 entries written so far. 2011/03/18 04:00:01|262144 entries written so far. 2011/03/18 04:00:01| Finished. Wrote 273823 entries. 2011/03/18 04:00:01| Took 0.1 seconds (3321643.5 entries/sec). 2011/03/18 04:00:01| logfileRotate: /var/log/squid/store.log 2011/03/18 04:00:01| logfileRotate (stdio): /var/log/squid/store.log 2011/03/18 04:00:01| logfileRotate: /var/log/squid/access.log 2011/03/18 04:00:01| logfileRotate (stdio): /var/log/squid/access.log 2011/03/18 04:00:01| helperOpenServers: Starting 1 'storeurl.pl' processes 2011/03/18 04:00:01| ipcCreate: fork: (12) Cannot allocate memory 2011/03/18 04:00:01| WARNING: Cannot run '/etc/squid/storeurl.pl' process. Any suggestions would be much appreciated. Thanks, Winfield
Re: [squid-users] no-cache , no-store
Hi amos
Thanks for your reply.
By turning setting session.cache_limiter off in php.ini would i be
able to cache these php generated pages??
Do you think my squid.conf is correct to cache the pages?? i only get
a lot of TCP MISS in my access.log :(
This is my squid.conf:
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports !Safe_ports
http_access allow localnet
http_access allow localhost
http_access allow all
http_access deny all
visible_hostname www.xxx.com
http_port 80 accel defaultsite=www.xxx.com
http_port 80 accel ignore-cc
cache_peer x.x.x.x parent 80 0 no-query originserver
emulate_httpd_log on
redirect_rewrites_host_header off
forwarded_for on
cache_dir ufs /usr/local/squid/var/cache 1000 16 256
cache_mem 256 MB
maximum_object_size_in_memory 128 KB
coredump_dir /usr/local/squid/var/cache
logformat combined %a %ui %un [%tl] %rm %ru HTTP/%rv %Hs %st
%{Referer}h %{User-Agent}h %Ss:%Sh
access_log /var/log/squid/access.log combined
logfile_rotate 10
refresh_pattern ^ftp: 144020% 10080
refresh_pattern ^gopher:14400% 1440
refresh_pattern -i (/cgi-bin/|\?) 00% 0
refresh_pattern . 0 20% 4320
refresh_pattern -i \.(html|htm|css|js)$ 1440 50% 40320
refresh_pattern -i \.php$ 1440 100% 40320 override-expire
override-lastmod reload-into-ims
refresh_pattern -i php\? 1440 100% 40320 override-expire
override-lastmod reload-into-ims
refresh_pattern -i \.xml$ 15 100% 20 override-expire override-lastmod
reload-into-ims
refresh_pattern . 1440 40% 40320
On Thu, Mar 17, 2011 at 11:26 PM, Amos Jeffries squ...@treenet.co.nz wrote:
On 18/03/11 16:08, N3O wrote:
Hello
i'm using squid 3.1.11 as a reverse proxy.
Is it possible to cache pages that show the no-cache, no-store
directives in their http headers?
no-cache do get cached. It only means that existing cached copies are not
to be sent to the requestor.
no-store is set on pages which are absolutely not allowed to be stored to
any long-term media. ie cached.
Server: Apache/2.0.52 (Red Hat)
Set-Cookie: PHPSESSID=de2721c82ebc2be4b9a388d2e6e3d66c; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private, no-cache, no-store, proxy-revalidate,
no-transform
In this case private indicates that the object contains some private
information. Caching this on a reverse-proxy will result in the cached copy
and thus the private information to be sent to all visitors. Major personal
info leakage usually resulting.
Are you fighting with PHP defaults? the php.ini setting
session.cache_limiter can be turned to not add things. The app NEEDS to be
setting its own correctly with that off, may off-the-shelf seem to rely on
the defaults.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.11
Beta testers wanted for 3.2.0.5
Re: [squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
After issuing the command gpasswd -a proxy winbindd_priv wbinfo -a username returns sucess for challenge/response but not for plain text. No error given sudo wbinfo -a this.user Enter this.user's password: plaintext password authentication failed Could not authenticate user this.user with plaintext password Enter this.user's password: challenge/response password authentication succeeded No error info in winbind log as well. Regards On 18 March 2011 17:14, Go Wow gow...@gmail.com wrote: Thanks Amos. I was going to try with cache_effective_user setting in squid.conf but I will try this config first. Will update you guys. Regards On 18 March 2011 17:06, Amos Jeffries squ...@treenet.co.nz wrote: On 19/03/11 00:15, Go Wow wrote: There is a script in /etc/init.d/winbind I tried editing it but still no luck. I check /etc/init.d/smbd but there is no mentioning about winbind. On 18 March 2011 15:02, Alex Crow wrote: On 18/03/11 10:47, Go Wow wrote: Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards Check /etc/init.d/winbind (or /etc/init.d/samba if you don't have separate scripts for winbind) to make sure it does not set permissions on the directory. Some distributions seem to do this, I think it might even be in upstream Samba. Just comment it out if it's doing it - it seems a stupid think to put in an init script to me. Cheers Alex The correct configuration is detailed here: http://wiki.squid-cache.org/ConfigExamples/Authenticate/Ntlm#winbind_privileged_pipe_permissions One major gotcha: RHEL and a few other OS patch a hard-coded value for this directive. So that removing it from config still fails. In that case a full re-build without the distro patch is required. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] performance drop moving from 3.0 to 3.1?
Hi, Which directive of these should be just cache in 3.1? The one which used to be called no_cache back in Squid-2.2. Amos So cache deny all is the same as no_cache deny all? Alex
Re: [squid-users] Re: Squid 3.1 and winbind 3.4.7 permissions issue on winbindd_privileged
Winbind works properly , my bad I was issuing sudo wbinfo -a username where it should been sudo wbinfo -a domain\\username Thanks for help. Regard On 18 March 2011 19:22, Go Wow gow...@gmail.com wrote: After issuing the command gpasswd -a proxy winbindd_priv wbinfo -a username returns sucess for challenge/response but not for plain text. No error given sudo wbinfo -a this.user Enter this.user's password: plaintext password authentication failed Could not authenticate user this.user with plaintext password Enter this.user's password: challenge/response password authentication succeeded No error info in winbind log as well. Regards On 18 March 2011 17:14, Go Wow gow...@gmail.com wrote: Thanks Amos. I was going to try with cache_effective_user setting in squid.conf but I will try this config first. Will update you guys. Regards On 18 March 2011 17:06, Amos Jeffries squ...@treenet.co.nz wrote: On 19/03/11 00:15, Go Wow wrote: There is a script in /etc/init.d/winbind I tried editing it but still no luck. I check /etc/init.d/smbd but there is no mentioning about winbind. On 18 March 2011 15:02, Alex Crow wrote: On 18/03/11 10:47, Go Wow wrote: Just to kill my curiosity and resolve the issue I added proxy and root user to winbindd_priv group as well. But still damn winbind wont start. Regards Check /etc/init.d/winbind (or /etc/init.d/samba if you don't have separate scripts for winbind) to make sure it does not set permissions on the directory. Some distributions seem to do this, I think it might even be in upstream Samba. Just comment it out if it's doing it - it seems a stupid think to put in an init script to me. Cheers Alex The correct configuration is detailed here: http://wiki.squid-cache.org/ConfigExamples/Authenticate/Ntlm#winbind_privileged_pipe_permissions One major gotcha: RHEL and a few other OS patch a hard-coded value for this directive. So that removing it from config still fails. In that case a full re-build without the distro patch is required. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
[squid-users] Squid in HA.
Hey Guys, Could you confirm what would be the scenario for Squid working in HA ?., Thanks.,
Re: [squid-users] Squid in HA.
Hey Guys, Could you confirm what would be the scenario for Squid working in HA ?., That depends on your setup ... we have squid running in a resource group with winbind (for authorization against AD), the respective internal gateway IP, and pingd resources that monitor external access and DNS resolution. If one of these resources fails to run or the internet/DNS is not reachable, the whole group will migrate to the second server. Setups are identical otherwise, we synchronize configuration files of squid etc via separate methodds (rsync via ssh), so we do not need to use a distributed filesystem for this. This is a version 1 heartbeat setup; we are currently experimenting with pacemaker and corosync but are still struggling to put everything together on a CentOS 5.5 box. HTH, Jakob Curdes
Re: [squid-users] Squid in HA.
My scenario is to use two Squids working as forwarding proxy : SquidA and SquidB. If SquidA fails users should be switched to the SquidB. If I decide to go with PAC files the workstation is the one that decide where to go. My concern is, where should I store the PAC file so that It can also be redundant let say saved in two places ? Thanks.!. -Original Message- From: Jakob Curdes Sent: Friday, March 18, 2011 9:49 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Squid in HA. Hey Guys, Could you confirm what would be the scenario for Squid working in HA ?., That depends on your setup ... we have squid running in a resource group with winbind (for authorization against AD), the respective internal gateway IP, and pingd resources that monitor external access and DNS resolution. If one of these resources fails to run or the internet/DNS is not reachable, the whole group will migrate to the second server. Setups are identical otherwise, we synchronize configuration files of squid etc via separate methodds (rsync via ssh), so we do not need to use a distributed filesystem for this. This is a version 1 heartbeat setup; we are currently experimenting with pacemaker and corosync but are still struggling to put everything together on a CentOS 5.5 box. HTH, Jakob Curdes
[squid-users] youtube safety mode
I had been asked if this is possible and doing a search through the mailing list and google, I could only find a howto for SafeSquid. Is it possible to do this in transparent mode using squid? If so, can someone point me to a doc on how to accomplish this? Thank you, Jon
[squid-users] Time-based shift of bandwidth from delay pool
Hi I've got a number of delay pools,one of which is only really used during daytime. Is there anyway for me to take the bandwidth allocated to that daytime pool and assign it to the other pools? My specs are: 2.6.18-1ubuntu3 on Ubuntu 8.04. Best regards Dayo
[squid-users] Re: squid as forward proxy for portal run on tomcat
Hi Amos, thanks for your response. I'll try to clarify. I want my browser (a client's browser) to always go through a squid proxy for accessing any website (target application). This is because I have an icap service working on the data. Thus to my understanding this is a forward proxy. Since I want it to work for both http and https sites, I configured squid to work with ssl-bump as shown above. I have tested this configuration, by setting firefox proxy settings to go to squid on port 3128, and it seems to work fine :) Now I have an additional target application. This application happens to be a portal that is run on tomcat. Furthermore, it is a tomcat that I configured the security settings for. Thus I have browser - squid - portal (run on tomcat). To my understanding this is still part of the same forward proxy? am I wrong here? Unfortunately, on this particular setting I get the failure I showed above. From cache.log: -BEGIN SSL SESSION PARAMETERS- MHECAQECAgMBBAIANQQg0b4mR/aJ5Vez5HNh6dSwUL4vs/d+v+ceEwKpWxHdFoME MI3ZqOI/+MjpLLsjIoFchf9dxA/wD9aoZZgrbiq6GRtvOTWRRFeaQA1KFfVgmFo7 FaEGAgRNgfR5ogQCAgEspAIEAA== -END SSL SESSION PARAMETERS- 2011/03/17 07:46:01| SSL unknown certificate error 18 in /C=IL/ST=NA/L=NA/O=IBM/OU=HRL/CN=Magen 2011/03/17 07:46:01| fwdNegotiateSSL: Error negotiating SSL connection on FD 13: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0) I guess I am still understanding something badly, please point me to it. Thanks, Ariel. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-as-forward-proxy-for-portal-run-on-tomcat-tp3383986p3388175.html Sent from the Squid - Users mailing list archive at Nabble.com.
Re: [squid-users] Squid in HA.
On Fri, 18 Mar 2011, Edouard Zorrilla wrote: My scenario is to use two Squids working as forwarding proxy : SquidA and SquidB. If SquidA fails users should be switched to the SquidB. this is very similar to what I have. What I do is to run two systems with squid on each system. I have the vis-hostname set the same for both of them, and that name resolves to a VIP that moves between the two systems if one fails. I use heartbeat (http://linux-ha.org) to manage the VIP and detect that a system has failed. by having squid running all the time on both boxes, the failover is very fast. however, by having squid running on each system, anything that happens on one system is not known by the other system, so if you do authentication or anything like that, when a failover happens users will need to re-authenticate. Also, the cache will be empty and have to be rebuilt. David Lang If I decide to go with PAC files the workstation is the one that decide where to go. My concern is, where should I store the PAC file so that It can also be redundant let say saved in two places ? Thanks.!. -Original Message- From: Jakob Curdes Sent: Friday, March 18, 2011 9:49 AM To: squid-users@squid-cache.org Subject: Re: [squid-users] Squid in HA. Hey Guys, Could you confirm what would be the scenario for Squid working in HA ?., That depends on your setup ... we have squid running in a resource group with winbind (for authorization against AD), the respective internal gateway IP, and pingd resources that monitor external access and DNS resolution. If one of these resources fails to run or the internet/DNS is not reachable, the whole group will migrate to the second server. Setups are identical otherwise, we synchronize configuration files of squid etc via separate methodds (rsync via ssh), so we do not need to use a distributed filesystem for this. This is a version 1 heartbeat setup; we are currently experimenting with pacemaker and corosync but are still struggling to put everything together on a CentOS 5.5 box. HTH, Jakob Curdes
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On Fri, 18 Mar 2011, Amos Jeffries wrote: On 18/03/11 21:54, da...@lang.hm wrote: On Fri, 18 Mar 2011, Amos Jeffries wrote: snip Some are offset by optimizations and fixes later, so its not cut-n-dry. Work is underway by Alex and Co. to identify the problems. We all work on ways to grab performance back when found. Most of these optimizations won't make it into 3.1, but 3.2 hopes to be better. any feel for how 3.2 is doing (how close is it to no longer being 'release candidate', which for some strange reason scares management types ;-) So far its looking like no earlier than end-April. FWIW we have some selective 3.2 builds running happily in production. They just need a bit of testing before use. thanks, this helps a lot, I didn't know if it was 'probably sometime in the next few weeks' or 'possibly sometime in 2012' April timeframe gives me a lot of useful information. what areas are still being worked on? In my case I specifically just need the access control portion, so if some caching modes are still showing bugs it wouldn't affect me, but if SMP configurations are still having bugs it would. David Lang
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On Fri, 18 Mar 2011, da...@lang.hm wrote: On Fri, 18 Mar 2011, Amos Jeffries wrote: On 18/03/11 10:05, da...@lang.hm wrote: ping, any comments on this? excluding acl's, cache_peer* and *direct config entries (~500 lines worth, all IP, servername, port# or url_regex based) Tested with or without all those ACLs? They do make a difference to speed, even the fast ACL tests. I would expect them to, but my issue isn't with the overall speed, but rather with the relative speed of the two versions when running the same ruleset. It appears that 3.1 is significantly slower under these conditions than 3.0. pid_filename /var/run/squid.pid cache_dir null /tmp client_db off cache_access_log syslog squid NP: Squid needs a syslog format spec. Same as you would use in the syslog config. syslog:daemon.1 or some such. And the directive name is now just access_log it is documented as the facility and severity being optional, and the format spec is given 'squid' There is about 30% CPU load increase as well as the raw speed drop. That 30% is IMO what you are measuring. When topping out the CPU it obviously can't handle many more RPS. * adding IPv6 support - copying, checking version and text'ifying larger IPs a lot is SLOW. - looking up DNS twice ( and A) is relatively slower. - failover when connecting via a network with broken IPv6 connectivity results slower server connect times. any transit network blocking ICMPv6 breaks *your* IP failover. 3.1 was compiled without IPv6 support (I'll report all the config options in the morning) Ok, the config options for 3.0 and 3.1 are: From dl...@digitalinsight.com Fri Mar 18 14:47:00 2011 Date: Fri, 18 Mar 2011 14:46:59 -0700 (PDT) From: David Lang dl...@digitalinsight.com To: da...@lang.hm Subject: squid config It was created by Squid Web Proxy configure 3.0.STABLE12, which was generated by GNU Autoconf 2.62. Invocation command line was $ ./configure --prefix= --bindir=/usr/squid/bin --sbindir=/usr/squid/sbin --libexecdir=/usr/squid/libexec --datadir=/usr/squid/share --sysconfdir=/etc/squid --localstatedir=/usr/squid/var --mandir=/usr/squid/man --disable-ident-lookups --enable-default-err-language=English --enable-err-languages=English --disable-wccp --enable-kill-parent-hack --enable-gnuregex --disable-loadable-modules --enable-ssl --disable-translation --with-large-files --with-logdir=/var/log --enable-storeio=null --enable-err-languages=en en-us It was created by Squid Web Proxy configure 3.1.11, which was generated by GNU Autoconf 2.68. Invocation command line was $ ./configure --prefix= --bindir=/usr/squid/bin --sbindir=/usr/squid/sbin --libexecdir=/usr/squid/libexec --datadir=/usr/squid/share --sysconfdir=/etc/squid --localstatedir=/usr/squid/var --mandir=/usr/squid/man --disable-ident-lookups --disable-wccp --enable-kill-parent-hack --disable-loadable-modules --disable-ssl --disable-translation --with-large-files --with-logdir=/var/log --disable-ipv6 --with-filedescriptors=32768 is there anything in here that hurts the performance and I should remove (or anything I should have in here for best performance) David Lang
Re: [squid-users] ipcCreate: fork: (12) Cannot allocate memory
On 19/03/11 02:43, Winfield Henry wrote: Hi, Squid has started to NOT come back up after log rotate. Here is snippett from cache.log. Machine has 1G ram and cache_mem is set to 500MB, Squid uses fork() instead of vfork() to spawn helpers, on some OS the fork() implementation prevents extremely huge amounts of virtual memory being allocated (even though it is neither allocated nor used). The helper multiplexer has been created to get around this problem: ftp://ftp.squid-cache.org/pub/squid/contrib/helper-mux/ Details on how to use it are in the Squid-3.2 release notes: http://www.squid-cache.org/Versions/v3/3.2/RELEASENOTES.html (published as part of 3.2, but works on all squid-2.6 or later). PS. A few of use have looked at making Squid use vfork() but been defeated by the parent doing followup logics. If anyone is keen to attack the problem patches that work will be VERY welcome. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] ipcCreate: fork: (12) Cannot allocate memory
On Sat, 19 Mar 2011, Amos Jeffries wrote: On 19/03/11 02:43, Winfield Henry wrote: Hi, Squid has started to NOT come back up after log rotate. Here is snippett from cache.log. Machine has 1G ram and cache_mem is set to 500MB, Squid uses fork() instead of vfork() to spawn helpers, on some OS the fork() implementation prevents extremely huge amounts of virtual memory being allocated (even though it is neither allocated nor used). I think you mean that on some OS the form implementation 'results in' rather than 'prevents' on linux this is the 'overcommit' option, on by default in the kernel, but many people think it makes their systems more reliable to disable it. you can work around the problem by makeing sure the system has enough virtual memory available, usually by increasing the amount of swap space avialable. David Lang The helper multiplexer has been created to get around this problem: ftp://ftp.squid-cache.org/pub/squid/contrib/helper-mux/ Details on how to use it are in the Squid-3.2 release notes: http://www.squid-cache.org/Versions/v3/3.2/RELEASENOTES.html (published as part of 3.2, but works on all squid-2.6 or later). PS. A few of use have looked at making Squid use vfork() but been defeated by the parent doing followup logics. If anyone is keen to attack the problem patches that work will be VERY welcome. Amos
Re: [squid-users] no-cache , no-store
On 19/03/11 02:44, N3O wrote:
Hi amos
Thanks for your reply.
By turning setting session.cache_limiter off in php.ini would i be
able to cache these php generated pages??
You would yes, provided the PHP scripts generating them do not deny it.
Do you think my squid.conf is correct to cache the pages?? i only get
a lot of TCP MISS in my access.log :(
So what headers are being sent around now?
If you are testing with the refresh button be aware that it often
forces no-cache or at best revalidate to happen. Which results in an
IMS_MISS.
Pressing enter in the address bar is the best test of normal traffic
HIT/MISS behaviour.
This is my squid.conf:
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports !Safe_ports
http_access allow localnet
http_access allow localhost
http_access allow all
http_access deny all
visible_hostname www.xxx.com
http_port 80 accel defaultsite=www.xxx.com
http_port 80 accel ignore-cc
cache_peer x.x.x.x parent 80 0 no-query originserver
emulate_httpd_log on
redirect_rewrites_host_header off
forwarded_for on
cache_dir ufs /usr/local/squid/var/cache 1000 16 256
cache_mem 256 MB
maximum_object_size_in_memory 128 KB
coredump_dir /usr/local/squid/var/cache
logformat combined %a %ui %un [%tl] %rm %ru HTTP/%rv %Hs %st
%{Referer}h %{User-Agent}h %Ss:%Sh
access_log /var/log/squid/access.log combined
logfile_rotate 10
refresh_pattern ^ftp: 144020% 10080
refresh_pattern ^gopher:14400% 1440
refresh_pattern -i (/cgi-bin/|\?) 00% 0
refresh_pattern . 0 20% 4320
All these following refresh_patten do not work. The . pattern above
catches *everything*.
refresh_pattern -i \.(html|htm|css|js)$ 1440 50% 40320
refresh_pattern -i \.php$ 1440 100% 40320 override-expire
override-lastmod reload-into-ims
refresh_pattern -i php\? 1440 100% 40320 override-expire
override-lastmod reload-into-ims
refresh_pattern -i \.xml$ 15 100% 20 override-expire override-lastmod
reload-into-ims
refresh_pattern . 1440 40% 40320
On Thu, Mar 17, 2011 at 11:26 PM, Amos Jeffriessqu...@treenet.co.nz wrote:
On 18/03/11 16:08, N3O wrote:
Hello
i'm using squid 3.1.11 as a reverse proxy.
Is it possible to cache pages that show the no-cache, no-store
directives in their http headers?
no-cache do get cached. It only means that existing cached copies are not
to be sent to the requestor.
no-store is set on pages which are absolutely not allowed to be stored to
any long-term media. ie cached.
Server: Apache/2.0.52 (Red Hat)
Set-Cookie: PHPSESSID=de2721c82ebc2be4b9a388d2e6e3d66c; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: private, no-cache, no-store, proxy-revalidate,
no-transform
In this case private indicates that the object contains some private
information. Caching this on a reverse-proxy will result in the cached copy
and thus the private information to be sent to all visitors. Major personal
info leakage usually resulting.
Are you fighting with PHP defaults? the php.ini setting
session.cache_limiter can be turned to not add things. The app NEEDS to be
setting its own correctly with that off, may off-the-shelf seem to rely on
the defaults.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.11
Beta testers wanted for 3.2.0.5
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.11
Beta testers wanted for 3.2.0.5
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On 19/03/11 04:32, Alex Crow wrote: Hi, Which directive of these should be just cache in 3.1? The one which used to be called no_cache back in Squid-2.2. Amos So cache deny all is the same as no_cache deny all? Yes. They are identical, except that recently Squid will throw warnings at you for using the obsolete name no_cache. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On 19/03/11 10:44, da...@lang.hm wrote: On Fri, 18 Mar 2011, Amos Jeffries wrote: On 18/03/11 21:54, da...@lang.hm wrote: On Fri, 18 Mar 2011, Amos Jeffries wrote: snip Some are offset by optimizations and fixes later, so its not cut-n-dry. Work is underway by Alex and Co. to identify the problems. We all work on ways to grab performance back when found. Most of these optimizations won't make it into 3.1, but 3.2 hopes to be better. any feel for how 3.2 is doing (how close is it to no longer being 'release candidate', which for some strange reason scares management types ;-) So far its looking like no earlier than end-April. FWIW we have some selective 3.2 builds running happily in production. They just need a bit of testing before use. thanks, this helps a lot, I didn't know if it was 'probably sometime in the next few weeks' or 'possibly sometime in 2012' April timeframe gives me a lot of useful information. what areas are still being worked on? * RockStore shared cache for efficient SMP caching support. * Comm layer restructuring for IPv6 split-stack support * miscellaneous Bugs (34 important ones left today) libecap 0.2.0 support and the final expected cache manager SMP changes hit 3.2 yesterday. In my case I specifically just need the access control portion, so if some caching modes are still showing bugs it wouldn't affect me, but if SMP configurations are still having bugs it would. I'm not aware of anything major in SMP. Just the hassle of each worker currently needing its own separate cache. There are some crashes in auth, but with patches available to avoid that (not merged due to inefficiency). Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] performance drop moving from 3.0 to 3.1?
On 19/03/11 10:53, da...@lang.hm wrote: On Fri, 18 Mar 2011, da...@lang.hm wrote: On Fri, 18 Mar 2011, Amos Jeffries wrote: On 18/03/11 10:05, da...@lang.hm wrote: ping, any comments on this? excluding acl's, cache_peer* and *direct config entries (~500 lines worth, all IP, servername, port# or url_regex based) Tested with or without all those ACLs? They do make a difference to speed, even the fast ACL tests. I would expect them to, but my issue isn't with the overall speed, but rather with the relative speed of the two versions when running the same ruleset. It appears that 3.1 is significantly slower under these conditions than 3.0. pid_filename /var/run/squid.pid cache_dir null /tmp client_db off cache_access_log syslog squid NP: Squid needs a syslog format spec. Same as you would use in the syslog config. syslog:daemon.1 or some such. And the directive name is now just access_log it is documented as the facility and severity being optional, and the format spec is given 'squid' There is about 30% CPU load increase as well as the raw speed drop. That 30% is IMO what you are measuring. When topping out the CPU it obviously can't handle many more RPS. * adding IPv6 support - copying, checking version and text'ifying larger IPs a lot is SLOW. - looking up DNS twice ( and A) is relatively slower. - failover when connecting via a network with broken IPv6 connectivity results slower server connect times. any transit network blocking ICMPv6 breaks *your* IP failover. 3.1 was compiled without IPv6 support (I'll report all the config options in the morning) Ok, the config options for 3.0 and 3.1 are: From dl...@digitalinsight.com Fri Mar 18 14:47:00 2011 Date: Fri, 18 Mar 2011 14:46:59 -0700 (PDT) From: David Lang dl...@digitalinsight.com To: da...@lang.hm Subject: squid config It was created by Squid Web Proxy configure 3.0.STABLE12, which was generated by GNU Autoconf 2.62. Invocation command line was $ ./configure --prefix= --bindir=/usr/squid/bin --sbindir=/usr/squid/sbin --libexecdir=/usr/squid/libexec --datadir=/usr/squid/share --sysconfdir=/etc/squid --localstatedir=/usr/squid/var --mandir=/usr/squid/man --disable-ident-lookups --enable-default-err-language=English --enable-err-languages=English --disable-wccp --enable-kill-parent-hack --enable-gnuregex --disable-loadable-modules --enable-ssl --disable-translation --with-large-files --with-logdir=/var/log --enable-storeio=null --enable-err-languages=en en-us It was created by Squid Web Proxy configure 3.1.11, which was generated by GNU Autoconf 2.68. Invocation command line was $ ./configure --prefix= --bindir=/usr/squid/bin --sbindir=/usr/squid/sbin --libexecdir=/usr/squid/libexec --datadir=/usr/squid/share --sysconfdir=/etc/squid --localstatedir=/usr/squid/var --mandir=/usr/squid/man --disable-ident-lookups --disable-wccp --enable-kill-parent-hack --disable-loadable-modules --disable-ssl --disable-translation --with-large-files --with-logdir=/var/log --disable-ipv6 --with-filedescriptors=32768 is there anything in here that hurts the performance and I should remove (or anything I should have in here for best performance) Nothing in that lot. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] youtube safety mode
On 19/03/11 07:14, Test User wrote: I had been asked if this is possible and doing a search through the mailing list and google, I could only find a howto for SafeSquid. Is it possible to do this in transparent mode using squid? If so, can someone point me to a doc on how to accomplish this? What is this youtube safety mode you speak of? NP: SafeSquid is a system which is not related to Squid, just taking the brand name to boost their product. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
Re: [squid-users] Time-based shift of bandwidth from delay pool
On 19/03/11 07:37, Dayo Adewunmi wrote: Hi I've got a number of delay pools,one of which is only really used during daytime. Is there anyway for me to take the bandwidth allocated to that daytime pool and assign it to the other pools? Not as such. Pools do not hold any particular bandwidth. They are simply speed-caps on what a particular request may used for its duration (some milliseconds). It sounds like you want one set of pools during daytime and one set during non-daytime. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.11 Beta testers wanted for 3.2.0.5
[squid-users] documentation for --enable-forw-via-db
I see this config option in 3.2, but searching for this string (or subsets of it) on the wiki isn't finding anything. what does this option do? David Lang
