Re: [squid-users] Log Daemon (queue is too large)
On Thu, Aug 14, 2014 at 8:43 AM, Amos Jeffries squ...@treenet.co.nz wrote: Are you using the default Squid daemon or a custom one? Its the default Squid daemon. Can you reproduce this problem with the current 3.4 stable release? I have not tried just yet - its the first time I have come across this error. Searching the archives and Google turned up nothing. So I figured let me ask. Are you able to identify what the daemon helper is doing when it is loosing log lines? Unfortunately I did not think to see what the logfile-daemon was doing at the time and simply reconfigured squid. The next time it happens I'll be sure to do that. Thanks Amos. -- .warren
[squid-users] Log Daemon (queue is too large)
HI all, I noticed this error message (multiple entries) for yesterday and today on Squid 3.3.11 2014/08/13 00:01:06 kid1| Logfile: daemon:/util/var/squid/log/access.log: queue is too large; some log messages have been lost. Its not a very high utilized proxy so I was a little surprised this happened. I assume something may have caused a spike in traffic resulting in the log buffer filling up but whats concerning is that it never recovers until a -k reconfigure was issued, a -k rotate didnt help. So all log entries for yesterday and today are gone. Any ideas on why it doesn't recover and possibly what could have caused the issue? As looking at the access logs leading up to the event there is nothing that stands out. Thanks, Warren
Re: [squid-users] external_acl_type helper exiting
On Mon, Sep 30, 2013 at 5:46 PM, Amos Jeffries squ...@treenet.co.nz wrote: I can't help with the Go side of things unfortuately. It is still on my TODO list for learning. I am no pro myself :) Is it a timeout of the read() call or the stdin actually closing on timeout? The first one can possibly be caught explicitly in your code. If the error code is given and the buffer empty re-try the read(). There should be a different error code entirely when Squid closes the socket to shutdown the helper. I added code to re-try the read but it looks like stdin is actually been closed on timeout. I have re-searched Go for some indication of a timeout but can't find anything. On the Go OS package side (FreeBSD 8.3) there is also no indication of a timeout. I have added more debugging so lets see what else i can pick up on. Thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On Fri, Sep 27, 2013 at 4:53 PM, Warren Baker war...@decoy.co.za wrote: On Fri, Sep 27, 2013 at 4:27 PM, Amos Jeffries squ...@treenet.co.nz wrote: Arg. yes, too much wading through IP specs today. I meant rfc1738 encoding (Squid flavour of it anyway). Ah ok. So i need to cater for this. Thanks for the point in the right direction. Amos, the actual root cause is this: read /dev/stdin: operation timed out I don't seem to find a timeout value in Go on reading from stdin. Also since I have run the helper manually for days without an issue, it points to Squid timing it out. I can work around the timing out issue in the helper however Squid still sees that it gets no response from the helper and shuts the helper down and then restarts it. You have any ideas? Thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
Hi Eliezer, On Mon, Sep 30, 2013 at 3:06 PM, Eliezer Croitoru elie...@ngtech.co.il wrote: Just wondering if you can share the Golang source code? I can't as it is closed source. In a nutshell it a receives a request, checks a session table (in a nosql datastore) and responds. As to whether I have tried a basic one that just returns ok or error - the truth is I haven't. I have a similar Golang program which is used by the URL Rewriter and the problem is not exhibited there at all. I give a simplified example for the wiki if thats what you are leaning towards? did you tried it with no concurrency? Not yet - i had thought about that but figured i would ping the list to see if there were better ideas. I have switched off concurrency now and will see what happens. it seems to me like a very simple helper that I have written almost the same concept and can might be used to test the case.(in ruby) Yeah its not overly complicated as described above. Thanks for the help. -- .warren
Re: [squid-users] external_acl_type helper exiting
On Mon, Sep 30, 2013 at 5:11 PM, Eliezer Croitoru elie...@ngtech.co.il wrote: OK so a pesudo code of the helper would be nice. datastore used:(what nosql engine?) Redis. I assume it's a simple test of is the ip exists? in a case of yes then reply with OK in a case that it dosnt' exits reply with ERR. Well yes to a point, but the helper does more than just query does the ip exist?. The check includes certain profile specific checking, time based checking and a few other things. This is exactly what I was writing my helper for. The golang helper might be nice but how closed can it be?? See above. the pesudo is not a secret and the code can be reproduced in a matter of minutes\hours\days in a case that the programmer knows what he does... I am happy to provide a basic helper which queries redis to see if an IP is allowed or not and return the relevant result. I have mailed the WikiAdmin to create an account, I will then also add an example. thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On Thu, Sep 26, 2013 at 6:37 PM, Amos Jeffries squ...@treenet.co.nz wrote: It still might be Squid since there have been helper protocol changes in the last few versions. But you will only find that out from the helpers debug. Thanks Amos - it seems the helper was eventually receiving a EOF on stdin from Squid and thus the helper was exiting. Any idea on why input is been terminated? Thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On Fri, Sep 27, 2013 at 4:03 PM, Amos Jeffries squ...@treenet.co.nz wrote: Or because the data input to the helper contained a \0 byte that was not rfc1918 encoded. It must be this as Squid is not been reconfigured nor rotating log files. rfc1918? typo? thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On Fri, Sep 27, 2013 at 4:27 PM, Amos Jeffries squ...@treenet.co.nz wrote: Arg. yes, too much wading through IP specs today. I meant rfc1738 encoding (Squid flavour of it anyway). Ah ok. So i need to cater for this. Thanks for the point in the right direction. -- .warren
[squid-users] external_acl_type helper exiting
HI Guys, I have a problem where the helpers are exiting after a period of time in an external_acl_type setup. The acl is defined as follows: external_acl_type portal_page ipv4 children-idle=5 children-startup=5 ttl=60 negative_ttl=0 concurrency=20 %SRC /usr/local/bin/portal-authenticator portal-authenticator is a Golang program that runs indefinitely. I have turned on debug_options ALL,1 82,9 but that has not indicated why the helper exits. If i run the helper manually I can leave it for days without it exiting. I have bumped up debug_options to ALL,9 to see whether there is anything else I can see but so far nothing. Here is an extract from cache.log (the last log entry been related to the bug 3751): 2013/09/26 11:44:37.017 kid1| WARNING: portal_page #5 exited 2013/09/26 11:44:37.017 kid1| Too few portal_page processes are running (need 1/15) 2013/09/26 11:44:37.017 kid1| Starting new helpers 2013/09/26 11:44:37.017 kid1| helperOpenServers: Starting 1/15 'portal-authenticator' processes 2013/09/26 11:44:37.017 kid1| WARNING: no_suid: setuid(0): (1) Operation not permitted thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On Thu, Sep 26, 2013 at 4:33 PM, Amos Jeffries squ...@treenet.co.nz wrote: The debug options for Squid will not help with this. The helper is a completely separate program and all Squid can know is that the connection to it died before told it to shutdown. Exactly what your log snippet stated. If the helper has any form of debugging you need to enable that and the helpers stderr output will appear in squid cache.log. Good luck. Ok I had a feeling that might be the case. I'll modify the helper and see what I can dig up. I just thought it had something to do with squid since I could run the helper manually with no problem. Thanks for the help Amos. -- .warren
Re: [squid-users] Plain HTML site not opening through squid.
On Mon, Apr 15, 2013 at 7:00 AM, Vivek Sharma vi...@hqr.drdo.in wrote: I am facing this weird problem while trying to open a site mkm.drdo.gov.in = using squid 3.1.10. While the site works fine bypassing squid. While using squid - I am getting Zero Sized Reply on the client browser w= hereas my access.log says http://mkm.drdo.gov.in/ - NONE/- text/html Any help is highly solicited. Domain, mkm.drdo.gov.in, doesn't resolve for me. Checking other name servers, like Google's, also doesn't resolve it. -- .warren
Re: [squid-users] Re: Authentication problem
On Fri, Nov 23, 2012 at 12:05 PM, Amos Jeffries squ...@treenet.co.nz wrote: You are guessing right. As it documented in the wiki ... (seems to be down right now). Thanks Amos - that was my first point of reference, however it was down at the time so thought i might ping the list. The way around it is to combine userA and userB into one ACL, or to check auth and use the all-hack to prevent auth challenges on username mismatch. 1) acl users proxy_auth userA userB http_access allow users http_access deny all 2) acl login proxy_auth REQUIRED acl userA proxy_auth userA acl userB proxy_auth userB http_access deny !login http_access allow userA all http_access allow userB all http_access deny all Ahh yeah this now makes sense - i found reference to the 'all' hack but couldn't put a successful config together. Thanks for the info! -- .warren
[squid-users] Authentication problem
HI List I have squid-3.2.3 configured to make use of negotiate, and to authenticate certain users. However the following config doesn't work acl userA proxy_auth warren acl userB proxy_auth testb http_access allow userA http_access allow userB http_access deny all userA successfully authenticates and can browse. userB however obviously doesn't match userA acl and just gets presented with the cache denied page. Debugging the acl shows that it never moves passed the userA acl and doesn't continue onto authenticating the 'testb' user. What am I doing wrong here? thx -- .warren
[squid-users] Re: Authentication problem
On Thu, Nov 22, 2012 at 3:57 PM, Warren Baker war...@decoy.co.za wrote: I have squid-3.2.3 configured to make use of negotiate, and to authenticate certain users. However the following config doesn't work acl userA proxy_auth warren acl userB proxy_auth testb http_access allow userA http_access allow userB http_access deny all userA successfully authenticates and can browse. userB however obviously doesn't match userA acl and just gets presented with the cache denied page. Debugging the acl shows that it never moves passed the userA acl and doesn't continue onto authenticating the 'testb' user. I am guessing it is because testb user's browser fails with the challenge on userA http_access test and thus it gets failed completely. Is there a way around this? thx
Re: [squid-users] FATAL: Ipc::Mem::Segment::create failed to shm_open(/squid-squid-page-pool.shm): (13) Permission denied
On Mon, Sep 17, 2012 at 9:44 PM, Holmes, Michael A (Mike) mike.hol...@lsi.com wrote: I just added workers 4 to my working configuration and I get an shm_open error. I assume it is trying to write to the root directory /squid-squid-page-pool.shm as the user squid which will fail. How do I redirect this? Thanks Mike Squid Cache (Version 3.2.1): Terminated abnormally. CPU Usage: 0.060 seconds = 0.040 user + 0.020 sys Maximum Resident Size: 18560 KB Page faults with physical i/o: 0 2012/09/17 15:34:33 kid1| Creating Swap Directories 2012/09/17 15:38:02| storeDirWriteCleanLogs: Starting... 2012/09/17 15:38:02| Finished. Wrote 0 entries. 2012/09/17 15:38:02| Took 0.00 seconds ( 0.00 entries/sec). FATAL: Ipc::Mem::Segment::create failed to shm_open(/squid-squid-page-pool.shm): (13) Permission denied What OS + version? -- .warren
Re: [squid-users] Fw: [ERROR] commonUfsDirOpenSwapLog: Failed to open swap log
On Mon, Jul 23, 2012 at 8:02 AM, Anonymous eletters_m...@yahoo.com wrote: squid_enableYES If you installed Squid from ports then you will have a file called /usr/local/etc/rc.d/squid - which is the startup file. You can get the startup variables from this file by executing it and passing the option rcvar. # /usr/local/etc/rc.d/squid rcvar # squid # squid_enable=NO # (default: ) it should return what the startup variable is and what it's value it. In your configured setting, above, you are missing the '=' sign. So set it to squid_enable=YES and remove any of your other additional references to squid. -- .warren
Re: [squid-users] Fw: [ERROR] commonUfsDirOpenSwapLog: Failed to open swap log
On Mon, Jul 23, 2012 at 11:02 AM, Anonymous eletters_m...@yahoo.com wrote: Well! I have tried every mentioned method and still getting the same old error...! Send the contents (sanitized if needed) of your /etc/rc.conf and the contents of your /usr/local/etc/rc.d/ directory. -- .warren
Re: [squid-users] Fw: [ERROR] commonUfsDirOpenSwapLog: Failed to open swap log
On Mon, Jul 23, 2012 at 12:51 PM, Anonymous eletters_m...@yahoo.com wrote:
Send the contents (sanitized if needed) of your /etc/rc.conf
hostname=test
sshd_enable=YES
powerd_enable=YES
# Set dumpdev to AUTO to enable crash dumps, NO to disable
dumpdev=NO
pdnsd_enable=YES
apache22_enable=YES
#squid_enable=YES
Uncomment this
#/usr/local/sbin/squid
/usr/local/etc/rc.d/squid.sh
Remove this ^^^
***
ls /usr/local/etc/rc.d/
apache22htcachecleansquid.sh
bandwidthd.sh.samplepdnsd webmin
How did you install Squid? Ahh..looking further back in the thread I
see you have a 3.2 version for which there is no port yet.
So in that case you can either amend your squid.sh file to include
rc.subr and add the appropriate checks or use another squid startup
file and modify it to suit your needs.
Squid3.1 port startup file below, but should work fine for your install:
#!/bin/sh
#
# $FreeBSD: ports/www/squid31/files/squid.in,v 1.10 2011/08/31
21:09:18 flo Exp $
#
# PROVIDE: squid
# REQUIRE: LOGIN
# KEYWORD: shutdown
# Note:
# Set squid_enable=yes in either /etc/rc.conf, /etc/rc.conf.local or
# /etc/rc.conf.d/squid to activate Squid.
#
# Additional variables you can define in one of these files:
#
# squid_chdir: the directory into which the rc system moves into before
# starting Squid. Default: /var/squid
#
# squid_conf: The configuration file that Squid should use.
# Default: /usr/local/etc/squid/squid.conf
#
# squid_fib:The alternative routing table id that Squid should use.
# Default: none
# See setfib(1) for further details. Note that the setfib(2)
# system call is not available in FreeBSD versions prior to 7.1.
#
# squid_user: The user id that should be used to run the Squid master
# process. Default: squid.
# Note that you probably need to define squid_user=root if
# you want to run Squid in reverse proxy setups or if you want
# Squid to listen on a privileged port 1024.
#
# squid_pidfile:
# The name (including the full path) of the Squid
# master process' PID file.
# Default: /var/run/squid/squid.pid.
# You only need to change this if you changed the
# corresponding entry in your Squid configuration.
# squid_flags: Additional commandline arguments for Squid you might want to
# use. See squid(8) for further details.
#
squid_checkrunning() {
${command} ${command_args} ${squid_flags} -k check 2/dev/null
}
squid_setfib() {
sysctl net.fibs /dev/null 21 || return 0
if [ x${squid_fib} != xNONE ]; then
command=setfib -F ${squid_fib} ${command}
else
return 0
fi
}
squid_stop() {
echo Stopping ${name}.
${command} ${command_args} ${squid_flags} -k shutdown
run_rc_command poll
}
. /etc/rc.subr
name=squid
rcvar=${name}_enable
command=/usr/local/sbin/squid
extra_commands=reload
reload_cmd=${command} ${command_args} ${squid_flags} -k reconfigure
start_precmd=squid_setfib
stop_precmd=squid_checkrunning
stop_cmd=squid_stop
load_rc_config ${name}
squid_chdir=${squid_chdir:-/var/squid}
squid_conf=${squid_conf:-/usr/local/etc/squid/squid.conf}
squid_enable=${squid_enable:-NO}
squid_fib=${squid_fib:-NONE}
squid_pidfile=${squid_pidfile:-/var/run/squid/squid.pid}
squid_user=${squid_user:-squid}
pidfile=${squid_pidfile}
required_dirs=${squid_chdir}
# squid(8) will not start if ${squid_conf} is not present so try
# to catch that beforehand via ${required_files} rather than make
# squid(8) crash.
required_files=${squid_conf}
# Now make sure that we invoke squid with -f ${squid_conf}:
command_args=-f ${squid_conf}
run_rc_command $1
# Eof
--
.warren
Re: [squid-users] NTLM auth to remote server fails through squid
On Tue, Jul 17, 2012 at 4:43 AM, Amos Jeffries squ...@treenet.co.nz wrote: Please update to the latest beta though before doing more testing. 3.2.0.20 is out and the latest snapshot has some relevant bug fixes. I'm only seeing 3.2.0.18 and 3 daily auto-generated releases on http://www.squid-cache.org/Versions/v3/3.2/ . Am i missing something? thanks -- .warren
Re: [squid-users] cache videos/bittorrent
On Wed, May 16, 2012 at 12:55 PM, Mário Sérgio Candian mscand...@mastercabo.com.br wrote: Thanks for the answer Amos. I saw the solution of PeerApp and I thought that the squid could do the same as their software, perhaps with some modification. Some years ago, I used cachevideo with squid, but after Youtube changed that solution stopped working and now I don't know how it is. And the solution PeerApp could do it all. At least that's what they promise... PeerApp does cache p2p, http etc. however it is costs $$$ and does require some network changes. You can either set the PeerApp inline so that all traffic is sent through the device or do some policy routing to ensure only certain ports are sent or rather not sent through the PeerApp. Its only a L2 cache so you dont get the benefits of other technologies such as WCCP, nifty Squid ACLs, WPAD etc. Also, I am not sure how either PeerApp or Oversi handles p2p encrypted transfers. -- .warren
Re: [squid-users] Problem with swf streams from www.tv4play.se behind squid
On Fri, Mar 9, 2012 at 4:24 PM, Peter Olsson p...@leissner.se wrote: Hello! Hi. We can't access swf streams from www.tv4play.se behind squid since about a month ago. Earlier we could at least play some of them, but now it seems that none of them work through squid. We haven't changed anything in the squid version or squid configuration that would cause this. We have contacted their support, but they see no problems on their side. Does anyone know of any problems with swf through squid, either in general or specifically from www.tv4play.se? (These streams are only available from Sweden, so they are unfortunately not available for general testing.) For what its worth I don't have any problem streaming some of those videos but I am using v3.2.0.16. I streamed the Big Brother one for awhile with no problems. Thx -- .warren
Re: [squid-users] Host header forgery
On Mon, Feb 27, 2012 at 11:01 PM, Amos Jeffries squ...@treenet.co.nz wrote: 3.2.0.15+ will do a soft-fail type behaviour, which allows the request through but does not allow caching of the response and only relays the original destination IP. Which hides the problems from client visibility, at cost of some cache HITs. ok interesting - I assume this will be some config option? Not as such. There is host_verify_strict directive to *increase* the number of things validated, including forward-proxy traffic. Which is off by default so only the minimal checks are done. The risk of turning this off entirely is cache poisoning, which immediately spreads infection across the whole network. Since the action vector to do the initial infection is so trivial (a client running a website script can do it without knowing). That is too much risk to allow configuration. Ok that makes sense - thanks Amos. -- .warren
[squid-users] Host header forgery
HI all, Just a question with regards to 3.2.0.X and 'Host header forgery' detection. If there are multiple name servers specified on the client and squid and for whatever reason (timeout,network problem etc.) the client uses a different name server to resolve a site (eg facebook) to what Squid uses - then this problem seems to pop up. So since there is no guarantee on what name server the client uses I guess the only alternative is to enforce WPAD or browser settings. Thanks -- .warren
Re: [squid-users] Host header forgery
On Mon, Feb 27, 2012 at 12:58 PM, Amos Jeffries squ...@treenet.co.nz wrote: It is best to consider interception an action of last resort, for this any many other reasons. yeah of course. 3.2.0.15+ will do a soft-fail type behaviour, which allows the request through but does not allow caching of the response and only relays the original destination IP. Which hides the problems from client visibility, at cost of some cache HITs. ok interesting - I assume this will be some config option? Thanks Amos. -- .warren
