Re: [squid-users] external_acl_type helper exiting
On Mon, Sep 30, 2013 at 5:46 PM, Amos Jeffries wrote: > I can't help with the Go side of things unfortuately. It is still on my TODO > list for learning. I am no pro myself :) > Is it a timeout of the read() call or the stdin actually closing on timeout? > The first one can possibly be caught explicitly in your code. If the error > code is given and the buffer empty re-try the read(). There should be a > different error code entirely when Squid closes the socket to shutdown the > helper. I added code to re-try the read but it looks like stdin is actually been closed on timeout. I have re-searched Go for some indication of a timeout but can't find anything. On the Go OS package side (FreeBSD 8.3) there is also no indication of a timeout. I have added more debugging so lets see what else i can pick up on. Thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
I will try to update my helper to work with redis as backhand. Eliezer On 09/30/2013 10:18 PM, Warren Baker wrote: > On Mon, Sep 30, 2013 at 5:11 PM, Eliezer Croitoru > wrote: >> OK so a pesudo code of the helper would be nice. >> datastore used:(what nosql engine?) > > Redis. > >> I assume it's a simple test of "is the ip exists?" >> in a case of yes then reply with OK >> in a case that it dosnt' exits reply with ERR. > > Well yes to a point, but the helper does more than just query "does > the ip exist?". > The check includes certain profile specific checking, time based > checking and a few other things. > >> This is exactly what I was writing my helper for. >> The golang helper might be nice but how closed can it be?? > > See above. > >> the pesudo is not a secret and the code can be reproduced in a matter of >> minutes\hours\days in a case that the programmer knows what he does... > > I am happy to provide a basic helper which queries redis to see if an > IP is allowed or not and return the relevant result. > I have mailed the WikiAdmin to create an account, I will then also add > an example. > > thanks >
Re: [squid-users] external_acl_type helper exiting
On Mon, Sep 30, 2013 at 5:11 PM, Eliezer Croitoru wrote: > OK so a pesudo code of the helper would be nice. > datastore used:(what nosql engine?) Redis. > I assume it's a simple test of "is the ip exists?" > in a case of yes then reply with OK > in a case that it dosnt' exits reply with ERR. Well yes to a point, but the helper does more than just query "does the ip exist?". The check includes certain profile specific checking, time based checking and a few other things. > This is exactly what I was writing my helper for. > The golang helper might be nice but how closed can it be?? See above. > the pesudo is not a secret and the code can be reproduced in a matter of > minutes\hours\days in a case that the programmer knows what he does... I am happy to provide a basic helper which queries redis to see if an IP is allowed or not and return the relevant result. I have mailed the WikiAdmin to create an account, I will then also add an example. thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On 1/10/2013 12:55 a.m., Warren Baker wrote: On Fri, Sep 27, 2013 at 4:53 PM, Warren Baker wrote: On Fri, Sep 27, 2013 at 4:27 PM, Amos Jeffries wrote: Arg. yes, too much wading through IP specs today. I meant rfc1738 encoding (Squid flavour of it anyway). Ah ok. So i need to cater for this. Thanks for the point in the right direction. Amos, the actual root cause is this: read /dev/stdin: operation timed out I don't seem to find a timeout value in Go on reading from stdin. Also since I have run the helper manually for days without an issue, it points to Squid timing it out. I can work around the timing out issue in the helper however Squid still sees that it gets no response from the helper and shuts the helper down and then restarts it. You have any ideas? I can't help with the Go side of things unfortuately. It is still on my TODO list for learning. Is it a timeout of the read() call or the stdin actually closing on timeout? The first one can possibly be caught explicitly in your code. If the error code is given and the buffer empty re-try the read(). There should be a different error code entirely when Squid closes the socket to shutdown the helper. Amos
Re: [squid-users] external_acl_type helper exiting
OK so a pesudo code of the helper would be nice. datastore used:(what nosql engine?) I assume it's a simple test of "is the ip exists?" in a case of yes then reply with OK in a case that it dosnt' exits reply with ERR. This is exactly what I was writing my helper for. The golang helper might be nice but how closed can it be?? the pesudo is not a secret and the code can be reproduced in a matter of minutes\hours\days in a case that the programmer knows what he does... Eliezer On 09/30/2013 05:22 PM, Warren Baker wrote: > Hi Eliezer, > > On Mon, Sep 30, 2013 at 3:06 PM, Eliezer Croitoru > wrote: >> >> Just wondering if you can share the Golang source code? > > I can't as it is closed source. In a nutshell it a receives a request, > checks a session table (in a nosql datastore) and responds. > As to whether I have tried a basic one that just returns ok or error - > the truth is I haven't. I have a similar Golang program which is used > by the URL Rewriter and the problem is not exhibited there at all. > > I give a simplified example for the wiki if thats what you are leaning > towards? > >> did you tried it with no concurrency? > > Not yet - i had thought about that but figured i would ping the list > to see if there were better ideas. I have switched off concurrency now > and will see what happens. > >> it seems to me like a very simple helper that I have written almost the >> same concept and can might be used to test the case.(in ruby) > > Yeah its not overly complicated as described above. > > Thanks for the help. > >
Re: [squid-users] external_acl_type helper exiting
Hi Eliezer, On Mon, Sep 30, 2013 at 3:06 PM, Eliezer Croitoru wrote: > > Just wondering if you can share the Golang source code? I can't as it is closed source. In a nutshell it a receives a request, checks a session table (in a nosql datastore) and responds. As to whether I have tried a basic one that just returns ok or error - the truth is I haven't. I have a similar Golang program which is used by the URL Rewriter and the problem is not exhibited there at all. I give a simplified example for the wiki if thats what you are leaning towards? > did you tried it with no concurrency? Not yet - i had thought about that but figured i would ping the list to see if there were better ideas. I have switched off concurrency now and will see what happens. > it seems to me like a very simple helper that I have written almost the > same concept and can might be used to test the case.(in ruby) Yeah its not overly complicated as described above. Thanks for the help. -- .warren
Re: [squid-users] external_acl_type helper exiting
On 09/26/2013 02:42 PM, Warren Baker wrote: > HI Guys, > > I have a problem where the helpers are exiting after a period of time > in an external_acl_type setup. > The acl is defined as follows: > > external_acl_type portal_page ipv4 children-idle=5 children-startup=5 > ttl=60 negative_ttl=0 concurrency=20 %SRC > /usr/local/bin/portal-authenticator Just wondering if you can share the Golang source code? did you tried it with no concurrency? it seems to me like a very simple helper that I have written almost the same concept and can might be used to test the case.(in ruby) Eliezer > > portal-authenticator is a Golang program that runs indefinitely. I > have turned on debug_options ALL,1 82,9 but that has not indicated why > the helper exits. If i run the helper manually I can leave it for days > without it exiting. I have bumped up debug_options to ALL,9 to see > whether there is anything else I can see but so far nothing. > > Here is an extract from cache.log (the last log entry been related to > the bug 3751): > > 2013/09/26 11:44:37.017 kid1| WARNING: portal_page #5 exited > 2013/09/26 11:44:37.017 kid1| Too few portal_page processes are > running (need 1/15) > 2013/09/26 11:44:37.017 kid1| Starting new helpers > 2013/09/26 11:44:37.017 kid1| helperOpenServers: Starting 1/15 > 'portal-authenticator' processes > 2013/09/26 11:44:37.017 kid1| WARNING: no_suid: setuid(0): (1) > Operation not permitted > > > thanks >
Re: [squid-users] external_acl_type helper exiting
On Fri, Sep 27, 2013 at 4:53 PM, Warren Baker wrote: > On Fri, Sep 27, 2013 at 4:27 PM, Amos Jeffries wrote: >> Arg. yes, too much wading through IP specs today. I meant rfc1738 encoding >> (Squid flavour of it anyway). > > Ah ok. So i need to cater for this. > > Thanks for the point in the right direction. Amos, the actual root cause is this: read /dev/stdin: operation timed out I don't seem to find a timeout value in Go on reading from stdin. Also since I have run the helper manually for days without an issue, it points to Squid timing it out. I can work around the timing out issue in the helper however Squid still sees that it gets no response from the helper and shuts the helper down and then restarts it. You have any ideas? Thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On Fri, Sep 27, 2013 at 4:27 PM, Amos Jeffries wrote: > Arg. yes, too much wading through IP specs today. I meant rfc1738 encoding > (Squid flavour of it anyway). Ah ok. So i need to cater for this. Thanks for the point in the right direction. -- .warren
Re: [squid-users] external_acl_type helper exiting
On 28/09/2013 2:25 a.m., Warren Baker wrote: On Fri, Sep 27, 2013 at 4:03 PM, Amos Jeffries wrote: Or because the data input to the helper contained a \0 byte that was not rfc1918 encoded. It must be this as Squid is not been reconfigured nor rotating log files. rfc1918? typo? Arg. yes, too much wading through IP specs today. I meant rfc1738 encoding (Squid flavour of it anyway). Amos
Re: [squid-users] external_acl_type helper exiting
On Fri, Sep 27, 2013 at 4:03 PM, Amos Jeffries wrote: > > Or because the data input to the helper contained a \0 byte that was not > rfc1918 encoded. It must be this as Squid is not been reconfigured nor rotating log files. rfc1918? typo? thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On 28/09/2013 1:44 a.m., Warren Baker wrote: On Thu, Sep 26, 2013 at 6:37 PM, Amos Jeffries wrote: It still might be Squid since there have been helper protocol changes in the last few versions. But you will only find that out from the helpers debug. Thanks Amos - it seems the helper was eventually receiving a EOF on stdin from Squid and thus the helper was exiting. Any idea on why input is been terminated? Thanks Either Squid needs the helper to shutdown on reconfigure or log rotation. In which case you won't get any WARNING out of Squid, but may get the other lines about too few and starting another. Or because the data input to the helper contained a \0 byte that was not rfc1918 encoded. Amos
Re: [squid-users] external_acl_type helper exiting
On Thu, Sep 26, 2013 at 6:37 PM, Amos Jeffries wrote: > > It still might be Squid since there have been helper protocol changes in the > last few versions. But you will only find that out from the helpers debug. Thanks Amos - it seems the helper was eventually receiving a EOF on stdin from Squid and thus the helper was exiting. Any idea on why input is been terminated? Thanks -- .warren
Re: [squid-users] external_acl_type helper exiting
On 27/09/2013 2:43 a.m., Warren Baker wrote: On Thu, Sep 26, 2013 at 4:33 PM, Amos Jeffries wrote: The debug options for Squid will not help with this. The helper is a completely separate program and all Squid can know is that the connection to it died before told it to shutdown. Exactly what your log snippet stated. If the helper has any form of debugging you need to enable that and the helpers stderr output will appear in squid cache.log. Good luck. Ok I had a feeling that might be the case. I'll modify the helper and see what I can dig up. I just thought it had something to do with squid since I could run the helper manually with no problem. It still might be Squid since there have been helper protocol changes in the last few versions. But you will only find that out from the helpers debug. Amos
Re: [squid-users] external_acl_type helper exiting
On Thu, Sep 26, 2013 at 4:33 PM, Amos Jeffries wrote: > > The debug options for Squid will not help with this. The helper is a > completely separate program and all Squid can know is that the connection to > it died before told it to shutdown. Exactly what your log snippet stated. > > If the helper has any form of debugging you need to enable that and the > helpers stderr output will appear in squid cache.log. Good luck. Ok I had a feeling that might be the case. I'll modify the helper and see what I can dig up. I just thought it had something to do with squid since I could run the helper manually with no problem. Thanks for the help Amos. -- .warren
Re: [squid-users] external_acl_type helper exiting
On 26/09/2013 11:42 p.m., Warren Baker wrote: HI Guys, I have a problem where the helpers are exiting after a period of time in an external_acl_type setup. The acl is defined as follows: external_acl_type portal_page ipv4 children-idle=5 children-startup=5 ttl=60 negative_ttl=0 concurrency=20 %SRC /usr/local/bin/portal-authenticator portal-authenticator is a Golang program that runs indefinitely. I have turned on debug_options ALL,1 82,9 but that has not indicated why the helper exits. If i run the helper manually I can leave it for days without it exiting. I have bumped up debug_options to ALL,9 to see whether there is anything else I can see but so far nothing. The debug options for Squid will not help with this. The helper is a completely separate program and all Squid can know is that the connection to it died before told it to shutdown. Exactly what your log snippet stated. If the helper has any form of debugging you need to enable that and the helpers stderr output will appear in squid cache.log. Good luck. HTH Amos
[squid-users] external_acl_type helper exiting
HI Guys, I have a problem where the helpers are exiting after a period of time in an external_acl_type setup. The acl is defined as follows: external_acl_type portal_page ipv4 children-idle=5 children-startup=5 ttl=60 negative_ttl=0 concurrency=20 %SRC /usr/local/bin/portal-authenticator portal-authenticator is a Golang program that runs indefinitely. I have turned on debug_options ALL,1 82,9 but that has not indicated why the helper exits. If i run the helper manually I can leave it for days without it exiting. I have bumped up debug_options to ALL,9 to see whether there is anything else I can see but so far nothing. Here is an extract from cache.log (the last log entry been related to the bug 3751): 2013/09/26 11:44:37.017 kid1| WARNING: portal_page #5 exited 2013/09/26 11:44:37.017 kid1| Too few portal_page processes are running (need 1/15) 2013/09/26 11:44:37.017 kid1| Starting new helpers 2013/09/26 11:44:37.017 kid1| helperOpenServers: Starting 1/15 'portal-authenticator' processes 2013/09/26 11:44:37.017 kid1| WARNING: no_suid: setuid(0): (1) Operation not permitted thanks -- .warren
