[squid-users] Memory Leaks after OS upgrade
Hi, after upgrading to opensuse 13.1, squid3.4 (same with current 3.HEAD) leaks memory after a short time and stops responding to snmp queries(not sure if that's related). In a test setup i can reproduce the issue as follows: Test with 100 rps: Everything good Test with 250 rps: After some minutes, only 100 to 200 requessts pass, memory increases with constant speed, some snmp queries fail (timeout) or take longer to process, nothing in log Test with 500 rps: Only around 100 queries pass, memory increases faster, most snmp queries fail, nothing in logs Same with 1 or multiple workers. In cachemgr, i see that Cumulative allocated volume continues to increase to values like 7 GB while actual VIRT mem is 300MB, RSS 160MB Any ideas how to fix this or otherwise how to get more debug information? -- Best Regards Nils Hügelmann
Re: [squid-users] Delay Pools with Digest and External Auth
Thanks, i've made it working using a modification of your recommendations.
I summarize my solution in case others have a similar problem:
- Class 5 Delay Pools used (limit by Tag)
- External Auth helper program assigns username as EXT_TAG
- When Digest is used, there is a dummy helper that just assigns
username as EXT_TAG
- Dummy helper is activated using "http_access allow proxyauth
digest_tagger"
- Classification in multiple delay pools is done via other external_auth
ACLs
- These external_auths are activated (to circumvent slow/fast acl
issues) using "http_access allow EXTACLNAME !all"
- These external_auths need to interpret both the external_auth header
and the digest callback to get the username
Best Regards
Nils
Am 13.05.2013 02:32, schrieb Amos Jeffries:
> On 12/05/2013 8:03 a.m., Nils Hügelmann wrote:
>> Hi,
>>
>> I want to use both Digest Auth and External Auth ("simpleheaderauth")
>> for authentification, and need to assign different delay pools to single
>> users based on another external_acl (premiumcheck).
>>
>> So i have (stripped down for readibility)
>>
>> -
>> external_acl_type simpleheaderauth %>{Proxy-Authorization} simpleauth
>> external_acl_type premiumcheck %>{Proxy-Authorization} premium
>> auth_param digest program digestauth
>>
>> acl proxyauth proxy_auth REQUIRED
>> acl simpleheaderauth_passed external simpleheaderauth
>> acl premiumcheck_passed external premiumcheck
>>
>> # activate additional external acls
>> http_access allow premiumcheck_passed !all
>> http_access allow freethrottled_passed !all
>>
>> http_access allow simpleheaderauth_passed
>> http_access allow proxyauth
>> http_access deny !proxyauth
>>
>> http_access deny all
>> -
>>
>> Which works fine in regards to access control, one can either login via
>> "simpleheaderauth" (external_acl) or via "digestauth" (auth_param).
>>
>> I want to have 2 bandwidth limit levels.
>>
>> Situation from here is as follows:
>>
>> When using simpleheaderauth:
>> - EXT_USER is available (username passed from simpleheaderauth
>> external_acl)
>> - Tag is available (tag passed from simpleheaderauth external_acl)
>> - premiumcheck_passed is properly set
>>
>> When using digestauth:
>> - LOGIN is available (username passed from auth_param)
>> - Tag is not available
>> - premiumcheck_passed is not usable
>>
>> Delay pools need to work per individual user, so only class 5 pools (
>> tagrate ) or class 4 pools ( aggregate, network, individual, user )
>> would be possible.
>>
>> As simpleheaderauth has no user defined, and digestauth has no tag, my
>> first attempt for delay_pools was to create 2 sets of pools with 2
>> classes each:
>>
>> -
>> delay_class 1 5
>> delay_class 2 5
>> delay_class 3 4
>> delay_class 4 4
>>
>> # 1st set for simpleheaderauth
>> delay_parameters 2 2097152/2097152
>> delay_access 2 allow simpleheaderauth_passed premiumcheck_passed
>>
>> delay_parameters 1 76800/76800
>> delay_access 1 deny premiumcheck_passed
>> delay_access 1 allow simpleheaderauth_passed
>>
>> # 2nd set for digestauth
>> delay_parameters 4 -1/-1 -1/-1 -1/-1 2097152/2097152
>> delay_access 4 allow premiumcheck_passed
>>
>> delay_parameters 3 -1/-1 -1/-1 -1/-1 76800/76800
>> delay_access 3 deny premiumcheck_passed
>> delay_access 3 allow all
>> -
>>
>> 1. Can one somehow simplify this by making Tag available for digest, or
>> making class 4 username available for external_acl?
>
> I have work lined up on the TODO list for implementing tag on auth
> interfaces in the next Squid versions.
> If you are able to assist with sponsoring that I can divert some time
> back towards it.
>
> However, ...
>
> Alternative #1:
> * make your simple and premium helper lookups produce tags indicating
> those levels.
> * create a dummy external ACL helper lookup test which always
> responds "OK tag=digest-auth". Call it only after proxyauth ACL has
> succeeded doing digest.
>
> eg:
> external_acl_type digestauth %LOGIN basic_fake_auth
> acl digest_tagger external digestauth
>
> http_access allow proxyauth digest_tagger
>
> You can then use "tag" type ACLs for delay_access.
>
>
>> 2. The problem with my attempt is that premiumcheck_passed is not
>> evaluated when usind digestauth. Every digestauth user is assigned to
>> pool 3, while simpleheaderauth users are properly assigned based on
>> premiumcheck_passed. How can i solve this?
>
> You have isolated the problem pretty accurately. It's root cause is
> the mismatch between delay_access being "fast" ACL check and the tests
> you are using being "slow" group ACL.
>
> Amos
[squid-users] Delay Pools with Digest and External Auth
Hi,
I want to use both Digest Auth and External Auth ("simpleheaderauth")
for authentification, and need to assign different delay pools to single
users based on another external_acl (premiumcheck).
So i have (stripped down for readibility)
-
external_acl_type simpleheaderauth %>{Proxy-Authorization} simpleauth
external_acl_type premiumcheck %>{Proxy-Authorization} premium
auth_param digest program digestauth
acl proxyauth proxy_auth REQUIRED
acl simpleheaderauth_passed external simpleheaderauth
acl premiumcheck_passed external premiumcheck
# activate additional external acls
http_access allow premiumcheck_passed !all
http_access allow freethrottled_passed !all
http_access allow simpleheaderauth_passed
http_access allow proxyauth
http_access deny !proxyauth
http_access deny all
-
Which works fine in regards to access control, one can either login via
"simpleheaderauth" (external_acl) or via "digestauth" (auth_param).
I want to have 2 bandwidth limit levels.
Situation from here is as follows:
When using simpleheaderauth:
- EXT_USER is available (username passed from simpleheaderauth
external_acl)
- Tag is available (tag passed from simpleheaderauth external_acl)
- premiumcheck_passed is properly set
When using digestauth:
- LOGIN is available (username passed from auth_param)
- Tag is not available
- premiumcheck_passed is not usable
Delay pools need to work per individual user, so only class 5 pools (
tagrate ) or class 4 pools ( aggregate, network, individual, user )
would be possible.
As simpleheaderauth has no user defined, and digestauth has no tag, my
first attempt for delay_pools was to create 2 sets of pools with 2
classes each:
-
delay_class 1 5
delay_class 2 5
delay_class 3 4
delay_class 4 4
# 1st set for simpleheaderauth
delay_parameters 2 2097152/2097152
delay_access 2 allow simpleheaderauth_passed premiumcheck_passed
delay_parameters 1 76800/76800
delay_access 1 deny premiumcheck_passed
delay_access 1 allow simpleheaderauth_passed
# 2nd set for digestauth
delay_parameters 4 -1/-1 -1/-1 -1/-1 2097152/2097152
delay_access 4 allow premiumcheck_passed
delay_parameters 3 -1/-1 -1/-1 -1/-1 76800/76800
delay_access 3 deny premiumcheck_passed
delay_access 3 allow all
-
1. Can one somehow simplify this by making Tag available for digest, or
making class 4 username available for external_acl?
2. The problem with my attempt is that premiumcheck_passed is not
evaluated when usind digestauth. Every digestauth user is assigned to
pool 3, while simpleheaderauth users are properly assigned based on
premiumcheck_passed. How can i solve this?
Thanks
Nils Hügelmann
[squid-users] ACL::ChecklistMatches: result for %NAME% is -1
Hello,
I've sent this message a few days ago but got no reply yet. Should i try
it somewhere else?
-
I get random DENIEDs, but do not find the cause of it.
Config:
external_acl_type simpleheaderauth ttl=600 concurrency=500 children=1
%>{Proxy-Authorization} mono /opt/squidhelpers/authhelper.exe simpleauth
acl simpleheaderauth_passed external simpleheaderauth
http_access allow simpleheaderauth_passed
Log:
--
2012/10/02 20:06:40.901| aclmatchAclList: async=0 nodeMatched=0
async_in_progress=0 lastACLResult() = 0 finished() = 0
2012/10/02 20:06:40.901| ACLChecklist::preCheck: 0x7f703fbe1178 checking
'http_access allow simpleheaderauth_passed'
2012/10/02 20:06:40.901| ACLList::matches: checking simpleheaderauth_passed
2012/10/02 20:06:40.901| ACL::checklistMatches: checking
'simpleheaderauth_passed'
2012/10/02 20:06:40.901| ACL::ChecklistMatches: result for
'simpleheaderauth_passed' is -1
2012/10/02 20:06:40.901| ACLList::matches: result is false
2012/10/02 20:06:40.901| aclmatchAclList: 0x7f703fbe1178 returning false
(AND list entry failed to match)
2012/10/02 20:06:40.901| ACL::FindByName 'simpleheaderauth_passed'
2012/10/02 20:06:40.901| ACLChecklist::asyncInProgress: 0x7f703fbe1178
async set to 1
2012/10/02 20:06:40.901| ACLChecklist::asyncInProgress: 0x7f703fbe1178
async set to 0
2012/10/02 20:06:40.901| aclmatchAclList: async=1 nodeMatched=0
async_in_progress=0 lastACLResult() = 0 finished() = 0
2012/10/02 20:06:40.902| ACLChecklist::preCheck: 0x7f703fbe1178 checking
'http_access deny all'
-
With successful requests, i get a result "-1", and shortly after that a
similar log output with result "1".
With failed requests, i only get a "-1". The My auth helper log says
that the request was answered successfully.
Any ideas what the problem can be? If not, where else should i look for
the problem?
# squid -v
Squid Cache: Version 3.1.20
Best Regards
Nils Hügelmann
[squid-users] ACL::ChecklistMatches: result for %NAME% is -1
Hello,
I get random DENIEDs, but do not find the cause of it.
Config:
external_acl_type simpleheaderauth ttl=600 concurrency=500 children=1
%>{Proxy-Authorization} mono /opt/squidhelpers/authhelper.exe simpleauth
acl simpleheaderauth_passed external simpleheaderauth
http_access allow simpleheaderauth_passed
Log:
--
2012/10/02 20:06:40.901| aclmatchAclList: async=0 nodeMatched=0
async_in_progress=0 lastACLResult() = 0 finished() = 0
2012/10/02 20:06:40.901| ACLChecklist::preCheck: 0x7f703fbe1178 checking
'http_access allow simpleheaderauth_passed'
2012/10/02 20:06:40.901| ACLList::matches: checking simpleheaderauth_passed
2012/10/02 20:06:40.901| ACL::checklistMatches: checking
'simpleheaderauth_passed'
2012/10/02 20:06:40.901| ACL::ChecklistMatches: result for
'simpleheaderauth_passed' is -1
2012/10/02 20:06:40.901| ACLList::matches: result is false
2012/10/02 20:06:40.901| aclmatchAclList: 0x7f703fbe1178 returning false
(AND list entry failed to match)
2012/10/02 20:06:40.901| ACL::FindByName 'simpleheaderauth_passed'
2012/10/02 20:06:40.901| ACLChecklist::asyncInProgress: 0x7f703fbe1178
async set to 1
2012/10/02 20:06:40.901| ACLChecklist::asyncInProgress: 0x7f703fbe1178
async set to 0
2012/10/02 20:06:40.901| aclmatchAclList: async=1 nodeMatched=0
async_in_progress=0 lastACLResult() = 0 finished() = 0
2012/10/02 20:06:40.902| ACLChecklist::preCheck: 0x7f703fbe1178 checking
'http_access deny all'
-
With successful requests, i get a result "-1", and shortly after that a
similar log output with result "1".
With failed requests, i only get a "-1". The My auth helper log says
that the request was answered successfully.
Any ideas what the problem can be? If not, where else should i look for
the problem?
# squid -v
Squid Cache: Version 3.1.20
Best Regards
Nils Hügelmann
Re: [squid-users] Http status 200 on deny_info?
Yes, the file exists. And when i replace 200 with 403 it works. So i should file a bug for it? -- Regards Nils Hügelmann
[squid-users] Http status 200 on deny_info?
in the config the change message for 3.2: "Support the full range of 200-599 HTTP status codes. 3xx status only available when redirecting to a URI. Other status only available when supplying an error template body." but actually it doesn't accept 200 as http return code. deny_info 200:okbody url_selfcheck -> "FATAL: Bungled squid.conf" I have a scenario where i would like to return status 200 together with a predefined "error" message file for a http client. Should/will this work or not? Best Regards Nils Hügelmann
Re: [squid-users] squid non-accel default website
Hi Amos, are there any news about this? Thanks, Nils Hügelmann > On Wed, 12 May 2010 23:02:08 +0200, Nils Hügelmann > wrote: > > Hi Henrik, > > > > thanks for the answer, a fallback feature for direct requests would be > > great :-) > > > > regards > > nils > > > > Am 12.05.2010 22:38, schrieb Henrik Nordström: > >> tis 2010-05-11 klockan 17:04 +0200 skrev Nils Hügelmann: > >> > >> > >>> At the current state, it shows an "invalid URL" ... "while trying to > >>> retrieve the URL: /" error on direct access, which prevents using url > >>> rewriters(and deny_info too?!) so how to do this?... > >>> > >> You can't. > >> > >> The reason is because Squid really need to know if an request is being > >> proxied or accelerated as it have impact on how the request should be > >> processed, and HTTP requires web servers (including accelerators) to > >> also know how to process requests using full URL. > >> > >> Can't you move the proxy to a separate port, freeing up port 80 to be > >> used as a web server? > >> > >> But yes, I guess we could add support for fallback mode when seeing an > >> obvious webserver request on a proxy port instead of bailing out with > >> invalid request. > >> > > FYI: > There are some security holes opened when defaulting to intercept or > accel mode on supposedly forward traffic. > Mandrivia has supplied captive-portal 'splash' pages for 3.2 that can be > sent instead of the current invalid response page. If anyone can spare the > time to implement a bit of polish let me know please, there are only two > small'ish alterations needed to make this happen for 3.2. > > Amos
[squid-users] Squid doesn't check external_acl although needed?
Hi all, i have the following setup: [irrelevant parts left out] auth_param basic [...] external_acl_type premium children-startup=1 %LOGIN [...] acl premiumacl external premium acl authenticated proxy_auth REQUIRED #http_access allow premiumacl http_access allow authenticated http_access deny all delay_pools 2 delay_class 1 4 delay_class 2 4 delay_parameters 1 -1/-1 -1/-1 -1/-1 2000/10 delay_access 1 deny premiumacl delay_access 1 allow authenticated delay_access 1 deny all delay_parameters 2 -1/-1 -1/-1 -1/-1 20/1000 delay_access 2 allow premiumacl delay_access 2 deny all all users have to authenticate using the basic auth, and all users login names must be checked by the external_acl to determine the correct delay pool they belong to. Problem: it doesn't work in the above setup, although expected: all basic-authenticated users get the delay_pool 1, my external_acl helper is not queried. Only if i enable "http_access allow premiumacl" to make an unneccessary check for http_access, squid polls the external_acl and everything works fine.. I'm not sure if this is a bug or something i don't understand yet, please help me:-) Best Regards Nils Hügelmann
[squid-users] the server_persistent_connections + multiple tcp_outgoing_address issue
Hi all, i think it would be fine if one could use server_persistent_connections and multiple tcp_outgoing_address together. would it be a big deal to create multiple persistent connections with different outgoing addresses on demand, and route outgoing requests via the respective connection? Thanks, Nils Hügelmann
Re: [squid-users] unable to compile 3.2.0.2
Hi Jan / Hi list i have the same problem, see my bug report http://bugs.squid-cache.org/show_bug.cgi?id=3038 Regards Nils Hügelmann
Re: [squid-users] squid non-accel default website
Hi Henrik, thanks for the answer, a fallback feature for direct requests would be great :-) regards nils Am 12.05.2010 22:38, schrieb Henrik Nordström: > tis 2010-05-11 klockan 17:04 +0200 skrev Nils Hügelmann: > > >> At the current state, it shows an "invalid URL" ... "while trying to >> retrieve the URL: /" error on direct access, which prevents using url >> rewriters(and deny_info too?!) so how to do this?... >> > You can't. > > The reason is because Squid really need to know if an request is being > proxied or accelerated as it have impact on how the request should be > processed, and HTTP requires web servers (including accelerators) to > also know how to process requests using full URL. > > Can't you move the proxy to a separate port, freeing up port 80 to be > used as a web server? > > But yes, I guess we could add support for fallback mode when seeing an > obvious webserver request on a proxy port instead of bailing out with > invalid request. > > Regards > Henrik > >
[squid-users] squid non-accel default website
Hi, i have a non-accel non-transparent squid 3.1 running on port 80, and when someone accesses the proxy directly (via http://hostname or http://ip) i want the proxy to show an explanation website. At the current state, it shows an "invalid URL" ... "while trying to retrieve the URL: /" error on direct access, which prevents using url rewriters(and deny_info too?!) so how to do this?... Thanks Nils
