I appreciate very much your looking at my question. >Date: Thu, 30 May 2024 22:15:27 +1200 >From: Amos Jeffries <squ...@treenet.co.nz> >To: squid-users@lists.squid-cache.org >Subject: Re: [squid-users] can't explain 403 denied for authenticated > user >Message-ID: <96746157-da51-47db-8d52-d65239f27...@treenet.co.nz> >Content-Type: text/plain; charset=UTF-8; format=flowed > >On 25/05/24 07:28, Kevin wrote: >> Hi, >> >> We have 2 external ACLs that take a request's data (IP, authenticated >> username, URL, user-agent, etc) and uses that information to determine >> whether a user or host should be permitted to access that URL.?? It >> almost always works well, but we have a recurring occasional issue that >> I can't figure out. >> >> When it occurs, it's always around 4AM.?? This particular request occurs >> often - averages about once a second throughout the day. >> >> What we see is a "403 forbidden" for a (should be) permitted site from >> an authenticated user from the same IP/user and to the same site that >> gets a "202 connection established" every other time. > >Is maybe 4am the time when your auth system refreshes all nonce? > - thus making any currently in-use by the clients invalid until they >re-auth. You might see a mix of 403/401/407 in a bunch at such times.
My auth system is just squid with one flat file each for basic and digest authentication (we allow basic only if an application doesn't support digest). >Or maybe in a similar style one/some of the clients is broken and fails >to update its nonce before it expires at 4am? The client is a Java-based application. I can find out more from the team that wrote it. > - looking at which client agent and IP were getting the 403 and/or the >nonce which received 403 will give you hints about this possibility. Oh: it's only this one application from this one particular host. > >Or your network router(s) do garbage collection and terminate >long-running connections to free up TCP resources? > - thus forcing a lot of client re-connects at 4am, which may: > a) overload the auth system/helper, or > b) break a transaction that included nonce update for clients - >resulting in their next request being invalid nonce. I'm not aware of a 4AM connection-terminating task but will confirm. As mentioned (in answer to a), the auth system is just squid. Re: b - I hadn't thought of that. > > >Or maybe you have log processing software that does "squid -k restart" >instead of the proper "squid -k rotate" to get access to the log files? I hadn't checked that! But logrotate runs at 00:00 and the rotate script does use "squid -k rotate". >Or maybe your auth system has a limit on how large nonce-count can become? > - I notice that the working request has 0x2F uses and the forbidden >has 0x35 (suspiciously close to 50 in decimal) I don't know what that is. I didn't find a lot of helpful detail on squid's digest auth implementation. > >> >> The difference I see in the logs:? though all the digest auth info looks >> okay, the %un field in the log for the usual (successful) request is the >> authenticated username, while in the failed request, it's "-".?? So >> though there hasn't been an authentication error or "authentication >> required" in the log - and the username is in the authentication details >> in the log entry -? it seems like squid isn't recognizing that username >> as %un. > > >Be aware that a properly behaving client will *not* send credentials >until they are requested, after which it should *always* send >credentials on that same connection (even if they are not requested >explicitly). >That means some requests on a multiplex/pipeline/keep-alive connection >MAY arrive with credentials and be accepted(2xx)/denied(403) without >authentication having occured. Entirely due to your *_access directives >sequence. In these cases the log will show auth headers but no value for >%un and/or %ul. > > >> >> My squid.conf first tests a request to see if an unauthenticated request >> from a particular host is permitted.? That external ACL doesn't take a >> username as an argument.?? If that external ACL passes, the request is >> allowed. >> > >Please *show* the config lines rather than describing what you *think* >they do. Their exact ordering matters *a lot*. > Obfuscation of sensitive details is okay/expected so long as you makes >it easy for us to tell that value A and value B are different. > >FWIW; if your config file is *actually* containing only what you >described it is missing a number of things necessary to have a safe and >secure proxy. A look at your full config (without comments or empty >lines) will help us point out any unnoticed issues for you to consider >fixing. Understood. Here it is: acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl windows_net src 172.18.114.0/24 acl sample_host src 172.18.115.1/32 acl rsync port 873 acl SSL_ports port 443 acl SSL_ports port 873 #873 is rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 873 acl CONNECT method CONNECT acl PURGE method PURGE acl localhost src 127.0.0.1 http_access allow PURGE localhost http_access deny PURGE acl URN proto URN http_access deny URN http_access deny manager acl API_FIREFOX dstdomain api.profiler.firefox.com http_access deny API_FIREFOX acl ff_browser browser ^Mozilla/5\.0 acl rma_ua browser ^RMA/1\.0.*compatible;.RealMedia uri_whitespace encode acl trellix_phone_cloud dstdomain amcore-ens.rest.gti.trellix.com http_access deny trellix_phone_cloud external_acl_type host_based_filter children-max=15 ttl=0 %ACL %DATA %SRC %>rd %>rP /PATH/TO/FILTER/SCRIPT.py acl HostBasedRules external host_based_filter http_access allow HostBasedRules auth_param digest program /usr/lib/squid/digest_file_auth -c /etc/squid/passwd auth_param digest realm squid auth_param digest children 2 auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/basic_passwd auth_param basic children 2 auth_param basic realm squidb auth_param basic credentialsttl 2 hours acl auth_users proxy_auth REQUIRED external_acl_type custom_acl_db children-max=15 ttl=0 %ACL %DATA %ul %SRC %>rd %>rP %credentials /PATH/TO/FILTER/SCRIPT.py acl CustomAclDB external custom_acl_db http_access allow CustomAclDB acl CRLs url_regex "/etc/squid/conf.d/CRL_urls.txt" http_access allow CRLs deny_info 303:https://abc.def.com/ http_access deny all acl apache rep_header Server ^Apache icp_access allow localnet icp_access deny all http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localhost http_port 3128 coredump_dir /var/cache/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 logformat squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt logformat squidmime %ul %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt [%>h] [%<h] logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh logformat ip_port %ts %tu %>a %>p %<lp %<a %<p %dt %tr %un %>ru %Ss access_log daemon:/var/log/squid/access.log combined access_log daemon:/var/log/squid/network.log ip_port cache_log /var/log/squid/cache.log cache_store_log /var/log/squid/store.log access_log daemon:/var/log/squid/useragent.log useragent visible_hostname proxy.abc.com cache deny all > >> The next line in squid.conf is >> >> acl auth_users proxy_auth REQUIRED >> > >FYI the above just means that Squid is using authentication. It says >nothing about when the authentication will be (or not be) performed. > > >> ... and after that, the external ACL that takes the username as well as >> the other info. >> > > >HTH >Amos From: squid-users-requ...@lists.squid-cache.org To: "squid-users" <squid-users@lists.squid-cache.org> Sent: Thursday, May 30, 2024 8:00:01 AM Subject: squid-users Digest, Vol 117, Issue 31 I appreciate very much your looking at my question. >Date: Thu, 30 May 2024 22:15:27 +1200 >From: Amos Jeffries <squ...@treenet.co.nz> >To: squid-users@lists.squid-cache.org >Subject: Re: [squid-users] can't explain 403 denied for authenticated > user >Message-ID: <96746157-da51-47db-8d52-d65239f27...@treenet.co.nz> >Content-Type: text/plain; charset=UTF-8; format=flowed > >On 25/05/24 07:28, Kevin wrote: >> Hi, >> >> We have 2 external ACLs that take a request's data (IP, authenticated >> username, URL, user-agent, etc) and uses that information to determine >> whether a user or host should be permitted to access that URL.?? It >> almost always works well, but we have a recurring occasional issue that >> I can't figure out. >> >> When it occurs, it's always around 4AM.?? This particular request occurs >> often - averages about once a second throughout the day. >> >> What we see is a "403 forbidden" for a (should be) permitted site from >> an authenticated user from the same IP/user and to the same site that >> gets a "202 connection established" every other time. > >Is maybe 4am the time when your auth system refreshes all nonce? > - thus making any currently in-use by the clients invalid until they >re-auth. You might see a mix of 403/401/407 in a bunch at such times. My auth system is just squid with one flat file each for basic and digest authentication (we allow basic only if an application doesn't support digest). >Or maybe in a similar style one/some of the clients is broken and fails >to update its nonce before it expires at 4am? The client is a Java-based application. I can find out more from the team that wrote it. > - looking at which client agent and IP were getting the 403 and/or the >nonce which received 403 will give you hints about this possibility. Oh: it's only this one application from this one particular host. > >Or your network router(s) do garbage collection and terminate >long-running connections to free up TCP resources? > - thus forcing a lot of client re-connects at 4am, which may: > a) overload the auth system/helper, or > b) break a transaction that included nonce update for clients - >resulting in their next request being invalid nonce. I'm not aware of a 4AM connection-terminating task but will confirm. As mentioned (in answer to a), the auth system is just squid. Re: b - I hadn't thought of that. > > >Or maybe you have log processing software that does "squid -k restart" >instead of the proper "squid -k rotate" to get access to the log files? I hadn't checked that! But logrotate runs at 00:00 and the rotate script does use "squid -k rotate". >Or maybe your auth system has a limit on how large nonce-count can become? > - I notice that the working request has 0x2F uses and the forbidden >has 0x35 (suspiciously close to 50 in decimal) I don't know what that is. I didn't find a lot of helpful detail on squid's digest auth implementation. > >> >> The difference I see in the logs:? though all the digest auth info looks >> okay, the %un field in the log for the usual (successful) request is the >> authenticated username, while in the failed request, it's "-".?? So >> though there hasn't been an authentication error or "authentication >> required" in the log - and the username is in the authentication details >> in the log entry -? it seems like squid isn't recognizing that username >> as %un. > > >Be aware that a properly behaving client will *not* send credentials >until they are requested, after which it should *always* send >credentials on that same connection (even if they are not requested >explicitly). >That means some requests on a multiplex/pipeline/keep-alive connection >MAY arrive with credentials and be accepted(2xx)/denied(403) without >authentication having occured. Entirely due to your *_access directives >sequence. In these cases the log will show auth headers but no value for >%un and/or %ul. > > >> >> My squid.conf first tests a request to see if an unauthenticated request >> from a particular host is permitted.? That external ACL doesn't take a >> username as an argument.?? If that external ACL passes, the request is >> allowed. >> > >Please *show* the config lines rather than describing what you *think* >they do. Their exact ordering matters *a lot*. > Obfuscation of sensitive details is okay/expected so long as you makes >it easy for us to tell that value A and value B are different. > >FWIW; if your config file is *actually* containing only what you >described it is missing a number of things necessary to have a safe and >secure proxy. A look at your full config (without comments or empty >lines) will help us point out any unnoticed issues for you to consider >fixing. Understood. Here it is: acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl windows_net src 172.18.114.0/24 acl sample_host src 172.18.115.1/32 acl rsync port 873 acl SSL_ports port 443 acl SSL_ports port 873 #873 is rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 873 acl CONNECT method CONNECT acl PURGE method PURGE acl localhost src 127.0.0.1 http_access allow PURGE localhost http_access deny PURGE acl URN proto URN http_access deny URN http_access deny manager acl API_FIREFOX dstdomain api.profiler.firefox.com http_access deny API_FIREFOX acl ff_browser browser ^Mozilla/5\.0 acl rma_ua browser ^RMA/1\.0.*compatible;.RealMedia uri_whitespace encode acl trellix_phone_cloud dstdomain amcore-ens.rest.gti.trellix.com http_access deny trellix_phone_cloud external_acl_type host_based_filter children-max=15 ttl=0 %ACL %DATA %SRC %>rd %>rP /PATH/TO/FILTER/SCRIPT.py acl HostBasedRules external host_based_filter http_access allow HostBasedRules auth_param digest program /usr/lib/squid/digest_file_auth -c /etc/squid/passwd auth_param digest realm squid auth_param digest children 2 auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/basic_passwd auth_param basic children 2 auth_param basic realm squidb auth_param basic credentialsttl 2 hours acl auth_users proxy_auth REQUIRED external_acl_type custom_acl_db children-max=15 ttl=0 %ACL %DATA %ul %SRC %>rd %>rP %credentials /PATH/TO/FILTER/SCRIPT.py acl CustomAclDB external custom_acl_db http_access allow CustomAclDB acl CRLs url_regex "/etc/squid/conf.d/CRL_urls.txt" http_access allow CRLs deny_info 303:https://abc.def.com/ http_access deny all acl apache rep_header Server ^Apache icp_access allow localnet icp_access deny all http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager http_access allow localhost http_port 3128 coredump_dir /var/cache/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 logformat squid %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt logformat squidmime %ul %ts.%03tu %6tr %>a %Ss/%>Hs %<st %rm %ru %un %Sh/%<A %mt [%>h] [%<h] logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh logformat ip_port %ts %tu %>a %>p %<lp %<a %<p %dt %tr %un %>ru %Ss access_log daemon:/var/log/squid/access.log combined access_log daemon:/var/log/squid/network.log ip_port cache_log /var/log/squid/cache.log cache_store_log /var/log/squid/store.log access_log daemon:/var/log/squid/useragent.log useragent visible_hostname proxy.abc.com cache deny all > >> The next line in squid.conf is >> >> acl auth_users proxy_auth REQUIRED >> > >FYI the above just means that Squid is using authentication. It says >nothing about when the authentication will be (or not be) performed. > > >> ... and after that, the external ACL that takes the username as well as >> the other info. >> > > >HTH >Amos
_______________________________________________ squid-users mailing list squid-users@lists.squid-cache.org https://lists.squid-cache.org/listinfo/squid-users
