[squid-users] squid restart

[Vieri]

Hi,

Around every hour or so, the Squid proxy client experience comes to a crawl.
It takes a very long time to load a simple web page.

This is a snapshot taken when this happens:

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Sat, 31 Oct 2020 10:43:21 GMT
Content-Type: text/plain;charset=utf-8
Expires: Sat, 31 Oct 2020 10:43:21 GMT
Last-Modified: Sat, 31 Oct 2020 10:43:21 GMT
X-Cache: MISS from inf-fw1
X-Cache-Lookup: MISS from inf-fw1:3128
Connection: close

Squid Object Cache: Version 5.0.4-20201013-r6b13b73d3
Build Info:
Service Name: squid
Start Time: Sat, 31 Oct 2020 04:50:45 GMT
Current Time:   Sat, 31 Oct 2020 10:43:21 GMT
Connection information for squid:
    Number of clients accessing cache:  320
    Number of HTTP requests received:   519828
    Number of ICP messages received:    0
    Number of ICP messages sent:    0
    Number of queued ICP replies:   0
    Number of HTCP messages received:   0
    Number of HTCP messages sent:   0
    Request failure ratio:   0.00
    Average HTTP requests per minute since start:   1474.3
    Average ICP messages per minute since start:    0.0
    Select loop called: 9044075 times, 2.339 ms avg
Cache information for squid:
    Hits as % of all requests:  5min: 2.1%, 60min: 2.5%
    Hits as % of bytes sent:    5min: -68.9%, 60min: -402.5%
    Memory hits as % of hit requests:   5min: 78.3%, 60min: 62.3%
    Disk hits as % of hit requests: 5min: 0.0%, 60min: 1.7%
    Storage Swap size:  29040 KB
    Storage Swap capacity:  88.6% used, 11.4% free
    Storage Mem size:   29212 KB
    Storage Mem capacity:   89.1% used, 10.9% free
    Mean Object Size:   17.31 KB
    Requests given to unlinkd:  11815
Median Service Times (seconds)  5 min    60 min:
    HTTP Requests (All):   0.04519  0.04776
    Cache Misses:  0.06286  0.06286
    Cache Hits:    0.0  0.0
    Near Hits: 0.04277  0.02317
    Not-Modified Replies:  0.0  0.0
    DNS Lookups:   0.0  0.0
    ICP Queries:   0.0  0.0
Resource usage for squid:
    UP Time:    21155.513 seconds
    CPU Time:   1334.166 seconds
    CPU Usage:  6.31%
    CPU Usage, 5 minute avg:    8.60%
    CPU Usage, 60 minute avg:   9.88%
    Maximum Resident Size: 4287872 KB
    Page faults with physical i/o: 0
Memory accounted for:
    Total accounted:   744703 KB
    memPoolAlloc calls: 136343652
    memPoolFree calls:  140190831
File descriptor usage for squid:
    Maximum number of file descriptors:   4096
    Largest file desc currently in use:   4009
    Number of file desc currently in use: 3997
    Files queued for open:   0
    Available number of file descriptors:   99
    Reserved number of file descriptors:   100
    Store Disk files open:   0
Internal Data Structures:
  1852 StoreEntries
  1849 StoreEntries with MemObjects
  1754 Hot Object Cache Items
  1678 on-disk objects

If I issue the '-k reconfigure' command then the user experience is "great 
again".

A data snapshot taken right after the latter command shows this:

# squidclient mgr:info
HTTP/1.1 200 OK
Server: squid
Mime-Version: 1.0
Date: Sat, 31 Oct 2020 10:48:40 GMT
Content-Type: text/plain;charset=utf-8
Expires: Sat, 31 Oct 2020 10:48:40 GMT
Last-Modified: Sat, 31 Oct 2020 10:48:40 GMT
X-Cache: MISS from inf-fw1
X-Cache-Lookup: MISS from inf-fw1:3128
Connection: close

Squid Object Cache: Version 5.0.4-20201013-r6b13b73d3
Build Info:
Service Name: squid
Start Time: Sat, 31 Oct 2020 10:46:51 GMT
Current Time:   Sat, 31 Oct 2020 10:48:40 GMT
Connection information for squid:
    Number of clients accessing cache:  179
    Number of HTTP requests received:   4663
    Number of ICP messages received:    0
    Number of ICP messages sent:    0
    Number of queued ICP replies:   0
    Number of HTCP messages received:   0
    Number of HTCP messages sent:   0
    Request failure ratio:   0.00
    Average HTTP requests per minute since start:   2575.3
    Average ICP messages per minute since start:    0.0
    Select loop called: 51220 times, 2.121 ms avg
Cache information for squid:
    Hits as % of all requests:  5min: 0.3%, 60min: 0.3%
    Hits as % of bytes sent:    5min: 3.6%, 60min: 3.6%
    Memory hits as % of hit requests:   5min: 60.0%, 60min: 60.0%
    Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
    Storage Swap size:  10292 KB
    Storage Swap capacity:  31.4% used, 68.6% free
    Storage Mem size:   10456 KB
    Storage Mem capacity:   31.9% used, 68.1% free
    Mean Object Size:   15.07 KB
    Reques

Re: [squid-users] squid restart

[Amos Jeffries]

On 1/11/20 12:02 am, Vieri wrote:

Hi,

Around every hour or so, the Squid proxy client experience comes to a crawl.
It takes a very long time to load a simple web page.


...



I guess the reason could be for this:

     Maximum number of file descriptors:   4096
     Largest file desc currently in use:   4009
     Number of file desc currently in use: 3997



Yes, that looks like it.



However, I set the following directive in squid.conf:

max_filedescriptors 65536



Are you using systemd, SysV or another init ?

systemd default limits has this as a known issue for initial startup.



It doesn't seem to be honored here unless I stop and restart the squid service 
again (/etc/init.d/squid restart from command line):

File descriptor usage for squid:
     Maximum number of file descriptors:   65535

It seems that if I run the same command (/etc/init.d/squid restart) from 
crontab, that ulimit is not honored. I guess that's the root cause of my issue 
because I am asking cron to restart Squid once daily. I'll try not to, but I 
was hoping to see if there was a reliable way to fully restart the Squid 
process.

Vieri



The init system restart command is the preferred one - it handles any 
system details that need updating. Alternatively, "squid -k restart" can 
be used.



Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[Amos Jeffries]

On 31/10/20 1:34 pm, roee klinger wrote:


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002



No need to name a *_port like this. The default name is the first 
parameter string ("3002" on this line).




http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to change it 
in squid.conf and also in htpassword and then I run "squid -k reconfigure", if 
I don't reconfigure the old user still has access to the proxy and the new one doesn't 
for about 30 minutes.



No need to restart for that change. The helper you have there will 
automatically detect changes to the htpassword file and reload it.


It is a little odd that the new user was not able to authenticate. Check 
that your test did not lookup and cache a non-existence result for them 
prior to being added.



The delay is due to the credentials being valid for a period of time. To 
reduce workload on the auth system Squid caches credential details for a 
while.


Set "auth_param basic credentialsttl " to shorter values to reduce the 
delay (default is 2hrs).




I am expecting to have 100s of users soon that will change credentials often, 
and also I would like to blacklist websites often and on the fly, so I was 
searching for a better way to manage this without reconfiguring every time, 
since sometimes a reconfigure can take up to 10-15 seconds.



This helper does not need a reconfigure at all as far as I can tell from 
the code.


All the reconfigure was doing for you previously was triggering an early 
prune of the records in the credentials cache. Probably why you saw 
about 30min delay instead of about 2hrs.




I am new to Squid and wasn't able to find any info on this, am I doing this 
currently or there is a better way to change users/ACLs on the fly without 
reloading Squid?



Config changes in squid.conf itself needs a reconfigure or sometimes a 
restart.



For auth and ACLs whose values that come into Squid from a helper it 
depends on the helper itself. Most can auto-detect changes to their 
background databases and not need anything from Squid to update the 
outputs. All helpers do have some form of caching of their results by 
Squid, so there are settings in squid.conf to tune that to your needs - 
as you can see from the auth issue above.



For ACLs with values that are expected to change often it is best to use 
an external_acl_type helper that manages the updates or fetches from 
somewhere the updates are handled without a reload.




Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Reverse proxying Exchange OWA wembail with SSL offloading

[Amos Jeffries]

On 31/10/20 3:20 am, Scott wrote:

On Sat, Oct 31, 2020 at 12:49:16AM +1300, Amos Jeffries wrote:

On 30/10/20 3:27 pm, Scott wrote:

On Thu, Oct 29, 2020 at 10:08:42PM +1300, Amos Jeffries wrote:

On 29/10/20 12:06 pm, Scott wrote:

On Wed, Oct 28, 2020 at 12:00:01PM +, squid-users-reques wrote:

Date: Thu, 29 Oct 2020 00:08:34 +1300
From: Amos Jeffries

On 28/10/20 5:25 pm, Scott wrote:


Here are the logs (first not working, followed by working).

Note this is the login attempt, not the loading of the initial page.  You'll
see in the NOT WORKING section that the browser does NOT return a cookie to
the server, which is where the problem may be.  Again, I'm not sure why - I'm
thinking perhaps the browser/javascript is rejecting the cookie as it's
missing the "secure" attribute (because the back-end is talking plain HTTP).



The complete absence of a cookie may be expected to break something.

The absence of a "secure" flag should only make the cookie vulnerable to
leaking. It should not affect anything depending on that cookies value.


Amos



After some more research and experimentation I've confirmed that my
suspicions are correct.

Recent browsers are no longer accepting cookies with the SameSite flag set
without the Secure flag set.

It's not an issue with Squid (although one Squid could solve - but I'm unsure
of the implications).


Implications are that the server may have intentionally used the
combination it did, no mistakes.

The server is given "Front-End-Https: On" so that it is aware the client
is using HTTPS and can set (or not) the secure flag appropriately to
what it needs. Squid is not aware of whether the cookie is safe to use
on HTTP or restrict to just HTTPS.




Here is a useful link:
https://docs.microsoft.com/en-us/office365/troubleshoot/miscellaneous/chrome-behavior-affects-applications

I tested Chrome 85 on Windows - the default settings DO NOT allow for these
cookies.  However after setting
Cookies without SameSite must be secure
to Disabled, these cookies are permitted and OWA works.

There are obvious implications for sites doing SSL offloading here.  Are
sites no longer doing SSL offload?  Or are reverse proxies adding the Secure
flag?



Neither. When a site frontend is entirely https:// with no http://
resources mixed in the Secure flag can be used by the server regardless
of what the internal connections are.


Amos


My point is that, assuming browsers are now enforcing SameSite cookies must
be secure, then doing SSL offload (whereby the origin server does NOT flag
the cookie as secure) will break.


But offloading does not mean the server omits the secure flag.

Servers which choose to send it when offloading work fine. Servers that
choose to omit it have problems with the Google paranoid interpretation
of security.


Aren't origin servers oblivious to SSL offloading?  I thought they just
happily accepted HTTP connections with no knowledge of any secure channel
between client and reverse proxy.



It varies.  Mozilla have a brief writeup about the headers used:





If that's correct then, although not proscribed by the RFC, it's unlikely
that a Set-Cookie header would ever contain a Secure flag because the server
would be saying "I insist this Cookie be transmitted securely, but here it is
over an insecure channel".



That assumes the only type of security is HTTPS. The channel between 
proxy and server may be secured via other means than TLS, even as it 
transmits "plain text" HTTP.


There should be server settings to administrate that. I do not use OWA 
or IIS though, so now sure if they are available for your case.





In this particular case, I use squid to do reverse proxy with SSL offload to
a MS Exchange server.


You also have a TLS connection to the exchange:443 peer.


I have both for testing, but only one is active at any one time (as
referenced in a cache_peer_access).  Yes I now use the 443 peer because
Chrome/IE/Edge + SSL Offloading no longer works.



Please understand that with Squid acting as a reverse-proxy "offloading" 
is still happening. It is the cert you put in the https_port in 
Squid.conf that the clients are dealing with, and Squid which is doing 
any client cert validation.



For working security on the cache_peer you should identify the CA cert 
which signed the peer server's TLS certificate. Load that into Squid as 
cache_peer tls-cafile=  and you can drop the DONT_VERIFY_PEER flag.





Notice that in the logs you showed transactions sent to that peer get
the secure flag set, despite the SSL offload being done by Squid at the
same time.




Not true AFAICT.  Any cookies you see in those logs with Secure set are shown
under the "WORKING" heading whose entries are:
2020/10/28 12:01:23.549 kid1| 11,2| http.cc(2263) sendRequest: HTTP Server 
local=squid-internal:62597 remote=exchange:443 FD 30 flags=1

No SSL offloading there.  Otherwise it would say

Re: [squid-users] Reverse proxying Exchange OWA wembail with SSL offloading

[Amos Jeffries]

On 31/10/20 3:20 am, Scott wrote:

On Sat, Oct 31, 2020 at 12:49:16AM +1300, Amos Jeffries wrote:

On 30/10/20 3:27 pm, Scott wrote:

On Thu, Oct 29, 2020 at 10:08:42PM +1300, Amos Jeffries wrote:

On 29/10/20 12:06 pm, Scott wrote:

On Wed, Oct 28, 2020 at 12:00:01PM +, squid-users-reques wrote:

Date: Thu, 29 Oct 2020 00:08:34 +1300
From: Amos Jeffries

On 28/10/20 5:25 pm, Scott wrote:


Here are the logs (first not working, followed by working).

Note this is the login attempt, not the loading of the initial page.  You'll
see in the NOT WORKING section that the browser does NOT return a cookie to
the server, which is where the problem may be.  Again, I'm not sure why - I'm
thinking perhaps the browser/javascript is rejecting the cookie as it's
missing the "secure" attribute (because the back-end is talking plain HTTP).



The complete absence of a cookie may be expected to break something.

The absence of a "secure" flag should only make the cookie vulnerable to
leaking. It should not affect anything depending on that cookies value.


Amos



After some more research and experimentation I've confirmed that my
suspicions are correct.

Recent browsers are no longer accepting cookies with the SameSite flag set
without the Secure flag set.

It's not an issue with Squid (although one Squid could solve - but I'm unsure
of the implications).


Implications are that the server may have intentionally used the
combination it did, no mistakes.

The server is given "Front-End-Https: On" so that it is aware the client
is using HTTPS and can set (or not) the secure flag appropriately to
what it needs. Squid is not aware of whether the cookie is safe to use
on HTTP or restrict to just HTTPS.




Here is a useful link:
https://docs.microsoft.com/en-us/office365/troubleshoot/miscellaneous/chrome-behavior-affects-applications

I tested Chrome 85 on Windows - the default settings DO NOT allow for these
cookies.  However after setting
Cookies without SameSite must be secure
to Disabled, these cookies are permitted and OWA works.

There are obvious implications for sites doing SSL offloading here.  Are
sites no longer doing SSL offload?  Or are reverse proxies adding the Secure
flag?



Neither. When a site frontend is entirely https:// with no http://
resources mixed in the Secure flag can be used by the server regardless
of what the internal connections are.


Amos


My point is that, assuming browsers are now enforcing SameSite cookies must
be secure, then doing SSL offload (whereby the origin server does NOT flag
the cookie as secure) will break.


But offloading does not mean the server omits the secure flag.

Servers which choose to send it when offloading work fine. Servers that
choose to omit it have problems with the Google paranoid interpretation
of security.


Aren't origin servers oblivious to SSL offloading?  I thought they just
happily accepted HTTP connections with no knowledge of any secure channel
between client and reverse proxy.



It varies.  Mozilla have a brief writeup about the headers used:





If that's correct then, although not proscribed by the RFC, it's unlikely
that a Set-Cookie header would ever contain a Secure flag because the server
would be saying "I insist this Cookie be transmitted securely, but here it is
over an insecure channel".



That assumes the only type of security is HTTPS. The channel between 
proxy and server may be secured via other means than TLS, even as it 
transmits "plain text" HTTP.


There should be server settings to administrate that. I do not use OWA 
or IIS though, so now sure if they are available for your case.





In this particular case, I use squid to do reverse proxy with SSL offload to
a MS Exchange server.


You also have a TLS connection to the exchange:443 peer.


I have both for testing, but only one is active at any one time (as
referenced in a cache_peer_access).  Yes I now use the 443 peer because
Chrome/IE/Edge + SSL Offloading no longer works.



Please understand that with Squid acting as a reverse-proxy "offloading" 
is still happening. It is the cert you put in the https_port in 
Squid.conf that the clients are dealing with, and Squid which is doing 
any client cert validation.



For working security on the cache_peer you should identify the CA cert 
which signed the peer server's TLS certificate. Load that into Squid as 
cache_peer tls-cafile=  and you can drop the DONT_VERIFY_PEER flag.





Notice that in the logs you showed transactions sent to that peer get
the secure flag set, despite the SSL offload being done by Squid at the
same time.




Not true AFAICT.  Any cookies you see in those logs with Secure set are shown
under the "WORKING" heading whose entries are:
2020/10/28 12:01:23.549 kid1| 11,2| http.cc(2263) sendRequest: HTTP Server 
local=squid-internal:62597 remote=exchange:443 FD 30 flags=1

No SSL offloading there.  Otherwise it would say

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[roee klinger]

Thanks Amos!

I updated "auth_param basic credentialsttl" according to your advice and it
is working great.

I am still having issues with the "tcp_outgoing_address 192.168.8.12
acl_for_user3002" part, you mentioned:
> For ACLs with values that are expected to change often it is best to use
> an external_acl_type helper that manages the updates or fetches from
> somewhere the updates are handled without a reload.

My script updates the authenticator successfully, but when I update "acl
acl_for_user3002 proxy_auth user2" to the new username I have to
reconfigure to take effect.
I read online for hours but to my best understanding external_acl_type are
for auth and access control, but they don't work for my needs I believe.

Is there any way to use external_acl_type in a way I don't understand to
solve this problem? Do I have to reconfigure every time I make changes to
an ACL in squid.conf?

Thanks again for your help.

On Sat, Oct 31, 2020 at 5:48 PM Amos Jeffries  wrote:

> On 31/10/20 1:34 pm, roee klinger wrote:
> > 
> > Hey,
> > I have Squid configured to send users to different outgoing interface
> like so:
> >
> > ..
> > auth_param basic program /usr/lib/squid/basic_ncsa_auth
> /etc/squid/htpassword
> > acl acl_for_user3002 proxy_auth user2
> > tcp_outgoing_address 192.168.8.12 acl_for_user3002
> > http_port 3002 name=3002
>
>
> No need to name a *_port like this. The default name is the first
> parameter string ("3002" on this line).
>
>
> > http_access allow authenticated
> > ..
> >
> > When I wanted to change the username:password for user2, I run a bash
> script to change it in squid.conf and also in htpassword and then I run
> "squid -k reconfigure", if I don't reconfigure the old user still has
> access to the proxy and the new one doesn't for about 30 minutes.
> >
>
> No need to restart for that change. The helper you have there will
> automatically detect changes to the htpassword file and reload it.
>
> It is a little odd that the new user was not able to authenticate. Check
> that your test did not lookup and cache a non-existence result for them
> prior to being added.
>
>
> The delay is due to the credentials being valid for a period of time. To
> reduce workload on the auth system Squid caches credential details for a
> while.
>
> Set "auth_param basic credentialsttl " to shorter values to reduce the
> delay (default is 2hrs).
>
>
> > I am expecting to have 100s of users soon that will change credentials
> often, and also I would like to blacklist websites often and on the fly, so
> I was searching for a better way to manage this without reconfiguring every
> time, since sometimes a reconfigure can take up to 10-15 seconds.
> >
>
> This helper does not need a reconfigure at all as far as I can tell from
> the code.
>
> All the reconfigure was doing for you previously was triggering an early
> prune of the records in the credentials cache. Probably why you saw
> about 30min delay instead of about 2hrs.
>
>
> > I am new to Squid and wasn't able to find any info on this, am I doing
> this currently or there is a better way to change users/ACLs on the fly
> without reloading Squid?
> >
>
> Config changes in squid.conf itself needs a reconfigure or sometimes a
> restart.
>
>
> For auth and ACLs whose values that come into Squid from a helper it
> depends on the helper itself. Most can auto-detect changes to their
> background databases and not need anything from Squid to update the
> outputs. All helpers do have some form of caching of their results by
> Squid, so there are settings in squid.conf to tune that to your needs -
> as you can see from the auth issue above.
>
>
> For ACLs with values that are expected to change often it is best to use
> an external_acl_type helper that manages the updates or fetches from
> somewhere the updates are handled without a reload.
>
>
>
> Amos
> ___
> squid-users mailing list
> squid-users@lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Best practice for adding or removing ACLs dynamically ?

[Eliezer Croitor]

Hey Roee,

I am trying to understand what part of squid.conf you want to be able to change 
without a reconfigure/reload?
If you have many users, ie above 50 you should probably not use a simple 
ncsa_auth although  it's possible and in more then one case is preferable.
You could probably write your own basic auth helper that will interact with a 
DB which will probably simplify your whole setup.
(You can use existing basic auth helpers with mysql or ldap)

As for the tcp_outgoing_address, it’s a whole different story.
Since it's a "fast" acl type the options to do something dynamic with it are an 
issue.
(Maybe eCAP/ICAP service or a "pre-cooked" note or other factor to the acl can 
be used)

I am pretty sure that if an authentication service can reply with a note ie 
connection annotation then it can be used for the address selection.
One issue with it is that It will be valid for the next X ttl 
seconds/minutes/hours.

I do believe that there should be a way to allow something like external_acl 
helper to affect this squid feature.
I was thinking that an eCAP or an ICAP service or an external_acl helper can 
add a note for a connection based on couple other factors like:
* src ip
* auth username
* request domain or request sni
* ...

So let say the proxy will have a set of 100 addresses, each will have a single 
specific matching acl for a request header or connection annotation/note.
This way the selection of a tcp_outgoing_address would be a little less complex 
the it is today.

I have couple other ideas for implementations which I have experimented with 
but the proxy admin need to learn how these work which might be
a bit complicated some times.

Eliezer

Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1...@gmail.com

-Original Message-
From: squid-users  On Behalf Of roee 
klinger
Sent: Saturday, October 31, 2020 2:35 AM
To: squid-users@lists.squid-cache.org
Subject: [squid-users] Best practice for adding or removing ACLs dynamically ?


Hey,
I have Squid configured to send users to different outgoing interface like so:

..
auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/htpassword
acl acl_for_user3002 proxy_auth user2
tcp_outgoing_address 192.168.8.12 acl_for_user3002
http_port 3002 name=3002
http_access allow authenticated
..

When I wanted to change the username:password for user2, I run a bash script to 
change it in squid.conf and also in htpassword and then I run "squid -k 
reconfigure", if I don't reconfigure the old user still has access to the proxy 
and the new one doesn't for about 30 minutes.

I am expecting to have 100s of users soon that will change credentials often, 
and also I would like to blacklist websites often and on the fly, so I was 
searching for a better way to manage this without reconfiguring every time, 
since sometimes a reconfigure can take up to 10-15 seconds.

I am new to Squid and wasn't able to find any info on this, am I doing this 
currently or there is a better way to change users/ACLs on the fly without 
reloading Squid?

Thanks,
Roee Klinger
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users