Re: [squid-users] Backing up squid cache and restoring it

2019-01-24 Thread Amos Jeffries 
On 24/01/19 10:50 pm, Arne-Tobias Rak wrote:
> Thank you for your help. I eventually managed to get it working. The
> problem was related to stopping the squid service using the -k argument.
> 
> Closing squid using
> 
> sudo service squid stop
> 
> allows me to restore previous cache contents without any issues.
> 

Hmm, that does not make sense to me.

Squid-3.x are not compatible with systemd. They only way to use Squid-3
with a .service setup is for that to be linked up to use the "-k
shutdown" commands internally. Even then systemd's default SIGKILL
behaviour and confusion over what PID it is supposed to be monitoring
can result in problems anyway.

Please switch to Squid-4 if you want to use systemd controls.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Windows ECAP Success! And question about upload

2019-01-24 Thread Alex Rousskov 
On 1/24/19 8:38 AM, Russel McDonald wrote:
> Hi, I now have Squid running on Windows with ECAP passing both HTTP and
> HTTPS stream decrypted to my adapter. But I'm only seeing the download
> stream. How do I configure Squid to see the upload stream?

To see requests, you need a REQMOD eCAP service (reqmod_precache). A
REQMOD service is similar to a RESPMOD service you apparently have
working, but it is used for HTTP requests instead of HTTP responses. The
same set of squid.conf directives is used for both REQMOD and RESPMOD
vectoring points. The service itself will need to be written to handle
HTTP request messages instead of HTTP response messages, but the
protocol-level differences are minor.

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] squid on openwrt: Possible to get rid of "... SECURITY ALERT: Host header forgery detected ..." msgs ?

2019-01-24 Thread Alex Rousskov 
On 1/23/19 6:44 PM, Amos Jeffries wrote:
> For now all we can do is take the warnings seriously and find ways to
> prevent the network behaviours that cause them. 

For the record, the above is an opinion rather than a fact or consensus.
There are, of course, other (and far more realistic/useful) things we
could do. For example, we could give the admin the choice of which
"forgeries" should be classified as false positives and treated
accordingly, and we could improve reporting of the "forgeries" so that
the reporting itself does not become a problem.


> The security issues this detection prevents are so nasty we consider
> the pain worth the price of avoiding those outcomes.

In many cases, it would be possible for admins to suffer virtually no
"pain" while still "preventing security issues" and following best
practices (such as keeping logging enabled).

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] ICAP and 403 Encapsulated answers (SSL denied domains)

2019-01-24 Thread Alex Rousskov 
On 1/23/19 3:17 AM, FredB wrote:

> I found nothing in documentation about client_persistent_connections off
> impact, do you think this can be problematic with high load ?

Yes, disabling client-to-Squid persistent connections can increase load
on the proxy server. In SslBump environments that bump many connections,
such an increase can be drastic. IIRC, it may also break some
authenticaiton mechanisms that rely on connection persistency.

Alex.


___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] squid hanging in 100% steal

2019-01-24 Thread Eliezer Croitoru 
You can try the latest squid with my repo at:
http://ngtech.co.il/repo/amzn/1/

http://ngtech.co.il/repo/amzn/1/x86_64/squid-4.5-1.amzn1.x86_64.rpm
http://ngtech.co.il/repo/amzn/1/x86_64/squid-helpers-4.5-1.amzn1.x86_64.rpm

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: elie...@ngtech.co.il



-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Marc
Sent: Thursday, January 24, 2019 14:24
To: squid-users@lists.squid-cache.org
Subject: [squid-users] squid hanging in 100% steal

Hi,

For some reason my squid sometimes hangs (after weeks of running
smoothly) in 100% steal, until I kill the proces and restart it, after which 
the proces will again run stable for weeks.

It's running on a AWS EC2 instance, squid version:
squid-3.5.20-10.34.amzn1.x86_64 , see below for some debugging info.
Any idea what could be the problem here ? Thanks!

top:
[11:56:49][root@ip-172-31-9-138 ~]# top
top - 11:57:11 up 218 days, 17:36,  1 user,  load average: 1.06, 1.17, 1.09
Tasks:  81 total,   2 running,  79 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.5%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 95.2%st
Mem:501220k total,   405748k used,95472k free,65512k buffers
Swap:0k total,0k used,0k free,88948k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
29963 squid 20   0  290m 171m 7472 R 99.9 35.1 672:59.73 squid
1 root  20   0 19648 2480 2148 S  0.0  0.5   0:02.05 init


vmstat:
[11:57:39][root@ip-172-31-9-138 ~]# vmstat 1 procs ---memory-- 
---swap-- -io --system-- -cpu-
 r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id wa st
 1  1  0  95408  65536  8905200 0 411  0  0 99  0  0
 1  0  0  95408  65536  8904000 0 4   56   36  5  0  0  0 95
 2  0  0  95408  65536  8904000 0 0   54   18  5  0  0  0 95
 1  0  0  95408  65536  8904000 0 0   57   30  5  0  0  0 95
 1  0  0  95408  65536  8904000 0 4   52   25  5  0  0  0 95
 3  0  0  95408  65536  8904000 0 0   52   14  6  0  0  0 94
 1  0  0  95408  65536  8904000 0 0   50   26  4  0  0  0 96
 2  0  0  95408  65536  8904000 0 0   53   21  6  0  0  0 94
 1  0  0  95408  65540  8903600 012   62   38  5  0  0  0 95
 2  0  0  95408  65540  8904000 036   55   14  5  0  0  0 95
 1  0  0  95408  65540  8904000 0 0   51   34  5  0  0  0 95

gdb:
[11:55:07][root@ip-172-31-9-138 ~]# sudo gdb -n -batch -ex backtrace -pid 29963 
[Thread debugging using libthread_db enabled] Using host libthread_db library 
"/lib64/libthread_db.so.1".
0x007bca52 in
CbcPointer::operator=(CbcPointer
const&) ()
#0  0x007bca52 in
CbcPointer::operator=(CbcPointer
const&) ()
#1  0x007bc3d4 in Comm::AcceptLimiter::kick() ()
#2  0x00721867 in AsyncCall::make() ()
#3  0x007259e2 in AsyncCallQueue::fireNext() ()
#4  0x00725e20 in AsyncCallQueue::fire() ()
#5  0x005b0089 in EventLoop::runOnce() ()
#6  0x005b0178 in EventLoop::run() ()
#7  0x006192cc in SquidMain(int, char**) ()
#8  0x00514b3b in main ()

strace:
[11:52:51][root@ip-172-31-9-138 ~]# strace -t -s 8192 -f -p 29963 Process 29963 
attached
11:53:00 accept(10, {sa_family=AF_INET6, sin6_port=htons(45756), 
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0, 
sin6_scope_id=0}, [28]) = 16
11:53:00 getsockname(16, {sa_family=AF_INET6, sin6_port=htons(3128), 
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0, 
sin6_scope_id=0}, [28]) = 0
11:53:00 fcntl(16, F_GETFD) = 0
11:53:00 fcntl(16, F_SETFD, FD_CLOEXEC) = 0
11:53:00 fcntl(16, F_GETFL) = 0x2 (flags O_RDWR)
11:53:00 fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK) = 0
11:53:00 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 23
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENODEV (No such device)
11:53:00 ioctl(23, SIOCGIFCONF, {120, {{"lo", {AF_INET, 
inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("")}}, {"eth1", 
{AF_INET, inet_addr("")) = 0
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such device or 
address)
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such device or 
address)
11:53:00 close(23)  = 0
11:53:00 epoll_ctl(5, EPOLL_CTL_DEL, 27, {0, {u32=27, u64=4294967323}}) = 0
11:53:00 close(27)  = 0
11:53:03 accept(10, {sa_family=AF_INET6, sin6_port=htons(50050), 
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0, 
sin6_scope_id=0}, [28]) = 23
11:53:03 getsockname(23, {sa_family=AF_INET6, sin6_port=htons(3128), 
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0, 
sin6_scope_id=0}, [28]) = 0
11:53:03 fcntl(23, F_GETFD) = 0
11:53:03 fcntl(23,

[squid-users] Windows ECAP Success! And question about upload

2019-01-24 Thread Russel McDonald 
Hi, I now have Squid running on Windows with ECAP passing both HTTP and HTTPS 
stream decrypted to my adapter. But I'm only seeing the download stream. How do 
I configure Squid to see the upload stream? For example, I'd like to go to 
https:\\www.twitter.com, log in, tweet "Ulysses 12345" and have my adapter 
change that to "Grant 12345" so that "Grant 12345" is what gets posted. A 
squid.conf change? Or instead do I need reverse proxy as well.
Russel___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] squid on openwrt: Possible to get rid of "... SECURITY ALERT: Host header forgery detected ..." msgs ?

2019-01-24 Thread Leonardo Rodrigues 

Em 23/01/2019 06:22, reinerotto escreveu:

Running squid 4.4 on very limited device, unfortunately quite a lot of
messages: "... SECURITY ALERT: Host header forgery detected ... "  show up.
Unable to eliminate real cause of this issue (even using iptables to redir
all DNS requests to one dnsmasq does not help), these annoying messages tend
to fill up cache.log, which is kept in precious RAM.
Is there an "official" method to suppress these messages ?
Or can you please give a hint, where to apply a (hopefully) simple patch ?





    I have some OpenWRT boxes running squid 3.5 and cache_log simply 
goes null ... i do have access log enabled, with scripts to rotate, 
export to another server (where log analyzis are done) and keep just a 
minimum on the box itself, as storage is a big problem on these boxes.




--


Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertru...@solutti.com.br
My SPAMTRAP, do not email it



___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] squid hanging in 100% steal

2019-01-24 Thread Amos Jeffries 
On 25/01/19 1:24 am, Marc wrote:
> Hi,
> 
> For some reason my squid sometimes hangs (after weeks of running
> smoothly) in 100% steal, until I kill the proces and restart it, after
> which the proces will again run stable for weeks.

What does "100% steal" mean?

> 
> It's running on a AWS EC2 instance, squid version:
> squid-3.5.20-10.34.amzn1.x86_64 , see below for some debugging info.
> Any idea what could be the problem here ? Thanks!
> 
> top:
> [11:56:49][root@ip-172-31-9-138 ~]# top
> top - 11:57:11 up 218 days, 17:36,  1 user,  load average: 1.06, 1.17, 1.09
> Tasks:  81 total,   2 running,  79 sleeping,   0 stopped,   0 zombie
> Cpu(s):  4.5%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 95.2%st
> Mem:501220k total,   405748k used,95472k free,65512k buffers
> Swap:0k total,0k used,0k free,88948k cached
> 
>   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
> 29963 squid 20   0  290m 171m 7472 R 99.9 35.1 672:59.73 squid
> 1 root  20   0 19648 2480 2148 S  0.0  0.5   0:02.05 init
> 
> 
> vmstat:
> [11:57:39][root@ip-172-31-9-138 ~]# vmstat 1
> procs ---memory-- ---swap-- -io --system-- 
> -cpu-
>  r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id wa 
> st
>  1  1  0  95408  65536  8905200 0 411  0  0 99  0 
>  0
>  1  0  0  95408  65536  8904000 0 4   56   36  5  0  0  0 
> 95
>  2  0  0  95408  65536  8904000 0 0   54   18  5  0  0  0 
> 95
>  1  0  0  95408  65536  8904000 0 0   57   30  5  0  0  0 
> 95
>  1  0  0  95408  65536  8904000 0 4   52   25  5  0  0  0 
> 95
>  3  0  0  95408  65536  8904000 0 0   52   14  6  0  0  0 
> 94
>  1  0  0  95408  65536  8904000 0 0   50   26  4  0  0  0 
> 96
>  2  0  0  95408  65536  8904000 0 0   53   21  6  0  0  0 
> 94
>  1  0  0  95408  65540  8903600 012   62   38  5  0  0  0 
> 95
>  2  0  0  95408  65540  8904000 036   55   14  5  0  0  0 
> 95
>  1  0  0  95408  65540  8904000 0 0   51   34  5  0  0  0 
> 95
> 
> gdb:
> [11:55:07][root@ip-172-31-9-138 ~]# sudo gdb -n -batch -ex backtrace -pid 
> 29963
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> 0x007bca52 in
> CbcPointer::operator=(CbcPointer
> const&) ()
> #0  0x007bca52 in
> CbcPointer::operator=(CbcPointer
> const&) ()
> #1  0x007bc3d4 in Comm::AcceptLimiter::kick() ()
> #2  0x00721867 in AsyncCall::make() ()
> #3  0x007259e2 in AsyncCallQueue::fireNext() ()
> #4  0x00725e20 in AsyncCallQueue::fire() ()
> #5  0x005b0089 in EventLoop::runOnce() ()
> #6  0x005b0178 in EventLoop::run() ()
> #7  0x006192cc in SquidMain(int, char**) ()
> #8  0x00514b3b in main ()
> 

This looks like it may be one of the symptoms of
 which was fixed in
Squid-4.3 release.

Please try the current Squid-4 release to see if the issue is already
resolved. v3.5 is no longer supported, so if it is a bug we will need
traces and replication using the current Squid (v4 or v5) version to
have a realistic chance of anyone being able to fix it.

Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] squid hanging in 100% steal

2019-01-24 Thread Marc 
Hi,

For some reason my squid sometimes hangs (after weeks of running
smoothly) in 100% steal, until I kill the proces and restart it, after
which the proces will again run stable for weeks.

It's running on a AWS EC2 instance, squid version:
squid-3.5.20-10.34.amzn1.x86_64 , see below for some debugging info.
Any idea what could be the problem here ? Thanks!

top:
[11:56:49][root@ip-172-31-9-138 ~]# top
top - 11:57:11 up 218 days, 17:36,  1 user,  load average: 1.06, 1.17, 1.09
Tasks:  81 total,   2 running,  79 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.5%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si, 95.2%st
Mem:501220k total,   405748k used,95472k free,65512k buffers
Swap:0k total,0k used,0k free,88948k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
29963 squid 20   0  290m 171m 7472 R 99.9 35.1 672:59.73 squid
1 root  20   0 19648 2480 2148 S  0.0  0.5   0:02.05 init


vmstat:
[11:57:39][root@ip-172-31-9-138 ~]# vmstat 1
procs ---memory-- ---swap-- -io --system-- -cpu-
 r  b   swpd   free   buff  cache   si   sobibo   in   cs us sy id wa st
 1  1  0  95408  65536  8905200 0 411  0  0 99  0  0
 1  0  0  95408  65536  8904000 0 4   56   36  5  0  0  0 95
 2  0  0  95408  65536  8904000 0 0   54   18  5  0  0  0 95
 1  0  0  95408  65536  8904000 0 0   57   30  5  0  0  0 95
 1  0  0  95408  65536  8904000 0 4   52   25  5  0  0  0 95
 3  0  0  95408  65536  8904000 0 0   52   14  6  0  0  0 94
 1  0  0  95408  65536  8904000 0 0   50   26  4  0  0  0 96
 2  0  0  95408  65536  8904000 0 0   53   21  6  0  0  0 94
 1  0  0  95408  65540  8903600 012   62   38  5  0  0  0 95
 2  0  0  95408  65540  8904000 036   55   14  5  0  0  0 95
 1  0  0  95408  65540  8904000 0 0   51   34  5  0  0  0 95

gdb:
[11:55:07][root@ip-172-31-9-138 ~]# sudo gdb -n -batch -ex backtrace -pid 29963
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x007bca52 in
CbcPointer::operator=(CbcPointer
const&) ()
#0  0x007bca52 in
CbcPointer::operator=(CbcPointer
const&) ()
#1  0x007bc3d4 in Comm::AcceptLimiter::kick() ()
#2  0x00721867 in AsyncCall::make() ()
#3  0x007259e2 in AsyncCallQueue::fireNext() ()
#4  0x00725e20 in AsyncCallQueue::fire() ()
#5  0x005b0089 in EventLoop::runOnce() ()
#6  0x005b0178 in EventLoop::run() ()
#7  0x006192cc in SquidMain(int, char**) ()
#8  0x00514b3b in main ()

strace:
[11:52:51][root@ip-172-31-9-138 ~]# strace -t -s 8192 -f -p 29963
Process 29963 attached
11:53:00 accept(10, {sa_family=AF_INET6, sin6_port=htons(45756),
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 16
11:53:00 getsockname(16, {sa_family=AF_INET6, sin6_port=htons(3128),
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 0
11:53:00 fcntl(16, F_GETFD) = 0
11:53:00 fcntl(16, F_SETFD, FD_CLOEXEC) = 0
11:53:00 fcntl(16, F_GETFL) = 0x2 (flags O_RDWR)
11:53:00 fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK) = 0
11:53:00 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 23
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENODEV (No such device)
11:53:00 ioctl(23, SIOCGIFCONF, {120, {{"lo", {AF_INET,
inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("")}},
{"eth1", {AF_INET, inet_addr("")) = 0
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:00 close(23)  = 0
11:53:00 epoll_ctl(5, EPOLL_CTL_DEL, 27, {0, {u32=27, u64=4294967323}}) = 0
11:53:00 close(27)  = 0
11:53:03 accept(10, {sa_family=AF_INET6, sin6_port=htons(50050),
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 23
11:53:03 getsockname(23, {sa_family=AF_INET6, sin6_port=htons(3128),
inet_pton(AF_INET6, ":::", _addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 0
11:53:03 fcntl(23, F_GETFD) = 0
11:53:03 fcntl(23, F_SETFD, FD_CLOEXEC) = 0
11:53:03 fcntl(23, F_GETFL) = 0x2 (flags O_RDWR)
11:53:03 fcntl(23, F_SETFL, O_RDWR|O_NONBLOCK) = 0
11:53:03 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 25
11:53:03 ioctl(25, SIOCGARP, 0x7ffd21abeaa0) = -1 ENODEV (No such device)
11:53:03 ioctl(25, SIOCGIFCONF, {120, {{"lo", {AF_INET,
inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("")}},
{"eth1", {AF_INET, inet_addr("")) = 0
11:53:03 ioctl(25, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:03 ioctl(25, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)

Re: [squid-users] Backing up squid cache and restoring it

2019-01-24 Thread Arne-Tobias Rak 
Thank you for your help. I eventually managed to get it working. The 
problem was related to stopping the squid service using the -k argument.


Closing squid using

sudo service squid stop

allows me to restore previous cache contents without any issues.


Am 17.01.2019 um 18:22 schrieb Alex Rousskov:

On 1/17/19 6:47 AM, Arne-Tobias Rak wrote:


my goal is to restore a previous cache state in squid 3.x running on
Ubuntu 16.04.

So far I have tried to create a copy of the /var/spool/squid and
/var/log/squid folders.
When restoring the cache, I first shutdown squid using
/sudo squid -k shutdown//
//sudo service squid stop -k
/and then restore the previously copied folder contents. I then start
squid again using
/sudo service squid start./
Unfortunately, this does not restore the previous cache contents, as the
spool/squid/swap.state file is modified during squid startup.

Modification of swap.state upon startup is not incompatible with cache
contents preservation.

In a clean shutdown context, the swap.state* files are essentially an
optimization. You may preserve/restore them if you want to speed up
building of the restored cache index OR you can delete them (and Squid
will slowly build a new cache index from scratch). Just do not leave the
newer swap.state* files when trying to restore the old cache.

If you preserve/restore the old swap.state* files, you may need to
preserve their timestamps as well.


If you need further help, please share cache.log entries related to
cache_dir loading and indicate why you think the old cache contents is
not preserved. Sharing your cache_dir configuration may also help.


Cheers,

Alex.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users