Re: [squid-users] file descriptors leak
Em 28/11/2015 22:46, André Janna escreveu:
I took another network trace this time both at Squid and Windows
client ends.
cache.log:
2015/11/27 11:30:55.610 kid1| SECURITY ALERT: Host header forgery
detected on local=177.43.198.106:443 remote=192.168.64.4:61802 FD 5465
flags=33 (local IP does not match any domain IP)
--
network trace at Squid side
client connects
11:30:55.604870 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [S],
seq 1701554341, win 8192, options [mss 1460,nop,wscale
8,nop,nop,sackOK], length 0
11:30:55.604992 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags
[S.], seq 3125704417, ack 1701554342, win 29200, options [mss
1460,nop,nop,sackOK,nop,wscale 7], length 0
11:30:55.605766 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
ack 1, win 256, length 0
client sends SSL hello
11:30:55.606242 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[P.], seq 1:198, ack 1, win 256, length 197
11:30:55.606306 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, length 0
client OS sends TCP keep-alive packets
11:31:05.607191 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:05.607231 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:31:15.608966 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:15.609005 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:31:25.614527 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:25.614589 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
client sends FIN
11:31:29.384280 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[F.], seq 198, ack 1, win 256, length 0
11:31:29.421787 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, length 0
client OS sends TCP keep-alive packets
11:31:39.417426 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:39.417489 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
11:31:49.425366 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:49.425443 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
11:31:59.426153 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:59.426233 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
it continues this way until I powered off Windows client after
three hours ...
--
network trace at Windows client side
client connects
11:30:34.894242 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [S],
seq 1701554341, win 8192, options [mss 1460,nop,wscale
8,nop,nop,sackOK], length 0
11:30:34.898234 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags
[S.], seq 3125704417, ack 1701554342, win 29200, options [mss
1460,nop,nop,sackOK,nop,wscale 7], length 0
11:30:34.898298 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
ack 1, win 256, length 0
client sends SSL hello
11:30:34.898712 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[P.], seq 1:198, ack 1, win 256, length 197
11:30:34.899479 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, length 0
client OS sends TCP keep-alive packets
11:30:44.899271 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:30:44.899986 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:30:54.900495 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:30:54.901323 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:31:04.905731 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:04.906560 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
client sends FIN
11:31:08.675299 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[F.], seq 198, ack 1, win 256, length 0
11:31:08.713746 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, length 0
client OS sends TCP keep-alive packets
11:31:18.708086 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:18.708917 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
11:31:28.715600 IP 192.168.64.4.6180
Re: [squid-users] file descriptors leak
Citando Amos Jeffries :
So, the first place to look is not Squid I think. But why at least 6 of
those ACK packets did not make it back to the client. That needs
resolving first to esure that the TCP level is operating correctly.
Only then if the problem remains looking at Squid, the use of port 443
indicates it is the crypto process is possibly waiting for something and
not closing the port on a 0-byte read(2) operation.
I took another network trace this time both at Squid and Windows client ends.
cache.log:
2015/11/27 11:30:55.610 kid1| SECURITY ALERT: Host header forgery
detected on local=177.43.198.106:443 remote=192.168.64.4:61802 FD 5465
flags=33 (local IP does not match any domain IP)
--
network trace at Squid side
client connects
11:30:55.604870 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [S],
seq 1701554341, win 8192, options [mss 1460,nop,wscale
8,nop,nop,sackOK], length 0
11:30:55.604992 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags
[S.], seq 3125704417, ack 1701554342, win 29200, options [mss
1460,nop,nop,sackOK,nop,wscale 7], length 0
11:30:55.605766 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
ack 1, win 256, length 0
client sends SSL hello
11:30:55.606242 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[P.], seq 1:198, ack 1, win 256, length 197
11:30:55.606306 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, length 0
client OS sends TCP keep-alive packets
11:31:05.607191 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:05.607231 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:31:15.608966 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:15.609005 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:31:25.614527 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:25.614589 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
client sends FIN
11:31:29.384280 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[F.], seq 198, ack 1, win 256, length 0
11:31:29.421787 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, length 0
client OS sends TCP keep-alive packets
11:31:39.417426 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:39.417489 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
11:31:49.425366 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:49.425443 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
11:31:59.426153 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 198:199, ack 1, win 256, length 1
11:31:59.426233 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 199, win 237, options [nop,nop,sack 1 {198:199}], length 0
it continues this way until I powered off Windows client after
three hours ...
--
network trace at Windows client side
client connects
11:30:34.894242 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [S],
seq 1701554341, win 8192, options [mss 1460,nop,wscale
8,nop,nop,sackOK], length 0
11:30:34.898234 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags
[S.], seq 3125704417, ack 1701554342, win 29200, options [mss
1460,nop,nop,sackOK,nop,wscale 7], length 0
11:30:34.898298 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
ack 1, win 256, length 0
client sends SSL hello
11:30:34.898712 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[P.], seq 1:198, ack 1, win 256, length 197
11:30:34.899479 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, length 0
client OS sends TCP keep-alive packets
11:30:44.899271 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:30:44.899986 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:30:54.900495 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:30:54.901323 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
11:31:04.905731 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags [.],
seq 197:198, ack 1, win 256, length 1
11:31:04.906560 IP 177.43.198.106.443 > 192.168.64.4.61802: Flags [.],
ack 198, win 237, options [nop,nop,sack 1 {197:198}], length 0
client sends FIN
11:31:08.675299 IP 192.168.64.4.61802 > 177.43.198.106.443: Flags
[F.], seq 198, ack 1, win 256, length
Re: [squid-users] file descriptors leak
On 27/11/2015 7:36 a.m., André Janna wrote:
>
> Assinatura
> Em 24/11/2015 00:54, Amos Jeffries escreveu:
>> FYI: unless you have a specific need for 3.5 you should be fine with
>> the 3.4 squid3 package that is available for Jesse from Debian
>> backports. The alternative is going the other way and upgrading right
>> to the latest 3.5 snapshot (and/or 4.0 snapshot) to see if it is one
>> of the CONNECT or TLS issues we have fixed recently.
> I'm using version 3.5 because 3.4 doesn't have ssl::server_name acl.
> Debian package is not built with openssl because of licensing issues so
> I rebuilt Debian testing 3.5 source package on Debian Jessie.
> This Squid installation is in production now and cannot be easily
> migrated. But I'll perform another installation for testing in the near
> future.
>
>> Neither. So it is time to move away from lsof and start using packet
>> capture to get a full-body packet trace to find out what exact packets
>> are happening on at least one affected TCP connection.
>>
>> If possible identifying one of these connections from its SYN onwards
>> would be great, but if not then a 20min period of activity on an
>> existing one might still how more hints.
>>
> I did a test using a Windows laptop client with IP address 192.168.64.4,
> connected via wifi.
> I browsed a few https sites until triggering Squid "local IP does not
> match any domain IP" error.
> This error appeared when I was trying to open Yahoo home page. Browser
> redirected to https://br.yahoo.com/?p=us but page remained blank.
> Please note that this error appears randomly: opening the same site in
> another browser tab succeeded.
>
> cache.log:
> 2015/11/26 13:54:45.471 kid1| SECURITY ALERT: Host header forgery
> detected on local=206.190.56.191:443 remote=192.168.64.4:58887 FD 17244
> flags=33 (local IP does not match any domain IP)
>
> After a couple of minutes this connection disappeared from Windows
> netstat command output. Afterward I powered off Windows laptop.
>
> Tcpdump on Squid box:
> 13:54:45.410907 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [S],
> seq 1831867, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
> length 0
> 13:54:45.411000 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [S.],
> seq 3695298276, ack 1831868, win 29200, options [mss
> 1460,nop,nop,sackOK,nop,wscale 7], length 0
client 192.168.64.4:58887 connects.
> 13:54:45.411630 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> ack 1, win 256, length 0
> 13:54:45.412490 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [P.],
> seq 1:185, ack 1, win 256, length 184
> 13:54:45.412573 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, length 0
client sends 184 bytes of data.
> 13:54:55.439709 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:54:55.439761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:05.439965 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:05.440022 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:15.445667 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:15.445737 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:25.447281 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:25.447351 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:35.494936 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:35.495005 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:45.491694 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:45.491761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:55.492158 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:55.492208 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
client sends 1 byte of data - in 7x packets.
The recipient sends an ACK each and every time, but the client just
keeps repeating itself.
> 14:01:58.242748 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [F.],
> seq 185, ack 1, win 256, length 0
> 14:01:58.279916 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 186, win 237, length 0
... until the client changes and sends a FIN.
>
> Netstat output on Squid box:
> # date; netstat -tno | grep 192.168.64.4
> Thu Nov 26 13:59
Re: [squid-users] file descriptors leak
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
27.11.15 0:36, André Janna пишет:
>
> Assinatura
> Em 24/11/2015 00:54, Amos Jeffries escreveu:
>> FYI: unless you have a specific need for 3.5 you should be fine with
the 3.4 squid3 package that is available for Jesse from Debian
backports. The alternative is going the other way and upgrading right to
the latest 3.5 snapshot (and/or 4.0 snapshot) to see if it is one of the
CONNECT or TLS issues we have fixed recently.
> I'm using version 3.5 because 3.4 doesn't have ssl::server_name acl.
> Debian package is not built with openssl because of licensing issues
so I rebuilt Debian testing 3.5 source package on Debian Jessie.
> This Squid installation is in production now and cannot be easily
migrated. But I'll perform another installation for testing in the near
future.
>
>> Neither. So it is time to move away from lsof and start using packet
>> capture to get a full-body packet trace to find out what exact packets
>> are happening on at least one affected TCP connection.
>>
>> If possible identifying one of these connections from its SYN onwards
>> would be great, but if not then a 20min period of activity on an
>> existing one might still how more hints.
>>
> I did a test using a Windows laptop client with IP address
192.168.64.4, connected via wifi.
> I browsed a few https sites until triggering Squid "local IP does not
match any domain IP" error.
> This error appeared when I was trying to open Yahoo home page. Browser
redirected to https://br.yahoo.com/?p=us but page remained blank.
> Please note that this error appears randomly: opening the same site in
another browser tab succeeded.
>
> cache.log:
> 2015/11/26 13:54:45.471 kid1| SECURITY ALERT: Host header forgery
detected on local=206.190.56.191:443 remote=192.168.64.4:58887 FD 17244
flags=33 (local IP does not match any domain IP)
It's so commonplace that even Wiki long time ago there article:
http://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery
>
> After a couple of minutes this connection disappeared from Windows
netstat command output. Afterward I powered off Windows laptop.
>
> Tcpdump on Squid box:
> 13:54:45.410907 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [S],
seq 1831867, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
> 13:54:45.411000 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags
[S.], seq 3695298276, ack 1831868, win 29200, options [mss
1460,nop,nop,sackOK,nop,wscale 7], length 0
> 13:54:45.411630 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
ack 1, win 256, length 0
> 13:54:45.412490 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags
[P.], seq 1:185, ack 1, win 256, length 184
> 13:54:45.412573 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, length 0
> 13:54:55.439709 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:54:55.439761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:05.439965 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:55:05.440022 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:15.445667 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:55:15.445737 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:25.447281 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:55:25.447351 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:35.494936 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:55:35.495005 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:45.491694 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:55:45.491761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:55.492158 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
> 13:55:55.492208 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 14:01:58.242748 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags
[F.], seq 185, ack 1, win 256, length 0
> 14:01:58.279916 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 186, win 237, length 0
>
> Netstat output on Squid box:
> # date; netstat -tno | grep 192.168.64.4
> Thu Nov 26 13:59:40 BRST 2015
> tcp6 1 0 172.16.10.22:3126 192.168.64.4:58887
CLOSE_WAIT off (0.00/0/0)
>
> And after 2 hours and a half netstat output is still the same:
> #
Re: [squid-users] file descriptors leak
Assinatura
Em 24/11/2015 00:54, Amos Jeffries escreveu:
FYI: unless you have a specific need for 3.5 you should be fine with
the 3.4 squid3 package that is available for Jesse from Debian
backports. The alternative is going the other way and upgrading right
to the latest 3.5 snapshot (and/or 4.0 snapshot) to see if it is one
of the CONNECT or TLS issues we have fixed recently.
I'm using version 3.5 because 3.4 doesn't have ssl::server_name acl.
Debian package is not built with openssl because of licensing issues so
I rebuilt Debian testing 3.5 source package on Debian Jessie.
This Squid installation is in production now and cannot be easily
migrated. But I'll perform another installation for testing in the near
future.
Neither. So it is time to move away from lsof and start using packet
capture to get a full-body packet trace to find out what exact packets
are happening on at least one affected TCP connection.
If possible identifying one of these connections from its SYN onwards
would be great, but if not then a 20min period of activity on an
existing one might still how more hints.
I did a test using a Windows laptop client with IP address 192.168.64.4,
connected via wifi.
I browsed a few https sites until triggering Squid "local IP does not
match any domain IP" error.
This error appeared when I was trying to open Yahoo home page. Browser
redirected to https://br.yahoo.com/?p=us but page remained blank.
Please note that this error appears randomly: opening the same site in
another browser tab succeeded.
cache.log:
2015/11/26 13:54:45.471 kid1| SECURITY ALERT: Host header forgery
detected on local=206.190.56.191:443 remote=192.168.64.4:58887 FD 17244
flags=33 (local IP does not match any domain IP)
After a couple of minutes this connection disappeared from Windows
netstat command output. Afterward I powered off Windows laptop.
Tcpdump on Squid box:
13:54:45.410907 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [S],
seq 1831867, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
length 0
13:54:45.411000 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [S.],
seq 3695298276, ack 1831868, win 29200, options [mss
1460,nop,nop,sackOK,nop,wscale 7], length 0
13:54:45.411630 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
ack 1, win 256, length 0
13:54:45.412490 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [P.],
seq 1:185, ack 1, win 256, length 184
13:54:45.412573 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, length 0
13:54:55.439709 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:54:55.439761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
13:55:05.439965 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:55:05.440022 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
13:55:15.445667 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:55:15.445737 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
13:55:25.447281 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:55:25.447351 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
13:55:35.494936 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:55:35.495005 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
13:55:45.491694 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:55:45.491761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
13:55:55.492158 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
seq 184:185, ack 1, win 256, length 1
13:55:55.492208 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
14:01:58.242748 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [F.],
seq 185, ack 1, win 256, length 0
14:01:58.279916 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
ack 186, win 237, length 0
Netstat output on Squid box:
# date; netstat -tno | grep 192.168.64.4
Thu Nov 26 13:59:40 BRST 2015
tcp6 1 0 172.16.10.22:3126 192.168.64.4:58887
CLOSE_WAIT off (0.00/0/0)
And after 2 hours and a half netstat output is still the same:
# date; netstat -tno | grep 192.168.64.4
Thu Nov 26 16:32:37 BRST 2015
tcp6 1 0 172.16.10.22:3126 192.168.64.4:58887
CLOSE_WAIT off (0.00/0/0)
Squid is still using the file descriptor.
# date; lsof -n | grep 192.168.64.4
Thu Nov 26 16:33:10 BRST 2015
squid
Re: [squid-users] file descriptors leak
Just as a side note you should know that tcpdump on a busy server needs bigger buffer size to prevent the drop of captured packets. Eliezer On 24/11/2015 04:54, Amos Jeffries wrote: If possible identifying one of these connections from its SYN onwards would be great, but if not then a 20min period of activity on an existing one might still how more hints. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
On 24/11/2015 7:45 a.m., André Janna wrote: > > Assin Em 22/11/2015 16:25, Eliezer Croitoru escreveu: >> Hey Andre, >> >> There are couple things to the picture. >> It's not only squid that is the "blame". >> It depends on what your OS tcp stack settings are. >> To verify couple things you can try to use the netstat tool. >> run the command "netstat -nto" to see what is the timers status. >> You can then see how long will a new connection stay in the >> established state. >> It might be the squid settings but if the client is not there it could >> be because of some tcp tunable kernel settings. > > Hi Eliezer and Amos, > my kernel is a regular Debian Jessie kernel using the following tcp values. > tcp_keepalive_time: 7200 > tcp_keepalive_intvl: 25 > tcp_keepalive_probes: 9 > tcp_retries1: 3 > tcp_retries2: 15 > tcp_fin_timeout: 60 > So in my understanding the longest timeout is set to 2 hours and a few > minutes for keepalive connections. Okay. It is not always your kernel Squid machine. I've seen one mobile network where the Ethernet<->radio modem was interpreting the radio being alive as TCP keep-alive needing to stay alive. So just having the phones connected to the network would keep everything active. IIRC the only fix for that scenario is reducing Squid's client_lifetime value. FYI: unless you have a specific need for 3.5 you should be fine with the 3.4 squid3 package that is available for Jesse from Debian backports. The alternative is going the other way and upgrading right to the latest 3.5 snapshot (and/or 4.0 snapshot) to see if it is one of the CONNECT or TLS issues we have fixed recently. > > Today I monitored file descriptors 23 and 24 on my box during 5 hours > and lsof always showed: > squid 6574 proxy 23u IPv6 5320944 > 0t0TCP 172.16.10.22:3126->192.168.90.35:34571 (CLOSE_WAIT) > squid 6574 proxy 24u IPv6 5327276 > 0t0TCP 172.16.10.22:3126->192.168.89.236:49435 (ESTABLISHED) > while netstat always showed: > tcp6 1 0 172.16.10.22:3126 192.168.90.35:34571 > CLOSE_WAIT 6574/(squid-1) off (0.00/0/0) > tcp6 0 0 172.16.10.22:3126 192.168.89.236:49435 > ESTABLISHED 6574/(squid-1) off (0.00/0/0) > > The "off" flag in netstat output tells that for these sockets keepalive > and retransmission timers are disabled. Oooh. That should mean 30sec timout and then RST. Not even a whole minute of idle time. > Right now netstat shows 15,568 connections on squid port 3126 and only > 107 have timer set to a value other than "off". > > I read that connections that are in CLOSE_WAIT state don't have any tcp > timeout, it's Squid that must close the socket. Squid closes the socket/FD as soon as it received the FIN or RST that began the CLOSE_WAIT state. Unless it was Squid closing that began it. > > About the connections in ESTABLISHED state, I monitored the connection > to mobile device 192.168.89.236 using "tcpdump -i eth2 -n host > 192.168.89.236" during 2 hours and a half. > Tcpdump didn't record any packet and netstat is still displaying: > tcp6 1 0 172.16.10.22:3126 192.168.90.35:34571 > CLOSE_WAIT 6574/(squid-1) off (0.00/0/0) > tcp6 0 0 172.16.10.22:3126 192.168.89.236:49435 > ESTABLISHED 6574/(squid-1) off (0.00/0/0) > > So unfortunately I still don't understand why Squid or the kernel don't > close these sockets. Neither. So it is time to move away from lsof and start using packet capture to get a full-body packet trace to find out what exact packets are happening on at least one affected TCP connection. If possible identifying one of these connections from its SYN onwards would be great, but if not then a 20min period of activity on an existing one might still how more hints. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
Assin Em 22/11/2015 16:25, Eliezer Croitoru escreveu: Hey Andre, There are couple things to the picture. It's not only squid that is the "blame". It depends on what your OS tcp stack settings are. To verify couple things you can try to use the netstat tool. run the command "netstat -nto" to see what is the timers status. You can then see how long will a new connection stay in the established state. It might be the squid settings but if the client is not there it could be because of some tcp tunable kernel settings. Hi Eliezer and Amos, my kernel is a regular Debian Jessie kernel using the following tcp values. tcp_keepalive_time: 7200 tcp_keepalive_intvl: 25 tcp_keepalive_probes: 9 tcp_retries1: 3 tcp_retries2: 15 tcp_fin_timeout: 60 So in my understanding the longest timeout is set to 2 hours and a few minutes for keepalive connections. Today I monitored file descriptors 23 and 24 on my box during 5 hours and lsof always showed: squid 6574 proxy 23u IPv6 5320944 0t0TCP 172.16.10.22:3126->192.168.90.35:34571 (CLOSE_WAIT) squid 6574 proxy 24u IPv6 5327276 0t0TCP 172.16.10.22:3126->192.168.89.236:49435 (ESTABLISHED) while netstat always showed: tcp6 1 0 172.16.10.22:3126 192.168.90.35:34571 CLOSE_WAIT 6574/(squid-1) off (0.00/0/0) tcp6 0 0 172.16.10.22:3126 192.168.89.236:49435 ESTABLISHED 6574/(squid-1) off (0.00/0/0) The "off" flag in netstat output tells that for these sockets keepalive and retransmission timers are disabled. Right now netstat shows 15,568 connections on squid port 3126 and only 107 have timer set to a value other than "off". I read that connections that are in CLOSE_WAIT state don't have any tcp timeout, it's Squid that must close the socket. About the connections in ESTABLISHED state, I monitored the connection to mobile device 192.168.89.236 using "tcpdump -i eth2 -n host 192.168.89.236" during 2 hours and a half. Tcpdump didn't record any packet and netstat is still displaying: tcp6 1 0 172.16.10.22:3126 192.168.90.35:34571 CLOSE_WAIT 6574/(squid-1) off (0.00/0/0) tcp6 0 0 172.16.10.22:3126 192.168.89.236:49435 ESTABLISHED 6574/(squid-1) off (0.00/0/0) So unfortunately I still don't understand why Squid or the kernel don't close these sockets. Regards, André ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
On 23/11/2015 7:25 a.m., Eliezer Croitoru wrote: > Hey Andre, > > There are couple things to the picture. > It's not only squid that is the "blame". > It depends on what your OS tcp stack settings are. > To verify couple things you can try to use the netstat tool. > run the command "netstat -nto" to see what is the timers status. > You can then see how long will a new connection stay in the established > state. > It might be the squid settings but if the client is not there it could > be because of some tcp tunable kernel settings. Eliezer is right. The TCP layer itself should be terminating the connection within a short time (30sec default) after the clients last packet. Even if you use the TCP level keep-alive feature, that works by ensuring small packets go back and forth between the Squid device and the user device to keep the router state alive. Something is making the TCP stack itself think the client device is still connected *and active* on the network. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
Hey Andre, There are couple things to the picture. It's not only squid that is the "blame". It depends on what your OS tcp stack settings are. To verify couple things you can try to use the netstat tool. run the command "netstat -nto" to see what is the timers status. You can then see how long will a new connection stay in the established state. It might be the squid settings but if the client is not there it could be because of some tcp tunable kernel settings. In any case a Linux machine can handle up to ... very very high number of open connections idle or not so in a case you are starting to run out of them try to upper the limit by 50% or even 100% and monitor your machine netstat for abnormal too long connections. All The Bests, Eliezer On 22/11/2015 19:18, André Janna wrote: Update: Squid released file descriptors after about 24 hours, I suppose because expired client_lifetime which is set to 1 day default value. Regards, André ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
Citando André Janna: Squid is still using file descriptors 12 and 14 (and a lot of others) for the same connections as yesterday, although the mobile devices it was connected to have not been online in our network for at least 15 hours. Update: Squid released file descriptors after about 24 hours, I suppose because expired client_lifetime which is set to 1 day default value. Regards, André ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
Citando Amos Jeffries : CONNECT requests with tunnels can be particularly long lived, mobiles and their applications stay active for weeks on end with few outward signs of what is happening inside the encrypted tunnel. The only way to be sure the connection is finished with is when one of the client or server remote endpoints closes it. * The port 55815 connection _was_ closed sometimes within 15 mintes of the lsof being run. * The port 52288 connection is still being used. Given the timespan between those messages and the lsof, it is also possible they were closed and reopened in between. If you have a lot of ports in active use, then re-use of closed ones becomes more likely. Though I suspect it is just persistence doing what it is designed to do. You will need a more detailed trace of the entire time period to know. 11 more hours have passed since my last "lsof" and cause it's Sunday I'm sure that no device have been connected to 192.168.x.x network at least for 15 hours. Right now lsof output is the same as 11 hours before: squid 32490 proxy 12u IPv6 4065613 0t0 TCP 172.16.10.22:3126->192.168.93.113:55815 (CLOSE_WAIT) squid 32490 proxy 14u IPv6 4097822 0t0 TCP 172.16.10.22:3126->192.168.90.207:52288 (ESTABLISHED) ... Squid is still using file descriptors 12 and 14 (and a lot of others) for the same connections as yesterday, although the mobile devices it was connected to have not been online in our network for at least 15 hours. Is it by design? Raising file descriptors limit is the only solution? The maximum number of file descriptors in my installation is set to 65535. Is there any drawback to increasing this number let's say by a factor of ten? Regards, André ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] file descriptors leak
On 22/11/2015 4:10 p.m., André Janna wrote: > I'm running Squid 3.5.10 on Debian Jessie and after some hours of execution > it runs out of file descriptors. > Squid is listening on port 3125, 3126 and 3127. > Port 3126 is used for intercepting, via iptables redirect, https > connections mostly from mobile devices like smartphones. On this port is > active ssl-bump but I'm not decrypting https traffic, only "peek" to get > https server host name. > Port 3125 is used for intercepting http connections of the same mobile > devices whose https traffic is intercepted on port 3126. > Port 3127 is used for clients configured to use a proxy. > Leaked file descriptors are all related to connection on port 3126 (https > intercept ssl-bump). > A sample output of lsof command gives: > > squid 32490proxy 12u > IPv64065613 0t0TCP > 172.16.10.22:3126->192.168.93.113:55815 (CLOSE_WAIT) > squid 32490proxy 14u > IPv64097822 0t0TCP > 172.16.10.22:3126->192.168.90.207:52288 (ESTABLISHED) > ... > > where 172.16.10.22 is an IP address of my Squid installation and > 192.168.x.x are mobile devices. > Is seems that this condition is triggered by "local IP does not match any > domain IP" error logged by Squid in cache.log, but I'm not sure if all > stuck connections are caused by this kind of error. > For the 2 connections of the sample above the related cache.log errors are: > > 2015/11/21 12:57:51.229 kid1| SECURITY ALERT: Host header forgery > detected on local=23.0.163.57:443 remote=192.168.93.113:55815 FD 12 > flags=33 (local IP does not match any domain IP) > 2015/11/21 13:59:44.230 kid1| SECURITY ALERT: Host header forgery > detected on local=198.144.127.162:443 remote=192.168.90.207:52288 FD 14 > flags=33 (local IP does not match any domain IP) > > "lsof" sample output was taken more that 10 hours after Squid logged these > errors and it shows that Squid is still holding connections open, using a > lot of file descriptors. In HTTP connections stay open and get used for many requests. You are however assuming that the connection actually contains HTTP. There is no guarantee of that without bumping the decrypt and parsing the content inside. There are a number of non-HTTP/1.1 protocols that use port 443 to bypass proxy and firewall security. CONNECT requests with tunnels can be particularly long lived, mobiles and their applications stay active for weeks on end with few outward signs of what is happening inside the encrypted tunnel. The only way to be sure the connection is finished with is when one of the client or server remote endpoints closes it. * The port 55815 connection _was_ closed sometimes within 15 mintes of the lsof being run. * The port 52288 connection is still being used. Given the timespan between those messages and the lsof, it is also possible they were closed and reopened in between. If you have a lot of ports in active use, then re-use of closed ones becomes more likely. Though I suspect it is just persistence doing what it is designed to do. You will need a more detailed trace of the entire time period to know. PS. don't confuse file descriptors with ports. There as an absolute maximum of 64K ports per IP on each device. But sockets FD can reach millions, if you run out of FD just configure more to be allowed. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] file descriptors leak
I'm running Squid 3.5.10 on Debian Jessie and after some hours of execution it runs out of file descriptors. Squid is listening on port 3125, 3126 and 3127. Port 3126 is used for intercepting, via iptables redirect, https connections mostly from mobile devices like smartphones. On this port is active ssl-bump but I'm not decrypting https traffic, only "peek" to get https server host name. Port 3125 is used for intercepting http connections of the same mobile devices whose https traffic is intercepted on port 3126. Port 3127 is used for clients configured to use a proxy. Leaked file descriptors are all related to connection on port 3126 (https intercept ssl-bump). A sample output of lsof command gives: squid 32490 proxy 12u IPv6 4065613 0t0 TCP 172.16.10.22:3126->192.168.93.113:55815 (CLOSE_WAIT) squid 32490 proxy 14u IPv6 4097822 0t0 TCP 172.16.10.22:3126->192.168.90.207:52288 (ESTABLISHED) ... where 172.16.10.22 is an IP address of my Squid installation and 192.168.x.x are mobile devices. Is seems that this condition is triggered by "local IP does not match any domain IP" error logged by Squid in cache.log, but I'm not sure if all stuck connections are caused by this kind of error. For the 2 connections of the sample above the related cache.log errors are: 2015/11/21 12:57:51.229 kid1| SECURITY ALERT: Host header forgery detected on local=23.0.163.57:443 remote=192.168.93.113:55815 FD 12 flags=33 (local IP does not match any domain IP) 2015/11/21 13:59:44.230 kid1| SECURITY ALERT: Host header forgery detected on local=198.144.127.162:443 remote=192.168.90.207:52288 FD 14 flags=33 (local IP does not match any domain IP) "lsof" sample output was taken more that 10 hours after Squid logged these errors and it shows that Squid is still holding connections open, using a lot of file descriptors. Regards, André --- my squid.conf --- acl localnet src 10.0.0.0/8 # RFC1918 possible internal network acl localnet src 172.16.0.0/12 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl CONNECT method CONNECT acl squid-internal-static url_regex ^http://nat-academico:3127/squid-internal-static/ acl e2guardian localport 3127 follow_x_forwarded_for allow localhost http_access allow squid-internal-static http_access allow localhost manager http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access deny to_localhost http_access allow localhost http_access allow localnet e2guardian include /etc/squid/transparent-blacklist.conf include /etc/squid/transparent-whitelist.conf http_access allow transparent-whitelist-http http_access deny transparent-blacklist-http http_access allow localnet http_access deny all http_port 3127 http_port 3125 intercept https_port 3126 cert=/etc/ssl/certs/nat-academico.crt key=/etc/ssl/private/services.key intercept ssl-bump acl step1 at_step SslBump1 ssl_bump peek step1 all ssl_bump splice transparent-whitelist-https ssl_bump terminate transparent-blacklist-https cache_dir ufs /var/spool/squid 1 16 256 coredump_dir /var/spool/squid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 dns_v4_first on ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
