[squid-users] forward_max_tries 1 has no effect

[Ivan Larionov]

Hello.

We have an issue with squid when it tries to re-forward / retry failed
request even when forward_max_tries is set to 1. The situation when it
happens is when there's no response, parent just closes the connection.

Relevant parts of configuration so you understand the architecture:

cache_peer 127.0.0.1 parent 18070 0 no-query no-digest no-netdb-exchange
name=proxy
never_direct allow all
negative_ttl 0 seconds
forward_max_tries 1
retry_on_error off

The traffic flow from tcpdump is like this:

squid to parent
GET http://HOST/
parent to squid
ACK

waiting (no response for ~40 seconds)

parent to squid
FIN, ACK
squid to parent
FIN, ACK
parent to squid
FIN

Immediately after that:

squid to parent
GET http://HOST/ (again)

Debug logs from ALL,2:

…
http.cc(2229) sendRequest: HTTP Server local=127.0.0.2:46867 remote=
127.0.0.1:18070 FD 26 flags=1
http.cc(2230) sendRequest: HTTP Server REQUEST:
-
GET http://HOST:12345/ HTTP/1.1
…
http.cc(1299) continueAfterParsingHeader: WARNING: HTTP: Invalid Response:
No object data received for http://HOST:12345/ AKA HOST/
FwdState.cc(655) handleUnregisteredServerEnd: self=0x430a438*2
err=0x445fcf8 http://HOST:12345/
http.cc(2229) sendRequest: HTTP Server local=127.0.0.2:34417 remote=
127.0.0.1:18070 FD 26 flags=1
http.cc(2230) sendRequest: HTTP Server REQUEST:
-
GET http://HOST:12345/ HTTP/1.1
…

It doesn't happen 100% times. Sometimes squid returns 502 after the 1st
try, sometimes it retries once. Also I haven't seen more than 1 retry.

Could it be a bug? We'd really like to disable these retries.

Squid Cache: Version 3.5.27
Service Name: squid
configure options:  '--program-prefix=' '--prefix=/usr'
'--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
'--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid'
'--includedir=/usr/include' '--datadir=/usr/share'
'--sharedstatedir=/usr/com' '--localstatedir=/var'
'--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll'
'--enable-removal-policies=heap,lru' '--enable-storeio=aufs,rock'
'--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
'--with-large-files' '--with-maxfd=16384' '--enable-htcp'

-- 
With best regards, Ivan Larionov.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] url_rewrite_program and ACLs

[Amos Jeffries]

On 23/11/17 03:33, Vieri wrote:



From: Amos Jeffries 


If we assume that each request opens a new connection and they are not
closed until TCP times out on the socket we do get numbers much more
like that 11K+ you are seeing.

That implies that ICAP transactions are probably not finishing



completely.


I'll have to look into this asap. Quick question: if I restart c-icap shouldn't I see a 
drop in open FD numbers if it were c-icap's "fault"?

I restarted c-icap (stop+start), but the open FDs are the same.



Maybe, but there have been a few bugs where Squid being tricked into 
waiting on something arriving that was never coming left sockets in the 
TCP half-open CLOSE_WAIT state indefinitely.


Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] url_rewrite_program and ACLs

[Vieri]

From: Amos Jeffries 
>
> If we assume that each request opens a new connection and they are not 
> closed until TCP times out on the socket we do get numbers much more 
> like that 11K+ you are seeing.
> 
> That implies that ICAP transactions are probably not finishing 

> completely.

I'll have to look into this asap. Quick question: if I restart c-icap shouldn't 
I see a drop in open FD numbers if it were c-icap's "fault"?

I restarted c-icap (stop+start), but the open FDs are the same.

Thanks,

Vieri
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] block user agent

[Amos Jeffries]

On 22/11/17 23:48, Vieri wrote:


From: Amos Jeffries 


If you place that after the default "deny CONNECT !SSL_ports", and
before your UA checks, AND if you are using ssl_bump on the allowed
tunnels then you can relatively safely use "allow CONNECT".

Just be careful that the CONNECT allowed by that are always handled
safely by the ssl_bump rules you have.
   Meaning that you either bump or terminate traffic you are not sure is
okay, splice if you are reasonably sure, etc. it is a balancing effort
between "splice as much as possible" and "terminate if unsure of the
traffic" advice.



As you say, I placed "allow CONNECT" after the default "deny CONNECT 
!SSL_ports", and before my UA checks. I'm also using:
ssl_bump stare all
ssl_bump bump all


Considering the following (taken from previous e-mail):

http_access deny intercepted !localnet
http_access deny interceptedssl !localnet
http_access deny explicit !ORG_all
http_access deny explicit SSL_ports

Would it be "safer" or "indifferent" to use the following right before the UA 
checks?

http_access allow CONNECT interceptedssl SSL_ports



All CONNECT transactions that get past that earlier line with !SSL_Ports 
will match SSL_Ports. So that part of the line is redundant.


The "CONNECT interceptedssl" is more restricted than just "CONNECT" - so 
is safer due to that yes. But also leaves some traffic open to the same 
denial problem you had earlier if non-UA CONNECT happen other ways. Up 
to you whether that is wanted or acceptible.



Amos
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Lots of "BUG 3279: HTTP reply without Date:" after update to squid-5.0.0-20171117-r4d27d0a

[joseph]

after investigating  bug was exist but hidden all update ar correct
https://bugs.squid-cache.org/show_bug.cgi?id=3279
bug 3279  not yet fixed in v5
hint if they adapt the v3 patch will solve the problem :)



-
** 
* Crash to the future  
**
--
Sent from: 
http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-Users-f1019091.html
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] block user agent

[Vieri]

From: Amos Jeffries 
>
> If you place that after the default "deny CONNECT !SSL_ports", and 
> before your UA checks, AND if you are using ssl_bump on the allowed 
> tunnels then you can relatively safely use "allow CONNECT".
> 
> Just be careful that the CONNECT allowed by that are always handled 
> safely by the ssl_bump rules you have.
>   Meaning that you either bump or terminate traffic you are not sure is 
> okay, splice if you are reasonably sure, etc. it is a balancing effort 
> between "splice as much as possible" and "terminate if unsure of the 
> traffic" advice.


As you say, I placed "allow CONNECT" after the default "deny CONNECT 
!SSL_ports", and before my UA checks. I'm also using:
ssl_bump stare all
ssl_bump bump all


Considering the following (taken from previous e-mail):

http_access deny intercepted !localnet
http_access deny interceptedssl !localnet
http_access deny explicit !ORG_all
http_access deny explicit SSL_ports

Would it be "safer" or "indifferent" to use the following right before the UA 
checks?

http_access allow CONNECT interceptedssl SSL_ports


> Just FYI you would be a huge amount better off dropping the UA 
> fingerprinting. It's a _really_ simplistic idea about the HTTP world, 
> and it is partly because of that overly-simplistic nature and depending 
> on unreliable values that you are having so much more trouble than 
> normal admin face.


I'm aware that UA checks are not fully reliable, but in a big corporate 
environment it can reveal a lot of interested information.

I also know that some HTTP clients mimic others' user-agent strings or 
substrings. They can even sometimes dynamically change them.

However, in my particular case I could define a custom UA for our corporate 
browser allowed to go through Squid. For instance, Firefox can easily do that. 
Other browsers such as Edge seem not to.
In any case, it is not my intention to do so long-term. In short-term I found 
out that:

1) Squid logic *can* be understood :-)

2) some hosts may have HTTP clients that should be blocked even though the rest 
of the Squid rules were not programmed for that (so I couldn't know about it). 
A simple example: we may allow traffic to all microsoft sites, but some 
software may not necessarily be well installed/configured. I found that 
Microsoft Office may connect to an MS site to download or update software with 
a utility/service called 
OfficeClickToRun. Of course, generic rules in Squid.conf already blocked 
unauthorized downloads according to mimetypes or filetypes. However, some 
clients could be whitelisted and allowed to download (eg. from all MS sites). 
In this case, I would not necessarily want OfficeClickToRun to update. That 
could be done by identifying the dst domains, but they could change in time, 
and in any case would require more digging into. 


Adobe has similar http client behavior.


Anyway, it's informative to say the least, and can be used to improve the rest 
of the "standard" squid acl access rules.

I was also thinking of using custom HTTP headers such as X-MyCustomHeader: 
Whatever instead of UA strings. Custom headers can easily be added in Firefox, 
and other browsers such as Edge also seem to support that.

Anyway, I had a great time fiddling with Squid.
Thank you for your assistance.

Vieri
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Lots of "BUG 3279: HTTP reply without Date:" after update to squid-5.0.0-20171117-r4d27d0a

[Ralf Hildebrandt]

I was updating from squid-5.0.0-20171103-re3bf07f
to  squid-5.0.0-20171117-rd816577 (and I was also testing 
squid-5.0.0-20171117-r4d27d0a)

today and immediately found lots of "BUG 3279: HTTP reply without
Date:" messages in my log (cache_dir had been cleared upon start,
meaning I'm starting with a clean slate!):

2017/11/22 11:16:02| BUG 3279: HTTP reply without Date:
2017/11/22 11:16:02| StoreEntry->key: 2CD60200B515
2017/11/22 11:16:02| StoreEntry->next: 0x405f50e8
2017/11/22 11:16:02| StoreEntry->mem_obj: 0x35969af0
2017/11/22 11:16:02| StoreEntry->timestamp: 0
2017/11/22 11:16:02| StoreEntry->lastref: 1511345762
2017/11/22 11:16:02| StoreEntry->expires: 1511345762
2017/11/22 11:16:02| StoreEntry->lastModified_: -1
2017/11/22 11:16:02| StoreEntry->swap_file_sz: 0
2017/11/22 11:16:02| StoreEntry->refcount: 1
2017/11/22 11:16:02| StoreEntry->flags: 
REVALIDATE_ALWAYS,RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
2017/11/22 11:16:02| StoreEntry->swap_dirn: -1
2017/11/22 11:16:02| StoreEntry->swap_filen: -1
2017/11/22 11:16:02| StoreEntry->lock_count: 3
2017/11/22 11:16:02| StoreEntry->mem_status: 0
2017/11/22 11:16:02| StoreEntry->ping_status: 2
2017/11/22 11:16:02| StoreEntry->store_status: 1
2017/11/22 11:16:02| StoreEntry->swap_status: 0
2017/11/22 11:16:03| BUG 3279: HTTP reply without Date:
2017/11/22 11:16:03| StoreEntry->key: 13D60200B515
2017/11/22 11:16:03| StoreEntry->next: 0
2017/11/22 11:16:03| StoreEntry->mem_obj: 0x57a85d0
2017/11/22 11:16:03| StoreEntry->timestamp: 0
2017/11/22 11:16:03| StoreEntry->lastref: 1511345762
2017/11/22 11:16:03| StoreEntry->expires: 1511345763
2017/11/22 11:16:03| StoreEntry->lastModified_: -1
2017/11/22 11:16:03| StoreEntry->swap_file_sz: 0
2017/11/22 11:16:03| StoreEntry->refcount: 1
2017/11/22 11:16:03| StoreEntry->flags: 
RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
2017/11/22 11:16:03| StoreEntry->swap_dirn: -1
2017/11/22 11:16:03| StoreEntry->swap_filen: -1
2017/11/22 11:16:03| StoreEntry->lock_count: 3
2017/11/22 11:16:03| StoreEntry->mem_status: 0
2017/11/22 11:16:03| StoreEntry->ping_status: 2
2017/11/22 11:16:03| StoreEntry->store_status: 1
2017/11/22 11:16:03| StoreEntry->swap_status: 0
2017/11/22 11:16:03| BUG 3279: HTTP reply without Date:
2017/11/22 11:16:03| StoreEntry->key: 6AD60200B515
2017/11/22 11:16:03| StoreEntry->next: 0
2017/11/22 11:16:03| StoreEntry->mem_obj: 0x2fe3e2b0
2017/11/22 11:16:03| StoreEntry->timestamp: 0
2017/11/22 11:16:03| StoreEntry->lastref: 1511345763
2017/11/22 11:16:03| StoreEntry->expires: 1511345763
2017/11/22 11:16:03| StoreEntry->lastModified_: -1
2017/11/22 11:16:03| StoreEntry->swap_file_sz: 0
2017/11/22 11:16:03| StoreEntry->refcount: 1
2017/11/22 11:16:03| StoreEntry->flags: 
REVALIDATE_ALWAYS,RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
2017/11/22 11:16:03| StoreEntry->swap_dirn: -1
2017/11/22 11:16:03| StoreEntry->swap_filen: -1
2017/11/22 11:16:03| StoreEntry->lock_count: 3
2017/11/22 11:16:03| StoreEntry->mem_status: 0
2017/11/22 11:16:03| StoreEntry->ping_status: 2
2017/11/22 11:16:03| StoreEntry->store_status: 1
2017/11/22 11:16:03| StoreEntry->swap_status: 0

is this a known bug with the recent snapshot?


-- 
Ralf Hildebrandt   Charite Universitätsmedizin Berlin
ralf.hildebra...@charite.deCampus Benjamin Franklin
https://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users