Re: [squid-users] Error 503 accessing Instagram/facebook via IPv6
Hey, Is this a tproxy or intercept setup? Eliezer -Original Message- From: squid-users On Behalf Of marcelorodr...@graminsta.com.br Sent: Saturday, October 30, 2021 09:10 To: squid-users@lists.squid-cache.org Subject: [squid-users] Error 503 accessing Instagram/facebook via IPv6 Hi, I have been using squid for several years and am very grateful for the solution. Since last 3-4 days my customers haven't been able to access www.instagram.com and Facebook throug IPv6s that were already working as proxies for years. I only get 503 error after a time out. The strangest thing is that I can connect 20-30% of the attempts. I didn't make any changes to the VPSs. I switched IPv6s provider to a totally different subnet, changed several equipments, but it didn't work at all. If I access through the same VPS using the same IPs as squid is running using curl command but not going through the proxy It works. But through Squid it no longer accesses as before. This only happens with v6 IPs. On V4 Squid runs normally. I upgraded to Ubuntu 20.04 and Squid 5.2-10 but everything remains the same. My agency depends on it and I've already lost half of my clients. Could you please help me, even if I have to pay for support? ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid 5.2 Peer parent TCP connection to x.x.x.x/x failed
On 11/2/21 12:28 PM, David Touzeau wrote: > Ok, we will investigate on the Parent Proxy but it seems that when squid > child claim about TCP failed, it understand that the peer is dead Yes, that is the side effect of the deficiency I was talking about -- Squid inability to recognize that specific error response from the (indirect) parent Squid as something completely unrelated to the TCP connection to the next hop (and to the responding parent Squid ability to serve traffic in general). There are quite a few things that could be improved here for your and similar use cases. Alex. > Le 02/11/2021 à 16:17, Alex Rousskov a écrit : >> On 11/2/21 10:40 AM, David Touzeau wrote: >>> 2021/11/01 16:50:48.787 kid1| 93,3| >>> Http::Tunneler::handleReadyRead(conn9812727 local=127.0.0.1:23408 >>> remote=127.0.0.1:2320 FIRSTUP_PARENT) >>> 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: proto HTTP/1.1 >>> 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: status-code 503 >>> 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: reason-phrase >>> Service Unavailable >>> Server: squid >>> Date: Mon, 01 Nov 2021 15:50:48 GMT >>> X-Squid-Error: ERR_CONNECT_FAIL 110 >>> 2021/11/01 16:50:48.787 kid1| 83,3| bailOnResponseError: unsupported >>> CONNECT response status code >>> 2021/11/01 16:50:48.787 kid1| TCP connection to 127.0.0.1/2320 failed >> A parent[^1] proxy is a Squid proxy that cannot connect to the server in >> question. That Squid proxy responds with an HTTP 503 Error to your Squid >> CONNECT request. Your Squid logs the "TCP connection to ... failed" >> error that you were wondering about. >> >> This sequence highlights a deficiency in Squid CONNECT error handling >> code (and possibly cache_peer configuration abilities). Ideally, Squid >> should recognize Squid error responses coming from a parent HTTP proxy >> and avoid complaining about remote Squid-origin errors as if they are >> local Squid-parent errors. IIRC, some folks still insist on Squid >> complaining about the latter "within hierarchy" errors, but the former >> "external Squid-origin" errors are definitely not supposed to be >> reported to admins via level-0/1 messages in cache.log. >> >> >> HTH, >> >> Alex. >> >> [^1]: Direct or indirect parent -- I could not tell quickly but you >> should be able to tell by looking at addresses, configurations, and/or >> access logs. My bet is that it is an indirect parent if you are still >> using a load balancer between Squids. >> >> >> >>> Le 01/11/2021 à 15:53, Alex Rousskov a écrit : On 11/1/21 7:55 AM, David Touzeau wrote: > The Squid uses the loopback as a parent. > > The same problem occurs: > 06:19:47 kid1| TCP connection to 127.0.0.1/2320 failed > 06:15:13 kid1| TCP connection to 127.0.0.1/2320 failed > 06:14:41 kid1| TCP connection to 127.0.0.1/2320 failed > 06:14:38 kid1| TCP connection to 127.0.0.1/2320 failed > 06:13:15 kid1| TCP connection to 127.0.0.1/2320 failed > 06:11:23 kid1| TCP connection to 127.0.0.1/2320 failed > cache_peer 127.0.0.1 parent 2320 0 name=Peer11 no-query default > connect-timeout=3 connect-fail-limit=5 no-tproxy It is impossible to tell for sure what is going on because Squid does not (unfortunately; yet) report the exact reason behind these connection establishment failures or even the context in which a failure has occurred. You may be able to tell more by collecting/analyzing packet captures. Developers may be able to tell more if you share, say, ALL,5 debugging logs that show what led to the failure report. Alex. > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid 5.2 Peer parent TCP connection to x.x.x.x/x failed
Ok, we will investigate on the Parent Proxy but it seems that when squid child claim about TCP failed, it understand that the peer is dead and the whole surf is stopped during several times ( a squid -k reconfigure fix the issue quickly ) because it did not have any other path to forward the request. Le 02/11/2021 à 16:17, Alex Rousskov a écrit : On 11/2/21 10:40 AM, David Touzeau wrote: 2021/11/01 16:50:48.787 kid1| 93,3| Http::Tunneler::handleReadyRead(conn9812727 local=127.0.0.1:23408 remote=127.0.0.1:2320 FIRSTUP_PARENT) 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: proto HTTP/1.1 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: status-code 503 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: reason-phrase Service Unavailable Server: squid Date: Mon, 01 Nov 2021 15:50:48 GMT X-Squid-Error: ERR_CONNECT_FAIL 110 2021/11/01 16:50:48.787 kid1| 83,3| bailOnResponseError: unsupported CONNECT response status code 2021/11/01 16:50:48.787 kid1| TCP connection to 127.0.0.1/2320 failed A parent[^1] proxy is a Squid proxy that cannot connect to the server in question. That Squid proxy responds with an HTTP 503 Error to your Squid CONNECT request. Your Squid logs the "TCP connection to ... failed" error that you were wondering about. This sequence highlights a deficiency in Squid CONNECT error handling code (and possibly cache_peer configuration abilities). Ideally, Squid should recognize Squid error responses coming from a parent HTTP proxy and avoid complaining about remote Squid-origin errors as if they are local Squid-parent errors. IIRC, some folks still insist on Squid complaining about the latter "within hierarchy" errors, but the former "external Squid-origin" errors are definitely not supposed to be reported to admins via level-0/1 messages in cache.log. HTH, Alex. [^1]: Direct or indirect parent -- I could not tell quickly but you should be able to tell by looking at addresses, configurations, and/or access logs. My bet is that it is an indirect parent if you are still using a load balancer between Squids. Le 01/11/2021 à 15:53, Alex Rousskov a écrit : On 11/1/21 7:55 AM, David Touzeau wrote: The Squid uses the loopback as a parent. The same problem occurs: 06:19:47 kid1| TCP connection to 127.0.0.1/2320 failed 06:15:13 kid1| TCP connection to 127.0.0.1/2320 failed 06:14:41 kid1| TCP connection to 127.0.0.1/2320 failed 06:14:38 kid1| TCP connection to 127.0.0.1/2320 failed 06:13:15 kid1| TCP connection to 127.0.0.1/2320 failed 06:11:23 kid1| TCP connection to 127.0.0.1/2320 failed cache_peer 127.0.0.1 parent 2320 0 name=Peer11 no-query default connect-timeout=3 connect-fail-limit=5 no-tproxy It is impossible to tell for sure what is going on because Squid does not (unfortunately; yet) report the exact reason behind these connection establishment failures or even the context in which a failure has occurred. You may be able to tell more by collecting/analyzing packet captures. Developers may be able to tell more if you share, say, ALL,5 debugging logs that show what led to the failure report. Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] ftp_port and squidclamav
On 11/2/21 12:17 PM, Andrea Venturoli wrote: > On 10/12/21 16:51, Alex Rousskov wrote: > >> Squid has a configuration option to work around such adaptation service >> deficiencies: force_request_body_continuation. Please see if enabling >> that workaround helps in your environment: >> http://www.squid-cache.org/Doc/config/force_request_body_continuation/ > > Thanks, but that didn't work: with "force_request_body_continuation > allow ftptraffic", I'm able to delete remote files, create remote > directories, but file upload fails. Thank you for this update. If you want to solve this mystery, my original triage recommendation still applies (but now with the "force_request_body_continuation allow ftptraffic" in squid.conf). Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] ftp_port and squidclamav
On 10/12/21 16:51, Alex Rousskov wrote: Squid has a configuration option to work around such adaptation service deficiencies: force_request_body_continuation. Please see if enabling that workaround helps in your environment: http://www.squid-cache.org/Doc/config/force_request_body_continuation/ Thanks, but that didn't work: with "force_request_body_continuation allow ftptraffic", I'm able to delete remote files, create remote directories, but file upload fails. I'm back to adaptation_access service_req deny ftptraffic adaptation_access service_resp deny ftptraffic which works fine. bye & Thanks av. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid 5.2 Peer parent TCP connection to x.x.x.x/x failed
On 11/2/21 10:40 AM, David Touzeau wrote: > 2021/11/01 16:50:48.787 kid1| 93,3| > Http::Tunneler::handleReadyRead(conn9812727 local=127.0.0.1:23408 > remote=127.0.0.1:2320 FIRSTUP_PARENT) > 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: proto HTTP/1.1 > 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: status-code 503 > 2021/11/01 16:50:48.787 kid1| 74,5| parse: status-line: reason-phrase Service > Unavailable > Server: squid > Date: Mon, 01 Nov 2021 15:50:48 GMT > X-Squid-Error: ERR_CONNECT_FAIL 110 > 2021/11/01 16:50:48.787 kid1| 83,3| bailOnResponseError: unsupported CONNECT > response status code > 2021/11/01 16:50:48.787 kid1| TCP connection to 127.0.0.1/2320 failed A parent[^1] proxy is a Squid proxy that cannot connect to the server in question. That Squid proxy responds with an HTTP 503 Error to your Squid CONNECT request. Your Squid logs the "TCP connection to ... failed" error that you were wondering about. This sequence highlights a deficiency in Squid CONNECT error handling code (and possibly cache_peer configuration abilities). Ideally, Squid should recognize Squid error responses coming from a parent HTTP proxy and avoid complaining about remote Squid-origin errors as if they are local Squid-parent errors. IIRC, some folks still insist on Squid complaining about the latter "within hierarchy" errors, but the former "external Squid-origin" errors are definitely not supposed to be reported to admins via level-0/1 messages in cache.log. HTH, Alex. [^1]: Direct or indirect parent -- I could not tell quickly but you should be able to tell by looking at addresses, configurations, and/or access logs. My bet is that it is an indirect parent if you are still using a load balancer between Squids. > Le 01/11/2021 à 15:53, Alex Rousskov a écrit : >> On 11/1/21 7:55 AM, David Touzeau wrote: >> >>> The Squid uses the loopback as a parent. >>> >>> The same problem occurs: >>> 06:19:47 kid1| TCP connection to 127.0.0.1/2320 failed >>> 06:15:13 kid1| TCP connection to 127.0.0.1/2320 failed >>> 06:14:41 kid1| TCP connection to 127.0.0.1/2320 failed >>> 06:14:38 kid1| TCP connection to 127.0.0.1/2320 failed >>> 06:13:15 kid1| TCP connection to 127.0.0.1/2320 failed >>> 06:11:23 kid1| TCP connection to 127.0.0.1/2320 failed >>> cache_peer 127.0.0.1 parent 2320 0 name=Peer11 no-query default >>> connect-timeout=3 connect-fail-limit=5 no-tproxy >> It is impossible to tell for sure what is going on because Squid does >> not (unfortunately; yet) report the exact reason behind these connection >> establishment failures or even the context in which a failure has >> occurred. You may be able to tell more by collecting/analyzing packet >> captures. Developers may be able to tell more if you share, say, ALL,5 >> debugging logs that show what led to the failure report. >> >> Alex. > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid 5.2 Peer parent TCP connection to x.x.x.x/x failed
Hi,
Take time to enable the debug log an parsing the 10GB of logs
Here the piece of code:
2021/11/01 16:50:48.786 kid1| 33,5| AsyncCall.cc(30) AsyncCall: The
AsyncCall Server::clientWriteDone constructed, this=0x55849cb132b0
[call252226641]
2021/11/01 16:50:48.786 kid1| 5,5| Write.cc(37) Write: conn9813869
local=10.33.50.22:3128 remote=10.33.50.109:50157 FD 95 flags=1: sz 4529:
asynCall 0x55849cb132b0*1
2021/11/01 16:50:48.786 kid1| 5,5| ModEpoll.cc(118) SetSelect: FD 95,
type=2, handler=1, client_data=0x7f1caaa1a2d0, timeout=0
2021/11/01 16:50:48.786 kid1| 20,3| store.cc(467) unlock:
store_client::copy unlocking key 115EFC0099150100
e:=sXIV/0x55849dfec190*4
2021/11/01 16:50:48.786 kid1| 20,3| store.cc(467) unlock:
ClientHttpRequest::doCallouts-sslBumpNeeded unlocking key
115EFC0099150100 e:=sXIV/0x55849dfec190*3
2021/11/01 16:50:48.786 kid1| 28,4| FilledChecklist.cc(67)
~ACLFilledChecklist: ACLFilledChecklist destroyed 0x55849316fc88
2021/11/01 16:50:48.786 kid1| 28,4| Checklist.cc(197) ~ACLChecklist:
ACLChecklist::~ACLChecklist: destroyed 0x55849316fc88
2021/11/01 16:50:48.786 kid1| 84,5| helper.cc(1319)
StatefulGetFirstAvailable: StatefulGetFirstAvailable: Running servers 4
2021/11/01 16:50:48.786 kid1| 84,5| helper.cc(1344)
StatefulGetFirstAvailable: StatefulGetFirstAvailable: returning srv-Hlpr469
2021/11/01 16:50:48.786 kid1| 5,4| AsyncCall.cc(30) AsyncCall: The
AsyncCall helperStatefulHandleRead constructed, this=0x55848ad88730
[call252226642]
2021/11/01 16:50:48.786 kid1| 5,5| Read.cc(58) comm_read_base:
comm_read, queueing read for conn9811325 local=[::] remote=[::] FD 49
flags=1; asynCall 0x55848ad88730*1
2021/11/01 16:50:48.786 kid1| 5,5| ModEpoll.cc(118) SetSelect: FD 49,
type=1, handler=1, client_data=0x7f1caaa18a20, timeout=0
2021/11/01 16:50:48.786 kid1| 5,4| AsyncCallQueue.cc(61) fireNext:
leaving helperStatefulHandleRead(conn9811325 local=[::] remote=[::] FD
49 flags=1, data=0x5584982781c8, size=300, buf=0x558498dde700)
2021/11/01 16:50:48.786 kid1| 1,5| CodeContext.cc(60) Entering:
master25501192
2021/11/01 16:50:48.786 kid1| 5,3| IoCallback.cc(112) finish: called for
conn9812727 local=127.0.0.1:23408 remote=127.0.0.1:2320 FIRSTUP_PARENT
FD 85 flags=1 (0, 0)
2021/11/01 16:50:48.786 kid1| 93,3| AsyncCall.cc(97) ScheduleCall:
IoCallback.cc(131) will call Http::Tunneler::handleReadyRead(conn9812727
local=127.0.0.1:23408 remote=127.0.0.1:2320 FIRSTUP_PARENT FD 85
flags=1, data=0x55849b747e18) [call252202273]
2021/11/01 16:50:48.786 kid1| 5,5| Write.cc(69) HandleWrite: conn9813869
local=10.33.50.22:3128 remote=10.33.50.109:50157 FD 95 flags=1: off 0,
sz 4529.
2021/11/01 16:50:48.786 kid1| 5,5| Write.cc(89) HandleWrite: write()
returns 4529
2021/11/01 16:50:48.787 kid1| 5,3| IoCallback.cc(112) finish: called for
conn9813869 local=10.33.50.22:3128 remote=10.33.50.109:50157 FD 95
flags=1 (0, 0)
2021/11/01 16:50:48.787 kid1| 33,5| AsyncCall.cc(97) ScheduleCall:
IoCallback.cc(131) will call Server::clientWriteDone(conn9813869
local=10.33.50.22:3128 remote=10.33.50.109:50157 FD 95 flags=1,
data=0x55849e4c8218) [call252226641]
2021/11/01 16:50:48.787 kid1| 1,5| CodeContext.cc(60) Entering:
master25501192
2021/11/01 16:50:48.787 kid1| 93,3| AsyncCallQueue.cc(59) fireNext:
entering Http::Tunneler::handleReadyRead(conn9812727
local=127.0.0.1:23408 remote=127.0.0.1:2320 FIRSTUP_PARENT FD 85
flags=1, data=0x55849b747e18)
2021/11/01 16:50:48.787 kid1| 93,3| AsyncCall.cc(42) make: make call
Http::Tunneler::handleReadyRead [call252202273]
2021/11/01 16:50:48.787 kid1| 93,3| AsyncJob.cc(123) callStart:
Http::Tunneler status in: [state:w FD 85 job26507207]
2021/11/01 16:50:48.787 kid1| 5,3| Read.cc(93) ReadNow: conn9812727
local=127.0.0.1:23408 remote=127.0.0.1:2320 FIRSTUP_PARENT FD 85
flags=1, size 65535, retval 7782, errno 0
2021/01 16:50:48.787 kid1| 24,5| Tokenizer.cc(27) consume: consuming 1 bytes
2021/11/01 16:50:48.787 kid1| 24,5| Tokenizer.cc(27) consume: consuming
3 bytes
2021/11/01 16:50:48.787 kid1| 24,5| Tokenizer.cc(27) consume: consuming
1 bytes
2021/11/01 16:50:48.787 kid1| 24,5| Tokenizer.cc(27) consume: consuming
19 bytes
2021/11/01 16:50:48.787 kid1| 24,5| Tokenizer.cc(27) consume: consuming
2 bytes
2021/11/01 16:50:48.787 kid1| 74,5| ResponseParser.cc(224) parse:
status-line: retval 1
2021/11/01 16:50:48.787 kid1| 74,5| ResponseParser.cc(225) parse:
status-line: proto HTTP/1.1
2021/11/01 16:50:48.787 kid1| 74,5| ResponseParser.cc(226) parse:
status-line: status-code 503
2021/11/01 16:50:48.787 kid1| 74,5| ResponseParser.cc(227) parse:
status-line: reason-phrase Service Unavailable
2021/11/01 16:50:48.787 kid1| 74,5| ResponseParser.cc(228) parse:
Parser: bytes processed=34
2021/11/01 16:50:48.787 kid1| 74,5| Parser.cc(192) grabMimeBlock: mime
header (0-171) {Server: squid^M
Mime-Version: 1.0^M
Date: Mon, 01 Nov 2021 15:50:48 GMT^M
Content-Type: text/html;charset=utf-8^M
Content-Length: 7577^M
