Re: [squid-users] assertion failed: comm.cc:178: "fd_table[conn->fd].halfClosedReader != NULL"
Better to use: # Adobe/Java and other updates acl adobe_java_updates urlpath_regex "/usr/local/squid/etc/urlregex.updates" # Youtube & CDN store rewrite ACLs acl store_rewrite_list urlpath_regex \/(watch\?|get_video|videoplayback\?) \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|webp|flv|f4f|mp4)\? \/ads\? acl store_rewrite_list_web url_regex ^https?:\/\/([A-Za-z-]+[0-9]+)*\.[A-Za-z]*\.[A-Za-z]* ^https?:\/\/(khms|mt)[0-9]+\.google\.[a-z\.]+\/.* acl store_rewrite_list_path urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|webp|flv|f4f|mp4|ttf)$ acl store_rewrite_list_web_CDN url_regex ^https?:\/\/[a-z]+[0-9]\.google\.com doubleclick\.net akamaihd\.net range_offset_limit none adobe_java_updates store_rewrite_list store_rewrite_list_web store_rewrite_list_path store_rewrite_list_web_CDN Also be sure half_closed_clients parameter is default. 27.06.15 3:29, HackXBack пишет: acl Y-TUBE dstdomain .googlevideo.com range_offset_limit -1 Y-TUBE this conf make the assertion bug, we need a solution am still waiting Amos Thanks. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/assertion-failed-comm-cc-178-fd-table-conn-fd-halfClosedReader-NULL-tp4670979p4671919.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Mikrotik and Squid Transparent
On 27/06/2015 10:02 a.m., Alex Samad wrote: > Hi > > Sorry missing something here. > > I thought this was a mikrotek rtr , presumably acting as a default > gateway for the local lan to the internet. > it has a DNAT rule to capture all internet traffic that is port 80 > (and presumably at some point in time port 443) and it DNATS it to the > SQUID box. > > and there needs to be a special rule on the DGW to allow squid access > out to the internet with out resending it back to the squid and > creating a loop. > > from memory thats how I used to do this. unless the DGW is large > enough to run squid, then DNAT to the local box and onto squid. Yes, a lot of people used to do it that way. The problem was CVE-2009-0801 vulnerability allowed attackers script to send any request to Squid claiming an arbitrary server Host: header and get that content both delivered back as if it was to some other domain the client thought it was connecting to and injected into Squid cache for other clients to be affected by in the same way. That is no longer permitted since Squid-3.2. The DNAT can only happen once, and that must be on the Squid machine so Squid can lookup the NAT tables and unmangle the original dst-IP. You need to use routing rules on the Mikrotik (or tunnel sometimes works too) to deliver the original client generated packet to the Squid machine without NAT changing the dst-IP:port details (SNAT is fine, but will cause lies about client IP in the access.log). > > Why would there be a DoS for SQUID on another box, the only resources > I can think of is the NAT table, maybe conntrack Like I said earlier "The dst-IP:port on the TCP packets entering the Squid machine is where Squid will send the outgoing server requests." If you block forwarding loops the outbound requests from Squid get an error page *always* because the outboudn traffic is going from Squid to be served by Squid (forwarding loop). If you disable the Via header forwarding loop protection Squid will just loop until all TCP port numbers on the machine are consumed sending new "outbound" connections that loop back Squid. Then no network connections will be available to Squid or any other software. The RAM associated with each connection may also be too much and cause the OS to force-shutdown Squid. So you get to pick between a DoS or a very nasty DoS. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Mikrotik and Squid Transparent
Hi Sorry missing something here. I thought this was a mikrotek rtr , presumably acting as a default gateway for the local lan to the internet. it has a DNAT rule to capture all internet traffic that is port 80 (and presumably at some point in time port 443) and it DNATS it to the SQUID box. and there needs to be a special rule on the DGW to allow squid access out to the internet with out resending it back to the squid and creating a loop. from memory thats how I used to do this. unless the DGW is large enough to run squid, then DNAT to the local box and onto squid. Why would there be a DoS for SQUID on another box, the only resources I can think of is the NAT table, maybe conntrack Alex On 26 June 2015 at 22:49, Amos Jeffries wrote: > On 27/06/2015 12:14 a.m., Alex Samad wrote: >> aren't squid and nat box different ? that was my presumption.. >> > > Best not to. > > The dst-IP:port on the TCP packets entering the Squid machine is where > Squid will send the outgoing server requests. If that dst-IP is the IP > of the Squid machine itself you get into big DoS-level trouble really fast. > > Amos > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] assertion failed: comm.cc:178: "fd_table[conn->fd].halfClosedReader != NULL"
acl Y-TUBE dstdomain .googlevideo.com range_offset_limit -1 Y-TUBE this conf make the assertion bug, we need a solution am still waiting Amos Thanks. -- View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/assertion-failed-comm-cc-178-fd-table-conn-fd-halfClosedReader-NULL-tp4670979p4671919.html Sent from the Squid - Users mailing list archive at Nabble.com. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] acl for redirect
Nevermind... I found another fix within e2guardian: etc/e2guardian/lists/urlregexplist Added this entry: # Disable Google SSL Search # allows e2g to filter searches properly "^https://www.google.[a-z]{2,6}(.*)"->"http://www.google.com/webhp?nord=1"; This means whenever google.com or www.google.com is typed in the address bar, it loads the insecure page and allows e2guardian to properly filter whatever search terms they type in. This does break other aspects such as google toolbars, using the search bar at upper right of many browsers with google as the set search engine, and other ways, but that is an issue we can live with. On 6/26/2015 5:12 AM, Amos Jeffries wrote: On 26/06/2015 8:40 p.m., FredB wrote: Mike, you can also to try the dev branch https://github.com/e2guardian/e2guardian/tree/develop SSLMITM works now. The request from the client is intercepted, a spoofed certificate supplied for the target site and an encrypted connection made back to the client. A separate encrypted connection to the target server is set up. The resulting http dencrypted stream is then filtered as normal. If that order of operations is correct then the e2guardian dev have made the same mistake we made back in Squid-3.2. client-first bumping opens a huge security vulnerability - by hiding issues on the server connection from the client it enables attackers to hijack the server connection invisibly. This is the reason the more difficult to get working server-first and peek-n-splice modes exist and are almost mandatory in Squid today. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] acl for redirect - re Amos
Amos, I would like to use e2guardian if possible, and after checking it out, http://www.google.com/webhp?nord=1 does force the insecure, but previous entries attempted just cause all searches to loop back to that same url instead of passing it along. We could use a regex option in squid, but since we want the rest of the sites to be handled normally through e2guardian, what acl entries would we use to set it up to only take effect on google.com? Essentially "if dstdomain = google.com then use acl blocklist /etc/squid/badwords". I have not used a 2 layer or referring acl setup before, but before now never needed to. Thank you so much for the help! Mike On 6/26/2015 0:29 AM, Amos Jeffries wrote: On 26/06/2015 2:36 a.m., Mike wrote: Amos, thanks for info. The primary settings being used in squid.conf: http_port 8080 # this port is what will be used for SSL Proxy on client browser http_port 8081 intercept https_port 8082 intercept ssl-bump connection-auth=off generate-host-certificates=on dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ssl/squid.pem key=/etc/squid/ssl/squid.key cipher=ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA128-SHA:AES128-SHA:RC4-SHA:HIGH:!aNULL:!MD5:!ADH sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/squid_ssl_db -M 16MB sslcrtd_children 50 startup=5 idle=1 ssl_bump server-first all ssl_bump none localhost Then e2guardian uses 10101 for the browsers, and uses 8080 for connecting to squid on the same server. Doesn;t matter. Due to TLS security requirements Squid ensures the TLS connection in re-encrypted on outgoing. I am doubtful eth nord works anymore since Googles own documentation for schools states that one must install a MITM proxy that does the traffic filtering - e2guardian is not one of those. IMO you should convert your e2guardian config into Squid ACL rules that can be applied to the bumped traffic without forcing http:// But if nord does work, so should the deny_info in Squid. Something like this probably: acl google dstdomain .google.com deny_info 301:http://%H%R?nord=1 google acl GwithQuery urlpath_regex ? deny_info 301:http://%H%R&nord=1 GwithQuery http_access deny google Gquery http_access deny google Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] acl for redirect - re Fred
Yes we already have that version installed, that is the version having these issues. [root@Server1 ~]# e2guardian -v e2guardian 3.0.4 On 6/26/2015 3:40 AM, FredB wrote: Mike, you can also to try the dev branch https://github.com/e2guardian/e2guardian/tree/develop SSLMITM works now. The request from the client is intercepted, a spoofed certificate supplied for the target site and an encrypted connection made back to the client. A separate encrypted connection to the target server is set up. The resulting http dencrypted stream is then filtered as normal. https://github.com/e2guardian/e2guardian/blob/develop/notes/ssl_mitm Fred ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Mikrotik and Squid Transparent
On 27/06/2015 12:14 a.m., Alex Samad wrote: > aren't squid and nat box different ? that was my presumption.. > Best not to. The dst-IP:port on the TCP packets entering the Squid machine is where Squid will send the outgoing server requests. If that dst-IP is the IP of the Squid machine itself you get into big DoS-level trouble really fast. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Mikrotik and Squid Transparent
aren't squid and nat box different ? that was my presumption.. On 25 June 2015 at 19:07, Amos Jeffries wrote: > On 25/06/2015 12:45 p.m., Alex Samad wrote: >> Hi >> >> why this, doesn't this block all traffic getting to the squid port. >> iptables -t mangle -A PREROUTING -p tcp --dport $SQUIDPORT -j DROP > > All external traffic yes. The NAT interception happens afterward and works. > > The point is that NAT intercept MUST only be done directly on the Squid > machine. A single external connection being accepted will result in a > forwarding loop DoS and the above protects against that. > >> >> >> what I would do to test is run tcpdump on the squid box and capture >> all traffic coming to it on the squid listening port, > > IIRC, you can't do that because tcpdump operates before NAT. It will not > show you the NAT'ed traffic arriving. > > Running Squid with -X or "debug_options ALL,9" would be better. You can > see in cache.log what Squid is receiving and what the NAT de-mangling is > actually doing. > > Amos > ___ > squid-users mailing list > squid-users@lists.squid-cache.org > http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] ufdbGuard 1.31-13 released
ufdbGuard, the fastest and free URL filter for Squid, has a new patch release. Patch 13 resolves: + new installation procedure for Solaris 10 and 11 - with much appreciated help from Yuri Voinov + various overblocking/underblocking issues with complex ACLs + redirection of URLs with HTTPS on Squid 3.4+ sometimes failed + and 8 more minor fixes See https://www.urlfilterdb.com/release-1.31.html for more information. ufdbGuard is Open Source Software and can be downloaded at http://sourceforge.net and https://www.urlfilterdb.com ufdbGuard works with URL databases in plaintext format and with the commercial database of URLfilterDB. Consider replacing squidGuard with ufdbGuard since squidGuard has no active development since 2010. Marcus ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Logging of 'indirect' requests, e.g. involving NAT or VPN
Antony Stone writes: > Imagine the following setup: > > Organisation has a bunch of servers (maybe at their office in a > server room, maybe in a data centre, doesn't matter which), some of > which have public IPs, but all of which have private IPs on an > internal subnet (for system management purposes, aside from anything > else). One of these servers is the squid proxy. Another server is > the VPN endpoint for remote client machines. Got it, makes sense, thanks. > Remote client connects to public IP of the VPN server, gets assigned a > 192.168.x.y address. Remote client is configured to use the Squid proxy > server. When it does so, its request (from 192.168.x.y) is routed from the > VPN endpoint to the Squid server (they can talk directly to each other > because > they're both on the same subnet, no NAT involved) and the Squid server then > sends the request out to the Internet to fetch a web page. > > The client IP logged by the Squid server in this scenario is 192.168.x.y Thanks, that helps a lot. > I repeat my recommendation - pick one of the 192.168.m.n addresses > you're seeing in the log files and ask whoever looks after this > network which machine has that address (or at least, what that > subnet range is used for) Will do. ht -- Henry S. Thompson, School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: h...@inf.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/ [mail from me _always_ has a .sig like this -- mail without it is forged spam] ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] acl for redirect
Thanks Amos, I will discuss this in more details with the dev of SSLMITM in E2 Fred ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] acl for redirect
On 26/06/2015 8:40 p.m., FredB wrote: > Mike, you can also to try the dev branch > https://github.com/e2guardian/e2guardian/tree/develop > SSLMITM works now. The request from the client is intercepted, a spoofed > certificate supplied for > the target site and an encrypted connection made back to the client. > A separate encrypted connection to the target server is set up. The > resulting > http dencrypted stream is then filtered as normal. If that order of operations is correct then the e2guardian dev have made the same mistake we made back in Squid-3.2. client-first bumping opens a huge security vulnerability - by hiding issues on the server connection from the client it enables attackers to hijack the server connection invisibly. This is the reason the more difficult to get working server-first and peek-n-splice modes exist and are almost mandatory in Squid today. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Reg - Squid can cache the chrome OS updates.
Thanks for your valuable information Amos. Regards, Nithi On Friday 26 June 2015 10:48 AM, Amos Jeffries wrote: On 26/06/2015 4:36 p.m., Squid List wrote: Hi, Is the Squid can cache Microsoft Updates and IOS Updates? If its cache means, please help me out for cache Chrome OS updates in latest squid version that is installed in CentOS 6.6. The short answer (FWIW): Squid can (and does) cache any HTTP content which is cacheable. With the exception of 206 responses and PUT request payloads. The long answer: Whether the cached content is used depends entirely on what the client requests. It has the power to request that cached content be ignored. Whether content is cacheable depends entirely on what the server delivers. It has the power to place limits on cache times up to and including stating an object is already stale (ie not usefully cached). There are also some mechanisms which when used MAY make content completely untrustworthy or and uncacheable: * connection based authentication (NTLM, Negotiate) * traffic interception (NAT, TPROXY, SSL-Bump) * broken Vary headers (though this causes caching when it shouldn't) * I hope that explains why you wont get a clear simple answer to your question. To help any further we will need information about; - what Squid version you are using (if its not the latest 3.5 please try an upgrade), - how its configured (squid.conf without the comment lines please), - how its being used (explicit forward-, reverse-, or interception proxy) - what exactly the request messages you are trying to make into HITs are ("debug_options 11,2" produces a traces of those), - what response messages the server is delivering on the MISS (the same 11,2 trace) - what Squid is logging for them (access.log entries) Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Logging of 'indirect' requests, e.g. involving NAT or VPN
On Friday 26 Jun 2015 at 10:42, Henry S. Thompson wrote: > Antony Stone writes: > > > > It's entirely plausible (I'd even say common) for VPN clients to get > > 192.168 addresses; also if there's a NATting router in the path > > and Squid is logging its address, that could easily be 192.168 > > Thanks for your input, but I'm still confused. My (perhaps naive) > understanding was that a VPN host or NATting router assigns local > subnet range IPs (e.g. 192.168... or 10.10...) to its clients, but > presents their traffic to the world, including any proxy, as if from > themselves, encapsulated using their own public, static, 'real' IP. > So I don't see how, for example "a NATting router['s] ... address" > could ever be 192.168... Imagine the following setup: Organisation has a bunch of servers (maybe at their office in a server room, maybe in a data centre, doesn't matter which), some of which have public IPs, but all of which have private IPs on an internal subnet (for system management purposes, aside from anything else). One of these servers is the squid proxy. Another server is the VPN endpoint for remote client machines. Remote client connects to public IP of the VPN server, gets assigned a 192.168.x.y address. Remote client is configured to use the Squid proxy server. When it does so, its request (from 192.168.x.y) is routed from the VPN endpoint to the Squid server (they can talk directly to each other because they're both on the same subnet, no NAT involved) and the Squid server then sends the request out to the Internet to fetch a web page. The client IP logged by the Squid server in this scenario is 192.168.x.y Alternatively, imagine the organisation has several office locations interconnected using MPLS or some similar private connectivity (ie: not over the Internet, or tunneled if it is over the Internet - the end result either way being that each office has a 192.168.a.0/24 subnet for its clients). One of the offices has a Squid server and a connection to the Internet; connections from clients at the other offices go over the private links to this office, via Squid, to the Internet. Again, in this setup Squid will see the true IP address of the clients, ie: 192.168.a.b because that's the only address the clients have, and with direct interconnects there's no need for NATting to a public IP along the way. I repeat my recommendation - pick one of the 192.168.m.n addresses you're seeing in the log files and ask whoever looks after this network which machine has that address (or at least, what that subnet range is used for) - I think it's going to turn out to be one of: a) a real client in something like the second scenario above b) a VPN client in the first scenario above c) an internal router in a variation of the second scenario above Regards, Antony. -- You can spend the whole of your life trying to be popular, but at the end of the day the size of the crowd at your funeral will be largely dictated by the weather. - Frank Skinner Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Logging of 'indirect' requests, e.g. involving NAT or VPN
Antony Stone writes: > On Friday 26 Jun 2015 at 09:51, Henry S. Thompson wrote: > >> > logs will show the IP address that reached squid, ie. the source >> > address of the connection. If that was NATted, squid will never know >> > (and thus is not able to log) the original address before the NAT. >> >> That's what I assumed, but in a log I've been working with for >> research purposes, 192.168 turns up -- how is this possible given >> what you say? > > It's entirely plausible (I'd even say common) for VPN clients to get > 192.168 addresses; also if there's a NATting router in the path > and Squid is logging its address, that could easily be 192.168 Thanks for your input, but I'm still confused. My (perhaps naive) understanding was that a VPN host or NATting router assigns local subnet range IPs (e.g. 192.168... or 10.10...) to its clients, but presents their traffic to the world, including any proxy, as if from themselves, encapsulated using their own public, static, 'real' IP. So I don't see how, for example "a NATting router['s] ... address" could ever be 192.168... ht -- Henry S. Thompson, School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: h...@inf.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/ [mail from me _always_ has a .sig like this -- mail without it is forged spam] ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Logging of 'indirect' requests, e.g. involving NAT or VPN
On Friday 26 Jun 2015 at 09:51, Henry S. Thompson wrote: > > logs will show the IP address that reached squid, ie. the source > > address of the connection. If that was NATted, squid will never know > > (and thus is not able to log) the original address before the NAT. > > That's what I assumed, but in a log I've been working with for > research purposes, 192.168 turns up -- how is this possible given > what you say? It's entirely plausible (I'd even say common) for VPN clients to get 192.168 addresses; also if there's a NATting router in the path and Squid is logging its address, that could easily be 192.168 I'd say your best way of working out what's happening is to pick such an address you see (frequently?) in the log files, and ask whoever's network this is what machine that address belongs to. Hope that helps, Antony. -- There are two possible outcomes: If the result confirms the hypothesis, then you've made a measurement. If the result is contrary to the hypothesis, then you've made a discovery. - Enrico Fermi Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Logging of 'indirect' requests, e.g. involving NAT or VPN
Leonardo Rodrigues writes: > Em 24/06/15 15:28, Henry S. Thompson escreveu: >> I've searched the documentation and mailing list archives w/o success, >> and am not competent to read the source, so asking here: what is >> logged as the 'remotehost' in Squid logs when a request that has been >> encapsulated, as in from a machine on a local network behind a router >> implementing NAT, or from a machine accessing the proxy via a VPN >> connection? > > logs will show the IP address that reached squid, ie. the source > address of the connection. If that was NATted, squid will never know > (and thus is not able to log) the original address before the NAT. That's what I assumed, but in a log I've been working with for research purposes, 192.168 turns up -- how is this possible given what you say? ht -- Henry S. Thompson, School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, SCOTLAND -- (44) 131 650-4440 Fax: (44) 131 650-4587, e-mail: h...@inf.ed.ac.uk URL: http://www.ltg.ed.ac.uk/~ht/ [mail from me _always_ has a .sig like this -- mail without it is forged spam] ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] acl for redirect
Mike, you can also to try the dev branch https://github.com/e2guardian/e2guardian/tree/develop SSLMITM works now. The request from the client is intercepted, a spoofed certificate supplied for the target site and an encrypted connection made back to the client. A separate encrypted connection to the target server is set up. The resulting http dencrypted stream is then filtered as normal. https://github.com/e2guardian/e2guardian/blob/develop/notes/ssl_mitm Fred ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users