Re: [squid-users] reverse proxies and Host request header
On 13/06/2015 2:08 a.m., Julianne Bielski wrote: I have a general question about the use of the http Host request header by reverse proxies. As I understand it, the Host request header is used by transparent forward proxies as a way to route a request to the correct origin server since, unlike with an explicit proxy, the host is not included in the URI portion of the http method line. However, reverse proxies are always transparent from the perspective of the client and the Host header is often used by the proxy to map to the correct back end origin server. The term transparent has been overloaded so much its meaningless by itself. Beyond what you are asking and Anthony already answered well. There is the key difference of DNS involvement between reverse and interception proxy. A DNS lookup is used by the client to find and explicitly contact the reverse proxy. The proxy is able to use that as a guarantee that if the Host header does not contain a name its pre-configured for handling, that it can/must reject the request entirely. The same guarnatee for reverse proxies allows it make free use of all the origin server features of HTTP without causing security problems to the client/user. Thus the long list of problems at http://wiki.squid-cache.org/SquidFaq/InterceptionProxy#Concepts_of_Interception_Caching does not apply to reverse proxies. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] reverse proxies and Host request header
On Friday 12 June 2015 at 17:49:38 (EU time), Julianne Bielski wrote: With virtual hosting, the client is asking for a virtual origin server's host and DNS enables the ip address for the physical host to respond. The virtual host still sees its hostname in the host header, not the physical host's. There is no virtual host to see its hostname in the header. There is a single real web server (which might be a virtual machine, but that's a different use of the term virtual, and doesn't matter here one way or the other), which can happily handle requests for multiple websites (which you might want to call hostnames, domains, etc). The single web server never sees its own hostname in the header (assuming that its real hostname is not one of the virtual names it serves web pages for), and the web service is quite happy with this. With a reverse proxy, the backend origin server doesn't see its Host name, it sees the proxy's, even though the proxy is an http client with respect to the origin server. The backend server sees the hostname (if you want to call it that) of the website for which it is supposed to serve pages. An example might help: www.example.com www.example.net www.example.org downloads.example.org may all have DNS entries pointing to a single IP address. The machine at that IP address (it doesn't matter what its own hostname is, so I won't even suggest what it might be) has a web server configured to respond to requests for any of those web sites, therefore clients get the expected answers, without necessarily realising that they're all served by the same machine at a single IP address. Now put a reverse proxy in the way - the above hostnames now resolve to the IP address of the reverse proxy, it's configured to pass the requests on to the appropriate backend server/s, and if that's just a single machine, it is configured exactly the same way as the original web server was - it responds to requests for any of the virtual site names. I hope that helps; Regards, Antony. -- A user interface is like a joke. If you have to explain it, it didn't work. Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
[squid-users] reverse proxies and Host request header
I have a general question about the use of the http Host request header by reverse proxies. As I understand it, the Host request header is used by transparent forward proxies as a way to route a request to the correct origin server since, unlike with an explicit proxy, the host is not included in the URI portion of the http method line. However, reverse proxies are always transparent from the perspective of the client and the Host header is often used by the proxy to map to the correct back end origin server. I also think they usually pass the Host header as-is to the origin server. This last piece puzzles me because it means that the origin server is being given a different host name than itself in the header. Is this behavior correct? Does it ever cause problems? J. Bielski ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] reverse proxies and Host request header
With virtual hosting, the client is asking for a virtual origin server's host and DNS enables the ip address for the physical host to respond. The virtual host still sees its hostname in the host header, not the physical host's. With a reverse proxy, the backend origin server doesn't see its Host name, it sees the proxy's, even though the proxy is an http client with respect to the origin server. From: Antony Stone antony.st...@squid.open.source.it To: squid-users@lists.squid-cache.org Date: 06/12/2015 10:14 AM Subject:Re: [squid-users] reverse proxies and Host request header Sent by:squid-users squid-users-boun...@lists.squid-cache.org On Friday 12 June 2015 at 16:08:59 (EU time), Julianne Bielski wrote: reverse proxies are always transparent from the perspective of the client and the Host header is often used by the proxy to map to the correct back end origin server. I also think they usually pass the Host header as-is to the origin server. This last piece puzzles me because it means that the origin server is being given a different host name than itself in the header. Is this behavior correct? Does it ever cause problems? How is this different from a normal web server serving multiple virtual host sites? The web server doesn't care who it is, it just cares which virtual host it's being asked to serve pages for. A reverse proxy in the way basically makes no difference. Regards, Antony. -- All generalisations are inaccurate. Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] reverse proxies and Host request header
On Friday 12 June 2015 at 16:08:59 (EU time), Julianne Bielski wrote: reverse proxies are always transparent from the perspective of the client and the Host header is often used by the proxy to map to the correct back end origin server. I also think they usually pass the Host header as-is to the origin server. This last piece puzzles me because it means that the origin server is being given a different host name than itself in the header. Is this behavior correct? Does it ever cause problems? How is this different from a normal web server serving multiple virtual host sites? The web server doesn't care who it is, it just cares which virtual host it's being asked to serve pages for. A reverse proxy in the way basically makes no difference. Regards, Antony. -- All generalisations are inaccurate. Please reply to the list; please *don't* CC me. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Reverse proxies...
I still have the GET internal://pc-03/squid-internal-periodic/store_digest problem though.. What problem? It's cache digest exchanges between the Squids.. Not really a problem but pc-03 (or it's corresponding IP) are nowhere in the conf files, and so I was just wondering how the same (internal://pc-03/) did end up in the 3 squids logs... If the 3 squids were really on 3 different servers, there would be 3 different hostnames/IPs in the logs, right? But no big deal as long as it works ^_^ Thx, JD
Re: [squid-users] Reverse proxies...
On mån, 2008-06-30 at 03:08 -0700, John Doe wrote: I still have the GET internal://pc-03/squid-internal-periodic/store_digest problem though.. What problem? It's cache digest exchanges between the Squids.. Not really a problem but pc-03 (or it's corresponding IP) are nowhere in the conf files, and so I was just wondering how the same (internal://pc-03/) did end up in the 3 squids logs... Odd.. If the 3 squids were really on 3 different servers, there would be 3 different hostnames/IPs in the logs, right? IP yes. Hostname is what they request.. IP is from where the request was sent. The hostname in the URL is what that IP requested.. or how it was rebuilt using the http_port options in accelerator mode.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Reverse proxies...
Indeed, my unique server has 3 IP aliases. Apaches: 127.0.0.1:8081 127.0.0.1:8082 127.0.0.1:8083 Squids: 192.168.17.11:80 192.168.17.12:80 192.168.17.13:80 I added udp_incoming_address 192.168.17.[11|12|13] respectively in each squid confs I don't need to change udp_outgoing_address, do I? (See netstat at the end of this post) squid1: 2008/06/27 10:39:20| Accepting accelerated HTTP connections at 192.168.17.11, port 80, FD 12. 2008/06/27 10:39:20| Accepting ICP messages at 192.168.17.11, port 3130, FD 13. 2008/06/27 10:39:20| Configuring Parent 127.0.0.1/8081/0 2008/06/27 10:39:20| Configuring Sibling 192.168.17.12/80/3130 2008/06/27 10:39:20| Configuring Sibling 192.168.17.13/80/3130 squid2: 2008/06/27 10:39:23| Accepting accelerated HTTP connections at 192.168.17.12, port 80, FD 12. 2008/06/27 10:39:23| Accepting ICP messages at 192.168.17.12, port 3130, FD 13. 2008/06/27 10:39:23| Configuring Parent 127.0.0.1/8082/0 2008/06/27 10:39:23| Configuring Sibling 192.168.17.11/80/3130 2008/06/27 10:39:23| Configuring Sibling 192.168.17.13/80/3130 squid3: 2008/06/27 10:39:27| Accepting accelerated HTTP connections at 192.168.17.13, port 80, FD 12. 2008/06/27 10:39:27| Accepting ICP messages at 192.168.17.13, port 3130, FD 13. 2008/06/27 10:39:27| Configuring Parent 127.0.0.1/8083/0 2008/06/27 10:39:27| Configuring Sibling 192.168.17.11/80/3130 2008/06/27 10:39:27| Configuring Sibling 192.168.17.12/80/3130 ### Purge everything on all squids and browse squid1... ### squid1: 1214557685.718 2 192.168.17.11 TCP_MISS/200 2329 GET http://192.168.17.11/ - FIRST_UP_PARENT/127.0.0.1 text/html 1214557685.891 2 192.168.17.11 TCP_MISS/200 4378 GET http://192.168.17.11/img/apache_header.gif - FIRST_UP_PARENT/127.0.0.1 image/gif 1214557685.716 RELEASE 00 0029 32AC333767667E106447F39486BD2C7F0 -1-1-1 unknown -1/0 NONE http://192.168.17.12/squid- internal-periodic/store_digest 1214557685.716 SWAPOUT 00 1BDEF453F2C0A866DD905F6C79CF1002 200 1214557685 1214555963 1214559563 application/cache-digest 147/147 GET http://192.168.17.12/squid-internal-periodic/store_digest 1214557685.717 SWAPOUT 00 0001 8A4E6E4ABE750B9AFCBBE4277ADA20FA 200 1214557685 1214468758 1214557985 text/html 2020/2020 GET http://192.168.17.11/ 1214557685.889 SWAPOUT 00 0002 8BC2233ECEF0C976FDD190FF1A652998 200 1214557685 1214468039 1214557985 image/gif 4084/4084 GET http://192.168.17.11/ img/apache_header.gif squid2: 1214557685.187 0 192.168.17.11 UDP_MISS/000 42 ICP_QUERY http://192.168.17.11/ - NONE/- - 1214557685.716 0 192.168.17.12 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214557685.716 0 192.168.17.11 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - squid3: 1214557685.331 0 192.168.17.11 UDP_MISS/000 42 ICP_QUERY http://192.168.17.11/ - NONE/- - 1214557685.716 0 192.168.17.11 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - 1214557745.716 0 192.168.17.13 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest apache1: 127.0.0.1 - - [27/Jun/2008:11:08:05 +0200] GET / HTTP/1.0 200 2020 127.0.0.1 - - [27/Jun/2008:11:08:05 +0200] GET /img/apache_header.gif HTTP/1.0 200 4084 ### Browse squid1 again... ### squid1: 1214557857.361 0 192.168.17.11 TCP_IMS_HIT/304 254 GET http://192.168.17.11/ - NONE/- text/html 1214557857.484122 192.168.17.11 TCP_IMS_HIT/304 239 GET http://192.168.17.11/img/apache_header.gif - NONE/- image/gif 1214557857.361 RELEASE -1 F935846DBCC888FE053A2D9A6E8B529B 304 1214557685 1214468758 1214561285 text/html -1/0 GET http://192.168.17.11/ 1214557857.484 RELEASE -1 8262DA350A386275C1F0FFE25DEDFD1F 304 1214557685 1214468039 1214561285 image/gif -1/0 GET http://192.168.17.11/img/apache_header.gif ### Browse squid2... ### squid1: 1214558005.099 0 192.168.17.12 UDP_MISS/000 42 ICP_QUERY http://192.168.17.12/ - NONE/- - 1214558005.774 0 192.168.17.11 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214558005.860 0 192.168.17.12 UDP_MISS/000 63 ICP_QUERY http://192.168.17.12/img/apache_header.gif - NONE/- - squid2: 1214558005.955181 192.168.17.12 TCP_MISS/200 2329 GET http://192.168.17.12/ - TIMEOUT_FIRST_UP_PARENT/127.0.0.1
Re: [squid-users] Reverse proxies...
Also, even if I have the following: 2008/06/27 10:39:23| Local cache digest enabled; rebuild/rewrite every 600/3600 The digests don't seem to be rebuild every 10 minutes (or is it silent?). And they seem to exchange digests after the ICP_QUERY... In fact, after 1 hour, I got: squid1: 1214563796.846 RELEASE -1 B3E1F4FCF08D694B1FD5E848C7533251 200 1214560196 1214560196 1214563796 application/cache-digest 145/145 GET internal://pc-03.iper/squid-internal-periodic/store_digest 1214564100.709 RELEASE 00 E2F7000207B9A6BD38C373925811D656 200 1214560261 1214560200 1214563800 application/cache-digest 146/146 GET http://192.168.17.12/squid-internal-periodic/store_digest 1214564100.709 SWAPOUT 00 0004 1BDEF453F2C0A866DD905F6C79CF1002 200 1214564100 1214563800 1214567400 application/cache-digest 146/146 GET http://192.168.17.12/squid-internal-periodic/store_digest 1214564160.709 RELEASE 00 0003 1B1CAAE973E8AF74C183BDC81300FD3B 200 1214560321 1214560203 1214563803 application/cache-digest 146/146 GET http://192.168.17.13/squid-internal-periodic/store_digest 1214564160.709 SWAPOUT 00 0005 6BB5F118D5D548296562668556E2FF74 200 1214564160 1214563803 1214567403 application/cache-digest 146/146 GET http://192.168.17.13/squid-internal-periodic/store_digest squid2: 1214564100.709 0 192.168.17.12 TCP_MEM_HIT/200 373 GET internal://pc-03.iper/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214563800.002 RELEASE -1 072835FF91677E6ECD367A054F6744BD 200 1214560200 1214560200 1214563800 application/cache-digest 146/146 GET internal://pc-03.iper/squid-internal-periodic/store_digest squid3: 1214564160.709 0 192.168.17.13 TCP_MEM_HIT/200 373 GET internal://pc-03.iper/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214563803.117 RELEASE -1 072835FF91677E6ECD367A054F6744BD 200 1214560203 1214560203 1214563803 application/cache-digest 146/146 GET internal://pc-03.iper/squid-internal-periodic/store_digest But still same problem... JD
Re: [squid-users] Reverse proxies...
John Doe wrote: Indeed, my unique server has 3 IP aliases. Apaches: 127.0.0.1:8081 127.0.0.1:8082 127.0.0.1:8083 Squids: 192.168.17.11:80 192.168.17.12:80 192.168.17.13:80 I added udp_incoming_address 192.168.17.[11|12|13] respectively in each squid confs I don't need to change udp_outgoing_address, do I? (See netstat at the end of this post) Maybe you do. Definitely tcp_outgoing_address might be a good idea too. Amos squid1: 2008/06/27 10:39:20| Accepting accelerated HTTP connections at 192.168.17.11, port 80, FD 12. 2008/06/27 10:39:20| Accepting ICP messages at 192.168.17.11, port 3130, FD 13. 2008/06/27 10:39:20| Configuring Parent 127.0.0.1/8081/0 2008/06/27 10:39:20| Configuring Sibling 192.168.17.12/80/3130 2008/06/27 10:39:20| Configuring Sibling 192.168.17.13/80/3130 squid2: 2008/06/27 10:39:23| Accepting accelerated HTTP connections at 192.168.17.12, port 80, FD 12. 2008/06/27 10:39:23| Accepting ICP messages at 192.168.17.12, port 3130, FD 13. 2008/06/27 10:39:23| Configuring Parent 127.0.0.1/8082/0 2008/06/27 10:39:23| Configuring Sibling 192.168.17.11/80/3130 2008/06/27 10:39:23| Configuring Sibling 192.168.17.13/80/3130 squid3: 2008/06/27 10:39:27| Accepting accelerated HTTP connections at 192.168.17.13, port 80, FD 12. 2008/06/27 10:39:27| Accepting ICP messages at 192.168.17.13, port 3130, FD 13. 2008/06/27 10:39:27| Configuring Parent 127.0.0.1/8083/0 2008/06/27 10:39:27| Configuring Sibling 192.168.17.11/80/3130 2008/06/27 10:39:27| Configuring Sibling 192.168.17.12/80/3130 ### Purge everything on all squids and browse squid1... ### squid1: 1214557685.718 2 192.168.17.11 TCP_MISS/200 2329 GET http://192.168.17.11/ - FIRST_UP_PARENT/127.0.0.1 text/html 1214557685.891 2 192.168.17.11 TCP_MISS/200 4378 GET http://192.168.17.11/img/apache_header.gif - FIRST_UP_PARENT/127.0.0.1 image/gif 1214557685.716 RELEASE 00 0029 32AC333767667E106447F39486BD2C7F0 -1-1-1 unknown -1/0 NONE http://192.168.17.12/squid- internal-periodic/store_digest 1214557685.716 SWAPOUT 00 1BDEF453F2C0A866DD905F6C79CF1002 200 1214557685 1214555963 1214559563 application/cache-digest 147/147 GET http://192.168.17.12/squid-internal-periodic/store_digest 1214557685.717 SWAPOUT 00 0001 8A4E6E4ABE750B9AFCBBE4277ADA20FA 200 1214557685 1214468758 1214557985 text/html 2020/2020 GET http://192.168.17.11/ 1214557685.889 SWAPOUT 00 0002 8BC2233ECEF0C976FDD190FF1A652998 200 1214557685 1214468039 1214557985 image/gif 4084/4084 GET http://192.168.17.11/ img/apache_header.gif squid2: 1214557685.187 0 192.168.17.11 UDP_MISS/000 42 ICP_QUERY http://192.168.17.11/ - NONE/- - 1214557685.716 0 192.168.17.12 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214557685.716 0 192.168.17.11 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - squid3: 1214557685.331 0 192.168.17.11 UDP_MISS/000 42 ICP_QUERY http://192.168.17.11/ - NONE/- - 1214557685.716 0 192.168.17.11 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - 1214557745.716 0 192.168.17.13 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest apache1: 127.0.0.1 - - [27/Jun/2008:11:08:05 +0200] GET / HTTP/1.0 200 2020 127.0.0.1 - - [27/Jun/2008:11:08:05 +0200] GET /img/apache_header.gif HTTP/1.0 200 4084 ### Browse squid1 again... ### squid1: 1214557857.361 0 192.168.17.11 TCP_IMS_HIT/304 254 GET http://192.168.17.11/ - NONE/- text/html 1214557857.484122 192.168.17.11 TCP_IMS_HIT/304 239 GET http://192.168.17.11/img/apache_header.gif - NONE/- image/gif 1214557857.361 RELEASE -1 F935846DBCC888FE053A2D9A6E8B529B 304 1214557685 1214468758 1214561285 text/html -1/0 GET http://192.168.17.11/ 1214557857.484 RELEASE -1 8262DA350A386275C1F0FFE25DEDFD1F 304 1214557685 1214468039 1214561285 image/gif -1/0 GET http://192.168.17.11/img/apache_header.gif ### Browse squid2... ### squid1: 1214558005.099 0 192.168.17.12 UDP_MISS/000 42 ICP_QUERY http://192.168.17.12/ - NONE/- - 1214558005.774 0 192.168.17.11 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214558005.860 0 192.168.17.12 UDP_MISS/000 63 ICP_QUERY http://192.168.17.12/img/apache_header.gif - NONE/- - squid2: 1214558005.955
Re: [squid-users] Reverse proxies...
- Original Message Indeed, my unique server has 3 IP aliases. Apaches: 127.0.0.1:8081 127.0.0.1:8082 127.0.0.1:8083 Squids: 192.168.17.11:80 192.168.17.12:80 192.168.17.13:80 I added udp_incoming_address 192.168.17.[11|12|13] respectively in each squid confs I don't need to change udp_outgoing_address, do I? (See netstat at the end of this post) Maybe you do. Definitely tcp_outgoing_address might be a good idea too. Amos Ok, I try that. But, the doc says: If udp_outgoing_address is set to 255.255.255.255 (the default) it will use the same socket as udp_incoming_address. So I thought I could leave the default... And my netstat did show: udp0 0 192.168.17.13:3130 0.0.0.0:* udp0 0 192.168.17.12:3130 0.0.0.0:* udp0 0 192.168.17.11:3130 0.0.0.0:* udp0 0 192.168.17.11:33134 0.0.0.0:* udp0 0 192.168.17.12:33136 0.0.0.0:* udp0 0 192.168.17.13:33137 0.0.0.0:* == So I did add 3 new IP aliases for the outgoings (outgoing ICP IPs do not show up in logs but they do in netstat): 2008/06/27 15:50:25| Accepting accelerated HTTP connections at 192.168.17.11, port 80, FD 14. 2008/06/27 15:50:25| Accepting ICP messages at 192.168.17.11, port 3130, FD 15. 2008/06/27 15:50:25| Outgoing ICP messages on port 3130, FD 16. 2008/06/27 15:50:25| Configuring Parent 127.0.0.1/8081/0 2008/06/27 15:50:25| Configuring Sibling 192.168.17.12/80/3130 2008/06/27 15:50:25| Configuring Sibling 192.168.17.13/80/3130 2008/06/27 15:50:28| Accepting accelerated HTTP connections at 192.168.17.12, port 80, FD 14. 2008/06/27 15:50:28| Accepting ICP messages at 192.168.17.12, port 3130, FD 15. 2008/06/27 15:50:28| Outgoing ICP messages on port 3130, FD 16. 2008/06/27 15:50:28| Configuring Parent 127.0.0.1/8082/0 2008/06/27 15:50:28| Configuring Sibling 192.168.17.11/80/3130 2008/06/27 15:50:28| Configuring Sibling 192.168.17.13/80/3130 2008/06/27 15:50:31| Accepting accelerated HTTP connections at 192.168.17.13, port 80, FD 14. 2008/06/27 15:50:31| Accepting ICP messages at 192.168.17.13, port 3130, FD 15. 2008/06/27 15:50:31| Outgoing ICP messages on port 3130, FD 16. 2008/06/27 15:50:31| Configuring Parent 127.0.0.1/8083/0 2008/06/27 15:50:31| Configuring Sibling 192.168.17.11/80/3130 2008/06/27 15:50:31| Configuring Sibling 192.168.17.12/80/3130 udp0 0 192.168.17.16:3130 0.0.0.0:* udp0 0 192.168.17.13:3130 0.0.0.0:* udp0 0 192.168.17.15:3130 0.0.0.0:* udp0 0 192.168.17.12:3130 0.0.0.0:* udp0 0 192.168.17.14:3130 0.0.0.0:* udp0 0 192.168.17.11:3130 0.0.0.0:* udp0 0 192.168.17.14:33213 0.0.0.0:* udp0 0 192.168.17.15:33214 0.0.0.0:* udp0 0 192.168.17.16:33215 0.0.0.0:* == Same test (fill the 3 squids then delete 1 object from squid1 and browse squid1) and same result (squid1 asks siblings but end up asking apache1): squid1: 1214575092.895 0 192.168.17.11 TCP_MEM_HIT/200 2338 GET http://192.168.17.11/ - NONE/- text/html 1214575093.043148 192.168.17.11 TCP_MISS/200 4378 GET http://192.168.17.11/img/apache_header.gif - FIRST_UP_PARENT/127.0.0.1 image/gif 1214575093.042 SWAPOUT 00 0004 8BC2233ECEF0C976FDD190FF1A652998 200 1214575093 1214468039 1214575393 image/gif 4084/4084 GET http://192.168.17.11/img/apache_header.gif squid2: 1214575093.033 0 192.168.17.14 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - 1214575099.628 0 192.168.17.12 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214575094.267 RELEASE 00 0004 F560EF12A0FB4963759C32554341CB170 -1-1-1 unknown -1/0 NONE http://192.168.17.13/squid-internal-periodic/store_digest 1214575094.267 SWAPOUT 00 0004 6BB5F118D5D548296562668556E2FF74 200 1214575094 1214574631 1214578231 application/cache-digest 147/147 GET http://192.168.17.13/squid-internal-periodic/store_digest squid3: 1214575092.501 0 192.168.17.14 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - 1214575094.267 0 192.168.17.13 TCP_MEM_HIT/200 374 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest 1214575099.628 RELEASE 00 0004 D03BA4AEAA228DAF8EA08C9197F055220 -1-1-1 unknown -1/0 NONE http://192.168.17.12/squid-internal-periodic/store_digest 1214575099.628 SWAPOUT 00 0004 1BDEF453F2C0A866DD905F6C79CF1002 200 1214575099 1214574628 1214578228 application/cache-digest 147/147 GET
Re: [squid-users] Reverse proxies...
My bad... I did not realize that the ICP_QUERY URL should have been the apache IPs and not the squids IPs... I was accessing directly my squids instead of using them as transparent proxies. Once setup as proxies in my browser, I get: ICP_QUERY http://127.0.0.1/img/apache_header.gif which will match for all squids. I still have the GET internal://pc-03/squid-internal-periodic/store_digest problem though.. JD
Re: [squid-users] Reverse proxies...
On lör, 2008-06-28 at 01:02 +1200, Amos Jeffries wrote: Maybe you do. Definitely tcp_outgoing_address might be a good idea too. It helps sanity. But only udp_incoming_address, not udp_outgoing_address. See the description of udp_outgoing_address for why.. ### Browse squid2... ### squid1: 1214558005.099 0 192.168.17.12 UDP_MISS/000 42 ICP_QUERY http://192.168.17.12/ - NONE/- - I would expect an UDP_HIT here. The object is in squid1.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Reverse proxies...
On fre, 2008-06-27 at 03:25 -0700, John Doe wrote: squid1: 1214557685.718 2 192.168.17.11 TCP_MISS/200 2329 GET http://192.168.17.11/ - FIRST_UP_PARENT/127.0.0.1 text/html ### Browse squid2... ### squid1: 1214558005.099 0 192.168.17.12 UDP_MISS/000 42 ICP_QUERY http://192.168.17.12/ - NONE/- - Now I see the error. The URL you use is server specific, and it fails because of this reason alone. You need to make a more real test, sending the same request, with the same Host header. I.e. if you use a browser you need to use the exact same URL in the url-bar, and direct it to the right server by other means. Alternatively, if you only host a single site, you can configure Squid with the defaultsite= http_port option telling your Squids what the actual site name is (not the per-server IP, that should NEVER be specified here...). Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Reverse proxies...
On fre, 2008-06-27 at 08:06 -0700, John Doe wrote: I still have the GET internal://pc-03/squid-internal-periodic/store_digest problem though.. What problem? It's cache digest exchanges between the Squids.. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Reverse proxies...
Thanks again for your answers. On my test server, I did setup 3 apaches on 127.0.0.1:[81|82|83] and 3 squids on 192.168.16.[11|12|13] Sorry, the post is a bit long... == Delete browser cache and start 3 squids with brand new cache... == == /var/log/squid/cache.log == 2008/06/26 11:10:54| Accepting accelerated HTTP connections at 192.168.17.11, port 80, FD 11. 2008/06/26 11:10:54| Accepting ICP messages at 0.0.0.0, port 3130, FD 12. 2008/06/26 11:10:54| WCCP Disabled. 2008/06/26 11:10:54| Configuring Parent 127.0.0.1/8081/0 2008/06/26 11:10:54| Configuring Sibling 192.168.17.12/80/3130 2008/06/26 11:10:54| Configuring Sibling 192.168.17.13/80/3130 2008/06/26 11:10:54| Ready to serve requests. == /var/log/squid2/cache.log == 2008/06/26 11:10:56| Accepting accelerated HTTP connections at 192.168.17.12, port 80, FD 11. 2008/06/26 11:10:56| Accepting ICP messages at 0.0.0.0, port 3130, FD 12. 2008/06/26 11:10:56| WCCP Disabled. 2008/06/26 11:10:56| Configuring Parent 127.0.0.1/8082/0 2008/06/26 11:10:56| Configuring Sibling 192.168.17.11/80/3130 2008/06/26 11:10:56| Configuring Sibling 192.168.17.13/80/3130 2008/06/26 11:10:56| Ready to serve requests. == /var/log/squid3/cache.log == 2008/06/26 11:10:57| Accepting accelerated HTTP connections at 192.168.17.13, port 80, FD 11. 2008/06/26 11:10:57| Accepting ICP messages at 0.0.0.0, port 3130, FD 12. 2008/06/26 11:10:57| WCCP Disabled. 2008/06/26 11:10:57| Configuring Parent 127.0.0.1/8083/0 2008/06/26 11:10:57| Configuring Sibling 192.168.17.11/80/3130 2008/06/26 11:10:57| Configuring Sibling 192.168.17.12/80/3130 2008/06/26 11:10:57| Ready to serve requests. == Access squid1... == == /var/log/squid1/access.log == 1214491088.291 1001 192.168.17.11 TCP_MISS/200 2329 GET http://192.168.17.11/ - TIMEOUT_FIRST_UP_PARENT/127.0.0.1 text/html 1214491089.292 1000 192.168.17.11 TCP_MISS/200 4378 GET http://192.168.17.11/img/apache_header.gif - TIMEOUT_FIRST_UP_PARENT/127.0.0.1 image/gif == /var/log/httpd/apache-8081.access_log == 127.0.0.1 - - [26/Jun/2008:16:38:08 +0200] GET / HTTP/1.0 200 2020 127.0.0.1 - - [26/Jun/2008:16:38:09 +0200] GET /img/apache_header.gif HTTP/1.0 200 4084 == /var/log/squid2/access.log == 1214491087.293 0 192.168.17.12 TCP_MEM_HIT/200 373 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest == /var/log/squid3/access.log == 1214491086.740 0 192.168.17.12 UDP_MISS/000 42 ICP_QUERY http://192.168.17.11/ - NONE/- - 1214491086.740 0 192.168.17.13 UDP_MISS/000 42 ICP_QUERY http://192.168.17.11/ - NONE/- - 1214491087.740 0 192.168.17.12 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - 1214491087.740 0 192.168.17.13 UDP_MISS/000 63 ICP_QUERY http://192.168.17.11/img/apache_header.gif - NONE/- - 1214491147.291 0 192.168.17.13 TCP_MEM_HIT/200 373 GET internal://pc-03/squid-internal-periodic/store_digest - NONE/- application/cache-digest == Access squid1 again... == == /var/log/squid1/access.log == 1214491262.555 0 192.168.17.11 TCP_REFRESH_HIT/304 189 GET http://192.168.17.11/ - FIRST_UP_PARENT/127.0.0.1 - 1214491262.936380 192.168.17.11 TCP_REFRESH_HIT/304 189 GET http://192.168.17.11/img/apache_header.gif - FIRST_UP_PARENT/127.0.0.1 - == /var/log/httpd/apache-8081.access_log == 127.0.0.1 - - [26/Jun/2008:16:41:02 +0200] GET / HTTP/1.0 304 - 127.0.0.1 - - [26/Jun/2008:16:41:02 +0200] GET /img/apache_header.gif HTTP/1.0 304 - == Access squid1 several times == == /var/log/squid1/access.log == 1214491330.658 0 192.168.17.11 TCP_IMS_HIT/304 253 GET http://192.168.17.11/ - NONE/- text/html 1214491331.150492 192.168.17.11 TCP_IMS_HIT/304 238 GET http://192.168.17.11/img/apache_header.gif - NONE/- image/gif 1214491393.377749 192.168.17.11 TCP_IMS_HIT/304 244 GET http://192.168.17.11/ - NONE/- text/html 1214491393.556 17 192.168.17.11 TCP_IMS_HIT/304 237 GET http://192.168.17.11/img/apache_header.gif - NONE/- image/gif 1214491409.228683 192.168.17.11 TCP_IMS_HIT/304 253 GET http://192.168.17.11/ - NONE/- text/html 1214491409.545317 192.168.17.11 TCP_IMS_HIT/304 238 GET http://192.168.17.11/img/apache_header.gif - NONE/- image/gif == Access squid1 again with shift-reload == ==
Re: [squid-users] Reverse proxies...
On tor, 2008-06-26 at 09:05 -0700, John Doe wrote: Thanks again for your answers. On my test server, I did setup 3 apaches on 127.0.0.1:[81|82|83] and 3 squids on 192.168.16.[11|12|13] On the same server, or different servers? I ask because from your logs it looks like you have not assigned an address for ICP, which means the ICP port listens on all addresses at once... 2008/06/26 11:10:54| Accepting ICP messages at 0.0.0.0, port 3130, FD 12. See the udp_incoming_port directive for how to control the address ICP listens on. Regards Henrik signature.asc Description: This is a digitally signed message part
Re: [squid-users] Reverse proxies...
Hi again, I must admit I am still a bit uncomfortable with access rules and other things in reverse proxy... Here's my conf again: +--- Squid1 --- Apache1 | | Internet ---+--- Squid2 --- Apache3 | . | . +--- SquidN --- ApacheN Each squid is a reverse proxy for one apache listening on an external IP. Squid (port 80) and apache (port 8080) are both on the same server for now. All squids are proxy-only siblings on an internal IP. Basicaly, I want the following: client ask squid. If it is a miss, squid ask siblings in proxy-only (in order not to cache the same objects N times through the N squids). Here are the confusing parts of the confs: Common to all confs: acl from_localnet src 192.168.0.0/16# Internal IPs icp_access allow from_localnet icp_access deny all #miss_access deny from_localnet # Do I need this miss? acl to_examplecom dstdomain .example.com http_access allow to_examplecom squid1.conf: http_port EXTIP1:80 accel defaultsite=www.example.com vhost #https_port EXTIP1:3129 defaultsite=www.example.com cert=squid.pem key=squid.key cache_peer EXTIP1 parent 8080 0 no-query originserver max-conn=64 name=apache1 front-end-https=auto cache_peer INTIP2 sibling 80 3130 proxy-only name=squid2 ... cache_peer INTIPN sibling 80 3130 proxy-only name=squidN cache_peer_access apache1 allow to_examplecom cache_peer_access apache1 deny all squid2.conf: http_port EXTIP2:80 accel defaultsite=www.example.com vhost #https_port EXTIP2:3129 defaultsite=www.example.com cert=squid.pem key=squid.key cache_peer EXTIP2 parent 8080 0 no-query originserver max-conn=64 name=apache2 front-end-https=auto cache_peer INTIP1 sibling 80 3130 proxy-only name=squid1 ... cache_peer INTIPN sibling 80 3130 proxy-only name=squidN cache_peer_access apache2 allow to_examplecom cache_peer_access apache2 deny all squidN.conf: http_port EXTIP3:80 accel defaultsite=www.example.com vhost #https_port EXTIPN:3129 defaultsite=www.example.com cert=squid.pem key=squid.key cache_peer EXTIPN parent 8080 0 no-query originserver max-conn=64 name=apacheN front-end-https=auto cache_peer INTIP1 sibling 80 3130 proxy-only name=squid1 cache_peer INTIP2 sibling 80 3130 proxy-only name=squid2 ... cache_peer INTIP[N-1] sibling 80 3130 proxy-only name=squid[N-1] cache_peer_access apacheN allow to_examplecom cache_peer_access apacheN deny all Questions: 1. Anything wrong in the confs above? Do I need some cache_peer_access for the siblings communication? 3. If squids sits right next to the apaches (on a gigabit switch), can I not use digests? 4. If I want to handle leeches (using wrong referer) with a redirector script, would the following work? acl examplecom_referer referer_regex example.com url_rewrite_program handleleech.php url_rewrite_children 5 url_rewrite_concurrency 0 url_rewrite_host_header off # Do I need this for a reverse proxy? url_rewrite_access allow !examplecom_referer url_rewrite_access deny all The redirector, based on some internal rules, will either send back a 302 or the document URL... 5. If I want the apaches to tell squid to cache for 1 week and squids to tell clients to cache for 1 month, can I just use header_replace like these? header_replace Expires Sun Jan 1 00:00:00 CET 2012 header_replace Cache-Control max-age=2678400 Or is it refresh_pattern with override-expire and override-lastmod? 6. Is there a way to prevent shift-reload from making squid ask apache when the object is still in its cache...? Something like that? refresh_pattern 0 0% 0 ignore-reload Thank you, JD
Re: [squid-users] Reverse proxies...
tis 2008-06-24 klockan 11:30 -0700 skrev John Doe: #miss_access deny from_localnet # Do I need this miss? No, sibling relations automatically enforce this by using a Cache-Control header sent by the requesting Squid. Questions: 1. Anything wrong in the confs above? Do I need some cache_peer_access for the siblings communication? Default for cache_peer_access is to consider the peer a candidate for all requests. 3. If squids sits right next to the apaches (on a gigabit switch), can I not use digests? cache digests is between the squids, and may be quite beneficial to you as ICP can then be skipped on known cache hits. 4. If I want to handle leeches (using wrong referer) with a redirector script, would the following work? Probably, but you may want to consider implementing this using an external acl instead of a redirector.. 5. If I want the apaches to tell squid to cache for 1 week and squids to tell clients to cache for 1 month, can I just useheader_replace like these? header_replace Expires Sun Jan 1 00:00:00 CET 2012 header_replace Cache-Control max-age=2678400 Better to use a combination of max-age and s-maxage. What they are intended for.. 6. Is there a way to prevent shift-reload from making squid ask apache when the object is still in its cache...? Something like that? refresh_pattern 0 0% 0 ignore-reload Yes. Regards Henrik
[squid-users] Reverse proxies...
Hi everybody, I need to setup something like this: +--- Squid1 --- Apache1 | | | proxyonly | | Internet ---+--- Squid2 --- Apache2 | | | proxyonly | | +--- Squid3 --- Apache3 All squids are siblings and each one only cache one server. Squids (port 80) and apaches (port 8080) are on the same server for now. Would these confs work? squid1.conf: http_port IP1:80 accel defaultsite=www.example.com vhost cache_peer IP1 parent 8080 0 no-query originserver name=ap ache front-end-https=auto cache_peer IP1 sibling 80 3130 proxy-only name=squid1 cache_peer IP2 sibling 80 3130 proxy-only name=squid2 cache_peer IP3 sibling 80 3130 proxy-only name=squid3 squid2.conf: http_port IP2:80 accel defaultsite=www.example.com vhost cache_peer IP2 parent 8080 0 no-query originserver name=apache front-end-https=auto cache_peer IP1 sibling 80 3130 proxy-only name=squid1 cache_peer IP2 sibling 80 3130 proxy-only name=squid2 cache_peer IP3 sibling 80 3130 proxy-only name=squid3 squid3.conf: http_port IP3:80 accel defaultsite=www.example.com vhost cache_peer IP3 parent 8080 0 no-query originserver name=apache front-end-https=auto cache_peer IP1 sibling 80 3130 proxy-only name=squid1 cache_peer IP2 sibling 80 3130 proxy-only name=squid2 cache_peer IP3 sibling 80 3130 proxy-only name=squid3 Do I have to remove the self sibling line? Do I miss any classic cache_peer options for a regular website (with https login page)? Thank you, JD
Re: [squid-users] Reverse proxies...
On ons, 2008-06-18 at 06:55 -0700, John Doe wrote: Do I have to remove the self sibling line? Preferably yes. It will probably still surive with it, but there will quite likely be some forwarding loops detected and additionally unneeded load on your servers. Do I miss any classic cache_peer options for a regular website (with https login page)? login=PASS perhaps? Needed if you are using proper HTTP authentication, but not needed if you are using forms based authentication + cookies.. Regards Henrik signature.asc Description: This is a digitally signed message part