Re: [squid-users] Controlling Cache Peer

[Amos Jeffries]

On 14/11/2016 2:58 p.m., creditu wrote:
> I'm having trouble understanding how to configure an accelerator to
> handle multiple IPs and backend servers.  In the past we used virtual
> IPs and a redirector script to  send the requests to a given backend. 
> Now we need to change to cache peer statements. 

What you need is cache_peer_access as documented at
 and
.


> 
> Given the following:
> 
> Squid listens on:
> 10.10.10.1 - www.example.com
> 10.10.10.2 - dev.example.com
> 
> For .1, there are 3 backend origin servers.
> For .2 there is only 1 backend origin servers.
> 
> The following config (right now we need to handle both http and https):
> https_port 10.10.10.1:443 accel defaultsite=www.example.com
> cert=/etc/squid/www.crt key=/etc/squid/www.key
> http_port 10.10.10.1:80 accel defaultsite=www.example.com
> 
> # For www.example.com
> cache_peer 192.168.1.2 parent 80 0 no-query originserver round-robin
> cache_peer 192.168.1.3 parent 80 0 no-query originserver round-robin
> cache_peer 192.168.1.4 parent 80 0 no-query originserver round-robin
> 
> This seems to work fine for 10.10.10.1 (www.example.com), but I'm stuck
> on how to handle 10.10.10.2 (dev.example.com)and tell it to send
> requests coming in to a different cach_peer (cache_peer 192.168.0.1
> parent 80 0 no-query originserver)?

Use cache_peer_access to only permit the www.example.com dstdomain.

Like so:
 acl site1 dstdomain www.example.com

 cache_peer_access 192.168.1.2 allow site1
 cache_peer_access 192.168.1.2 deny all

 cache_peer_access 192.168.1.3 allow site1
 cache_peer_access 192.168.1.3 deny all

 cache_peer_access 192.168.1.4 allow site1
 cache_peer_access 192.168.1.4 deny all


> 
> Just guessing, but can I do something like this along with the above:
> https_port 10.10.10.2:443 accel defaultsite=dev.example.com
> cert=/etc/squid/www.crt key=/etc/squid/www.key
> http_port 10.10.10.2:80 accel defaultsite=dev.example.com
> 
> cache_peer 192.168.0.1 parent 80 0 no-query originserver
> 

Follow that with cache_peer_access like above, but allowing access only
to the dev.example.com domain.


> If so, I'm unsure how to do the ACLs to direct the traffic to the
> correct backend servers.  Especially since for www.example.com I can not
> use the same name= statement for all three backends to construct the
> ACLs.

name= is just a label for the cache_peer link. It does not by itself do
anything like permissions. The default name= for any peer link is the
text you put in as IP/hostname Squid is to contact.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Controlling Cache Peer

[creditu]

I'm having trouble understanding how to configure an accelerator to
handle multiple IPs and backend servers.  In the past we used virtual
IPs and a redirector script to  send the requests to a given backend. 
Now we need to change to cache peer statements. 

Given the following:

Squid listens on:
10.10.10.1 - www.example.com
10.10.10.2 - dev.example.com

For .1, there are 3 backend origin servers.
For .2 there is only 1 backend origin servers.

The following config (right now we need to handle both http and https):
https_port 10.10.10.1:443 accel defaultsite=www.example.com
cert=/etc/squid/www.crt key=/etc/squid/www.key
http_port 10.10.10.1:80 accel defaultsite=www.example.com

# For www.example.com
cache_peer 192.168.1.2 parent 80 0 no-query originserver round-robin
cache_peer 192.168.1.3 parent 80 0 no-query originserver round-robin
cache_peer 192.168.1.4 parent 80 0 no-query originserver round-robin

This seems to work fine for 10.10.10.1 (www.example.com), but I'm stuck
on how to handle 10.10.10.2 (dev.example.com)and tell it to send
requests coming in to a different cach_peer (cache_peer 192.168.0.1
parent 80 0 no-query originserver)?

Just guessing, but can I do something like this along with the above:
https_port 10.10.10.2:443 accel defaultsite=dev.example.com
cert=/etc/squid/www.crt key=/etc/squid/www.key
http_port 10.10.10.2:80 accel defaultsite=dev.example.com

cache_peer 192.168.0.1 parent 80 0 no-query originserver

If so, I'm unsure how to do the ACLs to direct the traffic to the
correct backend servers.  Especially since for www.example.com I can not
use the same name= statement for all three backends to construct the
ACLs.  

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] [SOLVED] Re: TCP Outgoing Address ACL Problem

[Garri Djavadyan]

On 2016-11-13 23:09, jarrett+squid-us...@jarrettgraham.com wrote:

My problem is solved.


The solution may be useful for other users also. Please, post the 
solution, if possible. Thanks!


Garri
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] [SOLVED] Re: TCP Outgoing Address ACL Problem

[jarrett+squid-users]

Thanks Garry and Amos!  My problem is solved.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Fwd: Using Squid to redirect Steam CDNs using storeID_rewrite

[Amos Jeffries]

On 13/11/2016 12:50 p.m., Stefan Wickstrom wrote:
> Hello all,
> Apologies for the possibly incorrect format/posting of this query; I am new
> to this mode of discussion in relation to software.
> 
> I am attempting to use Squid, in combination with storeID rewrite, to
> redirect Steam CDN requests allowing multiple CDN requests to be served
> from the single Squid cache entry.

Well, firstly. Store-ID does not _redirect_ anything. It simply
de-duplicates cache objects by re-writing the location where certain
ones are stored, so it is different to where their URL would store it.

If a backend server needs to be contacted the one the clients was going
to will be contacted and the shared Store-ID location gets updated with
any new data that server produces.

> 
> /var/ipfire/proxy/advanced/acls/include.acl
> #squid.conf
> acl cacheDomain dstdomain .steampowered.com .edgesuite.net .steamstatic.com
> .steamcontent.com
> cache deny !cacheDomain
> store_id_program /usr/lib/squid/storeid_file_rewrite
> /etc/squid/storeid_rewrite.conf
> store_id_children 10 startup=3 idle=1 concurrency=0
> 
> /etc/squid/storeid_rewrite.conf
> ^http.*steam.*\.com\/(.*) http://steamupdates.squid.internal/$1
> 

I highly recommend that you make that pattern a LOT more targeted. The
presence of ".*" allows any URL that happens to include the word "steam"
and then ".com/" to have its final portion stored in your cache as "game
content".


> The issue appears to be stemming from how squid and storeID_rewrite
> interact; currently if I test the storeid_rewrite.conf with the following
> command:
> echo http://valve314.steamcontent.com/depot/255711/chunk/
> a3f17a1be9c7861cbc56d1098b8ede146e114391? | 
> /usr/lib/squid/storeid_file_rewrite
> /etc/squid/storeid_rewrite.conf
> I get in return:
> OK store-id=http://steamupdates.squid.internal/depot/255711/chunk/
> a3f17a1be9c7861cbc56d1098b8ede146e114391?
> 

NOTE: when your log contains URLs ending in '?' check that you have
turned off the strip_query_terms mechanism before debugging. Otherwise
that significant part of the URLs will be hidden from you, and your test
results may be different from expectations.



> This indicates that storeID rewrite is functioning and using my RegEx
> command to rewrite the URL into one that only contains the unique game and
> chunk IDs from the URL.

No, the new key contains anything that happened to follow the string
".com/".

eg.
http://haha.example.net/steam/gotcha.com/depot/255711/chunk/a3f17a1be9c7861cbc56d1098b8ede146e114391?boo


> The issue is when I test the entire system using
> the following process I see multiple entries into the squid cache for the
> specific game chunk ID:
> 
>- Remove /var/log/squid/access.log to ensure no previous attempts will
>appear in test
>- Clear the cache through ipFire webUI
>- Restart Squid cache service through ipFire webUI
>- Download game through Steam interface
>- Verify Squid cached the download chunks by grepping through both
>/var/log/squid/access.log and Squid cache for specific game chunk IDs (this
>is a spot check at best)
>- Change Steam CDN location through Steam UI
>- Delete Steam game local content
>- Re-download Steam game
>- Verify Squid cache using game chunk ID
> 
> The command used to grep against the Squid cache and it's results are as
> follows:
> squidclient -h 192.168.1.1 cache_object://192.168.1.1 mgr:objects | grep
> f37f9405e2f38417a226eac378ac3982223d2966?
> GET http://valve608.steamcontent.com/depot/26502/chunk/
> f37f9405e2f38417a226eac378ac3982223d2966?
> GET http://valve313.steamcontent.com/depot/26502/chunk/
> f37f9405e2f38417a226eac378ac3982223d2966?
> 

See the note above about strip_query_terms.

If a single byte of any of the query portion of those URLs is different
then your Store-ID keys for them will be likewise different. Since
*everything* following the ".com/" is part of the Store-ID key produced
by your pattern.


> This indicates that at some point in the process, Squid is generating a KEY
> entry for the chunk based off the original Steam CDN URL and NOT the
> RegEx'd URL storeID_rewrite is supposedly generating.

The mgr:objects report lists the client request that object was a
response to. 

So the test above just indicates that objects exist which were stored
from those URLs. Client requests are of course not for the Store-ID
keys, but (one of) the actual URLs for that object.


> I have attempted to determine how Squid is generating it's KEY entries for
> the chunks it is storing, but have had no luck (basing attempts off this white
> paper entry )

That is about using digests used to inform other proxies about what is
possibly still in cache. Nothing to do with the storage itself. And
should not contain Store-ID keys (if it does that is a bug).