On Mon, 28 Sep 2009 15:04:35 -0700 (PDT), Chris Hostetter <hossman_sq...@fucit.org> wrote: > Background Information... > > My company currently runs several "clusters" of application servers behind > load balancers, which are each in turn sitting behind a "cluster" of squid > machines configured as accelerators. each squid cluster is then sitting > behind a load balancer that is hit by our clients. > > To elaborate: The hostname appAA resolves to a load balancer which proxies > to appAA-squid1, appAA-squid2, appAA-squid2, etc... Each of the > appAA-squidX machines is configured as a standalone accelerator (using > cache_peer ... parent no-query originserver) for appAA-backend. > appAA-backend resolves to a load balancer which proxies to appAA-backend1, > appAA-backend2, appAA-backend3, etc... Likewise for appBB, appCC, appDD,
> etc... > > None of these squid instances know anything about each other. in the case > of appAA-squidX vs appBB-squidX this is a good thing, because the entire > point of isolating these apps is for QoS commitments, and ensuring that > heavy load or catastrophic failure on one app doesn't affect another app. > > In the case of appAA-squidX vs appAA-squidY it definitely seems like cache > peering would be advantageous here. > > > The Problem(s)... > > Our operations team is pretty adamant about software/configs deployed to > boxes in a clustering needing to be the same for every box in the cluster. > The goal is understandable: they don't want to need custom install steps > for every individual machine. So while my dev setup of a 5 machine squid > cluster each with 4 distinct "cache_peer ... sibling" lines works great so > far, i can't deploy a unique squid.conf for each machine in a cluster. > > I could probably put a hack into our build system to check the current > hostname at installation time and remove any cache_peer lines refering to > that hostname -- but before i jumped through those hoops i wanted to > sanity check that there wasn't an easier way to do this in squid. > > is there any easy way to reuse the same cache_peer config options on > multiple instances, but keep squid smart enough that it doesn't bother > trying to peer with itself? > > (I had a glimmer of an idea about using ACL rules for this, but didn't > work it through all the way because it seemed like at best that would > cause squid to deny requests from itself, not prevent it from attempting > the request in the first place) > > I have a hard time imaging that i'm the first person to have this problem, > but i couldn't find any obvious solutions in the mail archives. > > > A slightly bigger problem is what to do when the cluster changes, either > because a machine is removed for maintenance issues or because a machine > is added due to because of increases in load. In our current setup this > is a no-brainer: tell the load balancer when you add/remove a machine and > everything just works -- none of the boxes know anything about each other, > and they all run identical configs. > > In order to setup sibling peering, it seems like i would need to > deploy/reload configs (with an updated list of cache_peer directives) to > every machine in the cluster anytime a new box is added or removed in > order for all of the siblings to know about each other. > > Is there an easier way to coordinate the "discovery" of new siblings > automatically? > > An ideal situation would be to use a DNS hostname in the cache_peer line > that resolves to multiple IPs and have squid re-resolve the hostname > periodically and update the list of peers based on *all* the addresses > associated with that name -- but from what i can tell squid will just > picking a single address (DNS Round-Robin style). > > > Any advice or suggestions for managing peers like this would be > appreciated. The DNS way would indeed be nice. It's not possible in current Squid however, if anyone is able to sponsor some work it might be doable. With Squid-2.7 you can use the 'include' directive to split the squid.conf apart and contain the unique per-machine parts in a separate file to the shared parts. Amos
