Re: [CentOS] squid HA failover?

2009-02-06 Thread J Potter

> Yes, I normally want one server handling the full load to maximize the
> cache hits.  But the other one should be up and running.

So, active/standby. Easier config. Squid won't even be aware that  
heartbeat is running; just keep it running on both servers all the time.

See my install notes at bottom.


> And, because
> this is already in production as a mid-level cache working behind a
> loadbalancer, I would like to be able to keep answering on the M or N
> addresses, gradually reconfiguring the web servers in the farm to use
> address O instead of the loadbalancer VIP address.

Go for it. It'll work fine. You could get fancy and switch primary  
interface from M to O, and make M the VIP. Depends if you can accept  
the ~30 seconds of downtime and your tolerance for risk.


> I got the impression
> from some of the docs/tutorials that it was a bad idea to access the  
> M/N
> addresses directly.

In your case, it's only "bad" if M/N go down.


> Or does that only apply to services where it is
> important to only have one instance alive at a time or where you are
> replicating data?

Depends on the service and replication setup. If you had master/slave  
MySQL and connected to the slave, you'd see amnesia on the master.  
(That setup wouldn't allow for fail-back though, so, probably wouldn't  
see it.) Things like drbd protect you from concurrent mounting.


> Even after converting all of the farm to use the new
> address, I'll still want to be able to monitor the backup server to be
> sure it is still healthy.

Yup. And you'll want to monitor the active node and force-failover if  
the service fails. My config below doesn't take this into  
consideration; maybe other list lurkers can correct it to be better.  
The quick and dirty fix is to for each node to check if it is active,  
and if it is, if squid is not active, to then run 'service heartbeat  
restart' to failover to the other node. (I.e. once-a-minute cron job.)  
Not as pretty as it should be.

best,
Jeff


Replace 1.2.3.4 with your VIP ip address, and a.example.com and  
b.example.com with your FQDN hostnames.

server A ("a.example.com"):
yum -y install heartbeat
chkconfig --add heartbeat
chkconfig --level 345 heartbeat on

echo 'a.example.com IPaddr::1.2.3.4' > /etc/ha.d/haresources
echo "node a.example.com" > /etc/ha.d/ha.cf
echo "node b.example.com" >> /etc/ha.d/ha.cf
echo "udpport 9000" >> /etc/ha.d/ha.cf
echo "bcast bond0" >> /etc/ha.d/ha.cf
echo "auto_failback off" >> /etc/ha.d/ha.cf
echo "logfile /var/log/ha-log" >> /etc/ha.d/ha.cf
echo "logfacility local0" >> /etc/ha.d/ha.cf
echo "auth 1" > /etc/ha.d/authkeys
echo "1 crc" >> /etc/ha.d/authkeys
chmod go-rwx /etc/ha.d/authkeys

server B ("b.example.com"):
yum -y install heartbeat
chkconfig --add heartbeat
chkconfig --level 345 heartbeat on

echo 'a.example.com IPaddr::1.2.3.4' > /etc/ha.d/haresources # yes,  
"a" again - that's the default host to run the service
echo "node a.example.com" > /etc/ha.d/ha.cf
echo "node b.example.com" >> /etc/ha.d/ha.cf
echo "udpport 9000" >> /etc/ha.d/ha.cf
echo "bcast bond0" >> /etc/ha.d/ha.cf
echo "auto_failback off" >> /etc/ha.d/ha.cf
echo "logfile /var/log/ha-log" >> /etc/ha.d/ha.cf
echo "logfacility local0" >> /etc/ha.d/ha.cf
echo "auth 1" > /etc/ha.d/authkeys
echo "1 crc" >> /etc/ha.d/authkeys
chmod go-rwx /etc/ha.d/authkeys

# This assumes:
# 1) your network is bond0, not eth0
# 2) you are on a private network where you don't care about  
security, otherwise see http://www.linux-ha.org/authkeys
# Make sure udpport isn't in use by any other instances; or, use mcast.

On server A:
service heartbeat start
# Then, check your log files (/var/log/ha-log and /var/log/messages).
# Ping the virtual IP.

On server B:
service heartbeat start
# check your log files

On server A:
service heartbeat restart

On server B:
ifconfig -a
# Check if the interface is now runing on server B.

You can monitor current active node with arp -- the mac address will  
switch to match the physical interface that the VIP is running on.

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


Re: [CentOS] squid HA failover?

2009-02-06 Thread Les Mikesell
J Potter wrote:
> Assuming server A with IP M, server B with IP N, and DNS entry X  
> currently pointing at IP M:
> 
> 1) Add heartbeat on servers A and B, with heartbeat managing a new IP  
> address O (this is your virtual IP -- nothing to do with VRRP, that's  
> for your routers to failover your gateway).

Thanks - I know vrrp is different (and compatible with routers and other 
devices) but I thought it should work as an alternative.

> If you want active-standby on your squids, then have both squids bind  
> to 0.0.0.0 and you're done. The standby server will have squid  
> listening to requests, but since standby won't have the VIP O, it'll  
> just sit there. In this setup, heartbeat is only managing the VIP, but  
> no services.

Yes, I normally want one server handling the full load to maximize the 
cache hits.  But the other one should be up and running.  And, because 
this is already in production as a mid-level cache working behind a 
loadbalancer, I would like to be able to keep answering on the M or N 
addresses, gradually reconfiguring the web servers in the farm to use 
address O instead of the loadbalancer VIP address.  I got the impression 
from some of the docs/tutorials that it was a bad idea to access the M/N 
addresses directly.  Or does that only apply to services where it is 
important to only have one instance alive at a time or where you are 
replicating data?  Even after converting all of the farm to use the new 
address, I'll still want to be able to monitor the backup server to be 
sure it is still healthy.

-- 
   Les Mikesell
lesmike...@gmail.com

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


Re: [CentOS] squid HA failover?

2009-02-06 Thread J Potter

Assuming server A with IP M, server B with IP N, and DNS entry X  
currently pointing at IP M:

1) Add heartbeat on servers A and B, with heartbeat managing a new IP  
address O (this is your virtual IP -- nothing to do with VRRP, that's  
for your routers to failover your gateway).
2) If you want active-active load sharing on servers A and B, install  
pound on both server A and B, and in your pound config, point pound to  
IP M and IP N (same pound config on both servers),
3) Update your DNS to point entry X to IP M.

If you want active-standby on your squids, then have both squids bind  
to 0.0.0.0 and you're done. The standby server will have squid  
listening to requests, but since standby won't have the VIP O, it'll  
just sit there. In this setup, heartbeat is only managing the VIP, but  
no services.

If you want active-active on your squids, then have squid on server A  
bind to only IP M, squid on server B bind to only IP N, and pound  
configured to bind to only IP O. Heartbeat will need to be configured  
to start pound on failover (since IP O will only exist on one box at a  
time, so pound can't bind to it unless the interface is up).

Make sure you test the case where squid (or pound in active-active) on  
the server running VIP O crashes.

-Jeff
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


Re: [CentOS] squid HA failover?

2009-02-05 Thread Les Mikesell
nate wrote:
> 
>> I'm running a pair of squids as an internal cache for some intermediate
>> data used by a web server farm and currently doing failover by going
>> through an F5 bigip.  However, I'd like to avoid the bigip and use
>> heartbeat since there are only two machines.   I don't need to sync any
> 
> I use bigip even if there is only 1 machine, using heartbeat
> sounds like it will make it quite a bit more complicated,
> load balancers make life easier..since you already have a
> good one(a pair hopefully), keep using it!

We have a lot of other equipment behind it and I want the capacity for 
growth in the other pools.  I agree it does a good job and is hard to 
match where you need balancing, but this only needs failover.

> I use load balancers wherever possible even with 1 machine
> so I can make changes to the system(s) in the pool without
> impact. e.g. if I want to replace hardware, I can install the
> new hardware, add it to the pool and remove the old system.
> Of course it's ideal to always have at least 2.

A floating IP should serve the same purpose if you can swap it cleanly.

-- 
   Les Mikesell
lesmikes...@gmail.com

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


Re: [CentOS] squid HA failover?

2009-02-05 Thread nate
Les Mikesell wrote:
> I'm running a pair of squids as an internal cache for some intermediate
> data used by a web server farm and currently doing failover by going
> through an F5 bigip.  However, I'd like to avoid the bigip and use
> heartbeat since there are only two machines.   I don't need to sync any

I use bigip even if there is only 1 machine, using heartbeat
sounds like it will make it quite a bit more complicated,
load balancers make life easier..since you already have a
good one(a pair hopefully), keep using it!

I use load balancers wherever possible even with 1 machine
so I can make changes to the system(s) in the pool without
impact. e.g. if I want to replace hardware, I can install the
new hardware, add it to the pool and remove the old system.
Of course it's ideal to always have at least 2.

nate

___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos


[CentOS] squid HA failover?

2009-02-05 Thread Les Mikesell
I'm running a pair of squids as an internal cache for some intermediate 
data used by a web server farm and currently doing failover by going 
through an F5 bigip.  However, I'd like to avoid the bigip and use 
heartbeat since there are only two machines.   I don't need to sync any 
content since it is a fast-changing cache and either machine can handle 
everything.  Is it possible to add a floating IP with heartbeat and cut 
over to it slowly while also still using the existing IP's on that 
subnet?  The documentation I've seen implies that it is bad to use the 
real IP's.  If heartbeat can't do this, would it work to use vrrp?

-- 
   Les Mikesell
lesmikes...@gmail.com
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos