Re: [lopsa-tech] DHCP for Servers

[Tracy Reed]

On Tue, Sep 07, 2010 at 02:07:50PM -0400, Patrick Cable spake thusly:
> I am of the mindset that as long as the server gets the same address
> every time, I don't care how it gets configured, and if I'm going to

I worry about the DHCP server being down (due to power outage or
whatever causing everything to power cycle) and the servers not coming
up. Now you have a timing dependency. I also worry about the DHCP
server going down or dhcpd process goes away unnoticed (yes,
monitoring *should* catch this) and servers not being able to renew
their lease and dying unexpectedly perhaps many days after the dhcp
server went down.

If I had a large compute cluster or "cloud" etc. where all of the
machines were intended to be identical yet generic resources I would
probably dhcp them just because it seems pointless to manage IPs for
them all. Just have to keep the above issues in mind.

-- 
Tracy Reed
http://tracyreed.org


pgpARIYKaJ1h5.pgp
Description: PGP signature
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Matthew Barr]

On Tue, 7 Sep 2010, Atom Powers wrote:

> How do you make DNS highly available (or load balance it)?
> Most clients use the first server on the list and if it doesn't answer
> then it tries the next etc. But the time-out makes some operating
> systems (Windows) very sad.

First, by not using windows :)
Second, we experimented with the timeouts, and did the best we could. 
Certain services recovered much faster, IIRC, like SSH.  That was the 
biggest issue we had when we did have a power problem, and had to bring 
everything up from scratch. No DNS meant SSH was taking minutes to 
connect.

It's not perfect.  I've since moved on to another job, but we had planned 
to try using the load balancers to load balance DNS internally.  Yes, that 
adds a dependency on the LB's, but without those, we had no external 
connection to the world, and no way to serve traffic. They also come back 
reasonably quickly from power issues.


Matthew
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[david]

On Tue, 7 Sep 2010, Atom Powers wrote:


On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:

On Tue, 7 Sep 2010, Brodie, Kent wrote:
We actually put DHCP, DNS, and LDAP on single boxes, and considered those
our primary boot first boxes.  Almost nothing would work w/o those
services.  Each of them has some reasonable highly available option, and
made life easy.


How do you make DNS highly available (or load balance it)?
Most clients use the first server on the list and if it doesn't answer
then it tries the next etc. But the time-out makes some operating
systems (Windows) very sad.


have multiple DNS servers and make the IP address that clients use be a 
VIP that moves from server to server. With Linux this is pretty simple to 
do with linux-ha (heartbeat). The clients think that they only have one 
server to talk to, but that server is always up even if one box fails.


David Lang___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[david]

On Tue, 7 Sep 2010, Patrick Cable wrote:

> Regarding using DHCP to configure servers:
>
> I am of the mindset that as long as the server gets the same address
> every time, I don't care how it gets configured, and if I'm going to
> set up a DHCP infrastructure why not do it for everything. I believe
> that even though things may never change, the ability to make things
> easily changeable is something worthwhile.
>
> Others are of the mindset that server IPs change so infrequently that
> it is not worth putting in DHCP and risky to do so. What's the benefit
> if the addresses never change?
>
> I am looking at a network renumber in my future, so I was going to
> deploy this with the renumber.
>
> How do you feel about DHCP used to configure server addresses?
> Do you do it at your $ORGANIZATION? Was it split? Do you like it? What
> do you see as values/issues with this deployment?

We do it on some servers where the drives themselves are replicated at the 
block level (we started where all the systems had their boot drive on EMC 
and EMC replicated to our DR datacenter)

having the IP addresses be configured via DHCP allowed us to have the 
drive images be identical, but have the systems be on different networks 
when they boot up.

Prior to doing this, we just used the same IP addresses in production and 
DR, but that causes significant problems for managing the other devices 
(firewalls, switches, etc) that did not boot from the EMC.

It works if you have good people managing a decent DHCP server, we 
initially tried it with a windows AD server doing the DHCP and it was a 
fiasco.

other than things like this where you have value in using the same system 
image, I prefer to use config management tools (i.e. puppet/cfengine type 
things) to set the addresses as it eliminates one more thing that can go 
wrong when the systems boot up.

David Lang
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[David Nolan]

On Tue, Sep 7, 2010 at 4:08 PM, Atom Powers  wrote:
> On Tue, Sep 7, 2010 at 12:59 PM, Paul Graydon  wrote:
>> On 09/07/2010 09:24 AM, Atom Powers wrote:
>> On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:
>>
>> On Tue, 7 Sep 2010, Brodie, Kent wrote:
>> We actually put DHCP, DNS, and LDAP on single boxes, and considered those
>> our primary boot first boxes.  Almost nothing would work w/o those
>> services.  Each of them has some reasonable highly available option, and
>> made life easy.
>>
>> How do you make DNS highly available (or load balance it)?
>>
>> There are a few fancy tricks for doing load-balanced DNS stuff, things that
>> should reduce time-out occurrences .  The main one that springs to mind is
>> using Anycast, and route advertising through Zebra.
>>
>> http://ftp.isc.org/isc/pubs/tn/isc-tn-2004-1.htm
>
> Using OSPF for load balancing ... that's just crazy enough to be
> brilliant. (Unfortunately I'm trying to avoid adding any more crazy
> into the network, so I don't think this approach will work for me.)
>

You can also find a presentation on doing this here:
http://www.net.cmu.edu/pres/anycast/

I was working at CMU when we deployed anycast DNS.  There was a lot of
anxiety around deploying it at first, but once we got all of the
details right we loved it.

I've since deployed anycast DNS at my current employer, and then
applied the same anycast technology to both RADIUS and WINS to improve
the HA design of those services.

Benefits include:
Single DNS config to configure everywhere.  No more site specific DNS
server entries, etc.
Near real time recovery from server problems
Trivial maintenance windows for servers.  (Stop the OSPF daemon, wait
a few seconds for routing to reconverge, do your maintenance...)
Geographical load balancing.  (Adding a new office in India?  Add an
anycast server there.  If it fails DNS will redirect to the closest
server.)

-David

___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Paul Graydon]

  On 09/07/2010 10:08 AM, Atom Powers wrote:
> On Tue, Sep 7, 2010 at 12:59 PM, Paul Graydon  wrote:
>> On 09/07/2010 09:24 AM, Atom Powers wrote:
>> On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:
>>
>> On Tue, 7 Sep 2010, Brodie, Kent wrote:
>> We actually put DHCP, DNS, and LDAP on single boxes, and considered those
>> our primary boot first boxes.  Almost nothing would work w/o those
>> services.  Each of them has some reasonable highly available option, and
>> made life easy.
>>
>> How do you make DNS highly available (or load balance it)?
>>
>> There are a few fancy tricks for doing load-balanced DNS stuff, things that
>> should reduce time-out occurrences .  The main one that springs to mind is
>> using Anycast, and route advertising through Zebra.
>>
>> http://ftp.isc.org/isc/pubs/tn/isc-tn-2004-1.htm
> Using OSPF for load balancing ... that's just crazy enough to be
> brilliant. (Unfortunately I'm trying to avoid adding any more crazy
> into the network, so I don't think this approach will work for me.)
>
It is all crazy kinds of genius.  I'd love to implement it some day just 
to be able to say I've done it.
Other than that I'm not sure I'd see enough benefit from it to justify it.
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Paul Graydon]

 On 09/07/2010 09:59 AM, Paul Graydon wrote:

On 09/07/2010 09:24 AM, Atom Powers wrote:

On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:

On Tue, 7 Sep 2010, Brodie, Kent wrote:
We actually put DHCP, DNS, and LDAP on single boxes, and considered those
our primary boot first boxes.  Almost nothing would work w/o those
services.  Each of them has some reasonable highly available option, and
made life easy.

How do you make DNS highly available (or load balance it)?
Most clients use the first server on the list and if it doesn't answer
then it tries the next etc. But the time-out makes some operating
systems (Windows) very sad.

There are a few fancy tricks for doing load-balanced DNS stuff, things 
that should reduce time-out occurrences .  The main one that springs 
to mind is using Anycast, and route advertising through Zebra.


http://ftp.isc.org/isc/pubs/tn/isc-tn-2004-1.htm 



That kind of set-up was in the pipeline for a sysadmin team I was part 
of a few years back but when I left hadn't actively been worked on.


Paul

WTF?  Okay, I have a serious case of the Mondays..

Nothing to see here folks.
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Atom Powers]

On Tue, Sep 7, 2010 at 12:59 PM, Paul Graydon  wrote:
> On 09/07/2010 09:24 AM, Atom Powers wrote:
> On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:
>
> On Tue, 7 Sep 2010, Brodie, Kent wrote:
> We actually put DHCP, DNS, and LDAP on single boxes, and considered those
> our primary boot first boxes.  Almost nothing would work w/o those
> services.  Each of them has some reasonable highly available option, and
> made life easy.
>
> How do you make DNS highly available (or load balance it)?
>
> There are a few fancy tricks for doing load-balanced DNS stuff, things that
> should reduce time-out occurrences .  The main one that springs to mind is
> using Anycast, and route advertising through Zebra.
>
> http://ftp.isc.org/isc/pubs/tn/isc-tn-2004-1.htm

Using OSPF for load balancing ... that's just crazy enough to be
brilliant. (Unfortunately I'm trying to avoid adding any more crazy
into the network, so I don't think this approach will work for me.)

-- 
Perfection is just a word I use occasionally with mustard.
--Atom Powers--

___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Paul Graydon]

 On 09/07/2010 09:24 AM, Atom Powers wrote:

On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:

On Tue, 7 Sep 2010, Brodie, Kent wrote:
We actually put DHCP, DNS, and LDAP on single boxes, and considered those
our primary boot first boxes.  Almost nothing would work w/o those
services.  Each of them has some reasonable highly available option, and
made life easy.

How do you make DNS highly available (or load balance it)?
Most clients use the first server on the list and if it doesn't answer
then it tries the next etc. But the time-out makes some operating
systems (Windows) very sad.

There are a few fancy tricks for doing load-balanced DNS stuff, things 
that should reduce time-out occurrences .  The main one that springs to 
mind is using Anycast, and route advertising through Zebra. 



http://ftp.isc.org/isc/pubs/tn/isc-tn-2004-1.htm 



That kind of set-up was in the pipeline for a sysadmin team I was part 
of a few years back but when I left hadn't actively been worked on.


Paul
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Atom Powers]

On Tue, Sep 7, 2010 at 11:53 AM, Matthew Barr  wrote:
> On Tue, 7 Sep 2010, Brodie, Kent wrote:
> We actually put DHCP, DNS, and LDAP on single boxes, and considered those
> our primary boot first boxes.  Almost nothing would work w/o those
> services.  Each of them has some reasonable highly available option, and
> made life easy.

How do you make DNS highly available (or load balance it)?
Most clients use the first server on the list and if it doesn't answer
then it tries the next etc. But the time-out makes some operating
systems (Windows) very sad.

-- 
Perfection is just a word I use occasionally with mustard.
--Atom Powers--

___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Matthew Barr]

On Tue, 7 Sep 2010, Brodie, Kent wrote:

> The upside of course is- centralized management of host info.   The
> downside is that you have now added another dependency for all of the
> servers - so DHCP server robustness is absolutely critical.

True.  However, it's fairly easy to do HA on DHCP, when it's a pool of 
static reservations.  You don't have to fight over keeping track of which 
IP's are in use, etc- you can just have a few servers running duplicate 
confs, and which ever responds first, no problem.

I've set it up to generate the DHCP conf via Cobbler, then sync that file 
out to the NAS, which is then diffed, copied, verified, and loaded onto 
the prod DHCP servers.

We actually put DHCP, DNS, and LDAP on single boxes, and considered those 
our primary boot first boxes.  Almost nothing would work w/o those 
services.  Each of them has some reasonable highly available option, and 
made life easy.  We actually didn't run the masters on those boxes- those 
were slaves.  Masters tended to be VM's for DNS & DHCP. It was nice to 
shield the load from the masters, and kept things very consistant between 
the slave boxes.

Matthew
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Robert Hajime Lanning]

Brad Knowles wrote:
 > So long as the DHCP server infrastructure can be expected to be
 > 100% reliable, or at least no less reliable than the routers &
 > switches on your network and the DNS servers configured for use
 > by the servers, then I don't see what the problem is.
 >
 > You're already dependent on these other things like DNS servers,
 > routers, switches, etc... so it's not like your non-DHCP server
 > is guaranteed 100% uptime just because it doesn't depend on the
 > DHCP system.  If your DNS servers go south, then anything that
 > depends on them will go south -- including all the other servers
 > on your network.  If your routers and/or switches go south, so
 > does everything else.
 >
 > At least DHCP has the advantage that the protocol is typically
 > on used at boot time, and once you have successfully booted, you
 > don't need to talk to it again.

There is another caveat to this.  Error timeouts.

When routers, switches, and DNS recover, "the next request" tends to
work.  You want your DHCP clients to make that next request, after
recovery of service.

You would need to configure the DHCP clients to either not timeout
and exit, or fall back to known unique reachable IP addresses.
Otherwise network recovery means getting on every server's console.

You also want to make sure your DHCP server can handle the load of
the datacenter coming back from a total power outage.  Though if
your clients are configured to never timeout, they will eventually
recover, even if the DHCP server is overloaded.

-- 
END OF LINE
   --MCP
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Patrick Cable]

On Tue, Sep 7, 2010 at 2:31 PM, Brad Knowles  wrote:
> Even if the servers are manually configured and don't actually use DHCP to 
> obtain their IP address, it is still useful to record all their information 
> in the DHCP system -- that helps you record all the information regarding all 
> the IP addresses and machines that are in use, and avoid the potential of 
> handing out the same address to more than one machine.

In our case, we have a centrally maintained hosts table that our DNS
zones are generated from to mitigate that.

- Pat

___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Brad Knowles]

On Sep 7, 2010, at 1:07 PM, Patrick Cable wrote:

> Others are of the mindset that server IPs change so infrequently that
> it is not worth putting in DHCP and risky to do so. What's the benefit
> if the addresses never change?

Even if the servers are manually configured and don't actually use DHCP to 
obtain their IP address, it is still useful to record all their information in 
the DHCP system -- that helps you record all the information regarding all the 
IP addresses and machines that are in use, and avoid the potential of handing 
out the same address to more than one machine.

> I am looking at a network renumber in my future, so I was going to
> deploy this with the renumber.

That also works.

> How do you feel about DHCP used to configure server addresses?

So long as the DHCP server infrastructure can be expected to be 100% reliable, 
or at least no less reliable than the routers & switches on your network and 
the DNS servers configured for use by the servers, then I don't see what the 
problem is.

You're already dependent on these other things like DNS servers, routers, 
switches, etc... so it's not like your non-DHCP server is guaranteed 100% 
uptime just because it doesn't depend on the DHCP system.  If your DNS servers 
go south, then anything that depends on them will go south -- including all the 
other servers on your network.  If your routers and/or switches go south, so 
does everything else.

At least DHCP has the advantage that the protocol is typically on used at boot 
time, and once you have successfully booted, you don't need to talk to it again.

--
Brad Knowles 
LinkedIn Profile: 


___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Atom Powers]

On Tue, Sep 7, 2010 at 11:07 AM, Patrick Cable  wrote:
> Regarding using DHCP to configure servers:
>
> I am of the mindset that as long as the server gets the same address
> every time, I don't care how it gets configured, and if I'm going to
> set up a DHCP infrastructure why not do it for everything. I believe
> that even though things may never change, the ability to make things
> easily changeable is something worthwhile.

DHCP has the additional advantage of making it easy to reconfigure
DNS, NTP, router, etc. for the entire network at the same time.

> Others are of the mindset that server IPs change so infrequently that
> it is not worth putting in DHCP and risky to do so. What's the benefit
> if the addresses never change?

But if, for example, the entire network room looses power and your
DHCP servers are offline when everything else tries to boot you may
have trouble. Heaven forbid this should ever happen to you, but it
/can/ happen. Also, depending on your dhclient application and DHCP
servers your host may handle this situation differently.

> I am looking at a network renumber in my future, so I was going to
> deploy this with the renumber.

I just went through a network renumber, having DHCP for the servers helped.

I use static assignments for all hosts, servers and clients, managed
through my directory service. A new server will initially be
configured for DHCP; once it is operational the configuration
management system will reconfigure it to be either a static address or
a custom dhclient.conf that specifies an address but requests other
values from the DHCP server. (I also have multiple DHCP servers.)

Also, now that we are using virtual servers we have a lot more servers
to manage and they are a lot more dynamic. DHCP is has been very
useful in this environment.

-- 
Perfection is just a word I use occasionally with mustard.
--Atom Powers--
___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/

Re: [lopsa-tech] DHCP for Servers

[Brodie, Kent]

The upside of course is- centralized management of host info.   The
downside is that you have now added another dependency for all of the
servers - so DHCP server robustness is absolutely critical.

Personally, I'm of the thought that in our setup, the "less" dependant
of outside services a server needs- the better.

Thoughts to consider:  

* What happens when a network service outage occurs (one that would
isolate the dhcp box from some - or all - of the servers)
* Who manages network address space?  (in our case, it's NOT us--
network belongs to corporate I.T..)

There are certainly pros and cons to doing this - I've only hit a few. -
I would assume that in this forum, you'll find a pretty wide variety of
opinions..!

-Kent

-
Kent C. Brodie  
Department of Physiology   
(and) Human and Molecular Genetics Center
Medical College of Wisconsin
bro...@mcw.edu   +1 414 955 8590


___
Tech mailing list
Tech@lopsa.org
http://lopsa.org/cgi-bin/mailman/listinfo/tech
This list provided by the League of Professional System Administrators
 http://lopsa.org/