Entire Data-center go offline because something went wrong servicing the UPS in 'routine' maintenance. Several hundred physical, virtual, mainframe, etc.... that was also not a good night, last month.... :)
There's a big 'portal UPS in a trailer with cables going into the building right now and some poor security guard that gets to stand near it until a replacement goes in. Steven Peck http://www.blkmtn.org On Tue, Jan 18, 2011 at 3:51 PM, Michael B. Smith <mich...@smithcons.com>wrote: > I’ve seen four full cabinets (over 120 servers) in a data-center go > offline because of a power short that caused first caused one UPS to go > offline and then two more UPS’ to go offline because of overload. > > > > That was not a good night. > > > > Regards, > > > > Michael B. Smith > > Consultant and Exchange MVP > > http://TheEssentialExchange.com > > > > *From:* David Lum [mailto:david....@nwea.org] > *Sent:* Tuesday, January 18, 2011 3:06 PM > *To:* NT System Admin Issues > *Subject:* RE: DHCP reservations explained... > > > > If DNS stops responding, what’s working? > > > > Having said that, I see your point. We (%dayjob%) have 3 DNS servers and I > suppose you’re right, a perfect storm of all 3 being offline would prevent > other systems from being monitored. At %I.T.GarageClient% if DNS is down > then I’m already working one of the two or three servers that matter already > and my clients aren’t getting any work done anyway. > > > > Along these lines, what’s the worst chain of issues you’ve seen? During our > move we had three simultaneous SAN issues – fibre channel controller was > dead, two drives (in different containers thankfully) died, and a redundant > power supply in the SAN went out. What relies on this SAN? Our file shares, > Exchange, 80% of our SQL DB’s…. > > > > At the same time we have new audio-video and that the vendor neglected to > mention they have some multicast(?) turned on that flooded our switches, > making the servers that could run really spotty to hit from a PC. SAN guy > not happy, network guy not happy, but my DC’s were fine, lol. > > > > Dave > > > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Tuesday, January 18, 2011 11:37 AM > *To:* NT System Admin Issues > *Subject:* RE: DHCP reservations explained... > > > > I will always do one of two things: > > > > [1] use an IP address, or > > > > [2] have a rather complex hosts file on the server(s) running the > monitoring software. > > > > After all, if DNS stops responding, are you going to stop monitoring? > > > > Regards, > > > > Michael B. Smith > > Consultant and Exchange MVP > > http://TheEssentialExchange.com > > > > *From:* David Lum [mailto:david....@nwea.org] > *Sent:* Tuesday, January 18, 2011 2:06 PM > *To:* NT System Admin Issues > *Subject:* DHCP reservations explained... > > > > The other day someone commented that it seemed like a bit much that 50% of > my 100-ish servers have DHCP reservations - driving home yesterday I > realized another reason why I have it that way (because yes, I chew on these > questions and constantly evaluate why I do some process or another) - > because my fellow SE's have their server monitoring set up to look at > specific IP's instead of hostnames and I am unable to convince them > otherwise. If the server IP changes it hoses their tests and the > dependencies. > > > > It’s not how I set **MY** monitoring up for servers I maintain, but I have > posted that question here in fact and have seen differing opinions on > weather hostname or IP is preferred. > > *David Lum** **// *SYSTEMS ENGINEER > NORTHWEST EVALUATION ASSOCIATION > (Desk) 503.548.5229 *// *(Cell) 503.267.9764 > > > > > > > > > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe ntsysadmin > > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ > ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe ntsysadmin > ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
