I finally got around to reviewing and cleaning up my script for removing
devices from maintenance. See the description in my previous e-mail (below) to
see the details of what it does.
We run the script daily from cron, using a cron job like this in /etc/cron.d/:
[email protected]
SPECROOT=/opt/spectrum
# Move devices without Notes from Maintenance to Hibernation
@daily spectrum ( LOGDATE=$(date +\%Y\%m\%d-\%H\%M);
$SPECROOT/custom/bin/maintenance-fixer.pl 2>&1 >
$SPECROOT/custom/log/maintenance-fixer/maintenance-fixer-$LOGDATE.log )
The script typically takes a few minutes to run, but sometimes up to 15
minutes, in an environment with 16 SpectroSERVERs and over 6000 managed
devices. As previously mentioned, it relies on
Spectrum::CLI<http://search.cpan.org/~plonka/Spectrum-CLI-1.016/CLI.pm> to
greatly simplify the mess that is the Spectrum command line. If you haven't
installed Spectrum::CLI (probably via CPAN), you should do that before running
the script.
A few other key notes:
* I added .txt to the script name so stupid e-mail tools and clients
(like Outlook) don't try to strip the attachment and prevent you from accessing
it. You can remove the .txt extension if you want.
* Your Spectrum Main Location Server (MLS) must be set in the script.
Look for the constant variables near the top
* You can change the amount of time the script allows a device to be in
maintenance. The default is 3 days. Look for the constant variable near the
MLS at the top to change it.
* There is a spot in the get_landscapes() function where you can
specify landscapes that you don't want to process, in case you need that
functionality
* Enabling "Debug" ($DBG) in the script is like a verbose mode. It
will tell you what the script is doing as it goes.
--
Christopher P. Cashell
________________________________
EIT Platform Engineering
E-Mail: [email protected]
Infrastructure Monitoring, Management, and Automation Division EIT ~
Converging People and Technologies
West Corporation<http://www.west.com/>
________________________________
From: Cashell, Christopher P. [mailto:[email protected]]
Sent: Thursday, September 17, 2015 2:08 PM
To: spectrum
Cc: spectrum
Subject: RE:[spectrum] Removing devices from maintenance daily
I have a Perl script I wrote (written/used on Linux) that we run from cron for
that. It uses the Spectrum::CLI Perl module and pulls a list of all devices in
maintenance from all landscapes, checks to see if there is a notes field, and
if there is no notes field and the device has been in maintenance for more than
3 days, it removes the device from maintenance by putting it into hibernation.
For clarity, what it does:
1. Connect to MLS (configured in script) and pull a list of all landscapes
a. filter out fault-tolerant "secondary" landscapes based on precedence
b. filter out our dedicated TrapDirector landscape
2. Get a list of all devices in maintenance
3. Enrich the data for those devices (get model class, last polled time,
notes field, and hibernation status)
4. Filter out devices that we are considering "out of scope"
a. in hibernation
b. with an entry in the notes field
c. that don't have a "last polled" time
d. interfaces (no reliable way to determine "last polled" time)
e. devices in maintenance for less than 3 days
5. For any devices left, take them out of maintenance by putting them in
"Hibernation"
a. Hibernation will get them removed from Maintenance after 3 successful
polls
i. This prevents immediate "device unreachable" alerts for devices that
are offline
We run the script from cron daily. I need to review it to make sure there's no
confidential information in it, but I can probably post it shortly.
--
Christopher P. Cashell
________________________________
EIT Platform Engineering
E-Mail: [email protected]<mailto:[email protected]>
Infrastructure Monitoring, Management, and Automation Division EIT ~
Converging People and Technologies
West Corporation<http://www.west.com/>
________________________________
From: Stallcup, Phillip L [mailto:[email protected]]
Sent: Wednesday, September 16, 2015 9:11 AM
To: spectrum
Subject: [spectrum] Removing devices from maintenance daily
We have a problem with devices/interfaces being put in maintenance, various
reasons, but mostly to stop paging, and then being forgotten.
I have a Global Collaction that collects these, so they can be more easily
seen, however I'm looking for a way to remove everything that's in this GC from
maintenance once per day (say 8:00 am).
Anyone have any ideas?
Phillip Stallcup, CCNA
Enterprise Solutions Architect,
Information Technology
405.951.2088
[cid:[email protected]][cid:[email protected]]<http://integrisok.com/>
________________________________
This e-mail may contain identifiable health information that is subject to
protection under state and federal law. This information is intended to be for
the use of the individual named above. If you are not the intended recipient,
be aware that any disclosure, copying, distribution or use of the contents of
this information is prohibited and may be punishable by law. If you have
received this electronic transmission in error, please notify us immediately by
electronic mail (reply).
* --To unsubscribe from spectrum, send email to
[email protected]<mailto:[email protected]> with the body: unsubscribe spectrum
[email protected]<mailto:[email protected]>
* --To unsubscribe from spectrum, send email to
[email protected]<mailto:[email protected]> with the body: unsubscribe spectrum
[email protected]<mailto:[email protected]>
---
To unsubscribe from spectrum, send email to [email protected] with the body:
unsubscribe spectrum [email protected]
#!/usr/bin/env perl
# vim: ts=3 sw=3 et sm ai smd sc bg=dark
# perltidy -i=3 -vt=1 -vtc=2 -lp -cab=1 -b
#######################################################################
# maintenance-fixer.pl - Move devices without Notes from Maintenance to
# Hibernation
#######################################################################
# Written by Christopher Cashell <[email protected]>
#######################################################################
# Copyright (c) 2014, Christopher P. Cashell
#
# Permission to use, copy, modify, and/or distribute this software for
# any purpose with or without fee is hereby granted, provided that the
# above copyright notice and this permission notice appear in all
# copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL
# WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
# AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL
# DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR
# PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
# TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
# PERFORMANCE OF THIS SOFTWARE.
#######################################################################
use utf8;
use strict;
use warnings;
use File::Basename qw(basename);
use Carp qw(carp croak confess cluck);
# Using this module solves some issues we've seen with scripting against
# 32bit Spectrum on a 64bit system.
#
# You should install Spectum::CLI from CPAN unless you want to manually
# install the library and tell the script where it is.
use Spectrum::CLI;
use vars qw($VERSION);
$VERSION = '0.3';
# Set this to non-zero to enable Debugging messages for testing
my $DBG = 0;
if ($DBG) { use Data::Dumper; }
$0 = basename($0); # shorter messages
$| = 1; # Don't buffer our output
# If we don't have a SPECROOT environment variable, set it here.
$ENV{SPECROOT} = "/opt/spectrum" unless $ENV{SPECROOT};
# Set some "constants" for easier use later
# Note: We set the name of our Spectrum Main Location Server (MLS) here;
# This MUST be set so the script knows where to pull a list of landscapes
my %CONST = (
MLS => "x-mls-1", # Spectrum MLS
MAX_DAYS_MAINT => 3, # Days before dev comes out of maint
DEV_MAINT => 4,
PORT_MAINT => 9,
MC_PORT => 15,
MC_WA_SEGMENT => 7, );
# Common attributes we want to pull for devices we process
my %attr = ( condition => "0x1000a",
notes => "0x11564",
last_poll => "0x11620",
model_class => "0x11ee8",
port_condition => "0x10f1b",
not_hibernate => "0x12aca",
model_name => "0x1006e",
hostname => "0x210096",
domainname => "0x210097", );
foreach my $landscape ( get_landscapes( $CONST{MLS} ) ) {
warn "Landscape: " . $landscape . "\n" if $DBG;
my $cli = Spectrum::CLI->new( $landscape, { Verbose => 0 } )
or die "Could not connect to vnmshd:";
my @devs = get_devs($cli);
my @mdevs = filter_devs( $cli, @devs );
foreach my $hdev (@mdevs) {
my $result = enable_hibernate( $cli, $hdev );
do {
warn "Possible problem with: ";
warn Dumper($hdev);
} unless $result;
warn "Result for $hdev->{MName}: " . $result . "\n" if $DBG;
} ## end foreach my $hdev (@mdevs)
} ## end foreach my $landscape ( ( get_landscapes...
########################################################################
# Function Definitions
########################################################################
# get_devs() - Pulls a list of devices in maintenance
# Expects: landscape to process
# Returns: array of hashes
# [ { 'MHandle' => '0x1020c9', 'MName' => 'site01-switch01',
# 'MTypeHnd' => '0x2100ce', 'MTypeName' => 'CiscoNXOS' }, ]
sub get_devs {
my $cli = shift;
# Get the initial list of maintenance devs
my @devs = $cli->seek("attr=$attr{condition},val=$CONST{DEV_MAINT}");
# Enrich data for each device for later processing
foreach my $dev (@devs) {
my $mh = $dev->{MHandle};
$dev->{last_polled} = get_attr( $cli, $mh, $attr{last_poll} );
$dev->{notes} = get_attr( $cli, $mh, $attr{notes} );
$dev->{not_hibernate} = get_attr( $cli, $mh, $attr{not_hibernate} );
$dev->{model_class} = get_attr( $cli, $mh, $attr{model_class} );
# If we're a port, get port details
if ( $dev->{model_class} == $CONST{MC_PORT} ) {
$dev->{port_condition} = get_attr( $cli, $mh, $attr{port_condition} );
}
} ## end foreach my $dev (@devs)
return @devs;
} ## end sub get_devs
# get_attr()
# Expects: Spectrum::CLI object, model handle, attribute to get
# Returns: Attribute value
sub get_attr {
my $cli = shift;
my $mh = shift;
my $attr = shift;
my $values = ( $cli->show_attributes("attr=$attr mh=$mh") )[0];
return $values->{Value};
} ## end sub get_attr
# filter_devs()
# Expects: Spectrum::CLI object, Array of model hashes
# Returns: Array of model hashes
sub filter_devs {
my $cli = shift;
my @devs = @_;
my @return_devs;
foreach my $dev (@devs) {
next unless $dev->{not_hibernate}; # Filter non-devices/missing values
next unless $dev->{last_polled}; # Filter non-devices/missing values
next if $dev->{notes}; # Filter devices with a notes field
# Filter out entries that aren't real devices
# Model_Class 7 == WA_Segment
next if $dev->{model_class} == $CONST{MC_WA_SEGMENT};
# Filter out Ports that aren't individually in maintenance
# Model_Class 15 == Ports
# Port Condition 9 == Port Maintenance
if ( ( $dev->{model_class} == $CONST{MC_PORT} )
and ( $dev->{port_condition} != $CONST{PORT_MAINT} ) )
{
next;
}
# Currently, we need to filter out all interfaces; no way to
# cleanly check a "last polled time"
# Model_Class 15 == Ports
next if $dev->{model_class} == $CONST{MC_PORT};
# Skip devices already in Hibernate
next if $dev->{not_hibernate} eq "FALSE";
my $utime = time;
# Time Variance: 3 days (72 hours) in seconds
my $time_var = 60 * 60 * 24 * $CONST{MAX_DAYS_MAINT};
my $t = ( $utime - $dev->{last_polled} );
if ($DBG) {
warn "Old dev ($dev->{MHandle}): $t\n" if $t > $time_var;
warn "New dev ($dev->{MHandle}): $t\n" if $t < $time_var;
}
next if ( $t < $time_var );
# Anything that's left, we'll send back and operate on
push @return_devs, $dev;
} ## end foreach my $dev (@devs)
return @return_devs;
} ## end sub filter_devs
# enable_hibernate()
# Expects: Spectrum::CLI object, model hash
# Returns: Status
sub enable_hibernate {
my $cli = shift;
my $devs = shift;
print "Moving "
. $devs->{MName} . " ("
. $devs->{MTypeName}
. ") to Hibernation status.\n";
return
$cli->update( "mh=$devs->{MHandle}",
"attr=$attr{not_hibernate},val=FALSE" )
unless $DBG;
return "DBG: ~/vnmsh/update mh=$devs->{MHandle}"
. " attr=$attr{not_hibernate},val=FALSE" . "\n"
if $DBG;
} ## end sub enable_hibernate
# get_landscapes()
# Expects: name of the MLS landscape
# Returns: List of landscape names
sub get_landscapes {
my $mls = shift;
my $cli = Spectrum::CLI->new( $mls, { Verbose => 0 } )
or die "Could not connect to vnmshd:";
# Filter out "Secondary" (Fault-Tolerant) landscapes
my %landscapes = map { $_->{SSName}, $_->{LHandle} }
grep { $_->{Precedence} == 10 } $cli->show_landscapes;
# Exclude the NewTraps server - we don't want to process devices on
# this landscape.
return grep { not /x-newtraps-1/i } keys %landscapes;
} ## end sub get_landscapes
