Please review pull request #387: Tickets 4426 ext/regexp_nodes improvements opened by (ahpook)
Description:
OK, got a clean branch this time -
This set of commits adds support to the regexp_nodes ENC script for setting client environment. It also features doc improvements, better logging, and bug fixes based on the code we ran in production. We've since moved on to another database-driven node classifier but since there was at least some interest in this version it seemed worth dusting off the code and shining it up a bit.
- Opened: Mon Jan 23 08:25:10 UTC 2012
- Based on: puppetlabs:master (44c4d7135fca92784db9af94c9babe0039ebe2fe)
- Requested merge: ahpook:tickets-4426-regexp (2d96b903fc6d2f9f664a38a1523e0dfe0bc6d59d)
Diff follows:
diff --git a/ext/regexp_nodes/environment/development b/ext/regexp_nodes/environment/development
new file mode 100644
index 0000000..5135757
--- /dev/null
+++ b/ext/regexp_nodes/environment/development
@@ -0,0 +1,2 @@
+^dev-
+prod-canary
diff --git a/ext/regexp_nodes/parameters/environment/prod b/ext/regexp_nodes/parameters/environment/prod
deleted file mode 100644
index dd3f100..0000000
--- a/ext/regexp_nodes/parameters/environment/prod
+++ /dev/null
@@ -1 +0,0 @@
-\d{3}$
diff --git a/ext/regexp_nodes/parameters/environment/qa b/ext/regexp_nodes/parameters/environment/qa
deleted file mode 100644
index 85fccca..0000000
--- a/ext/regexp_nodes/parameters/environment/qa
+++ /dev/null
@@ -1,3 +0,0 @@
-^qa-
-^qa2-
-^qa3-
diff --git a/ext/regexp_nodes/parameters/service/prod b/ext/regexp_nodes/parameters/service/prod
new file mode 100644
index 0000000..dd3f100
--- /dev/null
+++ b/ext/regexp_nodes/parameters/service/prod
@@ -0,0 +1 @@
+\d{3}$
diff --git a/ext/regexp_nodes/parameters/service/qa b/ext/regexp_nodes/parameters/service/qa
new file mode 100644
index 0000000..85fccca
--- /dev/null
+++ b/ext/regexp_nodes/parameters/service/qa
@@ -0,0 +1,3 @@
+^qa-
+^qa2-
+^qa3-
diff --git a/ext/regexp_nodes/parameters/service/sandbox b/ext/regexp_nodes/parameters/service/sandbox
new file mode 100644
index 0000000..2aea747
--- /dev/null
+++ b/ext/regexp_nodes/parameters/service/sandbox
@@ -0,0 +1 @@
+^dev-
diff --git a/ext/regexp_nodes/regexp_nodes.rb b/ext/regexp_nodes/regexp_nodes.rb
index 8712155..2d0de8e 100644
--- a/ext/regexp_nodes/regexp_nodes.rb
+++ b/ext/regexp_nodes/regexp_nodes.rb
@@ -8,31 +8,44 @@
# regexp_nodes.rb <host>
#
# = Description
-# This classifier implements filesystem autoloading: It looks through classes
-# and parameters subdirectories, looping through each file it finds there - the
-# contents are a regexp-per-line which, if they match the hostname passed us as
-# ARGV[0], will cause a class or parameter named the same thing as the file to
-# be set.
+# This classifier implements filesystem autoloading: It looks through classes,
+# parameters, and environment subdirectories, looping through each file it
+# finds. Each file's contents are a regexp-per-line which, if they match the
+# hostname passed to the program as ARGV[0], sets a class, parameter value
+# or environment named the same thing as the file itself. At the end, the
+# resultant data structure is returned back to the puppet master process as
+# yaml.
+#
+# = Caveats
+# Since the files are read in directory order, multiple matches for a given
+# hostname in the parameters/ and environment/ subdirectories will return the
+# last-read value. (Multiple classes/ matches don't cause a problem; the
+# class is either incuded or it isn't)
+#
+# Unmatched hostnames in any of the environment/ files will cause 'production'
+# to be emitted; be aware of the complexity surrounding the interaction between
+# ENC and environments as discussed in https://projects.puppetlabs.com/issues/3910
#
# = Examples
-# In the distribution there are two subdirectories test_classes/ and
-# test_parameters, which are passed as parameters to MyExternalNode.new.
-# test_classes/database will set the 'database' class for any hostnames
-# matching %r{db\d{2}} (that is, 'db' followed by two digits) or with 'mysql'
-# anywhere in the hostname. Similarly, hosts beginning with 'www' or 'web'
-# or the hostname 'leterel' (my workstation) will be assigned the 'webserver'
-# class.
+# Based on the example files in the classes/ and parameters/ subdirectories
+# in the distribution, classes/database will set the 'database' class for
+# hosts matching %r{db\d{2}} (that is, 'db' followed by two digits) or with
+# 'mysql' anywhere in the hostname. Similarly, hosts beginning with 'www' or
+# 'web' or the hostname 'leterel' (my workstation) will be assigned the
+# 'webserver' class.
#
-# Under test_parameters/ there is one subdirectory 'environment' which
-# sets the a parameter called 'environment' to the value 'prod' for production
-# hosts (whose hostnames always end with three numbers for us), 'qa' for
+# Under parameters/ there is one subdirectory 'service' which
+# sets the a parameter called 'service' to the value 'prod' for production
+# hosts (whose hostnames always end with a three-digit code), 'qa' for
# anything that starts with 'qa-' 'qa2-' or 'qa3-', and 'sandbox' for any
-# development machines which are, naturally, named after Autechre songs.
+# development machines whose hostnames start with 'dev-'.
#
+# In the environment/ subdirectory, any hosts matching '^dev-' and a single
+# production host which serves as 'canary in the coal mine' will be put into
+# the development environment
#
# = Author
-# Eric Sorenson <[email protected]>
-
+# Eric Sorenson <[email protected]>
# we need yaml or there's not much point in going on
require 'yaml'
@@ -40,18 +53,25 @@
# Sets are like arrays but automatically de-duplicate elements
require 'set'
-# set up some nice logging
-require 'logger'
-# XXX flip this for production vs local sandbox
-# $LOG = Logger.new("/var/lib/puppet/log/extnodes.log")
-# $LOG.level = Logger::FATAL
-$LOG = Logger.new($stderr)
-$LOG.level = Logger::DEBUG
+# set up some syslog logging
+require 'syslog'
+Syslog.open('extnodes', Syslog::LOG_PID | Syslog::LOG_NDELAY, Syslog::LOG_DAEMON)
+# change this to LOG_UPTO(Sysslog::LOG_DEBUG) if you want to see everything
+# but remember your syslog.conf needs to match this or messages will be filtered
+Syslog.mask = Syslog::LOG_UPTO(Syslog::LOG_INFO)
+
+# Helper method to log to syslog; we log at level debug if no level is specified
+# since those are the most frequent calls to this method
+def log(message,level=:debug)
+ Syslog.send(level,message)
+end
+
-# paths for files we use will be relative to this directory
-# XXX flip this for production vs local sandbox
-# WORKINGDIR = "/var/lib/puppet/bin"
-WORKINGDIR = Dir.pwd
+# set our workingdir to be the directory we're executed from, regardless
+# of parent's cwd, symlinks, etc. via handy Pathname.realpath method
+require 'pathname'
+p = Pathname.new(File.dirname(__FILE__))
+WORKINGDIR = "#{p.realpath}"
# This class holds all the methods for creating and accessing the properties
# of an external node. There are really only two public methods: initialize
@@ -59,7 +79,7 @@
class ExternalNode
# Make these instance variables get/set-able with eponymous methods
- attr_accessor :classes, :parameters, :hostname
+ attr_accessor :classes, :parameters, :environment, :hostname
# initialize takes three arguments:
# hostname:: usually passed in via ARGV[0] but it could be anything
@@ -67,26 +87,29 @@ class ExternalNode
# classes
# parameterdir:: directory under WORKINGDIR to look for directories to set
# parameters
- def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/')
+ def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/', environmentdir = 'environment/')
# instance variables that contain the lists of classes and parameters
@hostname
- @classes = Set.new ["baseclass"]
- @parameters = Hash.new("unknown") # sets a default value of "unknown"
+ @classes = Set.new
+ @parameters = Hash.new("unknown") # sets a default value of "unknown"
+ @environment = "production"
self.parse_argv(hostname)
- self.match_classes(WORKINGDIR + "/#{classdir}")
- self.match_parameters(WORKINGDIR + "/#{parameterdir}")
+ self.match_classes("#{WORKINGDIR}/#{classdir}")
+ self.match_parameters("#{WORKINGDIR}/#{parameterdir}")
+ self.match_environment("#{WORKINGDIR}/#{environmentdir}")
end
# private method called by initialize which sanity-checks our hostname.
# good candidate for overriding in a subclass if you need different checks
def parse_argv(hostname)
- if hostname =~ /^([-\w]+?)\.([-\w\.]+)/ # non-greedy up to the first . is hostname
+ if hostname =~ /^([-\w]+?)\.([-\w\.]+)/ # non-greedy up to the first . is hostname
@hostname = $1
- elsif hostname =~ /^([-\w]+)$/ # sometimes puppet's @name is just a name
+ elsif hostname =~ /^([-\w]+)$/ # sometimes puppet's @name is just a name
@hostname = hostname
+ log("got shortname for [#{hostname}]")
else
- $LOG.fatal("didn't receive parsable hostname, got: [#{hostname}]")
+ log("didn't receive parsable hostname, got: [#{hostname}]",:err)
exit(1)
end
end
@@ -100,7 +123,7 @@ def to_yaml
else
parameters = self.parameters
end
- ({ 'classes' => classes, 'parameters' => parameters}).to_yaml
+ ({ 'classes' => classes, 'parameters' => parameters, 'environment' => environment}).to_yaml
end
# Private method that expects an absolute path to a file and a string to
@@ -112,27 +135,40 @@ def matched_in_patternfile?(filepath, matchthis)
begin
open(filepath).each { |l|
- pattern = %r{#{l.chomp!}}
+ l.chomp!
+
+ next if l =~ /^$/
+ next if l =~ /^#/
+
+ if l =~ /^\s*(\S+)/
+ m = Regexp.last_match
+ log("found a non-comment line, transforming [#{l}] into [#{m[1]}]")
+ l.gsub!(l,m[1])
+ else
+ next l
+ end
+
+ pattern = %r{#{l}}
patternlist << pattern
- $LOG.debug("appending [#{pattern}] to patternlist for [#{filepath}]")
+ log("appending [#{pattern}] to patternlist for [#{filepath}]")
}
rescue Exception
- $LOG.fatal("Problem reading #{filepath}: #{$ERROR_INFO}")
+ log("Problem reading #{filepath}: #{$!}",:err)
exit(1)
end
- $LOG.debug("list of patterns for #{filepath}: #{patternlist}")
+ log("list of patterns for #{filepath}: #{patternlist}")
if matchthis =~ Regexp.union(patternlist)
- $LOG.debug("matched #{$~.to_s} in #{matchthis}, returning true")
+ log("matched #{$~.to_s} in #{matchthis}, returning true")
return true
- else # hostname didn't match anything in patternlist
- $LOG.debug("#{matchthis} unmatched, returning false")
+ else # hostname didn't match anything in patternlist
+ log("#{matchthis} unmatched, returning false")
return nil
end
- end
+ end # def
# private method - takes a path to look for files, iterates through all
# readable, regular files it finds, and matches this instance's @hostname
@@ -142,41 +178,60 @@ def match_classes(fullpath)
filepath = "#{fullpath}/#{patternfile}"
next unless File.file?(filepath) and
File.readable?(filepath)
- $LOG.debug("Attempting to match [#{@hostname}] in [#{filepath}]")
+ next if patternfile =~ /^\./
+ log("Attempting to match [#{@hostname}] in [#{filepath}]")
if matched_in_patternfile?(filepath,@hostname)
@classes << patternfile.to_s
- $LOG.debug("Appended #{patternfile.to_s} to classes instance variable")
+ log("Appended #{patternfile.to_s} to classes instance variable")
end
end
end
+ # match_environment is similar to match_classes but it overwrites
+ # any previously set value (usually just the default, 'production')
+ # with a match
+ def match_environment(fullpath)
+ Dir.foreach(fullpath) do |patternfile|
+ filepath = "#{fullpath}/#{patternfile}"
+ next unless File.file?(filepath) and
+ File.readable?(filepath)
+ next if patternfile =~ /^\./
+ log("Attempting to match [#{@hostname}] in [#{filepath}]")
+ if matched_in_patternfile?(filepath,@hostname)
+ @environment = patternfile.to_s
+ log("Wrote #{patternfile.to_s} to environment instance variable")
+ end
+ end
+ end
+
+
# Parameters are handled slightly differently; we make another level of
# directories to get the parameter name, then use the names of the files
# contained in there for the values of those parameters.
#
- # ex: cat /var/lib/puppet/bin/parameters/environment/production
+ # ex: cat ./parameters/service/production
# ^prodweb
- # would set parameters["environment"] = "production" for prodweb001
+ # would set parameters["service"] = "production" for prodweb001
def match_parameters(fullpath)
Dir.foreach(fullpath) do |parametername|
filepath = "#{fullpath}/#{parametername}"
- next if File.basename(filepath) =~ /^\./ # skip over dotfiles
+ next if File.basename(filepath) =~ /^\./ # skip over dotfiles
next unless File.directory?(filepath) and
File.readable?(filepath) # skip over non-directories
- $LOG.debug "Considering contents of #{filepath}"
+ log("Considering contents of #{filepath}")
Dir.foreach("#{filepath}") do |patternfile|
secondlevel = "#{filepath}/#{patternfile}"
- $LOG.debug "Found parameters patternfile at #{secondlevel}"
+ log("Found parameters patternfile at #{secondlevel}")
next unless File.file?(secondlevel) and
File.readable?(secondlevel)
- $LOG.debug("Attempting to match [#{@hostname}] in [#{secondlevel}]")
+ log("Attempting to match [#{@hostname}] in [#{secondlevel}]")
if matched_in_patternfile?(secondlevel, @hostname)
@parameters[ parametername.to_s ] = patternfile.to_s
- $LOG.debug("Set @parameters[#{parametername.to_s}] = #{patternfile.to_s}")
+ log("Set @parameters[#{parametername.to_s}] = #{patternfile.to_s}")
end
end
end
@@ -198,10 +253,11 @@ def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/')
match = Regexp.last_match
hostclass = match[2]
- $LOG.debug("matched hostclass #{hostclass}")
+ log("matched hostclass #{hostclass}")
@parameters[ "hostclass" ] = hostclass
else
- $LOG.debug("hostclass couldn't figure out class from #{@hostname}")
+ log("couldn't figure out class from #{@hostname}",:warning)
+
end
end
@@ -210,6 +266,6 @@ def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/')
# Here we begin actual execution by calling methods defined above
-mynode = MyExternalNode.new(ARGV[0], classes = 'test_classes', parameters = 'test_parameters')
+mynode = MyExternalNode.new(ARGV[0])
puts mynode.to_yaml
-- You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to [email protected].
For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
