Please review pull request #386: Ticket 4426 regexp nodes opened by (ahpook)
Description:
The 'files changed' pane here on github shows the correct deltas but there are a lot of commits scooped up that aren't really relevant -- the hashes of the important commits are
d679dc5 Mised a Fixme: Add link to redmine ticket describing ENC/env interaction
338fbb2 Adds support for setting environment via regexp files
c065e2e Merged updates to regexp_nodes from our production fork
so feel free to bounce this if I'm Doin' It Wrong[tm]
- Opened: Mon Jan 23 07:56:04 UTC 2012
- Based on: puppetlabs:master (44c4d7135fca92784db9af94c9babe0039ebe2fe)
- Requested merge: ahpook:ticket-4426-regexp_nodes (f8af0c54bd875dd25977569dec23deb90ae998df)
Diff follows:
diff --git a/ext/regexp_nodes/environment/development b/ext/regexp_nodes/environment/development
new file mode 100644
index 0000000..5135757
--- /dev/null
+++ b/ext/regexp_nodes/environment/development
@@ -0,0 +1,2 @@
+^dev-
+prod-canary
diff --git a/ext/regexp_nodes/parameters/environment/prod b/ext/regexp_nodes/parameters/environment/prod
deleted file mode 100644
index dd3f100..0000000
--- a/ext/regexp_nodes/parameters/environment/prod
+++ /dev/null
@@ -1 +0,0 @@
-\d{3}$
diff --git a/ext/regexp_nodes/parameters/environment/qa b/ext/regexp_nodes/parameters/environment/qa
deleted file mode 100644
index 85fccca..0000000
--- a/ext/regexp_nodes/parameters/environment/qa
+++ /dev/null
@@ -1,3 +0,0 @@
-^qa-
-^qa2-
-^qa3-
diff --git a/ext/regexp_nodes/parameters/service/prod b/ext/regexp_nodes/parameters/service/prod
new file mode 100644
index 0000000..dd3f100
--- /dev/null
+++ b/ext/regexp_nodes/parameters/service/prod
@@ -0,0 +1 @@
+\d{3}$
diff --git a/ext/regexp_nodes/parameters/service/qa b/ext/regexp_nodes/parameters/service/qa
new file mode 100644
index 0000000..85fccca
--- /dev/null
+++ b/ext/regexp_nodes/parameters/service/qa
@@ -0,0 +1,3 @@
+^qa-
+^qa2-
+^qa3-
diff --git a/ext/regexp_nodes/parameters/service/sandbox b/ext/regexp_nodes/parameters/service/sandbox
new file mode 100644
index 0000000..2aea747
--- /dev/null
+++ b/ext/regexp_nodes/parameters/service/sandbox
@@ -0,0 +1 @@
+^dev-
diff --git a/ext/regexp_nodes/regexp_nodes.rb b/ext/regexp_nodes/regexp_nodes.rb
index 8712155..2d0de8e 100644
--- a/ext/regexp_nodes/regexp_nodes.rb
+++ b/ext/regexp_nodes/regexp_nodes.rb
@@ -8,31 +8,44 @@
# regexp_nodes.rb <host>
#
# = Description
-# This classifier implements filesystem autoloading: It looks through classes
-# and parameters subdirectories, looping through each file it finds there - the
-# contents are a regexp-per-line which, if they match the hostname passed us as
-# ARGV[0], will cause a class or parameter named the same thing as the file to
-# be set.
+# This classifier implements filesystem autoloading: It looks through classes,
+# parameters, and environment subdirectories, looping through each file it
+# finds. Each file's contents are a regexp-per-line which, if they match the
+# hostname passed to the program as ARGV[0], sets a class, parameter value
+# or environment named the same thing as the file itself. At the end, the
+# resultant data structure is returned back to the puppet master process as
+# yaml.
+#
+# = Caveats
+# Since the files are read in directory order, multiple matches for a given
+# hostname in the parameters/ and environment/ subdirectories will return the
+# last-read value. (Multiple classes/ matches don't cause a problem; the
+# class is either incuded or it isn't)
+#
+# Unmatched hostnames in any of the environment/ files will cause 'production'
+# to be emitted; be aware of the complexity surrounding the interaction between
+# ENC and environments as discussed in https://projects.puppetlabs.com/issues/3910
#
# = Examples
-# In the distribution there are two subdirectories test_classes/ and
-# test_parameters, which are passed as parameters to MyExternalNode.new.
-# test_classes/database will set the 'database' class for any hostnames
-# matching %r{db\d{2}} (that is, 'db' followed by two digits) or with 'mysql'
-# anywhere in the hostname. Similarly, hosts beginning with 'www' or 'web'
-# or the hostname 'leterel' (my workstation) will be assigned the 'webserver'
-# class.
+# Based on the example files in the classes/ and parameters/ subdirectories
+# in the distribution, classes/database will set the 'database' class for
+# hosts matching %r{db\d{2}} (that is, 'db' followed by two digits) or with
+# 'mysql' anywhere in the hostname. Similarly, hosts beginning with 'www' or
+# 'web' or the hostname 'leterel' (my workstation) will be assigned the
+# 'webserver' class.
#
-# Under test_parameters/ there is one subdirectory 'environment' which
-# sets the a parameter called 'environment' to the value 'prod' for production
-# hosts (whose hostnames always end with three numbers for us), 'qa' for
+# Under parameters/ there is one subdirectory 'service' which
+# sets the a parameter called 'service' to the value 'prod' for production
+# hosts (whose hostnames always end with a three-digit code), 'qa' for
# anything that starts with 'qa-' 'qa2-' or 'qa3-', and 'sandbox' for any
-# development machines which are, naturally, named after Autechre songs.
+# development machines whose hostnames start with 'dev-'.
#
+# In the environment/ subdirectory, any hosts matching '^dev-' and a single
+# production host which serves as 'canary in the coal mine' will be put into
+# the development environment
#
# = Author
-# Eric Sorenson <[email protected]>
-
+# Eric Sorenson <[email protected]>
# we need yaml or there's not much point in going on
require 'yaml'
@@ -40,18 +53,25 @@
# Sets are like arrays but automatically de-duplicate elements
require 'set'
-# set up some nice logging
-require 'logger'
-# XXX flip this for production vs local sandbox
-# $LOG = Logger.new("/var/lib/puppet/log/extnodes.log")
-# $LOG.level = Logger::FATAL
-$LOG = Logger.new($stderr)
-$LOG.level = Logger::DEBUG
+# set up some syslog logging
+require 'syslog'
+Syslog.open('extnodes', Syslog::LOG_PID | Syslog::LOG_NDELAY, Syslog::LOG_DAEMON)
+# change this to LOG_UPTO(Sysslog::LOG_DEBUG) if you want to see everything
+# but remember your syslog.conf needs to match this or messages will be filtered
+Syslog.mask = Syslog::LOG_UPTO(Syslog::LOG_INFO)
+
+# Helper method to log to syslog; we log at level debug if no level is specified
+# since those are the most frequent calls to this method
+def log(message,level=:debug)
+ Syslog.send(level,message)
+end
+
-# paths for files we use will be relative to this directory
-# XXX flip this for production vs local sandbox
-# WORKINGDIR = "/var/lib/puppet/bin"
-WORKINGDIR = Dir.pwd
+# set our workingdir to be the directory we're executed from, regardless
+# of parent's cwd, symlinks, etc. via handy Pathname.realpath method
+require 'pathname'
+p = Pathname.new(File.dirname(__FILE__))
+WORKINGDIR = "#{p.realpath}"
# This class holds all the methods for creating and accessing the properties
# of an external node. There are really only two public methods: initialize
@@ -59,7 +79,7 @@
class ExternalNode
# Make these instance variables get/set-able with eponymous methods
- attr_accessor :classes, :parameters, :hostname
+ attr_accessor :classes, :parameters, :environment, :hostname
# initialize takes three arguments:
# hostname:: usually passed in via ARGV[0] but it could be anything
@@ -67,26 +87,29 @@ class ExternalNode
# classes
# parameterdir:: directory under WORKINGDIR to look for directories to set
# parameters
- def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/')
+ def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/', environmentdir = 'environment/')
# instance variables that contain the lists of classes and parameters
@hostname
- @classes = Set.new ["baseclass"]
- @parameters = Hash.new("unknown") # sets a default value of "unknown"
+ @classes = Set.new
+ @parameters = Hash.new("unknown") # sets a default value of "unknown"
+ @environment = "production"
self.parse_argv(hostname)
- self.match_classes(WORKINGDIR + "/#{classdir}")
- self.match_parameters(WORKINGDIR + "/#{parameterdir}")
+ self.match_classes("#{WORKINGDIR}/#{classdir}")
+ self.match_parameters("#{WORKINGDIR}/#{parameterdir}")
+ self.match_environment("#{WORKINGDIR}/#{environmentdir}")
end
# private method called by initialize which sanity-checks our hostname.
# good candidate for overriding in a subclass if you need different checks
def parse_argv(hostname)
- if hostname =~ /^([-\w]+?)\.([-\w\.]+)/ # non-greedy up to the first . is hostname
+ if hostname =~ /^([-\w]+?)\.([-\w\.]+)/ # non-greedy up to the first . is hostname
@hostname = $1
- elsif hostname =~ /^([-\w]+)$/ # sometimes puppet's @name is just a name
+ elsif hostname =~ /^([-\w]+)$/ # sometimes puppet's @name is just a name
@hostname = hostname
+ log("got shortname for [#{hostname}]")
else
- $LOG.fatal("didn't receive parsable hostname, got: [#{hostname}]")
+ log("didn't receive parsable hostname, got: [#{hostname}]",:err)
exit(1)
end
end
@@ -100,7 +123,7 @@ def to_yaml
else
parameters = self.parameters
end
- ({ 'classes' => classes, 'parameters' => parameters}).to_yaml
+ ({ 'classes' => classes, 'parameters' => parameters, 'environment' => environment}).to_yaml
end
# Private method that expects an absolute path to a file and a string to
@@ -112,27 +135,40 @@ def matched_in_patternfile?(filepath, matchthis)
begin
open(filepath).each { |l|
- pattern = %r{#{l.chomp!}}
+ l.chomp!
+
+ next if l =~ /^$/
+ next if l =~ /^#/
+
+ if l =~ /^\s*(\S+)/
+ m = Regexp.last_match
+ log("found a non-comment line, transforming [#{l}] into [#{m[1]}]")
+ l.gsub!(l,m[1])
+ else
+ next l
+ end
+
+ pattern = %r{#{l}}
patternlist << pattern
- $LOG.debug("appending [#{pattern}] to patternlist for [#{filepath}]")
+ log("appending [#{pattern}] to patternlist for [#{filepath}]")
}
rescue Exception
- $LOG.fatal("Problem reading #{filepath}: #{$ERROR_INFO}")
+ log("Problem reading #{filepath}: #{$!}",:err)
exit(1)
end
- $LOG.debug("list of patterns for #{filepath}: #{patternlist}")
+ log("list of patterns for #{filepath}: #{patternlist}")
if matchthis =~ Regexp.union(patternlist)
- $LOG.debug("matched #{$~.to_s} in #{matchthis}, returning true")
+ log("matched #{$~.to_s} in #{matchthis}, returning true")
return true
- else # hostname didn't match anything in patternlist
- $LOG.debug("#{matchthis} unmatched, returning false")
+ else # hostname didn't match anything in patternlist
+ log("#{matchthis} unmatched, returning false")
return nil
end
- end
+ end # def
# private method - takes a path to look for files, iterates through all
# readable, regular files it finds, and matches this instance's @hostname
@@ -142,41 +178,60 @@ def match_classes(fullpath)
filepath = "#{fullpath}/#{patternfile}"
next unless File.file?(filepath) and
File.readable?(filepath)
- $LOG.debug("Attempting to match [#{@hostname}] in [#{filepath}]")
+ next if patternfile =~ /^\./
+ log("Attempting to match [#{@hostname}] in [#{filepath}]")
if matched_in_patternfile?(filepath,@hostname)
@classes << patternfile.to_s
- $LOG.debug("Appended #{patternfile.to_s} to classes instance variable")
+ log("Appended #{patternfile.to_s} to classes instance variable")
end
end
end
+ # match_environment is similar to match_classes but it overwrites
+ # any previously set value (usually just the default, 'production')
+ # with a match
+ def match_environment(fullpath)
+ Dir.foreach(fullpath) do |patternfile|
+ filepath = "#{fullpath}/#{patternfile}"
+ next unless File.file?(filepath) and
+ File.readable?(filepath)
+ next if patternfile =~ /^\./
+ log("Attempting to match [#{@hostname}] in [#{filepath}]")
+ if matched_in_patternfile?(filepath,@hostname)
+ @environment = patternfile.to_s
+ log("Wrote #{patternfile.to_s} to environment instance variable")
+ end
+ end
+ end
+
+
# Parameters are handled slightly differently; we make another level of
# directories to get the parameter name, then use the names of the files
# contained in there for the values of those parameters.
#
- # ex: cat /var/lib/puppet/bin/parameters/environment/production
+ # ex: cat ./parameters/service/production
# ^prodweb
- # would set parameters["environment"] = "production" for prodweb001
+ # would set parameters["service"] = "production" for prodweb001
def match_parameters(fullpath)
Dir.foreach(fullpath) do |parametername|
filepath = "#{fullpath}/#{parametername}"
- next if File.basename(filepath) =~ /^\./ # skip over dotfiles
+ next if File.basename(filepath) =~ /^\./ # skip over dotfiles
next unless File.directory?(filepath) and
File.readable?(filepath) # skip over non-directories
- $LOG.debug "Considering contents of #{filepath}"
+ log("Considering contents of #{filepath}")
Dir.foreach("#{filepath}") do |patternfile|
secondlevel = "#{filepath}/#{patternfile}"
- $LOG.debug "Found parameters patternfile at #{secondlevel}"
+ log("Found parameters patternfile at #{secondlevel}")
next unless File.file?(secondlevel) and
File.readable?(secondlevel)
- $LOG.debug("Attempting to match [#{@hostname}] in [#{secondlevel}]")
+ log("Attempting to match [#{@hostname}] in [#{secondlevel}]")
if matched_in_patternfile?(secondlevel, @hostname)
@parameters[ parametername.to_s ] = patternfile.to_s
- $LOG.debug("Set @parameters[#{parametername.to_s}] = #{patternfile.to_s}")
+ log("Set @parameters[#{parametername.to_s}] = #{patternfile.to_s}")
end
end
end
@@ -198,10 +253,11 @@ def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/')
match = Regexp.last_match
hostclass = match[2]
- $LOG.debug("matched hostclass #{hostclass}")
+ log("matched hostclass #{hostclass}")
@parameters[ "hostclass" ] = hostclass
else
- $LOG.debug("hostclass couldn't figure out class from #{@hostname}")
+ log("couldn't figure out class from #{@hostname}",:warning)
+
end
end
@@ -210,6 +266,6 @@ def initialize(hostname, classdir = 'classes/', parameterdir = 'parameters/')
# Here we begin actual execution by calling methods defined above
-mynode = MyExternalNode.new(ARGV[0], classes = 'test_classes', parameters = 'test_parameters')
+mynode = MyExternalNode.new(ARGV[0])
puts mynode.to_yaml
-- You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to [email protected].
For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en.
