I'll take a stab at some of this. Hopefully I'm correctly understanding
your issue.
Am I correct in the following? :
You define 3 environments development, staging, and production. These
environments are defined as such in Puppet but they are also separate
environments within your network, for the sake of clarity I'll call them
zones from here out?
Each zone has a Puppet Master server.
Each Puppet Master server has three environments defined development,
staging, and production. Each environment has the full git repository with
the applicable branch checked out.
The clients in each zone connect to the Puppet Master in their zone and
pull their configs from the corresponding environment. So a
staging_zone_client connects to staging_zone_master and pulls from the
staging_environment.
If that's correct then:
You already have three separate Puppet Masters so the environments are
redundant. As configured staging_zone_client can pull from
production_environment using --environment. One fix could be to define
only the production environment on each zone's Puppet Master and check out
the applicable branch in only the production environment. As long as you
never check out the production branch in development or staging then
clients from those zones couldn't pull the settings for production zone as
it's just not available. As long as they cannot connect to the other
zone's Puppet Master, preventable by network segmentation, certs etc...
Within each zone you could then define environments such as development and
testing for conducting those activities within each zone. So you'd have
staging/dev and staging/test branches checked out in those environments. I
guess you could extend that and create environments for each admin within
each zone that would allow the admin to use the --environment option for
clients to test their work within a zone. This would result in a lot of
environments, and probably a lot of branches, but you wouldn't need a test
Puppet Master for each admin.
I'd think this would introduce the problem of making it difficult to reuse
modules between zones as I'd think you'd end up basically managing three
completely different branches. Unless the sensitive data you're worried
about is not being stored in your puppet repo and you have no issues
merging changes made to the production branch into the development and
testing branches, plus your admins will have a lot of different topic
branches to deal with. Long run you'd probably want to move zone specific
settings out of your modules and use something like hiera so you can
standardize your modules across zones and just pull in the location
settings using hiera.
Hope I understood your problems correctly and this is helpful..
On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote:
Hello folks,
After some conversation on #puppet on Freenode IRC, Eric Sorenson
requested I repost the information and question here, so I am doing so
and hopefully it will all make sense...
We currently have a well-established and relatively complex Puppet
setup in place at my company and I'm in the process of trying to
streamline changes as well as implement better testing to ensure
minimal disruption or issues when making those changes. Some
information on the current situation:
- There are currently three environments: development, staging,
production. These are controlled via the '--environment' setting for
puppet in each client. All clients only belong to one environment and
do not move between them.
- We have a single Puppet configuration to manage all environments.
Various conditional statements based on environment, application type,
hostname, etc. control what each client receives for its
configuration.
- There are separate servers for each environment for security reasons
(primarily sensitive information that can only exist in the production
environment).
- The Puppet configuration maintained via a Git repo, currently on a
single branch.
- Each person on the admin team checks out own copy of the repo, make
changes, commits the changes, then updates each environment on the
Puppet servers for the changes to take effect.
There are several issues with this process, unfortunately:
- Every so often a configuration mistake will adversely affect an
entire environment, and much of the time is only noticed _after_ the
changes are pushed out. As a result, local changes tend to be made in
the development environment for testing and sometimes aren't committed
for a long time, leaving discrepancies between the environments which
can lead to other subtle issues.
- Less frequent but still occuring often enough, changes can still
have subtle issues which cause things to work in one environment and
break horribly in another; this is especially bad when the broken
environment is the production one.
- The configuration for a given
