Trevor, Thank you for the response; I believe you got the idea pretty well and while your suggestion makes sense, it is something we definitely can't follow through with right now; our configuration is massive and complex and having to maintain three different yet similar sets of configuration would be difficult and reduce our response time to necessary user changes (of which we get anywhere from 5-10 a day). It's just not feasible without a complete reworking of how we do things right now, and not at the top of our priority lists.
I do appreciate the input, however. Thank you. - Ken Lareau On Wed, Apr 18, 2012 at 12:28 PM, Trevor Smith <trevor.c.sm...@gmail.com>wrote: > I'll take a stab at some of this. Hopefully I'm correctly understanding > your issue. > > Am I correct in the following? : > > You define 3 environments development, staging, and production. These > environments are defined as such in Puppet but they are also separate > environments within your network, for the sake of clarity I'll call them > zones from here out? > > Each zone has a Puppet Master server. > > Each Puppet Master server has three environments defined development, > staging, and production. Each environment has the full git repository with > the applicable branch checked out. > > The clients in each zone connect to the Puppet Master in their zone and > pull their configs from the corresponding environment. So a > staging_zone_client connects to staging_zone_master and pulls from the > staging_environment. > > If that's correct then: > > You already have three separate Puppet Masters so the environments are > redundant. As configured staging_zone_client can pull from > production_environment using --environment. One fix could be to define > only the production environment on each zone's Puppet Master and check out > the applicable branch in only the production environment. As long as you > never check out the production branch in development or staging then > clients from those zones couldn't pull the settings for production zone as > it's just not available. As long as they cannot connect to the other > zone's Puppet Master, preventable by network segmentation, certs etc... > > Within each zone you could then define environments such as development > and testing for conducting those activities within each zone. So you'd > have staging/dev and staging/test branches checked out in those > environments. I guess you could extend that and create environments for > each admin within each zone that would allow the admin to use the > --environment option for clients to test their work within a zone. This > would result in a lot of environments, and probably a lot of branches, but > you wouldn't need a test Puppet Master for each admin. > > I'd think this would introduce the problem of making it difficult to reuse > modules between zones as I'd think you'd end up basically managing three > completely different branches. Unless the sensitive data you're worried > about is not being stored in your puppet repo and you have no issues > merging changes made to the production branch into the development and > testing branches, plus your admins will have a lot of different topic > branches to deal with. Long run you'd probably want to move zone specific > settings out of your modules and use something like hiera so you can > standardize your modules across zones and just pull in the location > settings using hiera. > > Hope I understood your problems correctly and this is helpful.. > > On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote: >> >> Hello folks, >> >> After some conversation on #puppet on Freenode IRC, Eric Sorenson >> requested I repost the information and question here, so I am doing so >> and hopefully it will all make sense... >> >> We currently have a well-established and relatively complex Puppet >> setup in place at my company and I'm in the process of trying to >> streamline changes as well as implement better testing to ensure >> minimal disruption or issues when making those changes. Some >> information on the current situation: >> >> - There are currently three environments: development, staging, >> production. These are controlled via the '--environment' setting for >> puppet in each client. All clients only belong to one environment and >> do not move between them. >> - We have a single Puppet configuration to manage all environments. >> Various conditional statements based on environment, application type, >> hostname, etc. control what each client receives for its >> configuration. >> - There are separate servers for each environment for security reasons >> (primarily sensitive information that can only exist in the production >> environment). >> - The Puppet configuration maintained via a Git repo, currently on a >> single branch. >> - Each person on the admin team checks out own copy of the repo, make >> changes, commits the changes, then updates each environment on the >> Puppet servers for the changes to take effect. >> >> There are several issues with this process, unfortunately: >> >> - Every so often a configuration mistake will adversely affect an >> entire environment, and much of the time is only noticed _after_ the >> changes are pushed out. As a result, local changes tend to be made in >> the development environment for testing and sometimes aren't committed >> for a long time, leaving discrepancies between the environments which >> can lead to other subtle issues. >> - Less frequent but still occuring often enough, changes can still >> have subtle issues which cause things to work in one environment and >> break horribly in another; this is especially bad when the broken >> environment is the production one. >> - The configuration for a given type of client is complex enough that >> to change a client to a different application type (what we primarily >> key most of our configuration off of, followed by the environment) to >> test against a server would require rebuilding the client, which is a >> 25-45 minute process; too slow for simple changes and even too slow >> for all but the most complex changes, given how many changes we make >> in a single day. >> - We allow our users to create local VMs that the development Puppet >> server can key off their names to create a given configuration, but >> since the configuration for the various environments is shared in a >> Puppet configuration, potential for users point their puppet agents to >> the production environment is a concern (due to the sensitive >> information there). >> >> After discussion with a few coworkers, the following process was laid >> out to try to implement to resolve these issues: >> >> - Create separate branches for each of the environments and have only >> the matching branch checked out on the primary Puppet servers; changes >> will be merged into the various branches one at a time to prevent >> unintentional changes in a given environment before testing can be >> done on that environment. >> - Ensure a client in a given environment can ONLY run against that >> configuration (e.g. disallow a client in the development environment >> requesting the production configuration). >> - Each person on the admin team will have a test server where they can >> create their own branches from the Git repo for the changes they're >> working and use their test server to test changes against existing >> clients in the various environments (preventing the need to build out >> a new test client(s) to validate each change). The existing clients >> would only be run in no-op mode against the test servers. >> >> The reason for each person on the admin team to have their own test >> server that has access to all the environments is considered since: >> >> - Having a different server for each environment would be affected by >> the tight hardware resources currently. >> - The need for having separate test servers would prevent needing to >> use the primary servers for testing, which is difficult due to >> multiple admins continuously making changes and needing to test them >> without disturbing the other admins' work, along with not affecting >> the current primary servers from being able to properly handle their >> existing clients. >> >> What this all boils down to is I'm trying to find a way to deal with a >> single test server trying to be able to communicate with existing >> clients in all the environments; most of the current configuration >> would work fine except for the cert issue, which is the sticking point >> at this time. Any solution on how to handle this in the most >> straightforward manner would greatly be appreciated, as my research >> has been leading to solutions far more complex than what I would like >> (such as load balancing for the CA or trying to synchronize the certs >> across the various systems). >> >> Hopefully this made sense and I can find someone who can give advice >> on how to proceed with this issue. Thanks in advance. >> >> - Ken Lareau >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/2BDCbntAOXoJ. > > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > -- - Ken Lareau -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
