Hi Ken,
Your site and your practices (and the problems that go with it) are
similar to our own. I've answered by describing what we think the way
forward for Puppet is for us, hopefully you find it helpful and get some
ideas :-)
I feel your pain when you say there are changes languishing in Dev for
too long that get inadvertently pushed out to other environments. Once I
accidentally rolled out a whole set of 'work in progress' changes that a
colleague of mine was committing and pushing to one environment (we use
Git) and I cut a new tag and pushed a whole heap of these unfinished
changes to the next environment up the chain.
After this I realised our current model really wasn't going to work in
the long run: this happened with only two Admins working on Puppet in
the last week when we employ almost a dozen who could potentially make
the same mistakes. While we're still stuck in this model now, we think
the way forward for us is to take a page out of our Development Team's
book and set up a fully fledged CI environment for Puppet. It would be
even more awesome if our Developer's CI was also our CI - but
considering they've got half a million unit tests and we've got zero,
we've got a long way to go.
How this plays out practically is that all our admins commit into our
Puppet repository and those changes start getting applied immediately to
our Dev/CI servers and they start getting tested against immediately
(hence the unit tests). Problems will always occur, but they will occur
almost immediately and we'll have feedback to fix it.
In response to your reply to Trevor:
Can you explain how you see maintaining three slightly different
environments with one Puppet Master and a firewall is more difficult
than three individual Puppet Masters? You've already got all the
differences written down in your existing manifests - different IP
addresses for different services, different users/passwords, etc, maybe
slightly different packages, or am I missing something?
Without knowing more about your site (so hopefully this suggestion is
relevant), here's what I would do: I would slowly start to encourage
people to separate out your data from your code using a shared
repository. Take a look at Hiera if you haven't already, I'd recommend
using it but other data sources would suffice as well.
So right now you've got your Dev data in your Dev modules, your Staging
data in your Staging modules, etc. Take one of your modules (lets say
DNS, it's easy), and put it's data (DNS server IP addresses) into a
shared data store that each of your Dev, Staging and Prod Puppet Masters
can access. Then change your Dev module to remove the data and pull it
from the data store. Then do Staging, the same data store but different
information, then finally Prod. If your three separate DNS modules don't
look *exactly* the same now, you've done something wrong. Do that for
every module and you've now got no reason not to run a single Puppet
Master. The changes would take some time to do every module but if you
encourage every admin to make a small step forward each change they
make, eventually you'd get there.
What kind of changes are you making 5-10 times a day? The same changes
for the same people? We have a situation here where our Developers
sometimes need to change the configuration of our application every
hour. Right now we don't manage those systems with Puppet because making
that many changes of the same thing over and over is a monumental waste
of our time.
I hack on Hiera a little bit to get some extra functionality from it,
mainly the ability to tier my data sources. What I plan on doing is to
have a low priority data store that's managed by Developers. Since all
our modules will (eventually) look to the data stores to configure
themselves, our Developers will be able to make their own system
configuration changes. Our Admins control the modules (we write *how*
it's done) and the higher priority data stores so we can override
whatever the Developers enter if need be (we don't want a Developer
changing the root password now). The Developers then trigger their own
Puppet run and they've reconfigured the system using the same method
(same Puppet modules) as Production so I don't have to worry about
things "being done differently in Dev".
-Luke
On 18/04/12 21:08, Ken Lareau wrote:
Trevor,
Thank you for the response; I believe you got the idea pretty well and
while your suggestion makes sense, it is something we definitely can't
follow through with right now; our configuration is massive and
complex and having to maintain three different yet similar sets of
configuration would be difficult and reduce our response time to
necessary user changes (of which we get anywhere from 5-10 a day).
It's just not feasible without a complete reworking of how we do
things right now, and not at the top of our priority lists.
I do appreciate the input, however. Thank you.
- Ken Lareau
On Wed, Apr 18, 2012 at 12:28 PM, Trevor Smith
<trevor.c.sm...@gmail.com <mailto:trevor.c.sm...@gmail.com>> wrote:
I'll take a stab at some of this. Hopefully I'm correctly
understanding your issue.
Am I correct in the following? :
You define 3 environments development, staging, and production.
These environments are defined as such in Puppet but they are
also separate environments within your network, for the sake of
clarity I'll call them zones from here out?
Each zone has a Puppet Master server.
Each Puppet Master server has three environments defined
development, staging, and production. Each environment has the
full git repository with the applicable branch checked out.
The clients in each zone connect to the Puppet Master in their
zone and pull their configs from the corresponding environment.
So a staging_zone_client connects to staging_zone_master and
pulls from the staging_environment.
If that's correct then:
You already have three separate Puppet Masters so the environments
are redundant. As configured staging_zone_client can pull from
production_environment using --environment. One fix could be to
define only the production environment on each zone's Puppet
Master and check out the applicable branch in only the production
environment. As long as you never check out the production branch
in development or staging then clients from those zones couldn't
pull the settings for production zone as it's just not available.
As long as they cannot connect to the other zone's Puppet Master,
preventable by network segmentation, certs etc...
Within each zone you could then define environments such as
development and testing for conducting those activities within
each zone. So you'd have staging/dev and staging/test branches
checked out in those environments. I guess you could extend that
and create environments for each admin within each zone that would
allow the admin to use the --environment option for clients to
test their work within a zone. This would result in a lot of
environments, and probably a lot of branches, but you wouldn't
need a test Puppet Master for each admin.
I'd think this would introduce the problem of making it difficult
to reuse modules between zones as I'd think you'd end up basically
managing three completely different branches. Unless the
sensitive data you're worried about is not being stored in your
puppet repo and you have no issues merging changes made to the
production branch into the development and testing branches, plus
your admins will have a lot of different topic branches to deal
with. Long run you'd probably want to move zone specific settings
out of your modules and use something like hiera so you can
standardize your modules across zones and just pull in the
location settings using hiera.
Hope I understood your problems correctly and this is helpful..
On Tuesday, April 17, 2012 10:34:43 PM UTC-4, Ken Lareau wrote:
Hello folks,
After some conversation on #puppet on Freenode IRC, Eric Sorenson
requested I repost the information and question here, so I am
doing so
and hopefully it will all make sense...
We currently have a well-established and relatively complex
Puppet
setup in place at my company and I'm in the process of trying to
streamline changes as well as implement better testing to ensure
minimal disruption or issues when making those changes. Some
information on the current situation:
- There are currently three environments: development, staging,
production. These are controlled via the '--environment'
setting for
puppet in each client. All clients only belong to one
environment and
do not move between them.
- We have a single Puppet configuration to manage all
environments.
Various conditional statements based on environment,
application type,
hostname, etc. control what each client receives for its
configuration.
- There are separate servers for each environment for security
reasons
(primarily sensitive information that can only exist in the
production
environment).
- The Puppet configuration maintained via a Git repo,
currently on a
single branch.
- Each person on the admin team checks out own copy of the
repo, make
changes, commits the changes, then updates each environment on
the
Puppet servers for the changes to take effect.
There are several issues with this process, unfortunately:
- Every so often a configuration mistake will adversely affect an
entire environment, and much of the time is only noticed
_after_ the
changes are pushed out. As a result, local changes tend to be
made in
the development environment for testing and sometimes aren't
committed
for a long time, leaving discrepancies between the
environments which
can lead to other subtle issues.
- Less frequent but still occuring often enough, changes can
still
have subtle issues which cause things to work in one
environment and
break horribly in another; this is especially bad when the broken
environment is the production one.
- The configuration for a given type of client is complex
enough that
to change a client to a different application type (what we
primarily
key most of our configuration off of, followed by the
environment) to
test against a server would require rebuilding the client,
which is a
25-45 minute process; too slow for simple changes and even too
slow
for all but the most complex changes, given how many changes
we make
in a single day.
- We allow our users to create local VMs that the development
Puppet
server can key off their names to create a given
configuration, but
since the configuration for the various environments is shared
in a
Puppet configuration, potential for users point their puppet
agents to
the production environment is a concern (due to the sensitive
information there).
After discussion with a few coworkers, the following process
was laid
out to try to implement to resolve these issues:
- Create separate branches for each of the environments and
have only
the matching branch checked out on the primary Puppet servers;
changes
will be merged into the various branches one at a time to prevent
unintentional changes in a given environment before testing
can be
done on that environment.
- Ensure a client in a given environment can ONLY run against
that
configuration (e.g. disallow a client in the development
environment
requesting the production configuration).
- Each person on the admin team will have a test server where
they can
create their own branches from the Git repo for the changes
they're
working and use their test server to test changes against
existing
clients in the various environments (preventing the need to
build out
a new test client(s) to validate each change). The existing
clients
would only be run in no-op mode against the test servers.
The reason for each person on the admin team to have their own
test
server that has access to all the environments is considered
since:
- Having a different server for each environment would be
affected by
the tight hardware resources currently.
- The need for having separate test servers would prevent
needing to
use the primary servers for testing, which is difficult due to
multiple admins continuously making changes and needing to
test them
without disturbing the other admins' work, along with not
affecting
the current primary servers from being able to properly handle
their
existing clients.
What this all boils down to is I'm trying to find a way to
deal with a
single test server trying to be able to communicate with existing
clients in all the environments; most of the current
configuration
would work fine except for the cert issue, which is the
sticking point
at this time. Any solution on how to handle this in the most
straightforward manner would greatly be appreciated, as my
research
has been leading to solutions far more complex than what I
would like
(such as load balancing for the CA or trying to synchronize
the certs
across the various systems).
Hopefully this made sense and I can find someone who can give
advice
on how to proceed with this issue. Thanks in advance.
- Ken Lareau
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/2BDCbntAOXoJ.
To post to this group, send email to puppet-users@googlegroups.com
<mailto:puppet-users@googlegroups.com>.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users%2bunsubscr...@googlegroups.com>.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
- Ken Lareau
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
Luke Bigum
Information Systems
Ph: +44 (0) 20 3192 2520
luke.bi...@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN
FX and CFDs are leveraged products that can result in losses exceeding
your deposit. They are not suitable for everyone so please ensure you
fully understand the risks involved. The information in this email is not
directed at residents of the United States of America or any other
jurisdiction where trading in CFDs and/or FX is restricted or prohibited
by local laws or regulations.
The information in this email and any attachment is confidential and is
intended only for the named recipient(s). The email may not be disclosed
or used by any person other than the addressee, nor may it be copied in
any way. If you are not the intended recipient please notify the sender
immediately and delete any copies of this message. Any unauthorised
copying, disclosure or distribution of the material in this e-mail is
strictly forbidden.
LMAX operates a multilateral trading facility. Authorised and regulated
by the Financial Services Authority (firm registration number 509778) and
is registered in England and Wales (number 06505809).
Our registered address is Yellow Building, 1A Nicholas Road, London, W11
4AN.
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
