For canaries, we have some 'dev' versions of services, i.e. 'wiki' and 'wikidev'. Because of that, we can both leave the 'dev' versions on a feature branch if we need to (or disable puppet entirely when we're down in the nitty gritty and don't want puppet to undo development efforts) AND have a tier in hiera where we can provide different values to the regular and dev versions. We mostly rely on the hiera tier for data because then there's less manual or even orchestration effort to test on those nodes, except when we want a run immediately.
Rob Nelson rnels...@gmail.com On Mon, Aug 22, 2016 at 12:49 PM, Chadwick Banning < chadwickbann...@gmail.com> wrote: > Another potential management approach I'm exploring... > > Because of the complexity of my current situation, where we have multiple > independent internal apps and each one of these has a more-or-less unique > set of operational environments, AND we have common Puppet code that > applies across everything, I'm looking into spinning off each app team into > their own module and using the Puppet 4 data-in-modules to allow them to > manage their own data/hierarchy. I'd then use the controlrepo as the > integration point for these internal modules as well as the container for > the common Puppet code. > > I'm just now exploring this idea so I'm not ready to say whether it's a > good idea or not! > > I like the idea of doing A/B testing on feature branches off of > production. In my situation, I'd have to have a pretty robust and visible > way to test and report success/fail before the security team buys into it. > Especially if this was automated such that canarys are moved to feature > environment, test is performed, and on success feature is merged and > canarys are moved back to production environment. > > On Mon, Aug 22, 2016 at 12:30 PM, Rich Burroughs <r...@richburroughs.com> > wrote: > >> I've managed a lot of configs for apps our developers build. At times >> when config changes were not backwards compatible I built in the equivalent >> of feature flags. I added a Boolean param that controlled if it was on or >> off, and built logic into the ERB/EPP template that used it. Then I could >> control the rollout by setting the value for that param in Hiera. >> >> Not my favorite thing to do but it worked :) And of course the bad part >> about feature flags is remembering to clean them up when the rollout is >> done. >> >> I'm really enjoying this discussion. I haven't done A/B or canary nodes >> with Puppet and I think some of this stuff is hard to figure out if you >> haven't seen it work. >> >> >> Rich >> >> On Mon, Aug 22, 2016 at 7:06 AM Rob Nelson <rnels...@gmail.com> wrote: >> >>> I think there are so many ways to do A/B deployments that it's probably >>> not wise to codify that into your Puppet environments layout. Here's a few >>> examples of how: >>> >>> * Use DNS/Load Balancer rules/etc to control which puppet masters >>> different agents connect to. Some of the masters have a different default >>> environment, to force some percentage of nodes to run against your feature >>> branch and receive the correct environment. After the feature is merged >>> into production, those masters default back to the `production` environment. >>> * Your ENC provides the puppet environment to check into, forcing some >>> percentage of nodes to receive the `feature` environment, and later to >>> return to the `production` environment. >>> * Use any orchestration solution (mcollective, ansible, vRO, etc) to >>> configure agents to check in against the `feature` environment, and later >>> to return to the `production` environment. >>> * Manually check in canary nodes against the `feature` environment, and >>> later to return to the `production` environment. >>> >>> That's just a sampling of methods in no particular order (although I >>> think we can all agree that the manual option is the least appealing) but >>> is by no means comprehensive. However, all of these methods let you do A/B >>> testing between production and a specific feature branch, rather than >>> against a long lived and divergent branch than production which may have >>> numerous changes that cause interaction in unexpected ways, which certainly >>> tests a lot more but makes it more difficult for diagnosing where failures >>> come from. Having your feature branches diverge from `production`, your >>> single source of truth, greatly limits the number of variables in play at >>> any given time and streamlines your pipeline. >>> >>> I would recommend a book on Continuous Integration and/or Continuous >>> Delivery as these are subjects far too deep and wide to properly cover here >>> in the detail they require. >>> >>> >>> Rob Nelson >>> rnels...@gmail.com >>> >>> On Sun, Aug 21, 2016 at 11:20 PM, Alex Samad <a...@samad.com.au> wrote: >>> >>>> On 21 August 2016 at 11:04, Chadwick Banning <chadwickbann...@gmail.com> >>>> wrote: >>>> > As Rob Nelson mentioned above, you can differentiate between >>>> operational >>>> > environments in Hiera as long as you have the appropriate facts >>>> available. >>>> > >>>> > If you differentiate Puppet environments and operational >>>> environments, then >>>> > it's easier to address staged rollouts in each appropriate context. >>>> Staged >>>> > rollouts of changes across *operational* environments may be done >>>> through >>>> > Hiera, and staged rollouts of Puppet code (usually common Puppet code >>>> that >>>> > cuts across operational environments) can be done through *Puppet* >>>> > environments. >>>> > >>>> > If your environment is simple enough...such as a single app with dev, >>>> > staging, and production operational environments, then equating a >>>> Puppet >>>> > environment to an operational environment is that much of an issue. >>>> For more >>>> > complex Puppet setups, equating them always creates issues IMHO. >>>> >>>> >>>> Okay I get it you have another branch in hiera lets say env as you have >>>> above. >>>> >>>> But they pull in the same profile class - lets say openssh. >>>> >>>> So lets say ssh come out with v3, you want to slowly roll out. >>>> >>>> Do you have a bit if then or case switch statement in your ssh profile >>>> class or do you create a new ssh class call ssh-v3 and then assign it >>>> to only the env you want to. >>>> >>>> seems like every time you want to make a change to a profile/class you >>>> need to create another one so you can control the roll out or you have >>>> a big if/then or switch case to do that for you. >>>> >>>> trying to persist with the 2 branches production and testing for now .. >>>> >>>> > >>>> > This topic is really interesting to me since I've run into it multiple >>>> > times, the last being very recent. >>>> > >>>> > On Saturday, August 20, 2016 at 6:39:03 PM UTC-4, Alex Samad wrote: >>>> >> >>>> >> On 20 August 2016 at 22:50, Chadwick Banning <chadwic...@gmail.com> >>>> wrote: >>>> >> > This is an issue I run into pretty regularly. If your Puppet >>>> >> > infrastructure >>>> >> > is even moderately complex, I'd recommend NOT equating a Puppet >>>> >> > environment >>>> >> > to an operational environment, operational environment being the >>>> groups >>>> >> > of >>>> >> > machines known as dev, qa, staging, etc. >>>> >> >>>> >> But how to you stage a roll out of an update. If you want it to go >>>> to >>>> >> dev then uat then prod ... or through some logical steps. >>>> >> >>>> >> presuming you have a common profile used by all. >>>> >> >>>> >> > >>>> >> > For instance, in my infrastructure we have 50+ different >>>> operational >>>> >> > environments. If I equate each one of these to a Puppet >>>> environment, I'd >>>> >> > need 50+ branches. While doable, this immediately becomes a >>>> nightmare if >>>> >> > I >>>> >> > have a change that applies to all or some of the operational >>>> >> > environments -- >>>> >> > say, changing something in my base profile. Now I have to a) hope >>>> all >>>> >> > 50+ >>>> >> > branches are somewhat in sync, and b) merge my change into *each* >>>> branch >>>> >> > 50+ >>>> >> > times. If the branches aren't in sync at all I very well might end >>>> up >>>> >> > having >>>> >> > to fix unique conflicts each time I merge. >>>> >> > >>>> >> > This is *not* a place where you want to end up. >>>> >> >>>> >> Yes agree sounds like it would be a nightmare >>>> >> >>>> >> > >>>> >> [snip] >>>> > >>>> > -- >>>> > You received this message because you are subscribed to a topic in the >>>> > Google Groups "Puppet Users" group. >>>> > To unsubscribe from this topic, visit >>>> > https://groups.google.com/d/topic/puppet-users/4YL6D4wwJww/ >>>> unsubscribe. >>>> > To unsubscribe from this group and all its topics, send an email to >>>> > puppet-users+unsubscr...@googlegroups.com. >>>> > To view this discussion on the web visit >>>> > https://groups.google.com/d/msgid/puppet-users/272fba59-9684 >>>> -44b1-8350-417893cbfb9e%40googlegroups.com. >>>> > For more options, visit https://groups.google.com/d/optout. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-users+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/puppet-users/CAJ%2BQ1PWS3LHBWE6Z-h8Newbo_R7qdh4qn- >>>> yADxqWMbJLv8Ga4Q%40mail.gmail.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/puppet-users/CAC76iT_1QCeuEMx0hdBf7QVfjYkBLctfsGGR8zbHAc >>> V8%2B3ppMg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/puppet-users/CAC76iT_1QCeuEMx0hdBf7QVfjYkBLctfsGGR8zbHAcV8%2B3ppMg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Puppet Users" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/puppet-users/4YL6D4wwJww/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> puppet-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/puppet-users/CAPGcbtB%3Dj-nKJgEj5VQkt2a%2BJd-Uf1ePcXdse% >> 2BV%3DdCAyh8YM3g%40mail.gmail.com >> <https://groups.google.com/d/msgid/puppet-users/CAPGcbtB%3Dj-nKJgEj5VQkt2a%2BJd-Uf1ePcXdse%2BV%3DdCAyh8YM3g%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Chadwick Banning > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-users/CAPwwW9GJ-aF5SWB5_WCc-ess35RtnHqKWOsGUBMxehcFJs7j8Q% > 40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-users/CAPwwW9GJ-aF5SWB5_WCc-ess35RtnHqKWOsGUBMxehcFJs7j8Q%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC76iT82kiDpeWKUCL0i_ff%2B%3DZs0WvcGGrJixmBj75ySJyKmnQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
