Not sure if you'd call it a "best" practice, but with a fairly standard control repo(1) and r10k'ish environments you can:
* branch * make your changes in the new branch/environment * move a few canary hosts into the new environment using your ENC(2), see how that goes * move a few more, see if you are comfortable with the change * repeat until you're comfortable * merge the control repo branch and move hosts into the live environment with your ENC Another alternative, that I have never used, is to temporarily configure your environment for slower agent run cycling, say hours instead of 30 minutes, that would slow things right down. Similarly to option one, you could do the change in two phases: * one change pushes up the code to make the change, but turned off behind a feature flag * another set of actions activates the change in different parts of your infrastructure (probably using some hiera key) It's late and Friday, I'm probably out of ideas by now. (1) https://github.com/puppetlabs/control-repo (2) https://puppet.com/docs/puppet/5.5/nodes_external.html On Tue, 2019-05-07 at 08:57 -0700, Iakov Gan wrote: > Hi, > > I wonder are there any best practices for deployment of changes in a > large puppet environment? > > Once we change puppet code all changes are applied within 30 minutes > on an environment of several thousand hosts across multiple > geographical zones. An automation power in its scaring beauty. > Surely it is tested in preproduction and dev envs before, but still, > there is a risk when applying it on all zones almost the same time. > > What I look for is to deploy changes progressively per equivalent of > Amazon's Availability Zone via a pipeline, in order to control the > propagation of system changes. Surely it will make a change slower > but it is a price to pay. > > Any best practice about it? > > Best regards, > Iakov > > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-users/18f24142-226b-4f10-98a5- > 6db992f4a005%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1557523314.16997.2.camel%40pobox.com. For more options, visit https://groups.google.com/d/optout.
