On Jun 29, 2015, at 8:43 AM, Raphaël Pinson <raphael.pin...@camptocamp.com> wrote: > > Hello, > > > I've activated caching on our Puppetservers, using the admin API to > invalidate the cache upon deploying new environments. However, this only > caches manifests, and catalogs still need to be compiled for every request. > > I'm thinking (at least in our case) it wouldn't be totally crazy to cache > catalogs on the master so long as: > > * manifests are not changed (this is taken care of by the r10k hook + admin > API) > * data do not change (same, since we deploy hiera data with r10k) > * facts do not change. > > > Obviously, *some* facts always change (uptime, memoryfree, swapfree, etc.), > but most of them don't. So the idea would be to add a parameter in > puppet.conf with the name of these facts that should be used as a basis for > invalidating the catalog, and use the other facts to decide when a catalog > should be recompiled. > > Is there already some kind of code doing that, or any opinion/feedback on > this idea?
This is something that our team at Puppet Labs has been working on a ton. It’s beneficial in the short term, for the kind of performance and other benefits you describe, but it’s also key in a bunch of other cool stuff we’re doing. The short answer is that in some ways it’s quite easy, but it also requires some changes to the core that aren’t necessarily as easy. Eric Sorenson is lead on the work (code-named Direct Puppet), so hopefully he’ll chime in with more details. The basic idea, though, is that we do a few things, all together (note that this is from memory, and I’m sure I’m missing pieces or getting some of them wrong): * Make cached catalogs more valuable by changing reference-by-url of files to reference-by-content (so updated files on the server don’t change catalog behavior) * Have the client always check to see if its catalog is still valid, or if it should download a new one; this will result in the client defaulting to reusing its catalog in most situations * Provide simple mechanisms on the server for indicating when catalogs are out of date. I believe in the first release or so we’re providing a big huge boolean that just resets all catalog staleness (e.g., if you push your code to the server, there will be a command you run on that server that resets the ‘latest catalog’ date, so all existing catalogs will be considered stale) All of this together should mean that clients only download catalogs when you’ve pushed code (or made some other change, and then reset catalog freshness), which means you can often dramatically simplify your canary testing mechanisms for some use cases (push code to server; update catalogs on a couple of hosts; update on all hosts if it works), you’ll get a huge performance boost because you’ll only compile when needed, and everything will just make more sense. AFAIK all of the above is in the core product and thus is OSS, so its lack of openness is just laziness on our part (it’s hard enough to explain it to 5 people, much less the whole list), but maybe this will prompt us to publish it a bit more. We’re also relatively early on getting it all lined up, so it’s still a bit in flux. There are some other pieces necessary to put it all together, but we’re not quite ready to talk about those yet. Hopefully the above is exciting enough. :) -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/62B25B6E-797A-4CA2-A610-5A9F555203CA%40puppetlabs.com. For more options, visit https://groups.google.com/d/optout.
