I was curious while reading this, it seems like we're touching on a few different potentials.
1) Sending catalog diffs. 2) Asynchronous catalog compilation. I think that #2 might be critical to scaling and sort of mirrors what Peter was doing with the CM Guard. Basically, instead of the Puppet client sending facts to the server when requested, you have a fact collector that the clients send their facts to on a regular basis. The compiler then uses the last good set of facts to compile the catalog (compiler farm?) and then ship it off to the delivery cache. The client would be requested to update its facts if it were outside of some set good threshold. I think that this would be amazing if implemented simply and without adding any new data transport layers. Thanks, Trevor On Fri, Jul 3, 2015 at 12:34 PM, Peter Meier <peter.me...@immerda.ch> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi All, > > > Is there already some kind of code doing that, or any > > opinion/feedback on this idea? > > I have written once a change management guard, that does some of the > stuff you mentioned and as a side-effect also includes caching - or I > abused caching for change management... > > https://github.com/duritong/puppet-cm_guard > > Its main target was a 2.7 installation and it's still being used there > (afaik). Nevertheless, it should also work on a 3.x installation, no > idea about 4.x. I never tested that to the extend and didn't touch the > code since 2 years, so I don't give any guarantees. Still I think it > might be interesting for the current discussion. > > I see 2 main points: > > 1. invalidations: As you mentioned there are tons of effects that can > play into invalidating the cache and imho it is highly depending on > the specific puppet setup and all the different features it uses. So, > for the above implementation, I took the approach that there is a > simple method that needs to be implemented and the contract is, that > puppet passes all the information it has about such a node (mainly all > the facts the client sent) and the method should either say true or > false to indicate if a recompile is needed. > And you as a user of that plugin would be able to do whatever you want > within that invalidatior, e.g. measuring outside temperature or roll a > dice. > For a potential general solution I see such a method of contributing > to the invalidation process as crucial. There should be a sane and > useable default, that works in 80% of the cases, but it is freely > tuneable to also be able to address the other 20% (or at least 18%). > Providing a way of hooking into the decision and contributing to the > decision making process with whatever you like, is the solution we > should imho aim for. > > 2. not everything is within the catalog: plugins, file sources, result > or side-effects of function calls. > > 2.1 Plugins are synced at a time when the catalog is not yet in the > game, nevertheless they potentially affect catalog compilation > (new facts, new types, etc.). For my use case I was able to > ignore that fact, as it was enough to make people aware of that > problem. But still the fact that with my solution the cached > catalog you get and the plugins you might have gotten earlier can > be out of sync might be a problem in some cases. > > 2.2 file sources: They are not part of the catalog and I > tried to use the static compiler to get the content of the > sources into the catalog so we just have one huge static blob. > However, this might not work in all cases or at least is tricky > (e.g. recursive file resources). > Not being able to reference a certain file source version to a > certain catalog might change a file without doing other required > changes, that would be within the new catalog and hence cause > other problems. Imho this is one of the biggest problems and we > should definitely try to adress it. Unfortunately, it is also > quite a hard problem. > > 2.3 result/side-effect of function calls: They are also contributing > to side-effects in a similar way as new plugins, but the change > of a result might also be predicatable through an external > invalidation hook. E.g. hiera files that changed etc. > > The more you dive into the topic the more very tricky problems are > showing up and imho the topic of caching catalogs is one of the harder > problems to solve. However, I think these hard parts are because we > would like to address all/most possible cases that we can think of at > the moment. So maybe, we should not try to come up with the > complete/ideal solution from the beginning on, but rather a solution > that works in most ideal cases, that have clear outlined restrictions > and limitations and work from there on forward tackling the much > harder problems, while still getting feedback from the early usage of > the already released but minimized feature set of such a solution. > Also making it possible for people to hook into that process will > allow them to play around with/extend the feature by themselves and > empowers them to contribute to the final solution. > > best > > peter > -----BEGIN PGP SIGNATURE----- > > iEYEARECAAYFAlWWuYIACgkQbwltcAfKi38o6gCgsfOb7pzGolocFtwYgMZFoLK9 > 7+kAn1+hZdTxOG0JyFc3MjZmx0KF4l4I > =pi3j > -----END PGP SIGNATURE----- > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-dev/5596B987.5090900%40immerda.ch > . > For more options, visit https://groups.google.com/d/optout. > -- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWuVD_AUVHOxex6J-0q4KxWsNo%3Dyyc%3DTeD-kO%2BJuL%2B6RA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
