On Aug 26, 2013, at 10:49 AM, Andy Parker <a...@puppetlabs.com> wrote:
> Adrien put a lot of effort into tracking down what was happening in #15106 > (Missing site.pp can cause error 'Cannot find definition Class'). That exact > issue, as described in that bug, has been fixed, but in the investigation > Adrien figured out that there are a lot of other problems that can crop up > (https://projects.puppetlabs.com/issues/15106#note-13). > > Basically it comes down to the way puppet tracks what is loaded, what can be > loaded, and when things need to be reloaded. When compiling a catalog from > manifests, the autoloader (for puppet types, not for ruby code) will be > invoked at various times to parse the .pp files that it thinks should contain > the types that are needed. At the same time it caches what it has already > parsed in a Puppet::Resource::TypeCollection, which throughout the code is > known as known_resource_types. There are also a few cases where the > TypeCollection will be cleared, even part way through a compile, that causes > it to start reloading things. > > Charlie Sharpsteen, Adrien, and I talked about this around a week ago, before > puppetconf and came to the conclusion that the current method of autoloading > puppet manifests and tracking known types is just untenable. There are > multiple points in the code where it loses track of the environment that it > is working with, trying to pass that information through (I tried it a few > days ago) ends up uncovering more issues. > > The conclusion that we came to was that the current lazy-loading of puppet > manifests needs to go away. Lazy loading makes all of the information to > correctly load types at the right time and from the right place very > difficult to keep track of (not intrinsically so, but in our current state). > > I think the system needs to change to eager loading of manifests (not > applying them all, but at least loading them all). For the development case, > this makes things maybe a little more expensive, but it should make the > stable, production case for manifests much faster, because it will rarely, if > ever need to look at the filesystem to find a type. > > Now the problem is that if we start going down this path, it becomes a large > change to the underlying architecture of the compiler. It will be > unnoticeable to most users from a manifest standpoint (unless somehow they > were able to rely on certain manifests never being loaded), however we may > need to make changes that will break code at the ruby level (maybe the > testing wrappers, maybe types and providers, probably some functions). > > I think something this large should be an ARM, but I wanted to put this out > here to get some feedback before working up an ARM. Maybe we are missing > something and we can salvage this without a larger change, but at the moment > I'm skeptical. This seems like a big conclusion for something that appears to have only been investigated relatively recently, so I'd at least like to make sure we understand it more deeply before finalizing. Is the problem that lazy loading itself is impossible, or that the implementation of lazy loading is the problem? I could easily see the latter. How would you handle reloading in this scenario? Do you have a sense for what the performance difference would be for someone using Puppet interactively, especially with a smaller config set (i.e., where they're only using a couple % of the code)? Is there some threshold you'd want to make sure we stay within? -- Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199 Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13 -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To post to this group, send email to puppet-dev@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
