On Mon, Aug 26, 2013 at 11:15 AM, Luke Kanies <[email protected]> wrote:
> On Aug 26, 2013, at 10:49 AM, Andy Parker <[email protected]> wrote: > > > The conclusion that we came to was that the current lazy-loading of puppet > manifests needs to go away. Lazy loading makes all of the information to > correctly load types at the right time and from the right place very > difficult to keep track of (not intrinsically so, but in our current state). > > I think the system needs to change to eager loading of manifests (not > applying them all, but at least loading them all). For the development > case, this makes things maybe a little more expensive, but it should make > the stable, production case for manifests much faster, because it will > rarely, if ever need to look at the filesystem to find a type. > > Now the problem is that if we start going down this path, it becomes a > large change to the underlying architecture of the compiler. It will be > unnoticeable to most users from a manifest standpoint (unless somehow they > were able to rely on certain manifests never being loaded), however we may > need to make changes that will break code at the ruby level (maybe the > testing wrappers, maybe types and providers, probably some functions). > > > This seems like a big conclusion for something that appears to have only > been investigated relatively recently, so I'd at least like to make sure we > understand it more deeply before finalizing. > > It has actually be a recurring problem that has come up at other times, in other forms, for other issues. A lot of the performance work has been hacks on top of the lazy loading functionality to make it perform better. There have been bugs around losing track of the environment before and not being able to find the classes. Sorry that I don't have the bug numbers available right now. > Is the problem that lazy loading itself is impossible, or that the > implementation of lazy loading is the problem? I could easily see the > latter. > > There is nothing wrong, in practice, with lazily loading in this case. In fact it is required for the ruby DSL, since simply the act of loading the .rb file can have consequences (ruby classes being created, files being read, etc.). A lot of the problems that we encounter around this are related to the implementation, however sometimes simply the more complicated implementation needed for lazy loading is part of the overhead of debugging. However, there are ways to implement this so that those complications are minimized. > How would you handle reloading in this scenario? > > I can think of two ways: track all of the files loaded and reload when something changes or is added, reload only when told. I lean towards the "reload only when told" for a production master, since then the runtime behavior of it is much more predictable in time (no more stating) and in catalogs produced (no more ending up with half of one manifest version and half of another). > Do you have a sense for what the performance difference would be for > someone using Puppet interactively, especially with a smaller config set > (i.e., where they're only using a couple % of the code)? Is there some > threshold you'd want to make sure we stay within? > > Not at the moment. I haven't tried this out yet. It all really depends on how fast the parser is (not the interpreter, just the parser). There of course would be work arounds to reduce the amount that it tried to load by setting up more specific modulepaths and such, but that would put the onus on the user, which is not ideal. I really wouldn't want to see any major slowdown in the normal "try something out" cycle, since puppet is already a bit slow there. However, I think this does open up avenues for fixing that as well. There could be a "build" step in front of the interactive step (maybe even transparent to the user) that deals with not reparsing manifests that aren't changing. That would then also gives a "compiled bytecode" manifest that could be deployed to masters reducing the work that a master needs to do even more. > -- > Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | > +1-615-594-8199 > Join us at PuppetConf 2013, August 22-23 in San Francisco - > http://bit.ly/pupconf13 > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-dev. > For more options, visit https://groups.google.com/groups/opt_out. > -- Andrew Parker [email protected] Freenode: zaphod42 Twitter: @aparker42 Software Developer -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
