On Wed, 2014-02-12 at 15:15 +0100, Henrik Lindberg wrote: > On 2014-12-02 9:43, Brice Figureau wrote: > > On Wed, 2014-02-12 at 00:13 +0100, Henrik Lindberg wrote: > >> On 2014-11-02 17:25, Brice Figureau wrote: > > >>> I'm currently working on marking all resources with a kind attribute > >>> that can be either node, class, definition or builtin. This property > >>> gets sent over the catalog (which requires a schema modification, so > >>> might not be mergeable in a 3.x release). This allows the client to not > >>> try to guess a given resource type. > >>> I'm marking those resources at creation time (so the settings resource > >>> for instance get marked as builtin), and try to lookup the type of the > >>> resource accordingly (defaulting to the previous way of doing things if > >>> the resource is not marked). > >>> > >>> Unfortunately, I discovered that Puppet::Resource was so central in > >>> Puppet that everyone creates new instances of those for various > >>> completely different task, and most of the time it's not possible to > >>> know in advance the resource kind. > >>> > >>> The last one I found is when looking up a resource in a given catalog, > >>> we create a blank resource of the given title to be able to get a > >>> properly munged title back to perform the look up. I find that > >>> particularly bad performance-wise, creating so many transient objects > >>> (might be better with latest ruby GC though). > >>> > >>> I hope to have a PR ready by the end of the next week end, but at least > >>> you know what I'm trying to do :) > >>> > >> > >> Discussed this with zaphond42 today and I want to try an approach that > >> is different - since we will be visiting all directories at least once > >> when looking for resource type implementations in ruby, we could just as > >> well just cache their names up front. That way, subsequent searches are > >> simply a hash lookup, and we can continue to lookup ruby plugin resource > >> types before user defined. > >> > >> This works when we change the semantics to perform a stale check only > >> before each request is processed. I am counting on that there are far > >> fewer plugin types than there are definitions. > >> Since the plan is to not load all of them upfront, but only record their > >> existence the entire operation should be fast enough even if it will map > >> unused types. > > > > That sounds great, if it solves the agent case too. IMHO we are focusing > > a bit too much on the master performance :) > > > > I think it will. > > >> Lookup of user defined definitions could also benefit from similar > >> caching of directory entries - with the goal of only visiting each > >> directory once, but it is a bit more complicated to implement. > > > > The current way of loading definition works fine, because it is done > > during parsing, so the resource lookup is just a hash lookup, it doesn't > > involve reparsing everything. In the agent, the known_resource_types is > > empty and that means we constantly hit the type loader. If the type > > loading scheme you propose also works in the agent, then we'll be mostly > > fine except we won't be able to cache the definitions resources types > > (and thus will hit the type loader every time). That was the root of my > > ideas of marking resources, because once we know what kind a resource > > is, everything becomes simpler, as you don't have to guess what kind of > > type your resource is, and it can be looked up more easily. > > > Not sure what you are saying about known_resource_type - are you saying > it is always empty? What I am proposing will result in 1 scan of all > plugin resource types - then when a lookup of a type name is performed > it first checks if this could be a plugin type (one hash lookup), if not > then it must be a user defined type. Is there something in that you > suspect will not work on the agent side?
I meant that on the agent known_resource_type is always empty. That's the root of the issue I discovered in PUP-1592: we're constantly calling Puppet::Type.type() because we can't cache the resource_type result. The reason is Puppet::Type.type() returns nil (and scans all the paths each time) because this is a definition, and the last resort check (known_resource_type.definition) also returns nil because this hash is valid only on the master, so ultimately we return nil (which is not an issue in the agent) which by definition renders the cache unuseful. So yes, your idea will make sure we don't scan the disk (except once), but we'll still not cache the resource type and call Puppet::Type.type everytime. If your solution makes it only a negative cache lookup, I think this is quite safe (on a performance standpoint). -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/1392221779.29754.30.camel%40arsenic.daysofwonder.com. For more options, visit https://groups.google.com/groups/opt_out.
