On Thu, Mar 13, 2014 at 5:19 PM, Henrik Lindberg <
[email protected]> wrote:
> Hi,
> we are just started to get more concrete on how to implement things for 4x
> and breaking it down into actionable items. If you have looked in Jira,
> there are currently 5 big issues in the epic "Biff the Catalog Builder" [1]
> - which is the goal (a new, better performing catalog builder (what is
> currently known as the "compiler") where we can fix many known issues that
> today are just to hard to implement.
>
> This time, I want to talk about the implementation of Scope, which is
> part of "(PUP-1832) Implement the Puppet 4.0 Runtime" [2].
>
> Currently scope has many responsibilities (too many):
>
> * it is classic computer language scope (what is visible "here")
> * for a class it also represents one aspect of "an instance of a class"
> (the attributes of the class are variables in that scope).
> * Inheritance is achieved by looking up and continuing the search for a
> variable in another "scope".
>
> Coming up with a new implementation is important to make scope perform
> well. Thus it is important to know:
>
> - write vs read ratio
> - unqualified vs. qualified lookup (i.e. reading $a:.b::x from within
> $a::b vs from other scopes)
> - typical nesting levels of named scopes
>
> We also have to decide if any of the relative name-space functionality
> should remain (i.e. reference to x::y is relative to potentially a series of
> other name spaces ("dynamic scoping"), or if it is always a global
> reference when it is qualified.
>
>
Other than using the search() function (which I'm not sure it even works),
do you have an example of this happening? I can't seem to cause it to
occur. If it doesn't really exist, then I'm not too concerned about
dropping it :). I know that the relative name lookup happens when finding
classes (include(b) can refer to a::b), but haven't been able to get it for
variables yet.
> The implementation idea we have in mind is that there is one global scope
> where all "qualified variables" are found/can be resolved, and that all
> other variables are in local scopes that nest. (Local scopes include
> ephemeral scopes for match variables).
>
> Given the numbers from measuring the read ratio, we (sort of already know,
> but still need to measure) need a fast route from any scope to the global -
> we know that a qualified variable is never resolved by any local scope so
> we can go straight to the global scope. (This way
> we do not have to traverse the chain up to the "parent most" scope (the
> global one). Local scopes are always local, there is no way to address the
> local variables from some other non-nested scope - essentially how the
> regular CPU stack works, or how variables in a language like C work).
>
> i.e. we have something like this in Scope
>
> Scope
> attr_reader :global_scope
> attr_reader :parent_scope
> # ...
> end
>
>
> The global scope keeps an index designed to be as fast as possible to
> resolve a qualified name to a value. The design of this index depends on
> the frequency of different types of lookup. If all qualified lookups are
> absolute it would simply be a hash of all absolute names to values (it
> really cannot be faster than that).
>
> The logic for lookup then becomes:
> - for un-qualified name, search up the parent chain (this chain does not
> reach the global scope), if still unresolved, look in global scope.
> - for qualified name, look up in global scope directly
>
> If we need to also consider relative namespaces (i.e. x::y could mean
> z::x::y, or a::b::c::x::y etc. we can then either probe in turn with each
> name (which is fine if the number of things to probe is low), or provide a
> reverse index where y is first looked up to get the next level of names,
> etc. (the idea being that this requires fewer operations to find the right
> one).
>
> IF we can completely remove the notion of relative namespacing we gain
> performance!
>
> The global scope, in addition to having the qualified names also needs to
> separate the names by "kind" since we can have the same name for different
> "kinds". We can now keep keep all named things in the global scope -
> functions, types, variables, etc. Global scope and loading are
> associated (more about loading in a later post) but it is worth noting
> that it may be of value to be able to record that there has already been an
> attempt of loading a particular name, and that there was nothing there to
> load...
>
> We are going to need the following kinds of scopes:
>
> * Global Scope - holding map from kind, to fully qualified name to value
> * Local Scope - holding variables that shadow parent scope
> * Ephemeral / Match Scope (read only) - when a match is made
> * Class Scope - the topmost scope for a class - needed because variable
> lookup in it, and its nested scope needs to lookup all class attributes
> (and defined them) via reading/setting variables.
> * Resource Scope - the topmost scope for a user defined resource type -
> needed because its parameters are available as read only variables.
>
> The resource scope simply makes the resource parameters available. It
> behaves as a local scope otherwise.
>
> The class scope looks up unqualified variables in the class itself, if not
> found there, it continues up the parent chain of scopes. If the class
> inherits from another, then, the parent scope is one that represents its
> super class.
>
> In class scope, setting a variable also means that it is set in global
> scope with the fully qualified name. This is where the logic around class
> private variables comes in. If it is private, it cannot be accessed from
> the outside (i.e. with a qualified name), and thus it
> is only set in the class / class-scope. This in turn brings up the issue
> of also supporting "protected" variables; only visible from within the
> class logic, and the logic in sub classes, and if subclasses should see
> private inherited variables or not (probably not).
>
> The above could probably do with some picture :-)
>
> Now, some questions...
>
> - Are there any particular performance concerns you think we need to be
> aware of?
> - Do you have concerns about things we missed? Something important scope
> needs to do?
> - Do you have metrics from your environment? (number of lookups of various
> kinds, etc)
> - What is your reaction to getting rid of dynamic/relative name
> resolution? (Breakage vs. sanity...)
>
> Regards
> - henrik
>
> Links
> ---
>
> [1]: https://tickets.puppetlabs.com/browse/PUP-1789
> [2]: https://tickets.puppetlabs.com/browse/PUP-1832
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/puppet-dev/lfthtr%24vnh%241%40ger.gmane.org.
> For more options, visit https://groups.google.com/d/optout.
>
--
Andrew Parker
[email protected]
Freenode: zaphod42
Twitter: @aparker42
Software Developer
*Join us at PuppetConf 2014, September 23-24 in San Francisco - *
http://bit.ly/pupconf14
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CANhgQXuPnHBoDDQwJXxReESeCJoKzN0NBGgKkQd9fhTVfvZqAg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
