On Wed, Aug 28, 2013 at 7:05 AM, Henrik Lindberg < henrik.lindb...@cloudsmith.com> wrote:
> I have read this, and the comments made to date, and it is somewhat > difficult to understand exactly what someone means as we use language that > is fuzzy (at least to me). > > Here is an attempt to define the terms (after that I have a proposal). > > Parsing > ------- > The part of the process that goes "from source text to AST model". > > Validation > ---------- > Checks/asserts the validity of the AST model. > > Loading > ------- > Resolves symbolic name to something that can be evaluated. (i.e. AST model > or Ruby, or whatever we may invent in the future). As an example this binds > the name of a hostclass to the block of code that is the class' body. > > Linking > ------- > Resolving name to object references. This is not done in puppet as a > separate static step, it is done while evaluating. > > Evaluation > ---------- > Evaluates the loaded logic (i.e. visits AST nodes and performs operations > or calls Ruby). > > Compilation > ----------- > The act of loading a given start point and evaluating it (and its > transitive dependencies) for the purpose of compiling a catalog. > > Deferred Evaluation > ------------------- > We have deferred evaluation of language constructs that define classes > (and custom resource types? I have to check) - or rather, when evaluated > they only define the mapping of symbolic name to code to evaluate on demand > (either a singleton evaluation (class), or a potentially multiple times > (resource). > > (In puppet a hostclass is not evaluated, instead there is a search for > instantiable objects, these are transitively instantiated on "loading". > Later it's "code" (body) is evaluated). > > (In contrast the term "lazy loading" throws me; what is it that is lazy? > The parsing, the binding of name to code, or the evaluation of the bound > code?). > "Lazy loading" as I was using it is exactly what it would mean in your terminology. The autoloader allows us to ask for a symbolic name and get the AST node for that back. It is "lazy" because the autoloader maintains a cache and will only populate the entry for a symbol when it is requested. At that point it performs the parse, validation, and finally the loading. There are other complications in it, but that is basically what it does. > > Proposal > ======== > To me, the problem we are discussing is that "autoloading" performs > evaluation of an unlinked model. The result therefore depends on the > transitive dependency graph of resolved links. We cache the result and then > try to figure out what needs to be invalidated based on a changed file. > > Nope, autoloading does not perform evaluation. Autoloading simply loads the file from disk and parses it, producing the AST and then returning the section of the parsed information that corresponds to the requested symbol. > At the other extreme, if we cache nothing, manifests are processed from > scratch for every request, we have a potential long startup. > > Yes, I don't think that is a good idea. > A simple solution is to cache the validated parse result. This is a simple > mapping from source URI (e.g. a file path) to an AST. This is always a 1:1 > mapping - the source and the AST are two different representations of > exactly the same thing. Then when we evaluate, we always evaluate > everything. There is one special case, when none of the files have changed > there is an opportunity to avoid recomputing the catalog, but it assumes > that no external data has changed. (There are several different ways to > deal with such optimizations including asking something external "have > something changed" to using "valid until" information in the external touch > points). > > Yes, those are optimizations that I think are out of scope for now. > Yet another problem is a change in files "mid-transaction". We could solve > that by performing a scan of the system, noting all potential URIs > affecting the result and their "expiration-timestamp" (no parsing takes > place). If we during the evaluation finds a change in timestamp we fail the > transaction (or restart it (backing off in time and having a cap on retries > if we want to be fancy)). > > This is a pretty serious problem with the common deployment mechanisms that people use, combined with the way that puppet is loading manifests from disk. I don't think failing the transaction and trying again is a good idea, unless we add all sorts of other mechanisms to make it so that custom functions can also be made transactional. > I use the term "URI affecting the result" to mean a reference to a .pp > source file, data-bindings in some form, or an external service (proving > say ENC data/bindings)), or similar. > > I think the above is a combination of "autoloading" and "load everything > up front". > > I would like to get rid of "import" because it is path based, not because > it "imports" (loads code). I.e. I think we should have a loader that > resolves symbolic names to URIs and loads evaluatable content. > This loader should be able to search for what to "run" without having to > resort to explicit "run this path" - if not then there is IMO something > missing in the language itself. I can live with the entry point being a > file (e.g. site.pp), or possibly a set of files if users for some reason > want to split a site.pp into multiple files. > > I don't understand. We have a loader that resolves symbolic names to URIs (that is part of the autoloader mechanism). We also have a mechanism that is for pulling in other manifests, during the "parse" phase, by filename (actually it also supports a URI-like scheme, but I'm not sure if that was intentional). This mechanism, as implemented, is causing problems. The idea behind it can be made to work, but I think that we can step back, make things less "auto", more "static", and end up with a system that is much easier to move forward with. We might move back into loading the AST at the time that the evaluator asks for a symbol, but that would be a memory optimization for the master, and a performance optimization for the interactive use. What I'm thinking (in terms of a workflow a user might perform): # produce a fully validated AST serialization $ puppet manifest compile --manifest site.pp --modulepath ~/modules:/etc/puppet/modules --output my-puppet-code.ast # load that validated AST as a particular environment into the master. The master would swap in the new code for that environment only at the start of a new compile, any existing compiles using the old code loaded for the environment would continue unchanged. $ puppet master load --pic my-puppet-code.pic --environment staging This is just an example of how it could be split up into multiple commands. This could also be done at the beginning of the catalog request instead of by the user. At the moment, I'm ignoring any data that would need to go along with this. Taking that into account might change things as separate commands. - henrik > > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to > puppet-dev+unsubscribe@**googlegroups.com<puppet-dev%2bunsubscr...@googlegroups.com> > . > To post to this group, send email to puppet-dev@googlegroups.com. > Visit this group at > http://groups.google.com/**group/puppet-dev<http://groups.google.com/group/puppet-dev> > . > For more options, visit > https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> > . > -- Andrew Parker a...@puppetlabs.com Freenode: zaphod42 Twitter: @aparker42 Software Developer *Join us at PuppetConf 2014, September 23-24 in San Francisco* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To post to this group, send email to puppet-dev@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
