Hi Jeremiah, On Wed, Mar 30, 2016 at 11:56 AM, <[email protected]> wrote:
> I write a lot of native types for our internal use at work. (I tell > people that if you are just using Exec in Puppet that ansible is one search > away in Google. ) Some of the Puppet code used with these types would be > very challenging to "pre-compile" in any way. > > I think a lot of my questions are just because I don't consider a .pp file > to usually be the unit of function in Puppet. I only consider the > resource, which is usually implemented in Ruby, and everything else is > dressing. > > Reading over this proposal raises questions about how it will fit into > Puppet, a language and environment very different from Python, Java or > other languages (Puppet 4 long leaving the domain-specific title by the > wayside.) > > Since you have to build an AST on a node-by-node basis I am having a hard > time telling the value of this verses something like a marker that > indicates the code has been parser validated. If your code doesn't need > node-by-node rebuild then perhaps the compiler could cache the results > instead? I know the adage about about three hard problems in Computer > Science still applies but what this is doing is creating and managing a > cache. > ASTs don't need to be built on a node-by-node basis (unless you meant manifest-by-manifest basis); an AST is just a representation of the manifest's source code. The XPP file format is simply an attempt to define a serialization format for the AST itself so that, say, all the manifest files in an environment could be parsed, validated, and the resulting ASTs saved in a format that is faster to read when a catalog for a node is being compiled all in one go and "upfront" (i.e. not during the catalog compilation for any one node). In compiler terminology, there's a "frontend" and a "backend". The frontend is responsible for producing an AST that the backend can evaluate or generate code from (in the case of Puppet, the backend directly evaluates ASTs to generate a catalog). Having a well-defined AST serialization format means we could potentially swap out another implementation for the compiler's frontend, which is one of the goals of this project and a first step towards a new compiler implementation. > Pre-compiling is translating one language into another because compiling > is changing one programing language, usually higher level, into another, > usually lower level. Puppet does something very different in my > understanding given above. It sounds as if the goat of the C++ parser is > not to produce a catalog of resources but instead a ruby program for the > agent to run. > The goal of this particular initiative is to enable the C++ parser (i.e. the frontend) to interop with the Ruby evaluation implementation (i.e. the backend). The Puppet code is not being pre-compiled, but pre-parsed/pre-validated; the C++ implementation will not (yet) evaluate any Puppet code or load custom types or functions defined in Ruby. The ultimate goal of the C++ implementation is to replace both the frontend and the backend for catalog compilation, but that's a ways off still as there needs to be design and implementation around loading custom types and functions from Ruby source to maintain backwards compatibility. > From the statements about a C++ Puppet parser is the target still just a > collection of resources? Or is the goal to eventually spit out something > other than what the Puppet server should send to a node? > The target is still the same thing we send to agents now: a resource catalog. That won't change when the C++ compiler implements the backend too. > Is the scope just 'closed' .pp files? That is classes where all variables > can be resolved without inquiry to facts? The behavior of languages that > support things like pre-compiled files is specific to how they do binding > of missing data. While this proposal punts on the 'serialization' format > the handling of binding is pretty central. That raises questions like: > > How can a compiled format deal with Puppet features that call APIs? Is > this for the defined types? How will it deal with the fact that the code > could radically change based on node data not even present at the time the > pre-compile is done? > > What happens if a precompiled module that depends on PuppetDB information > is moved to an environment without it? For that matter, is the format > intended to move between Puppet installations or just very similar ones? > > My model of Puppet is that the compiled catalog is just an ordered list of > resources which do not have any placeholders like variables or > sub-dividable parts. Classes in this model are just names that help you > find the resources. > > This format is relevant to recent discussions among people on IRC and in > the mailing lists and groups about getting and storing the catalog for a > node. This is historically something hard as an end user. > > The context of any given resource is the whole catalog and at a minimum > requires the dependent and depending classes in the graph tree. Otherwise > how does this deal with the unresolved parts? > The above seems to be confusing, understandably so, pre-compiling a resource catalog with pre-parsing a manifest. In terms of a language like Python, the "pre-compiled" pyc files are simply a representation of the source that is more efficient to load and execute than having to parse the Python source again. This is analogous to what we're calling pre-parsed XPP files: a representation of the Puppet source code that enables faster evaluation (i.e. during resource catalog "compilation") and also enables the existing Ruby implementation to load and evaluate even if a entirely different parser implementation was used to generate them. Hope this helps. Peter > > On Wednesday, March 30, 2016 at 11:24:46 AM UTC-5, Eric Sorenson wrote: >> >> Hi, I've just posted a new Puppet RFC that describes pre-parsed and >> pre-validated Puppet files, akin to '.pyc' files for Python. It's called >> XPP and the doc is open for comments here: >> >> >> https://docs.google.com/document/d/17SFn_2PJYcO5HjgA4R65a5ynR6_bng_Ak5W53KjM4F8/edit?usp=sharing >> >> Please comment inline on the doc, or come back to this thread if the >> conversation gets too involved (more than about 4-5 replies in a comment >> box on google docs becomes unwieldy) >> >> Once the commenting tapers off we'll incorporate changes into the spec >> and post it as markdown in the puppet-rfc repo: >> https://github.com/puppetlabs/puppet-rfc >> >> --eric0 >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CACZQQfOhsBHcfjdoh9%3DyZS4YGUXQtnj6HRBTqLZiRr_CiueeNQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
