Hi Jeremiah, On Wed, Mar 30, 2016 at 11:56 AM, <jeremiah.pow...@seagate.com> wrote:
> I write a lot of native types for our internal use at work. (I tell > people that if you are just using Exec in Puppet that ansible is one search > away in Google. ) Some of the Puppet code used with these types would be > very challenging to "pre-compile" in any way. > > I think a lot of my questions are just because I don't consider a .pp file > to usually be the unit of function in Puppet. I only consider the > resource, which is usually implemented in Ruby, and everything else is > dressing. > > Reading over this proposal raises questions about how it will fit into > Puppet, a language and environment very different from Python, Java or > other languages (Puppet 4 long leaving the domain-specific title by the > wayside.) > > Since you have to build an AST on a node-by-node basis I am having a hard > time telling the value of this verses something like a marker that > indicates the code has been parser validated. If your code doesn't need > node-by-node rebuild then perhaps the compiler could cache the results > instead? I know the adage about about three hard problems in Computer > Science still applies but what this is doing is creating and managing a > cache. > ASTs don't need to be built on a node-by-node basis (unless you meant manifest-by-manifest basis); an AST is just a representation of the manifest's source code. The XPP file format is simply an attempt to define a serialization format for the AST itself so that, say, all the manifest files in an environment could be parsed, validated, and the resulting ASTs saved in a format that is faster to read when a catalog for a node is being compiled all in one go and "upfront" (i.e. not during the catalog compilation for any one node). In compiler terminology, there's a "frontend" and a "backend". The frontend is responsible for producing an AST that the backend can evaluate or generate code from (in the case of Puppet, the backend directly evaluates ASTs to generate a catalog). Having a well-defined AST serialization format means we could potentially swap out another implementation for the compiler's frontend, which is one of the goals of this project and a first step towards a new compiler implementation. > Pre-compiling is translating one language into another because compiling > is changing one programing language, usually higher level, into another, > usually lower level. Puppet does something very different in my > understanding given above. It sounds as if the goat of the C++ parser is > not to produce a catalog of resources but instead a ruby program for the > agent to run. > The goal of this particular initiative is to enable the C++ parser (i.e. the frontend) to interop with the Ruby evaluation implementation (i.e. the backend). The Puppet code is not being pre-compiled, but pre-parsed/pre-validated; the C++ implementation will not (yet) evaluate any Puppet code or load custom types or functions defined in Ruby. The ultimate goal of the C++ implementation is to replace both the frontend and the backend for catalog compilation, but that's a ways off still as there needs to be design and implementation around loading custom types and functions from Ruby source to maintain backwards compatibility. > From the statements about a C++ Puppet parser is the target still just a > collection of resources? Or is the goal to eventually spit out something > other than what the Puppet server should send to a node? > The target is still the same thing we send to agents now: a resource catalog. That won't change when the C++ compiler implements the backend too. > Is the scope just 'closed' .pp files? That is classes where all variables > can be resolved without inquiry to facts? The behavior of languages that > support things like pre-compiled files is specific to how they do binding > of missing data. While this proposal punts on the 'serialization' format > the handling of binding is pretty central. That raises questions like: > > How can a compiled format deal with Puppet features that call APIs? Is > this for the defined types? How will it deal with the fact that the code > could radically change based on node data not even present at the time the > pre-compile is done? > > What happens if a precompiled module that depends on PuppetDB information > is moved to an environment without it? For that matter, is the format > intended to move between Puppet installations or just very similar ones? > > My model of Puppet is that the compiled catalog is just an ordered list of > resources which do not have any placeholders like variables or > sub-dividable parts. Classes in this model are just names that help you > find the resources. > > This format is relevant to recent discussions among people on IRC and in > the mailing lists and groups about getting and storing the catalog for a > node. This is historically something hard as an end user. > > The context of any given resource is the whole catalog and at a minimum > requires the dependent and depending classes in the graph tree. Otherwise > how does this deal with the unresolved parts? > The above seems to be confusing, understandably so, pre-compiling a resource catalog with pre-parsing a manifest. In terms of a language like Python, the "pre-compiled" pyc files are simply a representation of the source that is more efficient to load and execute than having to parse the Python source again. This is analogous to what we're calling pre-parsed XPP files: a representation of the Puppet source code that enables faster evaluation (i.e. during resource catalog "compilation") and also enables the existing Ruby implementation to load and evaluate even if a entirely different parser implementation was used to generate them. Hope this helps. Peter > > On Wednesday, March 30, 2016 at 11:24:46 AM UTC-5, Eric Sorenson wrote: >> >> Hi, I've just posted a new Puppet RFC that describes pre-parsed and >> pre-validated Puppet files, akin to '.pyc' files for Python. It's called >> XPP and the doc is open for comments here: >> >> >> https://docs.google.com/document/d/17SFn_2PJYcO5HjgA4R65a5ynR6_bng_Ak5W53KjM4F8/edit?usp=sharing >> >> Please comment inline on the doc, or come back to this thread if the >> conversation gets too involved (more than about 4-5 replies in a comment >> box on google docs becomes unwieldy) >> >> Once the commenting tapers off we'll incorporate changes into the spec >> and post it as markdown in the puppet-rfc repo: >> https://github.com/puppetlabs/puppet-rfc >> >> --eric0 >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CACZQQfOhsBHcfjdoh9%3DyZS4YGUXQtnj6HRBTqLZiRr_CiueeNQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
