On Tue, Aug 27, 2013 at 12:50 PM, Henrik Lindberg < henrik.lindb...@cloudsmith.com> wrote:
> > This is very interesting discussion. > > In Geppetto the approach is to parse everything up-front. This is not the > same as what is typically referred to as parsing in the puppet community > which seems to also involve evaluation and linking. We often talk about > "parse order" when we actually mean "evaluation order". > > Yes, I find this confusion of terminology makes it really hard to talk accurately about parts of the system. > Parsing is quite straight forward, simply turn the DSL source into an AST > and remember it. It is the linking (and evaluation) that is tricky, > especially when changes can take place to files mid-transaction. > > In Geppetto it is not good to keep all information about all files in > memory all the time and there a technique is used to populate an index with > references to the source positions where referable elements are located, > this index is used during linking. A graph of all dependencies is created > in a way that makes it possible to compute if a change to a "name => > element" change will have effect on other resolutions (it was missing, now > it exists, it existed was resolved, but should now resolve differently). > > Right. Geppetto encounters many of the same problems that the puppet master deals with right now because the files could be changing at any time and it needs to track that. I suspect Geppetto also has a similar problem to other Eclipse editors in which if you change the file on disk via another mechanism (edit in vim, for example) it kinda "freaks out" as it were. I think that has to do with the caching that Eclipse tries to do in order to not have to hit the disk as often. > The build system that kicks in on any change makes use of the dependency > information to link the resulting AST in the correct order. > Sometimes it is required for the builder to make an extra pass due to lack > of information (the dependencies were not yet computed), or when there is > circularity. > > Puppet is a very tricky language to link since many of the links can be > dynamic (not known until evaluation takes place). You can see this in > Geppetto; sometimes you have to do a "build clean" as the interactive state > does not know that issues where resolved. Also, in many cases it is not > possible to validate/link these constructs at all. > > Yes, the evaluation of the puppet language is not very static. There is a lot that is determined dynamically, but the AST itself is mostly static. I say mostly because there is always the possibility that a custom function manipulates the known_resource_types at runtime, but I really hope nothing is doing that. > In Geppetto, the majority of the computing time is to validate the links. > > With all of that said. In a Puppet Master we could parse all files to AST > but when we start evaluation we start over from the beginning every time > (check if file is stale, if so reparse it). Holding more state than that in > memory is very problematic (esp. when environments are involved). > > Yes, and getting that state correct has been the cause of a lot of problems and confusion :( > I was talking with Eric Dalén, and he said they tried running the master > in way where each catalog request got a fresh process. If I understood him > correctly this did not have much (if any) negative impact on performance !! > I can think of many reasons why, ruby is not the best at garbage > collection, when things are done in virgin state there are less things to > check (poor cache implementations are sometimes worse than no cache), > memory is better organized. I can also imagine that speedups that were > measured in the past may not be as relevant today when disks are much > faster (SSD even), and with speed improvements in the Ruby runtime. It is > high time to again measure where the bottlenecks are. > > So - Andy, I am very much looking forward to hearing about the > measurements you are doing on parse time. Are your running both old/new > parser and 1.8.7, 1.9.3 ? > I'm not doing the measurements. I'm trying to crowd source them :) Adding what ruby version is in use would also be interesting, but I didn't think of adding that into my little measurement script. > > Regards > - henrik > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to > puppet-dev+unsubscribe@**googlegroups.com<puppet-dev%2bunsubscr...@googlegroups.com> > . > To post to this group, send email to puppet-dev@googlegroups.com. > Visit this group at > http://groups.google.com/**group/puppet-dev<http://groups.google.com/group/puppet-dev> > . > For more options, visit > https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> > . > -- Andrew Parker a...@puppetlabs.com Freenode: zaphod42 Twitter: @aparker42 Software Developer *Join us at PuppetConf 2014, September 23-24 in San Francisco* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To post to this group, send email to puppet-dev@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-dev. For more options, visit https://groups.google.com/groups/opt_out.
