On 2013-29-08 20:18, Andy Parker wrote:
On Wed, Aug 28, 2013 at 10:42 AM, Henrik Lindberg
<henrik.lindb...@cloudsmith.com <mailto:henrik.lindb...@cloudsmith.com>>
wrote:
On 2013-28-08 19 <tel:2013-28-08%2019>:19, Andy Parker wrote:
What I'm thinking (in terms of a workflow a user might perform):
# produce a fully validated AST serialization
$ puppet manifest compile --manifest site.pp --modulepath
~/modules:/etc/puppet/modules --output my-puppet-code.ast
# load that validated AST as a particular environment into the
master. The master would swap in the new code for that
environment only
at the start of a new compile, any existing compiles using the
old code
loaded for the environment would continue unchanged.
$ puppet master load --pic my-puppet-code.pic --environment
staging
This is just an example of how it could be split up into multiple
commands. This could also be done at the beginning of the catalog
request instead of by the user.
At the moment, I'm ignoring any data that would need to go along
with
this. Taking that into account might change things as separate
commands.
Basically this is a copy of the AST into another form that then has
to be loaded (which means some form of parsing and resurrection of
the data). If so, Ruby marshal of the AST will be the fastest. I
think we can allow this to be both a user level command "compile" as
well as "on demand" (a check is made at transaction start that
source has not changed, if so it can be compiled on the fly).
Not always the case https://gist.github..com/aishfenton/480059
<https://gist.github.com/aishfenton/480059>
Did not know about Yajl - that looks promising.
Have to take into account that the objects still needs to be created
from the loaded Json to resurrect the AST though. Will be interesting to
do some benchmarks "textfile - lex - parse => AST" vs "Json file - parse
- deserialize => AST". (Guessing that the Yajl + serialization will be
faster, question is how much).
I think the exact way of serializing things can be discussed at another
time. What I'm interested in is whether anyone things that this kind of
"bulk parse and validate first, then eval" is a good move. My opinion is
that it would be good for the master, but as others point out would be a
problem for interactive (aka puppet apply) use. I think we can address
that by creating a compiler and object model of manifests such that we
can recompile only the parts needed for interactive use (I hesitate to
use the term "incremental compiler" because that can get a little
confusing).
Agree, serialization is not the most important part, except if the rest
of the architecture rests on assumptions that may not be true (measure,
measure...). (Have been bit before by the assumption that "a cache
holding binary transformations of source is faster than always reading
the source")
The two usage scenarios can perhaps be solved by two different URI
resolvers, one is on demand only, the other scans the world (predicts
what is needed first, externally requested URIs gets higher priority
than backlog to scan). The simpler one does not deal with changing files
at all for the sake of simplicity; if user changes things while a puppet
apply is running... well - don't do that.
I think it is safe to use Ruby marshal since the data is for a
specific runtime and it is completely managed by this runtime.
If we instead want off-line compilation and have sharing of bundles
(i.e. 'jar-like' things), Ruby Marshal is out of the question, and I
then doubt that a serialized AST is much faster to load than parsing
the language, but it depends to a large degree on the complexity of
difficult to lex/parse constructs (gazillions of interpolation
expressions).
A jar-like approach would mean that only the timestamp of the jar
needs to be checked, then reading its manifest in order to know its
content.
Yeah, that is the win of using a jar-like approach. A single stat to
know if we need to reload (or no stat at all and the reload only happens
on a signal)
Although good from a technical standpoint, it is not fun to have to
build it (if the building is exposed to users).
A signal is good to have even if the build is "incremental" (who says we
are going to always be bug free :-) and a "clean and start over" safety
is prudent (i.e. restart the service :))
- henrik
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.
