On 2014-19-06 23:25, John Bollinger wrote:
On Wednesday, June 18, 2014 6:07:02 PM UTC-5, henrik lindberg wrote:
On 2014-18-06 22:44, John Bollinger wrote:
>
> Puppet already has a tool for separating the details of a
resource from
> the decision about whether to include it in the catalog: virtual
> resources. Those decisions can be tied to whether a given class is
> declared by having that class realize or (especially) collect the
> resources. In many cases that can be done via a wrapper class or
> definition if the target class itself must not be modified.
>
> Here's an idea: Collectors are already processed very late, so maybe
> they are late enough -- or could be /made/ late enough -- to
provide the
> kind of context needed for safe reflection. One might then write
> something along these lines:
>
> Firewall::Rule<| title == 'whatever' and defined == 'firewall' |>
> @firewall::rule { "whatever":
> # ...
> }
>
> Of course, the collector wouldn't need actually to be close to the
> resource declaration, and it might be more general than just one
> resource, but it could be right there with the resource
declaration to
> provide something much like an if defined() effect.
>
The only thing this does is to move the problem to a new "phase".
Since the purpose is to add resources, naturally you can add defined
types, which in turn creates many resources. The set of rules then
operate on a moving target (constraint solving).
I'm thinking about something more restricted than it may seem I've
suggested: an ability only to reflect which *classes* are in the
catalog, in a context and time where no new classes can be added (but
resources can). That characterization draws on my position that classes
should not be considered resources, at least at DSL level, but you can
say instead "(but resources other than classes can [be added])" if you
prefer.
> For that to be safe though, it would be necessary to prevent the
> collected resources from adding classes to the catalog, probably by
> failing the catalog if any of them try to do so. That
restriction might
> be reserved for when the 'defined' key is used in the collector's
select
> expression, provided that such collectors were processed after
all others.
>
I don't think we absolutely have to protect classes from being
included..
In a way they are just resources only singletons.
The idea was that this limited form of reflection escapes
evaluation-order dependencies because it draws on information that will
not subsequently change -- to wit, which classes are in the catalog.
That's the reason for preventing new classes from being added.
A new mechanism is then needed, since it is not possible to statically
derive the set of classes that will actually be included in the catalog
and what parameters they will have. It is possible (given a few caveats)
to know the set of classes that can potentially be included, but that is
about it.
Now that I think about it, though, I'm no longer sure that would be
sufficient. I think this would introduce a new form of evaluation-order
dependency involving the effects of collectors, since the full set of
resources selected by any given collector can be determined reliably
only after all resources that will be added to the catalog have been
added, at least virtually. On the other hand, maybe this problem isn't
new -- it seems like collectors can already suffer from this problem on
their own.
And I think any solution to that problem creates its own problems.
Consider this artificial example:
define loop::a ($index = 0) {
@loop::a { "a$index": index => $index + 1 }
}
@loop::a { 'start': }
Loop::A<| |>
Obviously, you shouldn't write infinite loops, but my point is that on
the one hand, whether that is in fact an infinite loop depends on the
evaluator implementation, but on the other hand, an evaluator for which
it is NOT an infinite loop does not reliably collect all resources that
should be collected. (And on the third hand, it might not even be
possible to collect all the resources that should be collected.)
You can do loops now with resources. If you can also evaluate code as a
side effect of collection (or when dependencies are formed), one more
such looping possibility is added. IMO this makes things worse.
I am thinking that a module can specify constraints as a predicate
(either on other modules being present, other classes, or resources, or
indeed on some abstract "capability"), and if the predicate becomes
true
at the end of the catalog production, then it includes one class from
the module.
But if a class is added as a result of that mechanism, then it turns out
that the predicate wasn't evaluated at the end of catalog production
after all. That's not an issue for just one such constraint, but as
soon as you have two or more you are right back to having an
evaluation-order problem. And consider also the issue with collectors
that I raised above -- I think that probably applies here, too.
This adds a much asked for feature; autoinclude. Inclusion in the
catalog by just adding a module to the modulepath.
As opposed to adding "include 'mymodule'" to the top level of the
starting-point manifest? Or adding a file containing that to the
starting-manifest directory? What's hard about that? The latter could
even be arranged as part of module installation.
Or why not have a separate module path for autoinclude modules? Users
then select autoinclude behavior by where they put the module -- or even
by where they put a symlink to the module.
I think simple autoinclude is a relatively tractable problem, but I
guess you're looking at conditional autoinclusion. Honestly, I'm not
very comfortable with that. Moreover, I'm inclined to think that
whatever framework might be devised for evaluating which of the
available modules to "auto"include would be no simpler, better, or
easier than what we already have.
(This is the reverse
dependency I was talking about earlier).
In all of these cases, some kind of constraint solver ability is needed
across the new feature, queries and the forming of dependencies (since
you most certainly need to also have control over the order in which
the
"injected" classes/resources are later applied.
I think you're saying there needs to be a way to set up dependencies
between classes and resources added to the catalog during the "normal"
evaluation phase and those added as a result of this hypothetical coda.
That supposes, I guess, that those cannot be declared either during the
normal evaluation pass (e.g. via a collector) or during the additional
one. I'm uncertain whether that's true.
Dependencies are evaluated after all classes and resource have been
evaluated. An operand of a dependency may be a query (a.k.a collection).
Therefore, dependencies are added last (I think)). Until that point, all
references are unchecked "future references".
The issues with dependencies and conditional inclusion is that neither
side (the one that causes the inclusion as a side effect, or the side
that is being included) knows enough. (Maybe this is an imaginary
problem and dependencies are more coarse grained).
The other approach I can think of (I wrote about it earlier) is that
you
make explicit statements about things that must have been evaluated
before "you" can be evaluated (i.e. "I need these Promises fulfilled /
defined before I can say what I want", and modules are given the
ability
to define such promises. This (as John also points out) requires a
context in which a resolution of available Promises can take place
reliably). Maybe a Promise is nothing else but a class, but that you
can
tie your evaluation to the outcome to the evaluation of one or many
other classes. (This is an area I hope to be able to work on ideas
after
we have released puppet 4).
Basically that's a way to constrain evaluation order, right? Such a
mechanism would indeed address the core problem, and conditional
evaluation can falls out as a special case. It might even fit in easily
with the current (future) evaluator, as it seems like it has some
parallels with 'include'. So, yes, this seems like a promising
direction to explore.
Yes it is, and in a safe way. (I think this is the most promising, the
other ways to solve it that have come up all seem to result in a game of
Whack-a-Mole).
And, to expand on this idea a little. I think the exact same mechanism
is applicable when the resources are applied on an agent. i.e. it is an
evaluation plan, workflow, or orchestration if you like.
I doubt I will have time to work on these ideas before Puppet Conf - I
don't have this in presentable form yet, maybe something that can be
shared by then. Some of these ideas have implications that we are
thinking about and working into Puppet 4.0. We also considered these
ideas for the design of the next phase; the catalog builder (replacement
for the current compiler) - that works will take place during the 4x
series with the intent of releasing it as Puppet 5.0.
- henrik
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/lnvots%24eod%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.
