I'd certainly be interested in working this through more at PuppetConf if there's time.
I also don't see why this needs to wait until 5.0. It should be 100% backward compatible with the 4.x series in theory (it would be nice anyway). Thanks! Trevor On Thu, Jun 19, 2014 at 6:41 PM, Henrik Lindberg < [email protected]> wrote: > On 2014-19-06 23:25, John Bollinger wrote: > >> On Wednesday, June 18, 2014 6:07:02 PM UTC-5, henrik lindberg wrote: >> >> On 2014-18-06 22:44, John Bollinger wrote: >> > >> > Puppet already has a tool for separating the details of a >> resource from >> > the decision about whether to include it in the catalog: virtual >> > resources. Those decisions can be tied to whether a given class is >> > declared by having that class realize or (especially) collect the >> > resources. In many cases that can be done via a wrapper class or >> > definition if the target class itself must not be modified. >> > >> > Here's an idea: Collectors are already processed very late, so >> maybe >> > they are late enough -- or could be /made/ late enough -- to >> provide the >> > kind of context needed for safe reflection. One might then write >> > something along these lines: >> > >> > Firewall::Rule<| title == 'whatever' and defined == 'firewall' |> >> > @firewall::rule { "whatever": >> > # ... >> > } >> > >> > Of course, the collector wouldn't need actually to be close to the >> > resource declaration, and it might be more general than just one >> > resource, but it could be right there with the resource >> declaration to >> > provide something much like an if defined() effect. >> > >> The only thing this does is to move the problem to a new "phase". >> Since the purpose is to add resources, naturally you can add defined >> types, which in turn creates many resources. The set of rules then >> operate on a moving target (constraint solving). >> >> >> >> I'm thinking about something more restricted than it may seem I've >> suggested: an ability only to reflect which *classes* are in the >> >> catalog, in a context and time where no new classes can be added (but >> resources can). That characterization draws on my position that classes >> should not be considered resources, at least at DSL level, but you can >> say instead "(but resources other than classes can [be added])" if you >> prefer. >> >> > For that to be safe though, it would be necessary to prevent the >> > collected resources from adding classes to the catalog, probably by >> > failing the catalog if any of them try to do so. That >> restriction might >> > be reserved for when the 'defined' key is used in the collector's >> select >> > expression, provided that such collectors were processed after >> all others. >> > >> I don't think we absolutely have to protect classes from being >> included.. >> >> In a way they are just resources only singletons. >> >> >> >> The idea was that this limited form of reflection escapes >> evaluation-order dependencies because it draws on information that will >> not subsequently change -- to wit, which classes are in the catalog. >> That's the reason for preventing new classes from being added. >> >> A new mechanism is then needed, since it is not possible to statically > derive the set of classes that will actually be included in the catalog and > what parameters they will have. It is possible (given a few caveats) to > know the set of classes that can potentially be included, but that is about > it. > > > Now that I think about it, though, I'm no longer sure that would be >> sufficient. I think this would introduce a new form of evaluation-order >> dependency involving the effects of collectors, since the full set of >> resources selected by any given collector can be determined reliably >> only after all resources that will be added to the catalog have been >> added, at least virtually. On the other hand, maybe this problem isn't >> new -- it seems like collectors can already suffer from this problem on >> their own. >> >> And I think any solution to that problem creates its own problems. >> Consider this artificial example: >> >> define loop::a ($index = 0) { >> @loop::a { "a$index": index => $index + 1 } >> } >> >> @loop::a { 'start': } >> >> Loop::A<| |> >> >> Obviously, you shouldn't write infinite loops, but my point is that on >> the one hand, whether that is in fact an infinite loop depends on the >> evaluator implementation, but on the other hand, an evaluator for which >> it is NOT an infinite loop does not reliably collect all resources that >> should be collected. (And on the third hand, it might not even be >> possible to collect all the resources that should be collected.) >> >> > You can do loops now with resources. If you can also evaluate code as a > side effect of collection (or when dependencies are formed), one more such > looping possibility is added. IMO this makes things worse. > > > I am thinking that a module can specify constraints as a predicate >> (either on other modules being present, other classes, or resources, >> or >> indeed on some abstract "capability"), and if the predicate becomes >> true >> at the end of the catalog production, then it includes one class from >> the module. >> >> >> >> But if a class is added as a result of that mechanism, then it turns out >> that the predicate wasn't evaluated at the end of catalog production >> after all. That's not an issue for just one such constraint, but as >> soon as you have two or more you are right back to having an >> evaluation-order problem. And consider also the issue with collectors >> that I raised above -- I think that probably applies here, too. >> >> >> This adds a much asked for feature; autoinclude. Inclusion in the >> catalog by just adding a module to the modulepath. >> >> >> >> As opposed to adding "include 'mymodule'" to the top level of the >> starting-point manifest? Or adding a file containing that to the >> starting-manifest directory? What's hard about that? The latter could >> even be arranged as part of module installation. >> >> Or why not have a separate module path for autoinclude modules? Users >> then select autoinclude behavior by where they put the module -- or even >> by where they put a symlink to the module. >> >> I think simple autoinclude is a relatively tractable problem, but I >> guess you're looking at conditional autoinclusion. Honestly, I'm not >> very comfortable with that. Moreover, I'm inclined to think that >> whatever framework might be devised for evaluating which of the >> available modules to "auto"include would be no simpler, better, or >> easier than what we already have. >> >> (This is the reverse >> dependency I was talking about earlier). >> >> In all of these cases, some kind of constraint solver ability is >> needed >> across the new feature, queries and the forming of dependencies (since >> you most certainly need to also have control over the order in which >> the >> "injected" classes/resources are later applied. >> >> >> >> I think you're saying there needs to be a way to set up dependencies >> between classes and resources added to the catalog during the "normal" >> evaluation phase and those added as a result of this hypothetical coda. >> That supposes, I guess, that those cannot be declared either during the >> normal evaluation pass (e.g. via a collector) or during the additional >> one. I'm uncertain whether that's true. >> >> Dependencies are evaluated after all classes and resource have been > evaluated. An operand of a dependency may be a query (a.k.a collection). > Therefore, dependencies are added last (I think)). Until that point, all > references are unchecked "future references". > > The issues with dependencies and conditional inclusion is that neither > side (the one that causes the inclusion as a side effect, or the side that > is being included) knows enough. (Maybe this is an imaginary problem and > dependencies are more coarse grained). > > > The other approach I can think of (I wrote about it earlier) is that >> you >> make explicit statements about things that must have been evaluated >> before "you" can be evaluated (i.e. "I need these Promises fulfilled / >> defined before I can say what I want", and modules are given the >> ability >> to define such promises. This (as John also points out) requires a >> context in which a resolution of available Promises can take place >> reliably). Maybe a Promise is nothing else but a class, but that you >> can >> tie your evaluation to the outcome to the evaluation of one or many >> other classes. (This is an area I hope to be able to work on ideas >> after >> we have released puppet 4). >> >> >> >> Basically that's a way to constrain evaluation order, right? Such a >> mechanism would indeed address the core problem, and conditional >> evaluation can falls out as a special case. It might even fit in easily >> with the current (future) evaluator, as it seems like it has some >> parallels with 'include'. So, yes, this seems like a promising >> direction to explore. >> >> Yes it is, and in a safe way. (I think this is the most promising, the > other ways to solve it that have come up all seem to result in a game of > Whack-a-Mole). > > And, to expand on this idea a little. I think the exact same mechanism is > applicable when the resources are applied on an agent. i.e. it is an > evaluation plan, workflow, or orchestration if you like. > > I doubt I will have time to work on these ideas before Puppet Conf - I > don't have this in presentable form yet, maybe something that can be shared > by then. Some of these ideas have implications that we are thinking about > and working into Puppet 4.0. We also considered these ideas for the design > of the next phase; the catalog builder (replacement for the current > compiler) - that works will take place during the 4x series with the intent > of releasing it as Puppet 5.0. > > > - henrik > > -- > > Visit my Blog "Puppet on the Edge" > http://puppet-on-the-edge.blogspot.se/ > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-dev/lnvots%24eod%241%40ger.gmane.org. > > For more options, visit https://groups.google.com/d/optout. > -- Trevor Vaughan Vice President, Onyx Point, Inc (410) 541-6699 [email protected] -- This account not approved for unencrypted proprietary information -- -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoX_U-T0i5ZEuBQ1rH0LxT11yF5qDrxhs4gctf%3DYky4Jog%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
