Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Trevor Vaughan commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Just found this ticket, but I also would like to be able to say X comes after all Packages without actually realizing them. Making this a set of functions might be more straightforward and non-breaking. It's not as puppet-y but I think people are pretty used to functions doing magic at this point. The only use cases that I've ever had for collectors are: Making things come before/after other things Performing resource overrides (key in many cases) Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.191187.1494891335000.121336.1611259920031%40Atlassian.JIRA.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title John Bollinger commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax It sounds like there is some consensus forming. There is surely some diversity in how we each imagine the details, but we all seem to agree that a standalone query would be a desirable thing. At least, that's what I would characterize a collector that does not automatically realize to be. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.191187.1494891335000.4461.1580306462085%40Atlassian.JIRA.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Henrik Lindberg commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Something I have wanted for a long time is to have the concept of "query" as a separate thing; like a lambda if you like, a value that could be given to different functions, or be used in resource like in Ben's Manifold. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.191187.1494891335000.4172.1580291161496%40Atlassian.JIRA.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Ben Ford commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax John Bollinger, no. Sorry I wasn't more clear. What I want is for collectors to not implicitly realize, just the same as you do. More specifically, I want collectors to do exactly one thing – just collect resources into a thing that you can do something with. If you want to add relationships, then yes, you can add relationships to a collection. If you want to realize, then yes, you can realize a collection. But simply invoking the collector shouldn't do anything other than collect. That would be a bigger breaking change, since so much current code relies on the side effect, but that's what I'd like long term. I'd also like a better query language, since there's currently not a way to collect "all packages except for those tagged as puppet_enterprise". The manifold type is simply a bandaid to separate the relationship case from the realize case without drastic language changes today. It can also be used on older Puppet versions. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.191187.1494891335000.3615.1580247661655%40Atlassian.JIRA.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title John Bollinger commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax The issue is when using Puppet Enterprise for example there are packages that are virtual and depending upon how you use collectors this will cause some nasty behavior That's certainly an issue, Chris Denneen, but the issue we're discussing here is broader. How does one propose to continue to use Collectors (or PQL) to allow for the desired behavior that Yum gets updated before yumrepo which are all applied before all other packages. With current PE alone, one would need to write a more selective collector. Either additional resource titles could be excluded specifically, or perhaps tags could be brought to bear to exclude the Puppet Enterprise packages. If one is willing to engage third-party modules, then it looks like Ben's Manifold module can stand in for a collector in that particular case and many similar ones. As I already commented, the broader issue revolves around the fact that collectors serve dual purposes. Way back in Puppet 2, they served only to realize virtual resources and collect exported ones. They retain those functions today, but since P3 they also serve to group resources as operands for the chain operators and for parameter overrides. Presently, there is no way to obtain either of the latter two collector behaviors without the first, but there could be in Puppet 8 (or maybe even in Puppet 7?). I already suggested one way that could be implemented, but there are certainly others, including alternatives along the lines of Ben's module. The first iteration of this discussion established that there was still strong support for exported and virtual resources at that time. Perhaps that has since waned, but I, at least, still want to keep them. On the other hand, the specifics of current collector syntax and semantics are much more open to consideration, and from my perspective, that's where PQL, Datalog, Manifold, etc. could come into the picture. Some of those can already be used to address parts of the problem. Maybe PE should avoid the use of Virtual Resources? But what's to stop Datadog, Sensu, etc vendors from doing the same. Whether PE or anyone else should stop using virtual resources is a separate question. The specific problem Ben brought up would no longer manifest if PE did so, but that would not address the larger issue.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Chris Denneen commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax The issue is when using Puppet Enterprise for example there are packages that are virtual and depending upon how you use collectors this will cause some nasty behavior: Notice: /Stage[main]/Puppet_enterprise::Packages/Package[pe-postgresql-pglogical]/ensure: current_value 'purged', should be 'latest' (noop) Notice: /Stage[main]/Puppet_enterprise::Packages/Package[pe-postgresql-pgrepack]/ensure: current_value 'purged', should be 'latest' (noop) Notice: /Stage[main]/Puppet_enterprise::Packages/Package[pe-java-devel]/ensure: current_value 'purged', should be 'latest' (noop) Notice: /Stage[main]/Puppet_enterprise::Packages/Package[pe-activemq]/ensure: current_value 'purged', should be 'latest' (noop) Notice: /Stage[main]/Puppet_enterprise::Packages/Package[pe-razor-libs]/ensure: current_value 'purged', should be 'latest' (noop) Notice: /Stage[main]/Puppet_enterprise::Packages/Package[pe-razor-server]/ensure: current_value 'purged', should be 'latest' (noop) Notice: Class[Puppet_enterprise::Packages]: Would have triggered 'refresh' from 6 events This is caused by the above example (to make sure Package['yum'] -> Yumrepo<| |> -> Package<| |>) Yumrepo<| |>->Package<| title != 'yum' |> How does one propose to continue to use Collectors (or PQL) to allow for the desired behavior that Yum gets updated before yumrepo which are all applied before all other packages. (Reason Yum has to be updated first is because sometimes features being added to yumrepos ssl, parameters, etc won't work with old versions of yum so the package must be applied first) but you want the yumrepos to be applied before all other packages are attempted to be installed/upgraded. Maybe PE should avoid the use of Virtual Resources? But what's to stop Datadog, Sensu, etc vendors from doing the same. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title John Bollinger commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax The exported resources and collection use cases seems to be all solved by PQL and regular logic - do you agree? I'm not sure I do. In the first place, it's not clear to me how you have in mind to use PQL. Nothing provided by core Puppet is adequately replaced by anything other than another core Puppet feature or a feature of a module supported and maintained by Puppet, Inc.. There is no core function specifically for querying puppetdb. It appears that the puppetlabs/puppetdb module provides such a function, but as far as I can tell, that function is not documented, which I take as an indication that it is not an officially supported feature of the module. But supposing that the puppetdb_query() function were a documented, supported feature of puppetlabs/puppetdb, and that that was the intended mechanism for addressing the collection of exported resources, I suspect that it does not adequately support all use cases. In particular, I do not think it properly handles cases in which a node wants to collect a resource that it exports itself. Similar would apply to any other mechanism that gathers only data that are already recorded in puppetdb. There could be other mismatches, though none spring immediately to mind. Additionally, although collector syntax is a bit arcane, it at least feels relatively well integrated into the language. PQL is at least as arcane as collector syntax, but passing PQL strings to a function and then dealing with the resulting raw hash is nowhere near as integrated. Powerful, yes. Flexible, yes. Integrated, no. For virtual resources and overrides use cases the replacement is less obvious. I think it may require logic programming in the vein of Datalog / OPA. Which could be applied in a catalog post processing step. A difficulty is how to express those in the Puppet Language in a natural way. I'm not really up on Datalog or OPA, but I think I see why the former, at least, comes to mind. Certainly, replacing collector-based syntax with a more fully-featured declarative language can be expected to provide more flexibility, albeit at the cost of making the Puppet language more complex and less cohesive. Order-independence such as Datalog in particular provides would be well suited to Puppet use cases, too, if indeed Puppet could provide such a thing, but I doubt it can do, given that some of the use cases of interest (i.e. overriding) involve modifying the data set to which the language applies. Overall, it sure would be nice to have a consistent way to select resources from among those known / knowable to the catalog builder, instead of having to use separate mechanisms for exported and virtual resources. Collectors very nearly provide this. PQL could likely be extended to provide it, but it does not currently do so. Some new thing certainly could do so, and should do so if one is made.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Henrik Lindberg commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Since this thread has woken up again I like to point out some of the difficulties with the current collectors and what a replacement needs to consider. First, the collectors run after the normal compilation has finished - in this sense it is a kind of post processing of the catalog. Each collector is triggered once (and thus the outcome depends on the order they are executed in). This is done because of possible conflicts and endless loops (for example rule A sets x to 2 if it is < 2, and rule B sets x to 1 if it is >= 2). Earlier there was an issue with defaults values - if they were applied "at end of regular compilation" or "after collectors/overrides". This was changed so defaults are set before collectors kick in. If a new design simply allows lambdas to be called (on some kind of set of resources produced by some query function) in some kind of deferred way while compiling then the logic in the lambdas would need to be allowed to do overrides, they would need to run in some order, and would need to be protected from infinite loops etc. This is very difficult with general purpose puppet logic because it could do anything (override what is in catalog, and add to it). The deferred/lazy mode of collectors is why they cannot really return anything to the language, and it has some hoops to jump through because of the relationships that can be established between collector results (those are also done in a lazy way, and thus depend on the order of evaluation of the lazily evaluated collectors/overrides/relationships). The exported resources and collection use cases seems to be all solved by PQL and regular logic - do you agree? For virtual resources and overrides use cases the replacement is less obvious. I think it may require logic programming in the vein of Datalog / OPA. Which could be applied in a catalog post processing step. A difficulty is how to express those in the Puppet Language in a natural way. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Julian commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax removing such a good feature sounds not good to me. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.191187.1494891335000.831.1579985882084%40Atlassian.JIRA.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title John Bollinger commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Quoting myself: there are certainly aspects of the legacy system that would bear improvement. In particular, separating the diverse functions served by collectors from each other springs immediately to mind The error scenario you describe, Ben Ford, arises in part from no such separation having been performed, so that collectors still perform multiple largely unrelated functions. The manifold module's capabilities look interesting, but they don't solve that problem themselves. Supposing that you recognize that, I guess you are suggesting that manifold, or something substantially similar, be incorporated into core puppet in conjunction with collectors being made unacceptable operands for the chain operators. I think that would be the wrong direction. I would rather see collectors continue to work with the chain operators and for resource parameter overrides, but cease to (automatically) realize virtual resources or collect external resources, even though the latter are the original functions of the constructs. I think it makes sense for the language to have a general-purpose syntax to represent a set of resources. Collectors serve that purpose reasonably well, but manifolds not so much. It looks like manifolds do fine, within their limitations, as a stand-in for collectors for resource relationship purposes, but there remains collectors' usage for resource parameter overrides, and there may well be future applications, too. For example, once upon a time I suggested that collectors be allowed as property values, so that they could be used with the relationship metaparameters (the chain operators were introduced instead). Moreover, if we accept that a such a general-purpose group-selection syntax is sensible, then voila! it provides its own natural solution to the problem. Take away the automatic realization / collection function, and introduce instead explicit realization of resources matched by collectors. One might use the realize keyword to accomplish that, for example. Thus, one might be obliged to write ... realize Host<<| |>> realize User<| tag == 'x' |> ... instead of just ... Host<<| |>> User<| tag == 'x' |> ... if one wanted not only to collect resources that are otherwise in the catalog but also to add matching known / discoverable resources to the catalog. And one could make such realize expressions work in chain expressions and parameter override specifications, so that converting old code would just be a matter of adding one keyword in those places where it is wanted. As an aside: it does not appear that the present implementation of manifolds is as flexible as collectors with regard to specifying member resources. Collectors allow multiple criteria to be combined via logical operators, and they support search values of several non-string types, notably including resource references and undef, but it does not appear that manifolds do the same. I'm sure that's a solvable problem, but I'm uncertain what the costs of building and maintaining a solution would be.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title
Ben Ford commented on PUP-7541
Re: Explore removing export / collect / virtual / realize syntax
I'd like to re-open this conversation. I had another conversation with a community member today who used the same old Yumrepo<| |>->Package<| title != 'yum' |> pattern to lay down internal repositories before any packages were installed. No surprise, he knocked over his whole infrastructure when all Puppet Enterprise packages were installed on all nodes. I do currently have a better solution for that use case in the manifold module. It's an anchor-like resource type that allows you to make meta relationships on real resources only. Aka, it won't realize anything like a collector will. I'd far prefer for this to be baked into the language the way that contain() effectively replaced anchors, but this is far better than collectors can be. A relationship like that would look something like
manifold { 'packages':
type => 'package', # make relationships on packages
match=> 'title', # match their title parameter
pattern => 'yum', # match on title=yum
invert => true,# but invert so title != yum
relationship => before, # and this resource should come before them
}
yumrepo { 'internal':
ensure => 'present',
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Josh Cooper updated an issue Puppet / PUP-7541 Explore removing export / collect / virtual / realize syntax Change By: Josh Cooper Fix Version/s: PUP 6.0.0 Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Adam Gardner commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax I'd like to point out, the absence of exported resources in Puppet Forge modules seems like a pretty big red herring; I suspect (obviously I can't prove) that exported resources are most often created and consumed by organization-specific Puppet code that never gets published to forge.puppet.com. Certainly, that's how we're using them. That's not to say I'm not in favor of eventually deprecating these features, and the idea of being able to use PuppetDB as a key-value store without having to create dummy types (either instead of or in addition to collecting resources) has a lot of appeal. But for the moment I agree with Sean, John, etc, the current alternatives aren't quite there yet. At the very least, if you're going to deprecate these features, add functions to query PuppetDB into stdlib. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Craig Dunn commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax I think generally speaking the use of export/collect and virtualize/realize are in decline in the wild - there are certainly very few Forge modules that use the pattern and I think there are sufficient replacement patterns now that achieve the desired result, so I'm not against moving to deprecate that specific functionality - however, resource collectors also are used to set dependencies with chaining and this is something that would break a lot of stuff. I think PUP-6712 should be linked into this issue - as that would enable the retention of chaining functionality whilst also deprecating the collector syntax. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title John Bollinger commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Having just stumbled across this issue, I'd like to add my voice, too, to Sean's et al. I can understand and appreciate the desire to get rid of exported and virtual resource syntax from a future version of the language, but the features they provide and the manner in which they provide them have turned out to be powerful, general, and useful. I wouldn't be too upset to see the somewhat arcane language syntax by which these features are presently provided be replaced by function calls – though that would be disruptive – but simply removing all the things you can do with them without adequate replacement would be a tremendous step backward. Among the features I think need to have continued support are the ability provided by virtual and exported resources to specify the identity (including type) and attributes of a resource while deferring any decision about whether to add it to the (or any) catalog; the ability provided by exported resources to externalize the identity and attributes of such deferred resources so that they are available for use in other nodes' catalogs; the ability provided by collectors and existing functions to selectively add deferred resources of both varieties to catalogs, including the use of search expressions to designate the resources to add; the ability provided by collectors of both types to specify resource relationships in terms of search expressions; the ability provided by collectors of both types to override attributes of resources specified via search expressions. with respect to all the above uses of search expressions, substantial independence of manifest evaluation order. As long as this area is under scrutiny, however, there are certainly aspects of the legacy system that would bear improvement. In particular, separating the diverse functions served by collectors from each other springs immediately to mind, as does improving the consistency of these features across plugin and defined types (see, for example, PUP-6014 ). Clarifying and/or allowing control over the the applicability of resource defaults to deferred resources would also be advantageous. I'm sure there are other opportunities.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Geoff Nichols updated an issue Puppet / PUP-7541 Explore removing export / collect / virtual / realize syntax Change By: Geoff Nichols Team/s: Agent Platform Core Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title
David Schmitt commented on PUP-7541
Re: Explore removing export / collect / virtual / realize syntax
Adding my voice on the side of Sean, Hunter, and others here: We need to build a better replacement first, before we can think about deprecating (or worse, removing) functionality.
Some possibilities are:
replace dangerous side-effecting collectors with explicit function calls. This could leverage PQL for DB-side filtering, and lambdas for compiler-side munging.
Having a Set Of Resources as type in the language, with explicit versions of all the current implicit functionality around collectors will be much easier to explain and reason about.
Use the orchestrator's service resources to wrap up data from one node's context for publication.
One can make a good argument that arbitrary querying of node-foreign resources in the PuppetDB is dangerous[tm]. Pushing people towards existing concepts (maybe reworking them a bit in the process) might alleviate this.
Folks already can use empty {{define}}s to achieve the same. Maybe that is enough?
Provide ways to post-process the catalog after compilation, before sending it to the agent.
This could also address a boatload of other things folks are currently abusing native types and providers for.
All of these feel like things we "should" be able to develop throughout the Puppet 5 life cycle as experimental features, and release the out-of-band as they become ready.
And finally:
Put a tag on all planned deprecations above to collect data from the field through the Analytics project.
Without that we'll never know when we're done.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Gareth Rushgrove commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax A couple of points: The first a quick example of usage. The dummy_service module, which is commonly used when building Docker images using Puppet, uses the resource collectors to clobber all services in the catalog. https://github.com/puppetlabs/puppetlabs-dummy_service/blob/master/manifests/init.pp I'd love to see usage data here, given that most Puppet users won't see this issue. For instance it would be interesting to present information about the usage of these language features from a) Puppet modules on the Forge and b) Puppet code available in the Google BigQuery index. Both of those datasets are publicly available and might help identify how widely used these features are and for what usecases. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Hunter (Hunner) Haugen commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax I have not read the entire conversation yet (longthread is long) but generally agree with Sean Millichamp's major points. I also want to offer a few technical details. <||> collectors are often used to add resource dependencies when we are unsure of any/all of the declared resources. Similar to the autorequire et al in native types. The biggest example that comes to mind is https://github.com/puppetlabs/puppetlabs-tomcat/blob/1.7.0/manifests/instance/dependencies.pp as there can be an arbitrary number of tomcat instances with widely varying paths and many (or none) defined types using the various instances. Building out such a tree was not possible without collecting on a commonly-shared quality (that of the catalina_base or catalina_home values). From a module dev perspective, we cannot depend on the existence of the puppetdb service. We also cannot depend on the ability to export resources, though may offer solutions that can take advantage of such (eg https://github.com/puppetlabs/puppetlabs-haproxy/blob/1.5.0/manifests/backend.pp#L66). I don't think module devs care about virtual resources; they are too entangled with the collector realization and spaghetti-code anti-patterns that we just avoid them. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Sean Millichamp commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Erik Dalén Sean Millichamp when compiling the catalog on a host you essentially only have the facts for that host as input, and those same facts can be queried from another host using some puppetdb query function. I've reviewed our code exploring the idea of using only non-exported resources data from PuppetDB to generate and I don't think we could use just node facts for ANY of our use cases. All of them are also influenced in some small or large part by input from Hiera (again, in the context of the node exporting the resource). Now, you could take that data, write it out as facts, and then read it that way when those facts are then read in on the next run. But that introduces a whole new set of problems: you are now trusting the facts the client reports (which in some use cases may not be sufficient), you are making data available on the system and in PuppetDB fact queries which may not be information you want generally available, and - worst of all - it now takes two client runs to get the data in PuppetDB for use. Henrik Lindberg's idea of a key/value store as a substitute/replacement for exported resources would probably be workable. In fact, you can approximate the requirements with exported resources as they are now. If you create a defined type which has parameters but no code and export it you have essentially a basic data export/import process suitable for retrieval and import-side processing with puppetdbquery functions. Whatever key/value store was implemented to replace exported resources would still have to follow the basic rule that it was not committed to the store (presumably this would be PuppetDB) unless the catalog compile was successful. If this store also allowed you to - during the same catalog compile - read back any values already stored into it (even though they hadn't been committed yet) then this would also allow you to replace much of the virtual resource workflow too. The one scenario which sticks out in my mind that this hypothetical key/value store would not let you do is replicate the virtual resource use case where you can realize resources which haven't been processed by the compiler yet. I know there is this feeling that you should be able to wave your hands and fix ordering issues - and in theory I agree. However, in a complex environment with a lot of profile-level code that is being provided to uses who - by and large - don't maintain it, don't read it, and don't particularly want to dig into it, adding additional ordering requirements does ratchet up the required knowledge level on those users as to rules on how to interact with that code. The property of virtual resources where the order in which you realize it does not matter versus where you declare the resource is quite valuable to smooth and ease the user experience in larger and complex environments where you have a wide range of Puppet skill levels involved. As far as using APL/lookup as the way to import the resources / key/value store data. This might be a convenient approach in some cases, but I would certainly still want a way to retrieve from that key/value store in a fashion that wasn't able to be overridden from the normal hierarchy resolution structure by users of that data. And to Henrik Lindberg's assertion that the problem is that exported resources work on resources I have actually always seen that has generally a good thing. Wh
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Henrik Lindberg commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax As Erik Dalén points out, many export/collect use cases are easily solved by simply doing the work required on the importing side. What I think is most problematic with exported and imported Resources is that they have to be resources - and they always end up being realized when importing. There are issues with versions of the type, with visibility, with assignments of defaults (from exporting or importing side?) etc. Some of those look like unsolvable ambiguities (which is probably why those problems have never been fixed). If the export/import is needed (that is, a publish/subscribe model which is very valuable in general) it seems better to me that there is a key/value store that you can read and write to in a controlled way. That way, an export could be a function call, and an import could be done with APL or lookup. APL and lookup provides tremendous flexibility in defining what gets assigned. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Henrik Lindberg commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax The resolutions of defaults is similar - i.e. setting of attributes with low precedence. What you cannot do then is to enforce (with high precedence) that all, or all matching instances of a type of Resource must have a particular value. If we then accept that there is some kind of loop over them at the end - then that is basically the same as the virtual collector with override. Collectors with overrides can be supported in the way I am thinking about Compilation - it is simply a predicated setting of resource attributes at a certain precedence. I still would want to drop the "virtual" and "realize" concepts. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title
Henrik Lindberg commented on PUP-7541
Re: Explore removing export / collect / virtual / realize syntax
Erik Dalén using a lambda could work.
The way I look at this is that at the same time as we drop collections + overrides we would also resolve multiple request to set attributes in one and the same resource. Thus if you do:
File { 'foo' : mode => '0777' }
In a module that depends on the module where 'foo' was declared (or from the site/environment) that declaration would win over the one with lower precedence. (Likewise if two modules being siblings both declare conflicting values there would be an error.
Thus, override is never needed as an imperative (with order of evaluation apparent) thing - it is simply a matter of precedence.
Add Comment
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Sean Millichamp commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Eric Sorenson To Erik Dalén's point, it is possible to work around a removal of the exported resource feature by digging through the facts for a host and performing all of the logic on the "collection" side of the compile using the facts from the host that you would have done on the "export" side, and then do all of that logic in terms of the results of the fact query - but why? We already have a context in which code and data are evaluated in terms of the "export" side - the Puppet compile of the agent which would have exported it! For very simple cases of logic perhaps it doesn't matter too much, but when you are dealing with many layers of conditionals and complex logic to arrive at what would be the exported resource I think it is far more cleaner and sensible to do those operations in the actual compiling context of the system the resource is being generated for. In terms of introspection into your environment, I think there is value in being able to ask "show me all of the resources associated with node foo". In some cases, I'm not quite as interested as where those resources are being applied as I am in seeing which node they are related to. True, this could probably be solved with tagging - but the current system is quite elegant in this regard. In terms of error handling: consider the scenario where you have some bad input or some other conditional occurring for which you might fail the catalog compile. In terms of a single node, a catalog compile is unfortunate, but only impacts that node. If all of that logic is moved to the device/node where the resources are acted upon now you are left with the choice of failing the whole catalog - thus impacting every single node relying on that catalog run - or opting for some soft failure condition which might include leaving out the resource (and who knows what the impact of that might be?) Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You receiv
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title
Erik Dalén commented on PUP-7541
Re: Explore removing export / collect / virtual / realize syntax
Henrik Lindberg For the case of overriding a resource it would be nicer if there was a function that could take a lambda and call that at the end of the compilation.
So instead of
File<| title="foo" |>{ mode => '0777' }
you would do something like
call_at_end() || { File["foo"]{ mode => '0777' } }
That would allow much more powerful stuff, not sure how easy it would be to implement though.
Add Comment
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To un
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Erik Dalén commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax Sean Millichamp when compiling the catalog on a host you essentially only have the facts for that host as input, and those same facts can be queried from another host using some puppetdb query function. So any resource you could template up and export could instead be templated on the target host instead. So there isn't really any need to put the final resource into puppetdb and then collect it on a different host as you can just as well just create it on that host directly instead. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Eric Sorenson commented on PUP-7541 Re: Explore removing export / collect / virtual / realize syntax (I adjusted the title to be less didactic) How would we declare the equivalent of exported resources on host A that should never be on host A but are only intended for some other host where they are realized? We use that pattern quite a bit. Or, put another way, how would we get resources into PuppetDB without having them declared on the host without the exported resources syntax? Sean Millichamp can you talk a little more about this? What problem does this solve for you? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7541) Explore removing export / collect / virtual / realize syntax
Title: Message Title Eric Sorenson updated an issue Puppet / PUP-7541 Explore removing export / collect / virtual / realize syntax Change By: Eric Sorenson Summary: Remove Explore removing export / collect / virtual / realize syntax Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
