On 7 February 2017 at 10:44, Thomas Hallgren <[email protected]> wrote:
> Like Martin Afke above, I'm very interested in what could be done using > the Puppet Language alone. What use cases can be covered entirely that way? > As David L pointed out, many of these things can be done today in defines, or classes, already. What is not feasible to ever do with PL? > Talk to external APIs. > Would an API that made it possible to write types and providers PL and > hand over tiny tasks to another language (Ruby, C, Go, etc.) be of > interest? I.e. a clean cut boundary between the things possible > (appropriate) to implement with PL and the things that aren't? > I do not understand how extending the puppet dsl on the server helps with resource management on the agent. > Perhaps this is a first step in that direction? > In one possible reading, types are already are a extension point of the DSL, and this proposal makes that less painful, and easier to reason about. Cheers, David PS: If anyone is at ConfigurationManagementCamp in Gent these days, hit me up to talk about this! > > - thomas > > > On Tue, Feb 7, 2017 at 11:31 AM, David Schmitt <[email protected]> > wrote: > >> Hi Corey, >> >> thanks for looking into this. I've been travelling so accept my apologies >> for a late answer. >> >> >> On 5 February 2017 at 20:52, Corey Osman <[email protected]> wrote: >> >>> One improvement I would like to see with the provider API is around >>> getting the current state. I see many beginners including myself trying to >>> add way more complexity than required in the exists? method. >>> >>> One would think that you in order to figure out if the resource exists >>> that you should just query the system and DIY. And I'll pick on my own >>> code as an example: https://github.com/logicminds/ >>> bmclib/blob/0.1.0/lib/puppet/provider/bmc/ipmitool.rb#L34 >>> >>> However, this is not the case. This is the exact opposite of what you >>> need to do. Instead you need to query the API which is hard to understand >>> and not obvious at first because instinct tells you to get the state >>> yourself. >>> >>> https://github.com/logicminds/bmclib/blob/master/lib/puppet/ >>> provider/bmc/ipmitool.rb#L60 >>> >>> Another improvement which I hope would get addressed is self.prefetch. >>> While self.instances is pretty simple to understand the prefetch method >>> could probably use a better name and an easier way to implement. It was not >>> obvious at first how to implement. So many people don't implement these two >>> methods when they should making their provider more useful. >>> >> >> Exactly what this should address. The implementation of `get` is very >> straightforward (and, currently, required). >> >> The type checking looks pretty amazing and simple to understand and >>> implement in the type code. >>> >> >> Thanks! I hope that a provider author would never have to "implement" >> this. The SimpleResource will create a proper Type class underneath with >> all the validation and munging generated. >> >> >> >>> >>> Additionally, As Trevor mentioned I also have some projects >>> (puppet-debugger and puppet-retrospec) that utilize the current Puppet API >>> to introspec resources and query the catalog please so make sure this new >>> code doesn't break my projects. I'll be happy to test out any new API for >>> third party compatibility. >>> >> >> Thanks for the offer! For the time being, the implementation will sit >> completely on top of the unmodified puppet code to work on all puppet 4 >> versions out there. I also don't expect existing types to switch >> immediately, but I do hope that the new API makes implementations so much >> easier that people will switch quickly. >> >> Then it will be a matter of filling in the gaps for the special cases. >> >> Cheers, David S >> >> >> >>> >>> >>> Corey (NWOps) >>> >>> >>> >>> On Tuesday, January 31, 2017 at 8:04:19 AM UTC-8, David Schmitt wrote: >>>> >>>> Hi *, >>>> >>>> The type and provider API has been the bane of my existence since I >>>> [started writing native resources](https://github.com/ >>>> DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). >>>> Now, finally, we'll do something about it. I'm currently working on >>>> designing a nicer API for types and providers. My primary goals are to >>>> provide a smooth and simple ruby developer experience for both scripters >>>> and coders. Secondary goals were to eliminate server side code, and make >>>> puppet 4 data types available. Currently this is completely aspirational >>>> (i.e. no real code has been written), but early private feedback was >>>> encouraging. >>>> >>>> To showcase my vision, this [gist](https://gist.github.com >>>> /DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type]( >>>> https://github.com/puppetlabs/puppetlabs-apt/blob/mast >>>> er/lib/puppet/type/apt_key.rb) and [provider](https://github.com/ >>>> puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/ap >>>> t_key/apt_key.rb) ported over to my proposal. The second example there >>>> is a more long-term teaser on what would become possible with such an API. >>>> >>>> The new API, like the existing, has two parts: the implementation that >>>> interacts with the actual resources, a.k.a. the provider, and information >>>> about what the implementation is all about. Due to the different usage >>>> patterns of the two parts, they need to be passed to puppet in two >>>> different calls: >>>> >>>> The `Puppet::SimpleResource.implement()` call receives the >>>> `current_state = get()` and `set(current_state, target_state, noop)` >>>> methods. `get` returns a list of discovered resources, while `set` takes >>>> the target state and enforces those goals on the subject. There is only a >>>> single (ruby) object throughout an agent run, that can easily do caching >>>> and what ever else is required for a good functioning of the provider. The >>>> state descriptions passed around are simple lists of key/value hashes >>>> describing resources. This will allow the implementation wide latitude in >>>> how to organise itself for simplicity and efficiency. >>>> >>>> The `Puppet::SimpleResource.define()` call provides a data-only >>>> description of the Type. This is all that is needed on the server side to >>>> compile a manifest. Thanks to puppet 4 data type checking, this will >>>> already be much more strict (with less effort) than possible with the >>>> current APIs, while providing more automatically readable documentation >>>> about the meaning of the attributes. >>>> >>>> >>>> Details in no particular order: >>>> >>>> * All of this should fit on any unmodified puppet4 installation. It is >>>> completely additive and optional. Currently. >>>> >>>> * The Type definition >>>> * It is data-only. >>>> * Refers to puppet data types. >>>> * No code runs on the server. >>>> * This information can be re-used in all tooling around >>>> displaying/working with types (e.g. puppet-strings, console, ENC, etc.). >>>> * autorelations are restricted to unmodified attribute values and >>>> constant values. >>>> * No more `validate` or `munge`! For the edge cases not covered by >>>> data types, runtime checking can happen in the implementation on the agent. >>>> There it can use local system state (e.g. different mysql versions have >>>> different max table length constraints), and it will only fail the part of >>>> the resource tree, that is dependent on this error. There is already ample >>>> precedent for runtime validation, as most remote resources do not try to >>>> replicate the validation their target is already doing anyways. >>>> * It maps 1:1 to the capabilities of PCore, and is similar to the >>>> libral interface description (see [libral#1](https://github.com/ >>>> puppetlabs/libral/pull/2)). This ensures future interoperability >>>> between the different parts of the ecosystem. >>>> * Related types can share common attributes by sharing/merging the >>>> attribute hashes. >>>> * `defaults`, `read_only`, and similar data about attributes in the >>>> definition are mostly aesthetic at the current point in time, but will make >>>> for better documentation, and allow more intelligence built on top of this >>>> later. >>>> >>>> * The implementation are two simple functions `current_state = get()`, >>>> and `set(current_state, target_state, noop)`. >>>> * `get` on its own is already useful for many things, like puppet >>>> resource. >>>> * `set` receives the current state from `get`. While this is >>>> necessary for proper operation, there is a certain race condition there, if >>>> the system state changes between the calls. This is no different than what >>>> current implementations face, and they are well-equipped to deal with this. >>>> * `set` is called with a list of resources, and can do batching if it >>>> is beneficial. This is not yet supported by the agent. >>>> * the `current_state` and `target_state` values are lists of simple >>>> data structures built up of primitives like strings, numbers, hashes and >>>> arrays. They match the schema defined in the type. >>>> * Calling `r.set(r.get, r.get)` would ensure the current state. This >>>> should run without any changes, proving the idempotency of the >>>> implementation. >>>> * The ruby instance hosting the `get` and `set` functions is only >>>> alive for the duration of an agent transaction. An implementation can >>>> provide a `initialize` method to read credentials from the system, and >>>> setup other things as required. The single instance is used for all >>>> instances of the resource. >>>> * There is no direct dependency on puppet core libraries in the >>>> implementation. >>>> * While implementations can use utility functions, they are >>>> completely optional. >>>> * The dependencies on the `logger`, `commands`, and similar >>>> utilities can be supplied by a small utility library (TBD). >>>> >>>> * Having a well-defined small API makes remoting, stacking, proxying, >>>> batching, interactive use, and other shenanigans possible, which will make >>>> for a interesting time ahead. >>>> >>>> * The logging of updates to the transaction is only a sketch. See the >>>> usage of `logger` throughout the example. I've tried different styles for >>>> fit. >>>> * the `logger` is the primary way of reporting back information to >>>> the log, and the report. >>>> * results can be streamed for immediate feedback >>>> * block-based constructs allow detailed logging with little code >>>> ("Started X", "X: Doing Something", "X: Success|Failure", with one or two >>>> calls, and only one reference to X) >>>> >>>> * Obviously this is not sufficient to cover everything existing types >>>> and providers are able to do. For the first iteration we are choosing >>>> simplicity over functionality. >>>> * Generating more resource instances for the catalog during >>>> compilation (e.g. file#recurse or concat) becomes impossible with a pure >>>> data-driven Type. There is still space in the API to add server-side code. >>>> * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or >>>> should not be prefetched. While it might not be convenient, a provider >>>> could always return nothing on the `get()` and do a more customized enforce >>>> motion in the `set()`. >>>> * With current puppet versions, only "native" data types will be >>>> supported, as type aliases do not get pluginsynced. Yet. >>>> * With current puppet versions, `puppet resource` can't load the data >>>> types, and therefore will not be able to take full advantage of this. Yet. >>>> >>>> * There is some "convenient" infrastructure (e.g. parsedfile) that >>>> needs porting over to this model. >>>> >>>> * Testing becomes possible on a completely new level. The test library >>>> can know how data is transformed outside the API, and - using the shape of >>>> the type - start generating test cases, and checking the actions of the >>>> implementation. This will require developer help to isolate the >>>> implementation from real systems, but it should go a long way towards >>>> reducing the tedium in writing tests. >>>> >>>> >>>> What do you think about this? >>>> >>>> >>>> Cheers, David >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/puppet-dev/95e3ecb5-e796-442f-ae73-7c40b67c6bc7%40googlegroups.com >>> <https://groups.google.com/d/msgid/puppet-dev/95e3ecb5-e796-442f-ae73-7c40b67c6bc7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/puppet-dev/CALF7fHbbiA31_QMT2VEBYBN4rJHoUZ8Cf%2BHtpHpCRT >> 0yHZ4UVg%40mail.gmail.com >> <https://groups.google.com/d/msgid/puppet-dev/CALF7fHbbiA31_QMT2VEBYBN4rJHoUZ8Cf%2BHtpHpCRT0yHZ4UVg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-dev/CAO5TtCtdy3JigiRxehAc2FOG-8vNXDBrUjJyJdqm_UKTb4STXQ% > 40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-dev/CAO5TtCtdy3JigiRxehAc2FOG-8vNXDBrUjJyJdqm_UKTb4STXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALF7fHZysK%3D8c9gQ-G%3DXszmLjfjrahxY4e4Stcrm3K-70Jt7fw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
