On 7 February 2017 at 10:44, Thomas Hallgren <thomas.hallg...@puppet.com> wrote:
> Like Martin Afke above, I'm very interested in what could be done using > the Puppet Language alone. What use cases can be covered entirely that way? > As David L pointed out, many of these things can be done today in defines, or classes, already. What is not feasible to ever do with PL? > Talk to external APIs. > Would an API that made it possible to write types and providers PL and > hand over tiny tasks to another language (Ruby, C, Go, etc.) be of > interest? I.e. a clean cut boundary between the things possible > (appropriate) to implement with PL and the things that aren't? > I do not understand how extending the puppet dsl on the server helps with resource management on the agent. > Perhaps this is a first step in that direction? > In one possible reading, types are already are a extension point of the DSL, and this proposal makes that less painful, and easier to reason about. Cheers, David PS: If anyone is at ConfigurationManagementCamp in Gent these days, hit me up to talk about this! > > - thomas > > > On Tue, Feb 7, 2017 at 11:31 AM, David Schmitt <david.schm...@puppet.com> > wrote: > >> Hi Corey, >> >> thanks for looking into this. I've been travelling so accept my apologies >> for a late answer. >> >> >> On 5 February 2017 at 20:52, Corey Osman <co...@logicminds.biz> wrote: >> >>> One improvement I would like to see with the provider API is around >>> getting the current state. I see many beginners including myself trying to >>> add way more complexity than required in the exists? method. >>> >>> One would think that you in order to figure out if the resource exists >>> that you should just query the system and DIY. And I'll pick on my own >>> code as an example: https://github.com/logicminds/ >>> bmclib/blob/0.1.0/lib/puppet/provider/bmc/ipmitool.rb#L34 >>> >>> However, this is not the case. This is the exact opposite of what you >>> need to do. Instead you need to query the API which is hard to understand >>> and not obvious at first because instinct tells you to get the state >>> yourself. >>> >>> https://github.com/logicminds/bmclib/blob/master/lib/puppet/ >>> provider/bmc/ipmitool.rb#L60 >>> >>> Another improvement which I hope would get addressed is self.prefetch. >>> While self.instances is pretty simple to understand the prefetch method >>> could probably use a better name and an easier way to implement. It was not >>> obvious at first how to implement. So many people don't implement these two >>> methods when they should making their provider more useful. >>> >> >> Exactly what this should address. The implementation of `get` is very >> straightforward (and, currently, required). >> >> The type checking looks pretty amazing and simple to understand and >>> implement in the type code. >>> >> >> Thanks! I hope that a provider author would never have to "implement" >> this. The SimpleResource will create a proper Type class underneath with >> all the validation and munging generated. >> >> >> >>> >>> Additionally, As Trevor mentioned I also have some projects >>> (puppet-debugger and puppet-retrospec) that utilize the current Puppet API >>> to introspec resources and query the catalog please so make sure this new >>> code doesn't break my projects. I'll be happy to test out any new API for >>> third party compatibility. >>> >> >> Thanks for the offer! For the time being, the implementation will sit >> completely on top of the unmodified puppet code to work on all puppet 4 >> versions out there. I also don't expect existing types to switch >> immediately, but I do hope that the new API makes implementations so much >> easier that people will switch quickly. >> >> Then it will be a matter of filling in the gaps for the special cases. >> >> Cheers, David S >> >> >> >>> >>> >>> Corey (NWOps) >>> >>> >>> >>> On Tuesday, January 31, 2017 at 8:04:19 AM UTC-8, David Schmitt wrote: >>>> >>>> Hi *, >>>> >>>> The type and provider API has been the bane of my existence since I >>>> [started writing native resources](https://github.com/ >>>> DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). >>>> Now, finally, we'll do something about it. I'm currently working on >>>> designing a nicer API for types and providers. My primary goals are to >>>> provide a smooth and simple ruby developer experience for both scripters >>>> and coders. Secondary goals were to eliminate server side code, and make >>>> puppet 4 data types available. Currently this is completely aspirational >>>> (i.e. no real code has been written), but early private feedback was >>>> encouraging. >>>> >>>> To showcase my vision, this [gist](https://gist.github.com >>>> /DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type]( >>>> https://github.com/puppetlabs/puppetlabs-apt/blob/mast >>>> er/lib/puppet/type/apt_key.rb) and [provider](https://github.com/ >>>> puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/ap >>>> t_key/apt_key.rb) ported over to my proposal. The second example there >>>> is a more long-term teaser on what would become possible with such an API. >>>> >>>> The new API, like the existing, has two parts: the implementation that >>>> interacts with the actual resources, a.k.a. the provider, and information >>>> about what the implementation is all about. Due to the different usage >>>> patterns of the two parts, they need to be passed to puppet in two >>>> different calls: >>>> >>>> The `Puppet::SimpleResource.implement()` call receives the >>>> `current_state = get()` and `set(current_state, target_state, noop)` >>>> methods. `get` returns a list of discovered resources, while `set` takes >>>> the target state and enforces those goals on the subject. There is only a >>>> single (ruby) object throughout an agent run, that can easily do caching >>>> and what ever else is required for a good functioning of the provider. The >>>> state descriptions passed around are simple lists of key/value hashes >>>> describing resources. This will allow the implementation wide latitude in >>>> how to organise itself for simplicity and efficiency. >>>> >>>> The `Puppet::SimpleResource.define()` call provides a data-only >>>> description of the Type. This is all that is needed on the server side to >>>> compile a manifest. Thanks to puppet 4 data type checking, this will >>>> already be much more strict (with less effort) than possible with the >>>> current APIs, while providing more automatically readable documentation >>>> about the meaning of the attributes. >>>> >>>> >>>> Details in no particular order: >>>> >>>> * All of this should fit on any unmodified puppet4 installation. It is >>>> completely additive and optional. Currently. >>>> >>>> * The Type definition >>>> * It is data-only. >>>> * Refers to puppet data types. >>>> * No code runs on the server. >>>> * This information can be re-used in all tooling around >>>> displaying/working with types (e.g. puppet-strings, console, ENC, etc.). >>>> * autorelations are restricted to unmodified attribute values and >>>> constant values. >>>> * No more `validate` or `munge`! For the edge cases not covered by >>>> data types, runtime checking can happen in the implementation on the agent. >>>> There it can use local system state (e.g. different mysql versions have >>>> different max table length constraints), and it will only fail the part of >>>> the resource tree, that is dependent on this error. There is already ample >>>> precedent for runtime validation, as most remote resources do not try to >>>> replicate the validation their target is already doing anyways. >>>> * It maps 1:1 to the capabilities of PCore, and is similar to the >>>> libral interface description (see [libral#1](https://github.com/ >>>> puppetlabs/libral/pull/2)). This ensures future interoperability >>>> between the different parts of the ecosystem. >>>> * Related types can share common attributes by sharing/merging the >>>> attribute hashes. >>>> * `defaults`, `read_only`, and similar data about attributes in the >>>> definition are mostly aesthetic at the current point in time, but will make >>>> for better documentation, and allow more intelligence built on top of this >>>> later. >>>> >>>> * The implementation are two simple functions `current_state = get()`, >>>> and `set(current_state, target_state, noop)`. >>>> * `get` on its own is already useful for many things, like puppet >>>> resource. >>>> * `set` receives the current state from `get`. While this is >>>> necessary for proper operation, there is a certain race condition there, if >>>> the system state changes between the calls. This is no different than what >>>> current implementations face, and they are well-equipped to deal with this. >>>> * `set` is called with a list of resources, and can do batching if it >>>> is beneficial. This is not yet supported by the agent. >>>> * the `current_state` and `target_state` values are lists of simple >>>> data structures built up of primitives like strings, numbers, hashes and >>>> arrays. They match the schema defined in the type. >>>> * Calling `r.set(r.get, r.get)` would ensure the current state. This >>>> should run without any changes, proving the idempotency of the >>>> implementation. >>>> * The ruby instance hosting the `get` and `set` functions is only >>>> alive for the duration of an agent transaction. An implementation can >>>> provide a `initialize` method to read credentials from the system, and >>>> setup other things as required. The single instance is used for all >>>> instances of the resource. >>>> * There is no direct dependency on puppet core libraries in the >>>> implementation. >>>> * While implementations can use utility functions, they are >>>> completely optional. >>>> * The dependencies on the `logger`, `commands`, and similar >>>> utilities can be supplied by a small utility library (TBD). >>>> >>>> * Having a well-defined small API makes remoting, stacking, proxying, >>>> batching, interactive use, and other shenanigans possible, which will make >>>> for a interesting time ahead. >>>> >>>> * The logging of updates to the transaction is only a sketch. See the >>>> usage of `logger` throughout the example. I've tried different styles for >>>> fit. >>>> * the `logger` is the primary way of reporting back information to >>>> the log, and the report. >>>> * results can be streamed for immediate feedback >>>> * block-based constructs allow detailed logging with little code >>>> ("Started X", "X: Doing Something", "X: Success|Failure", with one or two >>>> calls, and only one reference to X) >>>> >>>> * Obviously this is not sufficient to cover everything existing types >>>> and providers are able to do. For the first iteration we are choosing >>>> simplicity over functionality. >>>> * Generating more resource instances for the catalog during >>>> compilation (e.g. file#recurse or concat) becomes impossible with a pure >>>> data-driven Type. There is still space in the API to add server-side code. >>>> * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or >>>> should not be prefetched. While it might not be convenient, a provider >>>> could always return nothing on the `get()` and do a more customized enforce >>>> motion in the `set()`. >>>> * With current puppet versions, only "native" data types will be >>>> supported, as type aliases do not get pluginsynced. Yet. >>>> * With current puppet versions, `puppet resource` can't load the data >>>> types, and therefore will not be able to take full advantage of this. Yet. >>>> >>>> * There is some "convenient" infrastructure (e.g. parsedfile) that >>>> needs porting over to this model. >>>> >>>> * Testing becomes possible on a completely new level. The test library >>>> can know how data is transformed outside the API, and - using the shape of >>>> the type - start generating test cases, and checking the actions of the >>>> implementation. This will require developer help to isolate the >>>> implementation from real systems, but it should go a long way towards >>>> reducing the tedium in writing tests. >>>> >>>> >>>> What do you think about this? >>>> >>>> >>>> Cheers, David >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Developers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-dev+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/puppet-dev/95e3ecb5-e796-442f-ae73-7c40b67c6bc7%40googlegroups.com >>> <https://groups.google.com/d/msgid/puppet-dev/95e3ecb5-e796-442f-ae73-7c40b67c6bc7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-dev+unsubscr...@googlegroups.com. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/puppet-dev/CALF7fHbbiA31_QMT2VEBYBN4rJHoUZ8Cf%2BHtpHpCRT >> 0yHZ4UVg%40mail.gmail.com >> <https://groups.google.com/d/msgid/puppet-dev/CALF7fHbbiA31_QMT2VEBYBN4rJHoUZ8Cf%2BHtpHpCRT0yHZ4UVg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-dev+unsubscr...@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-dev/CAO5TtCtdy3JigiRxehAc2FOG-8vNXDBrUjJyJdqm_UKTb4STXQ% > 40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-dev/CAO5TtCtdy3JigiRxehAc2FOG-8vNXDBrUjJyJdqm_UKTb4STXQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALF7fHZysK%3D8c9gQ-G%3DXszmLjfjrahxY4e4Stcrm3K-70Jt7fw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
