Hi Corey, thanks for looking into this. I've been travelling so accept my apologies for a late answer.
On 5 February 2017 at 20:52, Corey Osman <[email protected]> wrote: > One improvement I would like to see with the provider API is around > getting the current state. I see many beginners including myself trying to > add way more complexity than required in the exists? method. > > One would think that you in order to figure out if the resource exists > that you should just query the system and DIY. And I'll pick on my own > code as an example: https://github.com/logicminds/ > bmclib/blob/0.1.0/lib/puppet/provider/bmc/ipmitool.rb#L34 > > However, this is not the case. This is the exact opposite of what you > need to do. Instead you need to query the API which is hard to understand > and not obvious at first because instinct tells you to get the state > yourself. > > https://github.com/logicminds/bmclib/blob/master/lib/puppet/ > provider/bmc/ipmitool.rb#L60 > > Another improvement which I hope would get addressed is self.prefetch. > While self.instances is pretty simple to understand the prefetch method > could probably use a better name and an easier way to implement. It was not > obvious at first how to implement. So many people don't implement these two > methods when they should making their provider more useful. > Exactly what this should address. The implementation of `get` is very straightforward (and, currently, required). The type checking looks pretty amazing and simple to understand and > implement in the type code. > Thanks! I hope that a provider author would never have to "implement" this. The SimpleResource will create a proper Type class underneath with all the validation and munging generated. > > Additionally, As Trevor mentioned I also have some projects > (puppet-debugger and puppet-retrospec) that utilize the current Puppet API > to introspec resources and query the catalog please so make sure this new > code doesn't break my projects. I'll be happy to test out any new API for > third party compatibility. > Thanks for the offer! For the time being, the implementation will sit completely on top of the unmodified puppet code to work on all puppet 4 versions out there. I also don't expect existing types to switch immediately, but I do hope that the new API makes implementations so much easier that people will switch quickly. Then it will be a matter of filling in the gaps for the special cases. Cheers, David S > > > Corey (NWOps) > > > > On Tuesday, January 31, 2017 at 8:04:19 AM UTC-8, David Schmitt wrote: >> >> Hi *, >> >> The type and provider API has been the bane of my existence since I >> [started writing native resources](https://github.com/ >> DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). >> Now, finally, we'll do something about it. I'm currently working on >> designing a nicer API for types and providers. My primary goals are to >> provide a smooth and simple ruby developer experience for both scripters >> and coders. Secondary goals were to eliminate server side code, and make >> puppet 4 data types available. Currently this is completely aspirational >> (i.e. no real code has been written), but early private feedback was >> encouraging. >> >> To showcase my vision, this [gist](https://gist.github.com >> /DavidS/430330ae43ba4b51fe34bd27ddbe4bc7) has the [apt_key type]( >> https://github.com/puppetlabs/puppetlabs-apt/blob/ >> master/lib/puppet/type/apt_key.rb) and [provider](https://github.com/ >> puppetlabs/puppetlabs-apt/blob/master/lib/puppet/provider/ >> apt_key/apt_key.rb) ported over to my proposal. The second example there >> is a more long-term teaser on what would become possible with such an API. >> >> The new API, like the existing, has two parts: the implementation that >> interacts with the actual resources, a.k.a. the provider, and information >> about what the implementation is all about. Due to the different usage >> patterns of the two parts, they need to be passed to puppet in two >> different calls: >> >> The `Puppet::SimpleResource.implement()` call receives the >> `current_state = get()` and `set(current_state, target_state, noop)` >> methods. `get` returns a list of discovered resources, while `set` takes >> the target state and enforces those goals on the subject. There is only a >> single (ruby) object throughout an agent run, that can easily do caching >> and what ever else is required for a good functioning of the provider. The >> state descriptions passed around are simple lists of key/value hashes >> describing resources. This will allow the implementation wide latitude in >> how to organise itself for simplicity and efficiency. >> >> The `Puppet::SimpleResource.define()` call provides a data-only >> description of the Type. This is all that is needed on the server side to >> compile a manifest. Thanks to puppet 4 data type checking, this will >> already be much more strict (with less effort) than possible with the >> current APIs, while providing more automatically readable documentation >> about the meaning of the attributes. >> >> >> Details in no particular order: >> >> * All of this should fit on any unmodified puppet4 installation. It is >> completely additive and optional. Currently. >> >> * The Type definition >> * It is data-only. >> * Refers to puppet data types. >> * No code runs on the server. >> * This information can be re-used in all tooling around >> displaying/working with types (e.g. puppet-strings, console, ENC, etc.). >> * autorelations are restricted to unmodified attribute values and >> constant values. >> * No more `validate` or `munge`! For the edge cases not covered by data >> types, runtime checking can happen in the implementation on the agent. >> There it can use local system state (e.g. different mysql versions have >> different max table length constraints), and it will only fail the part of >> the resource tree, that is dependent on this error. There is already ample >> precedent for runtime validation, as most remote resources do not try to >> replicate the validation their target is already doing anyways. >> * It maps 1:1 to the capabilities of PCore, and is similar to the >> libral interface description (see [libral#1](https://github.com/ >> puppetlabs/libral/pull/2)). This ensures future interoperability between >> the different parts of the ecosystem. >> * Related types can share common attributes by sharing/merging the >> attribute hashes. >> * `defaults`, `read_only`, and similar data about attributes in the >> definition are mostly aesthetic at the current point in time, but will make >> for better documentation, and allow more intelligence built on top of this >> later. >> >> * The implementation are two simple functions `current_state = get()`, >> and `set(current_state, target_state, noop)`. >> * `get` on its own is already useful for many things, like puppet >> resource. >> * `set` receives the current state from `get`. While this is necessary >> for proper operation, there is a certain race condition there, if the >> system state changes between the calls. This is no different than what >> current implementations face, and they are well-equipped to deal with this. >> * `set` is called with a list of resources, and can do batching if it >> is beneficial. This is not yet supported by the agent. >> * the `current_state` and `target_state` values are lists of simple >> data structures built up of primitives like strings, numbers, hashes and >> arrays. They match the schema defined in the type. >> * Calling `r.set(r.get, r.get)` would ensure the current state. This >> should run without any changes, proving the idempotency of the >> implementation. >> * The ruby instance hosting the `get` and `set` functions is only alive >> for the duration of an agent transaction. An implementation can provide a >> `initialize` method to read credentials from the system, and setup other >> things as required. The single instance is used for all instances of the >> resource. >> * There is no direct dependency on puppet core libraries in the >> implementation. >> * While implementations can use utility functions, they are >> completely optional. >> * The dependencies on the `logger`, `commands`, and similar utilities >> can be supplied by a small utility library (TBD). >> >> * Having a well-defined small API makes remoting, stacking, proxying, >> batching, interactive use, and other shenanigans possible, which will make >> for a interesting time ahead. >> >> * The logging of updates to the transaction is only a sketch. See the >> usage of `logger` throughout the example. I've tried different styles for >> fit. >> * the `logger` is the primary way of reporting back information to the >> log, and the report. >> * results can be streamed for immediate feedback >> * block-based constructs allow detailed logging with little code >> ("Started X", "X: Doing Something", "X: Success|Failure", with one or two >> calls, and only one reference to X) >> >> * Obviously this is not sufficient to cover everything existing types and >> providers are able to do. For the first iteration we are choosing >> simplicity over functionality. >> * Generating more resource instances for the catalog during compilation >> (e.g. file#recurse or concat) becomes impossible with a pure data-driven >> Type. There is still space in the API to add server-side code. >> * Some resources (e.g. file, ssh_authorized_keys, concat) cannot or >> should not be prefetched. While it might not be convenient, a provider >> could always return nothing on the `get()` and do a more customized enforce >> motion in the `set()`. >> * With current puppet versions, only "native" data types will be >> supported, as type aliases do not get pluginsynced. Yet. >> * With current puppet versions, `puppet resource` can't load the data >> types, and therefore will not be able to take full advantage of this. Yet. >> >> * There is some "convenient" infrastructure (e.g. parsedfile) that needs >> porting over to this model. >> >> * Testing becomes possible on a completely new level. The test library >> can know how data is transformed outside the API, and - using the shape of >> the type - start generating test cases, and checking the actions of the >> implementation. This will require developer help to isolate the >> implementation from real systems, but it should go a long way towards >> reducing the tedium in writing tests. >> >> >> What do you think about this? >> >> >> Cheers, David >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/puppet-dev/95e3ecb5-e796-442f-ae73-7c40b67c6bc7%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-dev/95e3ecb5-e796-442f-ae73-7c40b67c6bc7%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALF7fHbbiA31_QMT2VEBYBN4rJHoUZ8Cf%2BHtpHpCRT0yHZ4UVg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
