Indeed... :) Ok, next question for you/all...
How could I differentiate between what is a resource creation and a resource modification? Have updated the netapp_user provider to prefetch/flush, however am struggling to differentiate between what is a resource creation, and what is a modification... Code is here: https://github.com/fatmcgav/fatmcgav-netapp/commit/66092978f4182c5474a60011db99ee2e3e12e689 Thoughts? Cheers Gavin On Tuesday, 2 April 2013 16:17:33 UTC+1, Luke Kanies wrote: > > That is awesome. It's amazing what a bit of optimization will do for you. > > On Apr 2, 2013, at 3:12 PM, fatmcgav <fatm...@gmail.com <javascript:>> > wrote: > > Yep, have been testing it today on my dev and production Puppet masters, > and seems to be working as expected... > > Have been very impressed with the performance improvements - Has dropped > the netapp_volume resource time from 40+ seconds to <0.1 seconds. So > MASSIVE improvements to be made from switching to the prefetch/flush > style... > > Cheers > Gav > > > On 2 April 2013 13:24, Luke Kanies <lu...@puppetlabs.com <javascript:>>wrote: > >> Looks good from a cursory glance. Hopefully it's all working for you. >> >> On Apr 2, 2013, at 9:52 AM, Gavin Williams <fatm...@gmail.com<javascript:>> >> wrote: >> >> As a quick update, I've pushed my latest set of code tweaks up - >> https://github.com/fatmcgav/fatmcgav-netapp/commit/7a7d18bf39cdbb04a3b0b5192929ec6a857c0a5e >> >> Need to tidy the code up a bit, but should be able to get the gist :) >> Have got a couple of defs which pull back the bulk of the information, and >> then a couple that need to be called on a volume by volume basis. >> All gets put together in instances, and provider'd up in prefetch. >> >> Comments welcome. >> >> Cheers >> Gavin >> >> On Friday, 29 March 2013 13:15:12 UTC, Gavin Williams wrote: >>> >>> Luke >>> >>> Cheers for that info, should prive useful... >>> >>> Guess I could implement a method for each property that pulls back all >>> that property values for all the volumes, and then match them up in >>> prefetch. >>> >>> Cheers >>> Gav >>> On 29 Mar 2013 12:49, "Luke Kanies" <lu...@puppetlabs.com <javascript:>> >>> wrote: >>> >>>> Hi Gavin, >>>> >>>> Yeah, there isn't as much consistency in API as I would like. >>>> >>>> The way it should work, I believe, is something like: >>>> >>>> * The Type should support an 'instances' method that returns a complete >>>> list of all instances of that type that exist on the host itself. This >>>> method should not require a catalog. >>>> >>>> * The provider should support a 'prefetch' method, which accepts a >>>> list of resources from the catalog and updates them with the correct >>>> provider instance, with appropriate data already filled in to reflect the >>>> system state. >>>> >>>> * Optionally, many providers implement (I think) their own 'instances' >>>> method, which returns all instances and doesn't require a catalog, to >>>> implement both of the above methods. >>>> >>>> I just looked at the Provider base class, though, and it's not at all >>>> clear from this. Part of the problem is that the system is smart enough >>>> to >>>> skip providers that don't support prefetch, and the way we denote 'doesn't >>>> support prefetch' is that it doesn't have the method. Given that, I left >>>> the method off of the base class. Because 'instances' isn't used for this >>>> kind of test, the base class ships with a stub method that just throws an >>>> error. >>>> >>>> So yeah, all that's confusing, and could really use some additional >>>> work to make it easier to understand, and to bring some consistency. >>>> >>>> On Mar 29, 2013, at 12:37 PM, fatmcgav <fatm...@gmail.com <javascript:>> >>>> wrote: >>>> >>>> Luke >>>> >>>> No worries. >>>> >>>> For the 2 I've converted soo far, it's made quite a performance >>>> difference. Thankfully can pull back all the resources with one NetApp api >>>> call, so it's drop the resource time from 5+ seconds to sub 1 second. >>>> However was a bit confused initially, as different providers seem to >>>> use different models... Some with just prefetch, some with just instances, >>>> some with both... >>>> >>>> However I'm just about to start the more complex netapp_volume >>>> provider, which has 4+ properties that have different api calls on top of >>>> the base resource list call. This is by far the slowest provider out of >>>> them with an average of 20+ seconds per run. >>>> So not sure of the best way to handle as yet. >>>> >>>> Cheers >>>> Gav >>>> On 29 Mar 2013 12:11, "Luke Kanies" <lu...@puppetlabs.com <javascript:>> >>>> wrote: >>>> >>>>> Hi Gavin, >>>>> >>>>> I'm glad to see you've sorted it out, sorry I didn't jump in before it >>>>> was all resolved. >>>>> >>>>> How has prefetching worked out for you, in terms of performance? >>>>> >>>>> On Mar 28, 2013, at 3:28 PM, Gavin Williams >>>>> <fatm...@gmail.com<javascript:>> >>>>> wrote: >>>>> >>>>> Afternoon >>>>> >>>>> Managed to find my issue.. Variable name re-use :( >>>>> >>>>> Defined 'qtrees' as an empty array on line 3, and then populated it >>>>> with a whole load of device output on line 18 :( >>>>> >>>>> So have defined a 'qtree_instances' array on the outside to contain my >>>>> output, with qtrees being used on the inside to hold the NetApp filer >>>>> response... Latest code in >>>>> Github<https://github.com/fatmcgav/fatmcgav-netapp/commit/1d51b1267466176db1a3d4e6ae32d9340a06fb56> >>>>> . >>>>> >>>>> Cheers >>>>> Gavin >>>>> >>>>> On Thursday, 28 March 2013 11:44:39 UTC, Gavin Williams wrote: >>>>>> >>>>>> Morning all >>>>>> >>>>>> Quick update... Looks like I managed to hack around the issue by >>>>>> adding the following: >>>>>> >>>>>> ... >>>>>> >>>>>> ap qtree_info >>>>>> >>>>>> # Check if it is a NaElement >>>>>> >>>>>> >>>>>> next unless qtree_info.respond_to?(:child_****get_string) >>>>>> >>>>>> # Pull out the qtree name. >>>>>> >>>>>> >>>>>> name = qtree_info.child_get_string("**q**tree") >>>>>> ... >>>>>> >>>>>> >>>>>> However this shows that I'm getting a total of 78 items processed, >>>>>> whereas the original array only contains 53 items... >>>>>> The additional items being processed are all like: '* >>>>>> #<Puppet::Type::Netapp_qtree::ProviderNetapp_qtree:*'. >>>>>> Have updated the gist with latest code and log file. >>>>>> >>>>>> Would like to understand where these are coming from, and if it's >>>>>> something I'm doing incorrectly? >>>>>> >>>>>> In the mean-time, following fixing that bug, the provider now seems >>>>>> to work as expected :) >>>>>> >>>>>> Cheers >>>>>> Gavin >>>>>> >>>>>> >>>>>> On Wednesday, 27 March 2013 17:32:23 UTC, Gavin Williams wrote: >>>>>>> >>>>>>> Afternoon all >>>>>>> >>>>>>> I've started working on converting a couple more of my NetApp >>>>>>> network device providers to use a prefetch/flush model, as can see >>>>>>> performance gains available, etc... >>>>>>> >>>>>>> Anyways, I'm having issues with my netapp_qtree provider. It would >>>>>>> appear that somehow, an additional *Puppet::Type...* row is getting >>>>>>> into an array and breaking things... >>>>>>> >>>>>>> Have created a gist here <https://gist.github.com/fatmcgav/5256240>with >>>>>>> the details, as the log file is quite long. >>>>>>> Also includes the *instances *and *prefetch* def's for my >>>>>>> netapp_qtree provider... >>>>>>> >>>>>>> As you can see on Line 337 of the log, the array contains 40 items, >>>>>>> however on line 734 *self.instances* is trying to process item >>>>>>> 41?!?! >>>>>>> What's also strange is that the item contents look like a Puppet >>>>>>> Type (*#<Puppet::Type::Netapp_qtree*)****, whereas all the others >>>>>>> in the array are NetApp specific items (*#<NaElement:*). >>>>>>> >>>>>>> So, any ideas??? >>>>>>> >>>>>>> Cheers >>>>>>> Gavin >>>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Puppet Developers" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to puppet-dev+...@**googlegroups.com <javascript:>. >>>>> To post to this group, send email to >>>>> puppe...@googlegroups.com<javascript:> >>>>> . >>>>> Visit this group at >>>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>>> . >>>>> For more options, visit >>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>>> . >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>>> +1-615-594-8199 >>>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>>> http://bit.ly/pupconf13 >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Puppet Developers" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>>>> topic/puppet-dev/74JW491YSAk/**unsubscribe?hl=en<https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en> >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> puppet-dev+...@**googlegroups.com <javascript:>. >>>>> To post to this group, send email to >>>>> puppe...@googlegroups.com<javascript:> >>>>> . >>>>> Visit this group at >>>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>>> . >>>>> For more options, visit >>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>>> . >>>>> >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Developers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-dev+...@**googlegroups.com <javascript:>. >>>> To post to this group, send email to puppe...@googlegroups.com<javascript:> >>>> . >>>> Visit this group at >>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >>>> +1-615-594-8199 >>>> Join us at PuppetConf 2013, August 22-23 in San Francisco - >>>> http://bit.ly/pupconf13 >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Puppet Developers" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>>> topic/puppet-dev/74JW491YSAk/**unsubscribe?hl=en<https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en> >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> puppet-dev+...@**googlegroups.com <javascript:>. >>>> To post to this group, send email to puppe...@googlegroups.com<javascript:> >>>> . >>>> Visit this group at >>>> http://groups.google.com/**group/puppet-dev?hl=en<http://groups.google.com/group/puppet-dev?hl=en> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-dev+...@googlegroups.com <javascript:>. >> To post to this group, send email to puppe...@googlegroups.com<javascript:> >> . >> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> >> >> >> -- >> Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | >> +1-615-594-8199 >> Join us at PuppetConf 2013, August 22-23 in San Francisco - >> http://bit.ly/pupconf13 >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Puppet Developers" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en >> . >> To unsubscribe from this group and all its topics, send an email to >> puppet-dev+...@googlegroups.com <javascript:>. >> To post to this group, send email to puppe...@googlegroups.com<javascript:> >> . >> Visit this group at http://groups.google.com/group/puppet-dev?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-dev+...@googlegroups.com <javascript:>. > To post to this group, send email to puppe...@googlegroups.com<javascript:> > . > Visit this group at http://groups.google.com/group/puppet-dev?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > -- > Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | > +1-615-594-8199 > Join us at PuppetConf 2013, August 22-23 in San Francisco - > http://bit.ly/pupconf13 > > -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To post to this group, send email to puppet-dev@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-dev?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
