On Fri, Sep 28, 2012 at 5:37 PM, jcbollinger <john.bollin...@stjude.org>wrote:
> > > On Friday, September 28, 2012 10:53:45 AM UTC-5, Jeremy wrote: >> >> >> The use of the "YAML.load(open(args[0]))" call was in fact to support >> both local and network files. In this case I'm actually giving an >> authenticated S3 bucket URL to retrieve the file as the engineers releasing >> the code also upload the deployment YAML file to the S3 bucket. The >> tarballs that are deployed are also in the S3 bucket and also pass >> authenticated URLs in the catalog with an expiration equal to the catalog >> expiration time. I'd like to eventually modify it to include retrieving the >> deployment file and storing it locally only when it's been modified but >> want to keep it in S3 as it allows my Puppet master to operate as a >> blackbox that engineers have no access to. If I control the deployment file >> locally they claim I'm the bottleneck slowing them down so as long as I >> give them the means to update it and the process flow is error free and >> only problems encountered are when they screw up the deployment file >> contents accountability is maintainable. >> > > > Given your target environment I can imagine why S3 may be attractive, but > if you yave not already done so then you should investigate whether it > provides the performance guarantees (and real-life performance) necessary > for the use to which you're putting it. > > Have you considered pulling over the deployment file to the master on a > periodic basis (such as via cron) so that it can always be local for your > module? > As the puppet master is also a client I've thought about setting it up as a file to retrieve and store locally. Though I could probably write a script to check for updates and have it ran from cron more frequently to ensure that the web app server isn't dependent on the master having run it's update. Don't want to give engineers even more reason to say the process takes too long. > >> >> My thought on the "smoking gun" is in having to make the parser function >> calls to try and determine the unique components and unique versions in the >> case of a component with multiple versions needing to be deployed. This was >> the quickest way I could find to get the deployment file format converted >> and ensure that I only defined a resource once avoiding the duplicate >> resource definition errors. As a result I'm calling the 2 functions which >> have to iterate through the entire YAML content merging then sorting for >> unique values separately. >> >> > > How big are the real deployment files? I wouldn't think that parsing and > processing even moderately large YAML files would be prohibitively > expensive in itself, especially when compared to the work the master must > perform to compile all the DSL code. In any case, you should be able to > test that against real data by wrapping a test harness around the innards > of your function. > > Looking at the report metrics I can see that successful runs show config retrieval taking up to 130 seconds but most common is around 110 seconds so not much difference. When it fails it usually fails with a "Could not retrieve catalog from remote server: execution expired" and a "Could not retrieve catalog; skipping run" error messages and then proceeds with the cached catalog. Currently the catalog has 370-390 resources defined with a change usually involving 170-180 resources. Cheers, > > John > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/AHoMhpuyhGkJ. > > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
