Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Josh Cooper commented on PUP-5714 Re: Add a 'git' checksum type for file resource sources If we improve/cache file metadata, then it'll likely be done as part of a code manager/r10k deployment (like we do for puppet generate types). So I'm going to close this as won't do. Add Comment This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.112551.1452820376000.53639.1623344880384%40Atlassian.JIRA.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Geoff Nichols updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Geoff Nichols Team: Agent Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Henrik Lindberg commented on PUP-5714 Re: Add a 'git' checksum type for file resource sources David Lutterkort This is another case where it is of value to compute things on a deploy (we could also pre-parse all .pp files that have changed - suggested in other tickets). The class information service invoked via puppet server (obtains a list of classes and all of their parameters) is yet another place where we would benefit. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Henrik Lindberg updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Henrik Lindberg Labels: triaged Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Kylo Ginsberg updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Kylo Ginsberg Sprint: Direct Puppet Mop Up Add Comment This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Kylo Ginsberg updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Kylo Ginsberg Sprint: PCP Triage Add Comment This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Kylo Ginsberg updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Kylo Ginsberg Sprint: PCP Triage Add Comment This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Josh Cooper updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Josh Cooper One of the downsides of static catalog compilation is that the master needs to checksum all the files referenced in a catalog ahead of time . Without static catalogs , to include their checksums in the catalog master computes checksums during every puppet agent run (across N nodes). With static catalogs , which can be intense on the master computes checksums once per compilation (across N nodes).Checksums are compute and i/o resources intensive . With Direct Puppet, we're providing a linkage between a git-managed backing store and the file content but aren't (yet) leveraging the fact that git inherently knows the checksum of every file at every commit-ref under its control. This seems like it'd be a good idea from a resource-conservation standpoint, plus it's bloody awesome.Implementing this would require either embedding/requiring git on the agent or re-creating the algorithm git uses to generate its checksums ( see http://git-scm.com/book/en/v2/Git-Internals-Git-Objects#Object-Storage for the gory details ), but the server could simply query the file-sync service store (for PE) or user-specified git backend (for OSS) during catalog compilation and save some work.This isn't slated for a particular release but came up out of some internal discussion and I wanted to capture it in a ticket for future prioritization. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Josh Cooper updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Josh Cooper Sprint: Direct Puppet Mop Up Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Henrik Lindberg updated an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Change By: Henrik Lindberg Scrum Team: Client Platform Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title David Lutterkort commented on PUP-5714 Re: Add a 'git' checksum type for file resource sources A middleground for avoiding redundant checksum computations would be to cache checksums server-side, and invalidate that cache everytime new code is pushed. The cache could either be populated lazily (i.e., only when the checksum for a certain file is actually needed) or eagerly (e.g., as part of updating code) This would incur one checksum calculation per file and code_id, rather than none, but would not require introducing a new checksum type into the agent. The cache could also be keyed off checksum type so that we can cache different kinds of checksums, e.g. md5 and mtime Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Michael Smith commented on PUP-5714 Re: Add a 'git' checksum type for file resource sources Should this be in the Direct Puppet epic? Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title
Josh Cooper commented on PUP-5714
Re: Add a 'git' checksum type for file resource sources
Some implementation notes:
Add a new type of checksum to the file type, e.g. checksum => 'git'. The git hash could be specified in the manifest, e.g. checksum_value => 'CAFE...', computed from a function, e.g. checksum_value => git_hash('modules/ntp/ntpd.conf'), or inlined as part of the static catalog process. In all cases, the checksum value is specified in the manifest at compilation time.
The agent needs to implement the git checksum, which amounts to computing the following hash:
SHA1(header + content)
where header is blob \0 where is the content length. The agent can then compare the current git checksum against the desired checksum, and the master would not need to calculate any checksums.
Add Comment
This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5714) Add a 'git' checksum type for file resource sources
Title: Message Title Eric Sorenson created an issue Puppet / PUP-5714 Add a 'git' checksum type for file resource sources Issue Type: Bug Assignee: Unassigned Created: 2016/01/14 5:12 PM Priority: Normal Reporter: Eric Sorenson One of the downsides of static catalog compilation is that the master needs to checksum all the files referenced in a catalog ahead of time, to include their checksums in the catalog, which can be intense on compute and i/o resources. With Direct Puppet, we're providing a linkage between a git-managed backing store and the file content but aren't (yet) leveraging the fact that git inherently knows the checksum of every file at every commit-ref under its control. This seems like it'd be a good idea from a resource-conservation standpoint, plus it's bloody awesome. Implementing this would require either embedding/requiring git on the agent or re-creating the algorithm git uses to generate its checksums ( see http://git-scm.com/book/en/v2/Git-Internals-Git-Objects#Object-Storage for the gory details ), but the server could simply query the file-sync service store (for PE) or user-specified git backend (for OSS) during catalog compilation and save some work. This isn't slated for a particular release but came up out of some internal discussion and I wanted to capture it in a ticket for future prioritization.
