Jira (PUP-2423) Filebucket server should warn, not fail, if checksum type is not supported

[Adrien Thebo (JIRA)]

Title: Message Title

Adrien Thebo commented on an issue   Re: Filebucket server should warn, not fail, if checksum type is not supported There are multiple touchpoints for how a Puppet master and agent when they have differing digest_algorithms. The first is how file resources back up to a remote filebucket, and the second is how Puppet enforces the state of a resource with a `puppet:///` source. To begin to address this issue I added a `disallowed_digest_algorithms` setting which takes a list of algorithms that cannot be used when interacting with the file bucket and metadata file server. By default this setting is empty which means that both md5 and sha256 algorithms will work out of the box. The following issues only come up when `disallowed_digest_algorithms` includes the digest_algorithm on a given agent. The first case with backup up files to a remote filebucket is complicated because the filebucket client queries the master to determine if the file has already been backed up (https://github.com/puppetlabs/puppet/blob/3.6.2/lib/puppet/file_bucket/dipper.rb#L48) which was added as part of #5838 (http://projects.puppetlabs.com/issues/5838). When a request is made to the file bucket using a disallowed algorithm that request cannot be answered, but we cannot just out and out fail a HEAD request. HEAD requests cannot return with a body so if a HEAD request fails we cannot return an error message in the body, so when we query the filebucket for a file with an invalid digest algorithm we can only return this message from the server: Error: Could not back up /tmp/local-bucketed-file: Error 400 on SERVER: Bad Request Error: Could not back up /tmp/local-bucketed-file: Error 400 on SERVER: Bad Request Error: /Stage[main]/Main/Node[default]/File[/tmp/local-bucketed-file]/content: change from {md5}dbeefb7ce205108d148794995023388f to {md5}00e4e9a0343ccd13d68af97ba3e955f1 failed: Could not back up /tmp/local-bucketed-file: Error 400 on SERVER: Bad Request Because we can only return this non-specific error, we cannot determine if an invalid digest was used, or if a valid digest algorithm was used but the associated checksum did not match the expected format (https://github.com/puppetlabs/puppet/blob/3.6.2/lib/puppet/indirector/file_bucket_file/file.rb#L109). As it stands we cannot indicate a specific error when responding to a HEAD request; we could modify the indirector to allow for richer semantics when raising errors from a terminus back into the v1 API, or expose the response object to a terminus but these are both nontrivial changes. Given that we cannot respond to the actual request, and we can't fail hard, we have two ways to respond - we can always return false, or we can always return true. If we always return false then we can fail gracefully when the client does a GET or POST, since those can return an error message on the body. Unfortunately if we take this approach, every time a POST is submitted the entire file will have to be uploaded before we can fail. The backup attempt will be run every time the agent runs on the given node, which means that we'll chew up bandwidth while there's a mismatch between server and agent. This problem is mitigated because only files that are changed will be uploaded to the filebucket, and resources that are in sync or created will not have to interact with the filebucket. If we always return true we bypass all concerns about bandwidth usage. Puppet catalogs will be applied without error which means that if Puppet is being used to manage puppet.conf and that file is being backed up, changes to that file can still be made. However we will silently stop backing up files to remote filebuckets on agents, which could be fairly dangerous if people are relying on this and it will violate the principle of least astonishment. I can see a couple of solutions for this. First off, we can add richer semantics to error conditions encountered in the indirector. We can't simply raise an HTTP error from the file_bucket_file/file terminus because the terminus code itself isn't necessarily coupled up to the v1 API, but we can add exceptions that indicate that a request was forbidden that the v1 API can translate to an HTTP 403. Alternately we can return error messages for HEAD requests in the header. This might mean adding custom code for HEAD requests to add errors to the response, or it could mean exposing information about an indirector response to terminuses. The file_metadata code doesn't use HEAD requests, so for all of those requests we can just raise exceptions as usual and generally the right thing happens. Add Comment   Puppet / PUP-2423 Filebucket server should warn, not fail, if checksum type is not supported PUP-1840 allows users to specify the digest algorithm to use for file bucketing and the file checksum parameter. However, if you reconfigure the master to use SHA256: {noformat} [main] digest_algorithm=sha256 {noformat} Then it will reject file bucket requests from agents that haven't switched over: {noformat} # puppet agent -td ... Error: ... This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.