Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title Josh Cooper commented on PUP-8300 Re: File resource can't handle HTTP redirects when server does not accept HEAD requests The issue with HEAD requests failing due to Amazon presigned URLs is filed as PUP-6380. The issue with duplicate HEAD requests has been resolved now that we're using the HTTP client for all network requests. The issue with checksums is filed as PUP-10368, and I'm going to close this as a duplicate. Add Comment This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-bugs/JIRA.228476.1514629492000.65779.1589914260135%40Atlassian.JIRA.
Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title Josh Cooper updated an issue Puppet / PUP-8300 File resource can't handle HTTP redirects when server does not accept HEAD requests Change By: Josh Cooper Sub-team: Coremunity Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title Mariusz Gronczewski commented on PUP-8300 Re: File resource can't handle HTTP redirects when server does not accept HEAD requests Note that if you already know checksum of file does not match you don't need to make any HEAD requests - as you already know you need to download new version of the file. You only need to do HEAD for resources without checksum. And in that case you could probably just do GET with if-modified-since header but that would be a problem if server does not support it (.. if that is even a problem ? I'm not sure, I'd imagine most do) Whenever puppet downloads a file, it should write to a tempfile, and verify the downloaded content matches the expected checksum. Note if the http server doesn't send the content-md5 header, then we have to fall back to last-modified, which won't ensure the downloaded file checksum is correct. At that point, just fail the resource with sensible error. That is horrible security disaster waiting to happen. Just think about it: User changes MD5 hash (which is currently broken) into something secure like sha256. Effect: any attacker can serve any file under that URL because sha256 is silently ignored when downloading file and only indication is same resource running over and over again. Software on web server changes to something that doesn't serve MD5 header - effect same as above, checksum silently ignored. Everything works, MD5 hash is checked - still vulnerable as causing MD5 collision is pretty trivial with current computing power. With the current implementation checksum + http(s) is utterly pointless. It should either: checksum tmp file before moving checksum while downloading - most (all?) checksum algorightms support incremental checksumming and that way will probably be a bit faster for big files. Add Comment
Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title Josh Cooper commented on PUP-8300 Re: File resource can't handle HTTP redirects when server does not accept HEAD requests The 4 requests are due to a bug. We first make a HEAD request to get the content-md5 or last-modified "checksums". Note the server must return one of these headers in order for puppet to not download the file every time it runs. Puppet also doesn't support ETag. For reasons unknown the code would make 2 HEAD requests. I eliminated the second request in https://github.com/puppetlabs/puppet/commit/deac457fe305b487614eb280fe9e67d518d13d97. If the destination file is not "insync" (it's missing, "checksums" don't match, or the server didn't send a checksum), then we try to download the file contents. We make another HEAD request apparently to see if we should be redirected, and then issue the GET. I don't believe this HEAD request is necessary, and should probably be eliminated. Seems more correct to just issue the GET request, and follow the redirect(s), if any. Whenever puppet downloads a file, it should write to a tempfile, and verify the downloaded content matches the expected checksum. Note if the http server doesn't send the content-md5 header, then we have to fall back to last-modified, which won't ensure the downloaded file checksum is correct. So for this ticket it sounds like there are two things: Eliminate the one extra HEAD request Improve documentation that the server must reply with Content-MD5 or Last-Modified in the HEAD response so that the agent doesn't download the file on every run. And we can keep PUP-6380 open for how to handle servers like S3 that don't allow HEAD requests. Does that make sense? Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title Mariusz Gronczewski commented on PUP-8300 Re: File resource can't handle HTTP redirects when server does not accept HEAD requests Okay, but why it does 3 HEADs then ? On my localhost server (no redirect, just nginx on localhost serving a directory), just requesting one file yields me access log like that: 127.0.0.1 - - [10/Jan/2018:10:42:18 +0100] "HEAD /rnd HTTP/1.1" 200 0 "-" "Ruby" 127.0.0.1 - - [10/Jan/2018:10:42:18 +0100] "HEAD /rnd HTTP/1.1" 200 0 "-" "Ruby" 127.0.0.1 - - [10/Jan/2018:10:42:18 +0100] "HEAD /rnd HTTP/1.1" 200 0 "-" "Ruby" 127.0.0.1 - - [10/Jan/2018:10:42:18 +0100] "GET /rnd HTTP/1.1" 200 1024 "-" "Ruby" then it downloads the file and saves it in destination... *without checking the checksum*. On next run, it does check the checksum of file on the disk, then re-downloads it: Notice: /Stage[main]/Main/File[/tmp/rnd]/checksum_value: checksum_value changed '34bcf0e1286630afc465ac5f6219cabc97f63567e9f206442d8314a451e406a3' to '19248907fa84d23721987abf90f9f7cbac44c547fd38ef2c0ce31e194eb4d9ed' Notice: Applied catalog in 0.12 seconds and that is repeated in every run. So currently checksum functionality is utterly broken with http (do you want me to make a separate ticket about it?) as having "bad" file on target will still download "bad" file to client as for this ticket, yes, that's a dupe Add Comment
Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title Josh Cooper commented on PUP-8300 Re: File resource can't handle HTTP redirects when server does not accept HEAD requests Thanks [~Mariusz Gronczewski] I think this is a duplicate of PUP-6380. Can you review, and close this ticket if that's the case? One note, puppet makes a HEAD request to get metadata about the file, so that it doesn't download the file if it doesn't need to. If it wasn't for that, then puppet could skip HEAD requests altogether. Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8300) File resource can't handle HTTP redirects when server does not accept HEAD requests
Title: Message Title
Mariusz Gronczewski updated an issue
Puppet / PUP-8300
File resource can't handle HTTP redirects when server does not accept HEAD requests
Change By:
Mariusz Gronczewski
If URL is redirect and server does not handle HEAD requests (like in that case Amazon API who returns 403 when URL after redirect gets HEAD request) , Puppet fails with unhelpful message:{noformat}/opt/puppetlabs/bin/puppet apply /tmp/1.ppNotice: Compiled catalog for hydra.devrandom.pl in environment production in 0.01 secondsError: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64Error: /Stage[main]/Main/File[/tmp/dpp]/ensure: change from 'absent' to 'present' failed: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64{noformat}no redirect file works fine{code:puppet} File { ensure => present }file { '/tmp/dpp':source => 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64',mode => "644",checksum => "sha256",checksum_value => '6bda5ca841bf47d81283ee5b73b030c3b82e33e6817d8bb36a1e1006c1b5dd81',backup => false,}file { '/tmp/rnd':source => 'http://192.168.1.1/rnd',mode => "644",checksum => "sha256",checksum_value => '19248907fa84d23721987abf90f9f7cbac44c547fd38ef2c0ce31e194eb4d9ed',backup => false,}{code}this was present in 4.8.x version too so it seems like something broken since http was added
Add Comment
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title Mariusz Gronczewski commented on PUP-8300 Re: File resource can't handle HTTP redirects It is because puppet does HEAD request first and amazon API responds with 403 to it Doing wget works fine (as does "just clicking link from browser") -> ᛯ wget https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64 --2018-01-09 09:35:09-- https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64 Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113 Connecting to github.com (github.com)|192.30.253.112|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256=AKIAIWNJYAX4CSVEH53A%2F20180109%2Fus-east-1%2Fs3%2Faws4_request=20180109T083509Z=300=c26f7fd7442edaacacaeda35e9d3781299389f0ffc2d7bf6e44b9ffbf5ad9a1e=host_id=0=attachment%3B%20filename%3Ddpp.aarch64=application%2Foctet-stream [following] --2018-01-09 09:35:09-- https://github-production-release-asset-2e65be.s3.amazonaws.com/68653753/d2ea368e-ed47-11e7-98f2-0c05ce59f7f9?X-Amz-Algorithm=AWS4-HMAC-SHA256=AKIAIWNJYAX4CSVEH53A%2F20180109%2Fus-east-1%2Fs3%2Faws4_request=20180109T083509Z=300=c26f7fd7442edaacacaeda35e9d3781299389f0ffc2d7bf6e44b9ffbf5ad9a1e=host_id=0=attachment%3B%20filename%3Ddpp.aarch64=application%2Foctet-stream
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title Josh Cooper commented on PUP-8300 Re: File resource can't handle HTTP redirects I verified puppet handles a 302 redirect when using source => 'https://github.com/puppetlabs/bolt/archive/0.13.0.tar.gz'. If I use source => 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64' then I get a 403 access denied. opening connection to github.com:443... opened starting SSL for github.com:443... SSL established <- "HEAD /puppetlabs/bolt/archive/0.13.0.tar.gz HTTP/1.1\r\nAccept: */*\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: github.com\r\n\r\n" -> "HTTP/1.1 302 Found\r\n" -> "Date: Tue, 09 Jan 2018 00:24:25 GMT\r\n" -> "Content-Type: text/html; charset=utf-8\r\n" -> "Connection: close\r\n" -> "Server: GitHub.com\r\n"
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title
Josh Cooper updated an issue
Puppet / PUP-8300
File resource can't handle HTTP redirects
Change By:
Josh Cooper
If URL is redirect, Puppet fails with unhelpful message:{noformat}/opt/puppetlabs/bin/puppet apply /tmp/1.ppNotice: Compiled catalog for hydra.devrandom.pl in environment production in 0.01 secondsError: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64Error: /Stage[main]/Main/File[/tmp/dpp]/ensure: change from 'absent' to 'present' failed: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64{noformat}no redirect file works fine{code:puppet} File { ensure => present }file { '/tmp/dpp':source => 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64',mode => "644",checksum => "sha256",checksum_value => '6bda5ca841bf47d81283ee5b73b030c3b82e33e6817d8bb36a1e1006c1b5dd81',backup => false,}file { '/tmp/rnd':source => 'http://192.168.1.1/rnd',mode => "644",checksum => "sha256",checksum_value => '19248907fa84d23721987abf90f9f7cbac44c547fd38ef2c0ce31e194eb4d9ed',backup => false,}{code :puppet }this was present in 4.8.x version too so it seems like something broken since http was added
Add Comment
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title
Josh Cooper updated an issue
Puppet / PUP-8300
File resource can't handle HTTP redirects
Change By:
Josh Cooper
If URL is redirect, Puppet fails with unhelpful message:{ { noformat} /opt/puppetlabs/bin/puppet apply /tmp/1.ppNotice: Compiled catalog for hydra.devrandom.pl in environment production in 0.01 secondsError: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64Error: /Stage[main]/Main/File[/tmp/dpp]/ensure: change from 'absent' to 'present' failed: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64 {noformat } } no redirect file works fine{ { code:puppet} File { ensure => present }file { '/tmp/dpp':source => 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64',mode => "644",checksum => "sha256",checksum_value => '6bda5ca841bf47d81283ee5b73b030c3b82e33e6817d8bb36a1e1006c1b5dd81',backup => false,}file { '/tmp/rnd':source => 'http://192.168.1.1/rnd',mode => "644",checksum => "sha256",checksum_value => '19248907fa84d23721987abf90f9f7cbac44c547fd38ef2c0ce31e194eb4d9ed',backup => false,} {code:puppet } } this was present in 4.8.x version too so it seems like something broken since http was added
Add Comment
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title Craig Gomes updated an issue Puppet / PUP-8300 File resource can't handle HTTP redirects Change By: Craig Gomes Team: Platform Core Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title Mariusz Gronczewski updated an issue Puppet / PUP-8300 File resource can't handle HTTP redirects Change By: Mariusz Gronczewski Attachment: 1.pp Add Comment This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8300) File resource can't handle HTTP redirects
Title: Message Title
Mariusz Gronczewski created an issue
Puppet / PUP-8300
File resource can't handle HTTP redirects
Issue Type:
Bug
Affects Versions:
PUP 5.3.3
Assignee:
Unassigned
Created:
2017/12/30 2:24 AM
Priority:
Normal
Reporter:
Mariusz Gronczewski
If URL is redirect, Puppet fails with unhelpful message:
{{/opt/puppetlabs/bin/puppet apply /tmp/1.pp Notice: Compiled catalog for hydra.devrandom.pl in environment production in 0.01 seconds Error: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64 Error: /Stage[main]/Main/File[/tmp/dpp]/ensure: change from 'absent' to 'present' failed: Could not retrieve information from environment production source(s) https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64}}
no redirect file works fine {{ File { ensure => present }
file { '/tmp/dpp': source => 'https://github.com/XANi/go-dpp/releases/download/v0.0.4/dpp.aarch64', mode => "644", checksum => "sha256", checksum_value => '6bda5ca841bf47d81283ee5b73b030c3b82e33e6817d8bb36a1e1006c1b5dd81', backup => false, }
file { '/tmp/rnd': source => 'http://192.168.1.1/rnd', mode => "644", checksum => "sha256", checksum_value => '19248907fa84d23721987abf90f9f7cbac44c547fd38ef2c0ce31e194eb4d9ed', backup => false, }
}}
this was present in 4.8.x version too so it seems like something broken since http was added
