Package: release.debian.org Severity: important Tags: stretch User: release.debian....@packages.debian.org Usertags: pu
git-annex in stretch is vulnerable to CVE-2018-10857 and CVE-2018-10859. This update is a minimal fix for those CVEs prepared by its upstream, Joey Hess: git-annex (6.20170101-1+deb9u2) stretch; urgency=high [ Joey Hess ] * CVE-2018-10857: - Added annex.security.allowed-url-schemes setting, which defaults to only allowing http, https, and ftp URLs. Note especially that file:/ is no longer enabled by default. - Removed annex.web-download-command, since its interface does not allow supporting annex.security.allowed-url-schemes across redirects. If you used this setting, you may want to instead use annex.web-options to pass options to curl. - git-annex will refuse to download content from the web, to prevent accidental exposure of data on private webservers on localhost and the LAN. This can be overridden with the annex.security.allowed-http-addresses setting. (The S3, glacier, and webdav special remotes are still allowed to download from the web.) * CVE-2018-10857 and CVE-2018-10859: - Refuse to download content, that cannot be verified with a hash, from encrypted special remotes (for CVE-2018-10859), and from all external special remotes (for CVE-2018-10857). In particular, URL and WORM keys stored on such remotes won't be downloaded. If this affects your files, you can run `git-annex migrate` on the affected files, to convert them to use a hash. - Added annex.security.allow-unverified-downloads, which can override the above. -- Sean Whitton <spwhit...@spwhitton.name> Fri, 22 Jun 2018 16:42:37 +0100 The security team have decided that because a point release is close and only certain usecases of git-annex are impacted, they won't issue a DSA. I have not yet uploaded to stretch-proposed-updates for two reasons: (i) although the fix is minimal and much smaller than the fix for the CVEs included in git-annex in unstable, the source debdiff (attached) is still quite large, so I thought I should wait for an explicit ACK from the release team (ii) there is already a +deb9u1 version of git-annex in stretch-security, but not stretch, responding to a different CVE. I have based my work on the +deb9u1 upload, and I assume that uploading my +deb9u2 to stretch-proposed-updates will cause it to take precedence over the import of the +deb9u1 upload. But in case things are more fragile than that, I'm holding back. Note that my debdiff is from +deb9u1 to +deb9u2. I ran the test suite and manually smoke tested my proposed update against stretch as the latter stood about a week ago, and plan to run the test suite again right before uploading to stretch-proposed-updates. Thank you for your attention! -- System Information: Debian Release: 9.4 APT prefers stable APT policy: (900, 'stable'), (500, 'stable-updates') Architecture: i386 (i686) Kernel: Linux 4.9.0-6-686-pae (SMP w/2 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) -- Sean Whitton
diff -Nru git-annex-6.20170101/debian/changelog git-annex-6.20170101/debian/changelog --- git-annex-6.20170101/debian/changelog 2017-10-26 15:28:29.000000000 +0100 +++ git-annex-6.20170101/debian/changelog 2018-06-22 16:42:37.000000000 +0100 @@ -1,3 +1,33 @@ +git-annex (6.20170101-1+deb9u2) stretch; urgency=high + + [ Joey Hess ] + * CVE-2018-10857: + - Added annex.security.allowed-url-schemes setting, which defaults + to only allowing http, https, and ftp URLs. Note especially that file:/ + is no longer enabled by default. + - Removed annex.web-download-command, since its interface does not allow + supporting annex.security.allowed-url-schemes across redirects. + If you used this setting, you may want to instead use annex.web-options + to pass options to curl. + - git-annex will refuse to download content from the web, to prevent + accidental exposure of data on private webservers on localhost and the + LAN. This can be overridden with the + annex.security.allowed-http-addresses setting. + (The S3, glacier, and webdav special remotes are still allowed to + download from the web.) + * CVE-2018-10857 and CVE-2018-10859: + - Refuse to download content, that cannot be verified with a hash, + from encrypted special remotes (for CVE-2018-10859), + and from all external special remotes (for CVE-2018-10857). + In particular, URL and WORM keys stored on such remotes won't + be downloaded. If this affects your files, you can run + `git-annex migrate` on the affected files, to convert them + to use a hash. + - Added annex.security.allow-unverified-downloads, which can override + the above. + + -- Sean Whitton <spwhit...@spwhitton.name> Fri, 22 Jun 2018 16:42:37 +0100 + git-annex (6.20170101-1+deb9u1) stretch-security; urgency=high * Non-maintainer upload by the Security Team. diff -Nru git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch --- git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,293 @@ +From: Joey Hess <jo...@joeyh.name> +Date: Thu, 21 Jun 2018 11:35:27 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 fe511617f5bde916f1bde799b5a51ddc9404eea1 +Subject: add retrievalSecurityPolicy + +This will be used to protect against CVE-2018-10859, where an encrypted +special remote is fed the wrong encrypted data, and so tricked into +decrypting something that the user encrypted with their gpg key and did +not store in git-annex. + +It also protects against CVE-2018-10857, where a remote follows a http +redirect to a file:// url or to a local private web server. While that's +already been prevented in git-annex's own use of http, external special +remotes, hooks, etc use other http implementations and could still be +vulnerable. + +The policy is not yet enforced, this commit only adds the appropriate +metadata to remotes. + +This commit was sponsored by Boyd Stephen Smith Jr. on Patreon. + +(cherry picked from commit 4315bb9e421f2c643e517d8982c6c35b1909c78b) + +--- + +--- git-annex-6.20170101.orig/Remote/BitTorrent.hs ++++ git-annex-6.20170101/Remote/BitTorrent.hs +@@ -57,6 +57,8 @@ gen r _ c gc = + , storeKey = uploadKey + , retrieveKeyFile = downloadKey + , retrieveKeyFileCheap = downloadKeyCheap ++ -- Bittorrent does its own hash checks. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = dropKey + , lockContent = Nothing + , checkPresent = checkKey +--- git-annex-6.20170101.orig/Remote/Bup.hs ++++ git-annex-6.20170101/Remote/Bup.hs +@@ -56,6 +56,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap buprepo ++ -- Bup uses git, which cryptographically verifies content ++ -- (with SHA1, but sufficiently for this). ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Ddar.hs ++++ git-annex-6.20170101/Remote/Ddar.hs +@@ -56,6 +56,8 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap ++ -- Unsure about this, safe default until Robie answers. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Directory.hs ++++ git-annex-6.20170101/Remote/Directory.hs +@@ -54,6 +54,7 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap dir chunkconfig ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/External.hs ++++ git-annex-6.20170101/Remote/External.hs +@@ -81,6 +81,11 @@ gen r u c gc + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = \_ _ _ -> return False ++ -- External special remotes use many http libraries ++ -- and have no protection against redirects to ++ -- local private web servers, or in some cases ++ -- to file:// urls. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/GCrypt.hs ++++ git-annex-6.20170101/Remote/GCrypt.hs +@@ -111,6 +111,7 @@ gen' r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = \_ _ _ -> return False ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Git.hs ++++ git-annex-6.20170101/Remote/Git.hs +@@ -147,6 +147,7 @@ gen r u c gc + , storeKey = copyToRemote new + , retrieveKeyFile = copyFromRemote new + , retrieveKeyFileCheap = copyFromRemoteCheap new ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = dropKey new + , lockContent = Just (lockKey new) + , checkPresent = inAnnex new +--- git-annex-6.20170101.orig/Remote/Glacier.hs ++++ git-annex-6.20170101/Remote/Glacier.hs +@@ -53,6 +53,9 @@ gen r u c gc = new <$> remoteCost gc ver + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap this ++ -- glacier-cli does not follow redirects and does ++ -- not support file://, so this is secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Helper/Special.hs ++++ git-annex-6.20170101/Remote/Helper/Special.hs +@@ -161,6 +161,14 @@ specialRemote' cfg c preparestorer prepa + (retrieveKeyFileCheap baser k f d) + -- retrieval of encrypted keys is never cheap + (\_ -> return False) ++ -- When encryption is used, the remote could provide ++ -- some other content encrypted by the user, and trick ++ -- git-annex into decrypting it, leaking the decryption ++ -- into the git-annex repository. Verifiable keys ++ -- are the main protection against this attack. ++ , retrievalSecurityPolicy = if isencrypted ++ then RetrievalVerifiableKeysSecure ++ else retrievalSecurityPolicy baser + , removeKey = \k -> cip >>= removeKeyGen k + , checkPresent = \k -> cip >>= checkPresentGen k + , cost = if isencrypted +--- git-annex-6.20170101.orig/Remote/Hook.hs ++++ git-annex-6.20170101/Remote/Hook.hs +@@ -47,6 +47,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap hooktype ++ -- A hook could use http and be vulnerable to ++ -- redirect to file:// attacks, etc. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/P2P.hs ++++ git-annex-6.20170101/Remote/P2P.hs +@@ -53,6 +53,7 @@ chainGen addr r u c gc = do + , storeKey = store u addr connpool + , retrieveKeyFile = retrieve u addr connpool + , retrieveKeyFileCheap = \_ _ _ -> return False ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = remove u addr connpool + , lockContent = Just (lock u addr connpool) + , checkPresent = checkpresent u addr connpool +--- git-annex-6.20170101.orig/Remote/Rsync.hs ++++ git-annex-6.20170101/Remote/Rsync.hs +@@ -69,6 +69,7 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap o ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/S3.hs ++++ git-annex-6.20170101/Remote/S3.hs +@@ -86,6 +86,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap ++ -- HttpManagerRestricted is used here, so this is ++ -- secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Tahoe.hs ++++ git-annex-6.20170101/Remote/Tahoe.hs +@@ -71,6 +71,8 @@ gen r u c gc = do + , storeKey = store u hdl + , retrieveKeyFile = retrieve u hdl + , retrieveKeyFileCheap = \_ _ _ -> return False ++ -- Tahoe cryptographically verifies content. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = remove + , lockContent = Nothing + , checkPresent = checkKey u hdl +--- git-annex-6.20170101.orig/Remote/Web.hs ++++ git-annex-6.20170101/Remote/Web.hs +@@ -46,6 +46,9 @@ gen r _ c gc = + , storeKey = uploadKey + , retrieveKeyFile = downloadKey + , retrieveKeyFileCheap = downloadKeyCheap ++ -- HttpManagerRestricted is used here, so this is ++ -- secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = dropKey + , lockContent = Nothing + , checkPresent = checkKey +--- git-annex-6.20170101.orig/Remote/WebDAV.hs ++++ git-annex-6.20170101/Remote/WebDAV.hs +@@ -64,6 +64,9 @@ gen r u c gc = new <$> remoteCost gc exp + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap ++ -- HttpManagerRestricted is used here, so this is ++ -- secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Types/Key.hs ++++ git-annex-6.20170101/Types/Key.hs +@@ -1,6 +1,6 @@ + {- git-annex Key data type + - +- - Copyright 2011-2016 Joey Hess <i...@joeyh.name> ++ - Copyright 2011-2018 Joey Hess <i...@joeyh.name> + - + - Licensed under the GNU GPL version 3 or higher. + -} +@@ -160,3 +160,13 @@ prop_isomorphic_key_decode f + normalfieldorder = fields `isPrefixOf` "smSC" + fields = map (f !!) $ filter (< length f) $ map succ $ + elemIndices fieldSep f ++ ++{- Is the Key variety backed by a hash, which allows verifying content? ++ - It does not have to be cryptographically secure against eg birthday ++ - attacks. ++ -} ++isVerifiable :: Key -> Bool ++isVerifiable k = case keyBackendName k of ++ "WORM" -> False ++ "URL" -> False ++ _ -> True +--- git-annex-6.20170101.orig/Types/Remote.hs ++++ git-annex-6.20170101/Types/Remote.hs +@@ -2,7 +2,7 @@ + - + - Most things should not need this, using Types instead + - +- - Copyright 2011-2014 Joey Hess <i...@joeyh.name> ++ - Copyright 2011-2018 Joey Hess <i...@joeyh.name> + - + - Licensed under the GNU GPL version 3 or higher. + -} +@@ -17,6 +17,7 @@ module Types.Remote + , Availability(..) + , Verification(..) + , unVerified ++ , RetrievalSecurityPolicy(..) + ) + where + +@@ -75,6 +76,8 @@ data RemoteA a = Remote { + -- Retrieves a key's contents to a tmp file, if it can be done cheaply. + -- It's ok to create a symlink or hardlink. + retrieveKeyFileCheap :: Key -> AssociatedFile -> FilePath -> a Bool, ++ -- Security policy for reteiving keys from this remote. ++ retrievalSecurityPolicy :: RetrievalSecurityPolicy, + -- Removes a key's contents (succeeds if the contents are not present) + removeKey :: Key -> a Bool, + -- Uses locking to prevent removal of a key's contents, +@@ -145,3 +148,29 @@ unVerified :: Monad m => m Bool -> m (Bo + unVerified a = do + ok <- a + return (ok, UnVerified) ++ ++-- Security policy indicating what keys can be safely retrieved from a ++-- remote. ++data RetrievalSecurityPolicy ++ = RetrievalVerifiableKeysSecure ++ -- ^ Transfer of keys whose content can be verified ++ -- with a hash check is secure; transfer of unverifiable keys is ++ -- not secure and should not be allowed. ++ -- ++ -- This is used eg, when HTTP to a remote could be redirected to a ++ -- local private web server or even a file:// url, causing private ++ -- data from it that is not the intended content of a key to make ++ -- its way into the git-annex repository. ++ -- ++ -- It's also used when content is stored encrypted on a remote, ++ -- which could replace it with a different encrypted file, and ++ -- trick git-annex into decrypting it and leaking the decryption ++ -- into the git-annex repository. ++ -- ++ -- It's not (currently) used when the remote could alter the ++ -- content stored on it, because git-annex does not provide ++ -- strong guarantees about the content of keys that cannot be ++ -- verified with a hash check. ++ -- (But annex.securehashesonly does provide such guarantees.) ++ | RetrievalAllKeysSecure ++ -- ^ Any key can be securely retrieved. diff -Nru git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch --- git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,154 @@ +From: Joey Hess <i...@joeyh.name> +Date: Mon, 18 Jun 2018 17:40:50 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 139539615478c1f618525715bef184d450b38558 +Subject: block url downloads by default + +git-annex will refuse to download content from the web, to prevent +accidental exposure of data on private webservers on localhost and the +LAN. This can be overridden with the +annex.security.allowed-http-addresses setting. + +This is the simplest possible fix for the security hole. A better fix +has been developed for newer versions of git-annex but would be a lot of +work to backport, and perhaps too big a diff. + +There are several sets of git-annex users who will be impacted +in different ways by this: + +* Users who have a git-annex repository but don't use the web special + remote. Unaffected. + +* Users who have a git-annex repository that is for private use only. + They will have to read enough docs to find the setting to allow + git annex addurl to work again. + +* Users who have a git-annex repositry that is shared with people they + don't fully trust. + They will not be able to use the web special remote with this version + of git-annex. They'll have to upgrade. + +The S3, glacier, and webdav special remotes are still allowed to +download from the web. There are other potential attacks involving the +web server they connect to redirecting to a local private web server, +and tricking them from downloading content from it which then leaks back +to the attacker. Those attacks are not addressed here, but they also +seem fairly unlikely. Further analysis is needed; preliminary analysis +of glacier-cli, for example, suggests it does not follow redirects and +so is not vulnerable to such attacks. + +--- + +--- git-annex-6.20170101.orig/Annex/Quvi.hs ++++ git-annex-6.20170101/Annex/Quvi.hs +@@ -11,8 +11,8 @@ module Annex.Quvi where + + import Annex.Common + import qualified Annex ++import Annex.Url + import Utility.Quvi +-import Utility.Url + + withQuviOptions :: forall a. Query a -> [QuviParams] -> URLString -> Annex a + withQuviOptions a ps url = do +@@ -21,7 +21,14 @@ withQuviOptions a ps url = do + liftIO $ a v (concatMap (\mkp -> mkp v) ps ++ opts) url + + quviSupported :: URLString -> Annex Bool +-quviSupported u = liftIO . flip supported u =<< quviVersion ++quviSupported u = ifM httpAddressesUnlimited ++ ( liftIO . flip supported u =<< quviVersion ++ -- Don't allow any url schemes to be used when ++ -- there's a limit on the allowed addresses, because ++ -- there is no way to prevent quvi from ++ -- redirecting to any address. ++ , return False ++ ) + + quviVersion :: Annex QuviVersion + quviVersion = go =<< Annex.getState Annex.quviversion +--- git-annex-6.20170101.orig/Annex/Url.hs ++++ git-annex-6.20170101/Annex/Url.hs +@@ -11,6 +11,7 @@ module Annex.Url ( + withUrlOptions, + getUrlOptions, + getUserAgent, ++ httpAddressesUnlimited, + ) where + + import Annex.Common +@@ -18,6 +19,8 @@ import qualified Annex + import Utility.Url as U + import qualified Build.SysConfig as SysConfig + ++import qualified Data.Set as S ++ + defaultUserAgent :: U.UserAgent + defaultUserAgent = "git-annex/" ++ SysConfig.packageversion + +@@ -30,7 +33,7 @@ getUrlOptions = mkUrlOptions + <$> getUserAgent + <*> headers + <*> options +- <*> (annexAllowedUrlSchemes <$> Annex.getGitConfig) ++ <*> urlschemes + where + headers = do + v <- annexHttpHeadersCommand <$> Annex.getGitConfig +@@ -38,6 +41,18 @@ getUrlOptions = mkUrlOptions + Just cmd -> lines <$> liftIO (readProcess "sh" ["-c", cmd]) + Nothing -> annexHttpHeaders <$> Annex.getGitConfig + options = map Param . annexWebOptions <$> Annex.getGitConfig ++ urlschemes = ifM httpAddressesUnlimited ++ ( annexAllowedUrlSchemes <$> Annex.getGitConfig ++ -- Don't allow any url schemes to be used when ++ -- there's a limit on the allowed addresses, because ++ -- there is no way to prevent curl or wget from ++ -- redirecting to any address. ++ , pure S.empty ++ ) ++ ++httpAddressesUnlimited :: Annex Bool ++httpAddressesUnlimited = ++ ("all" == ) . annexAllowedHttpAddresses <$> Annex.getGitConfig + + withUrlOptions :: (U.UrlOptions -> IO a) -> Annex a + withUrlOptions a = liftIO . a =<< getUrlOptions +--- git-annex-6.20170101.orig/NEWS ++++ git-annex-6.20170101/NEWS +@@ -4,6 +4,12 @@ git-annex (6.20170101-1+deb9u2) stretch- + URL schemes by default. You can enable other URL schemes, at your own risk, + using annex.security.allowed-url-schemes. + ++ A related security fix prevents git-annex from downloading content from ++ the web. This can be overridden with the ++ annex.security.allowed-http-addresses setting. ++ (The S3, glacier, and webdav special remotes are still allowed to ++ download from the web.) ++ + The annex.web-download-command configuration has been removed, + use annex.web-options instead. + +--- git-annex-6.20170101.orig/doc/git-annex.mdwn ++++ git-annex-6.20170101/doc/git-annex.mdwn +@@ -1253,6 +1253,21 @@ Here are all the supported configuration + Some special remotes support their own domain-specific URL + schemes; those are not affected by this configuration setting. + ++* `annex.security.allowed-http-addresses` ++ ++ By default, this version of git-annex refuses to download the content of ++ annexed files from the web. Newer versions of git-annex allow downloading ++ from the web, but only when the web server is not on a private IP address. ++ ++ To relax this security check and allow getting annexed files from ++ anywhere on the web, set this to "all". ++ ++ Think very carefully before changing this; there are security ++ implications. Anyone who can get a commit into your git-annex repository ++ could `git annex addurl` an url on a private http server, possibly ++ causing it to be downloaded into your repository and transferred to ++ other remotes, exposing its content. ++ + * `annex.secure-erase-command` + + This can be set to a command that should be run whenever git-annex diff -Nru git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch --- git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,27 @@ +From: Joey Hess <jo...@joeyh.name> +Date: Thu, 21 Jun 2018 14:14:56 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 acdcbe25bc02919a8c2241155e399f9ef9533c5f +Subject: don't assume boto will remain secure + +On second thought, best to default to being secure even if boto changes +http libraries to one that happens to follow redirects. + +(cherry picked from commit f1b29dbeb4277bf8a7febc795fe52e7b109c6d59) + +--- + +--- git-annex-6.20170101.orig/Remote/Glacier.hs ++++ git-annex-6.20170101/Remote/Glacier.hs +@@ -54,8 +54,10 @@ gen r u c gc = new <$> remoteCost gc ver + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap this + -- glacier-cli does not follow redirects and does +- -- not support file://, so this is secure. +- , retrievalSecurityPolicy = RetrievalAllKeysSecure ++ -- not support file://, as far as we know, but ++ -- there's no guarantee that will continue to be ++ -- the case, so require verifiable keys. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy diff -Nru git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch --- git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,438 @@ +From: Joey Hess <jo...@joeyh.name> +Date: Thu, 21 Jun 2018 13:34:11 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 cd7e9410a9835b63fb06ef5c99b31d91f111c4c3 +Subject: enforce retrievalSecurityPolicy + +Leveraged the existing verification code by making it also check the +retrievalSecurityPolicy. + +Also, prevented getViaTmp from running the download action at all when the +retrievalSecurityPolicy is going to prevent verifying and so storing it. + +Added annex.security.allow-unverified-downloads. A per-remote version +would be nice to have too, but would need more plumbing, so KISS. +(Bill the Cat reference not too over the top I hope. The point is to +make this something the user reads the documentation for before using.) + +A few calls to verifyKeyContent and getViaTmp, that don't +involve downloads from remotes, have RetrievalAllKeysSecure hard-coded. +It was also hard-coded for P2P.Annex and Command.RecvKey, +to match the values of the corresponding remotes. + +A few things use retrieveKeyFile/retrieveKeyFileCheap without going +through getViaTmp. +* Command.Fsck when downloading content from a remote to verify it. + That content does not get into the annex, so this is ok. +* Command.AddUrl when using a remote to download an url; this is new + content being added, so this is ok. + +This commit was sponsored by Fernando Jimenez on Patreon. + +(cherry picked from commit b657242f5d946efae4cc77e8aef95dd2a306cd6b) + +--- + +--- git-annex-6.20170101.orig/Annex/Content.hs ++++ git-annex-6.20170101/Annex/Content.hs +@@ -1,6 +1,6 @@ + {- git-annex file content managing + - +- - Copyright 2010-2015 Joey Hess <i...@joeyh.name> ++ - Copyright 2010-2018 Joey Hess <i...@joeyh.name> + - + - Licensed under the GNU GPL version 3 or higher. + -} +@@ -15,6 +15,7 @@ module Annex.Content ( + lockContentShared, + lockContentForRemoval, + ContentRemovalLock, ++ RetrievalSecurityPolicy(..), + getViaTmp, + getViaTmp', + checkDiskSpaceToGet, +@@ -74,7 +75,7 @@ import qualified Annex.Content.Direct as + import Annex.ReplaceFile + import Annex.LockPool + import Messages.Progress +-import Types.Remote (unVerified, Verification(..)) ++import Types.Remote (unVerified, Verification(..), RetrievalSecurityPolicy(..)) + import qualified Types.Remote + import qualified Types.Backend + import qualified Backend +@@ -294,19 +295,19 @@ lockContentUsing locker key a = do + {- Runs an action, passing it the temp file to get, + - and if the action succeeds, verifies the file matches + - the key and moves the file into the annex as a key's content. -} +-getViaTmp :: VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool +-getViaTmp v key action = checkDiskSpaceToGet key False $ +- getViaTmp' v key action ++getViaTmp :: RetrievalSecurityPolicy -> VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool ++getViaTmp rsp v key action = checkDiskSpaceToGet key False $ ++ getViaTmp' rsp v key action + + {- Like getViaTmp, but does not check that there is enough disk space + - for the incoming key. For use when the key content is already on disk + - and not being copied into place. -} +-getViaTmp' :: VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool +-getViaTmp' v key action = do ++getViaTmp' :: RetrievalSecurityPolicy -> VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool ++getViaTmp' rsp v key action = checkallowed $ do + tmpfile <- prepTmp key + (ok, verification) <- action tmpfile + if ok +- then ifM (verifyKeyContent v verification key tmpfile) ++ then ifM (verifyKeyContent rsp v verification key tmpfile) + ( do + moveAnnex key tmpfile + logStatus key InfoPresent +@@ -319,22 +320,45 @@ getViaTmp' v key action = do + -- On transfer failure, the tmp file is left behind, in case + -- caller wants to resume its transfer + else return False ++ where ++ -- Avoid running the action to get the content when the ++ -- RetrievalSecurityPolicy would cause verification to always fail. ++ checkallowed a = case rsp of ++ RetrievalAllKeysSecure -> a ++ RetrievalVerifiableKeysSecure ++ | isVerifiable key -> a ++ | otherwise -> ifM (annexAllowUnverifiedDownloads <$> Annex.getGitConfig) ++ ( a ++ , warnUnverifiableInsecure key >> return False ++ ) + + {- Verifies that a file is the expected content of a key. ++ - + - Configuration can prevent verification, for either a +- - particular remote or always. ++ - particular remote or always, unless the RetrievalSecurityPolicy ++ - requires verification. + - + - Most keys have a known size, and if so, the file size is checked. + - +- - When the key's backend allows verifying the content (eg via checksum), ++ - When the key's backend allows verifying the content (via checksum), + - it is checked. ++ - ++ - If the RetrievalSecurityPolicy requires verification and the key's ++ - backend doesn't support it, the verification will fail. + -} +-verifyKeyContent :: VerifyConfig -> Verification -> Key -> FilePath -> Annex Bool +-verifyKeyContent _ Verified _ _ = return True +-verifyKeyContent v UnVerified k f = ifM (shouldVerify v) +- ( verifysize <&&> verifycontent +- , return True +- ) ++verifyKeyContent :: RetrievalSecurityPolicy -> VerifyConfig -> Verification -> Key -> FilePath -> Annex Bool ++verifyKeyContent _ _ Verified _ _ = return True ++verifyKeyContent rsp v UnVerified k f = case rsp of ++ RetrievalVerifiableKeysSecure ++ | isVerifiable k -> verifysize <&&> verifycontent ++ | otherwise -> ifM (annexAllowUnverifiedDownloads <$> Annex.getGitConfig) ++ ( verifysize <&&> verifycontent ++ , warnUnverifiableInsecure k >> return False ++ ) ++ _ -> ifM (shouldVerify v) ++ ( verifysize <&&> verifycontent ++ , return True ++ ) + where + verifysize = case keySize k of + Nothing -> return True +@@ -345,6 +369,16 @@ verifyKeyContent v UnVerified k f = ifM + Nothing -> return True + Just verifier -> verifier k f + ++warnUnverifiableInsecure :: Key -> Annex () ++warnUnverifiableInsecure k = warning $ unwords ++ [ "Getting " ++ kv ++ " keys with this remote is not secure;" ++ , "the content cannot be verified to be correct." ++ , "(Use annex.security.allow-unverified-downloads to bypass" ++ , "this safety check.)" ++ ] ++ where ++ kv = keyBackendName k ++ + data VerifyConfig = AlwaysVerify | NoVerify | RemoteVerify Remote | DefaultVerify + + shouldVerify :: VerifyConfig -> Annex Bool +@@ -790,7 +824,7 @@ isUnmodified key f = go =<< geti + go (Just fc) = cheapcheck fc <||> expensivecheck fc + cheapcheck fc = anyM (compareInodeCaches fc) + =<< Database.Keys.getInodeCaches key +- expensivecheck fc = ifM (verifyKeyContent AlwaysVerify UnVerified key f) ++ expensivecheck fc = ifM (verifyKeyContent RetrievalAllKeysSecure AlwaysVerify UnVerified key f) + -- The file could have been modified while it was + -- being verified. Detect that. + ( geti >>= maybe (return False) (compareInodeCaches fc) +--- git-annex-6.20170101.orig/Command/Get.hs ++++ git-annex-6.20170101/Command/Get.hs +@@ -108,7 +108,7 @@ getKey' key afile = dispatch + | Remote.hasKeyCheap r = + either (const False) id <$> Remote.hasKey r key + | otherwise = return True +- docopy r witness = getViaTmp (RemoteVerify r) key $ \dest -> ++ docopy r witness = getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) key $ \dest -> + download (Remote.uuid r) key afile forwardRetry + (\p -> do + showAction $ "from " ++ Remote.name r +--- git-annex-6.20170101.orig/Command/Move.hs ++++ git-annex-6.20170101/Command/Move.hs +@@ -179,7 +179,7 @@ fromPerform src move key afile = ifM (in + go = notifyTransfer Download afile $ + download (Remote.uuid src) key afile forwardRetry $ \p -> do + showAction $ "from " ++ Remote.name src +- getViaTmp (RemoteVerify src) key $ \t -> ++ getViaTmp (Remote.retrievalSecurityPolicy src) (RemoteVerify src) key $ \t -> + Remote.retrieveKeyFile src key afile t p + dispatch _ False = stop -- failed + dispatch False True = next $ return True -- copy complete +--- git-annex-6.20170101.orig/Command/ReKey.hs ++++ git-annex-6.20170101/Command/ReKey.hs +@@ -83,7 +83,7 @@ linkKey file oldkey newkey = ifM (isJust + - This avoids hard linking to content linked to an + - unlocked file, which would leave the new key unlocked + - and vulnerable to corruption. -} +- ( getViaTmp' DefaultVerify newkey $ \tmp -> unVerified $ do ++ ( getViaTmp' RetrievalAllKeysSecure DefaultVerify newkey $ \tmp -> unVerified $ do + oldobj <- calcRepo (gitAnnexLocation oldkey) + linkOrCopy' (return True) newkey oldobj tmp Nothing + , do +--- git-annex-6.20170101.orig/Command/RecvKey.hs ++++ git-annex-6.20170101/Command/RecvKey.hs +@@ -13,6 +13,7 @@ import Annex.Action + import Annex + import Utility.Rsync + import Types.Transfer ++import Types.Remote (RetrievalSecurityPolicy(..)) + import Command.SendKey (fieldTransfer) + import qualified CmdLine.GitAnnexShell.Fields as Fields + +@@ -31,7 +32,9 @@ start key = fieldTransfer Download key $ + fromunlocked <- (isJust <$> Fields.getField Fields.unlocked) + <||> (isJust <$> Fields.getField Fields.direct) + let verify = if fromunlocked then AlwaysVerify else DefaultVerify +- ifM (getViaTmp verify key go) ++ -- This matches the retrievalSecurityPolicy of Remote.Git ++ let rsp = RetrievalAllKeysSecure ++ ifM (getViaTmp rsp verify key go) + ( do + -- forcibly quit after receiving one key, + -- and shutdown cleanly +--- git-annex-6.20170101.orig/Command/Reinject.hs ++++ git-annex-6.20170101/Command/Reinject.hs +@@ -44,7 +44,7 @@ startSrcDest (src:dest:[]) + | otherwise = notAnnexed src $ do + showStart "reinject" dest + next $ ifAnnexed dest +- (\key -> perform src key (verifyKeyContent DefaultVerify UnVerified key src)) ++ (\key -> perform src key (verifyKeyContent RetrievalAllKeysSecure DefaultVerify UnVerified key src)) + stop + startSrcDest _ = giveup "specify a src file and a dest file" + +--- git-annex-6.20170101.orig/Command/SetKey.hs ++++ git-annex-6.20170101/Command/SetKey.hs +@@ -33,7 +33,7 @@ perform file key = do + -- the file might be on a different filesystem, so moveFile is used + -- rather than simply calling moveAnnex; disk space is also + -- checked this way. +- ok <- getViaTmp DefaultVerify key $ \dest -> unVerified $ ++ ok <- getViaTmp RetrievalAllKeysSecure DefaultVerify key $ \dest -> unVerified $ + if dest /= file + then liftIO $ catchBoolIO $ do + moveFile file dest +--- git-annex-6.20170101.orig/Command/TestRemote.hs ++++ git-annex-6.20170101/Command/TestRemote.hs +@@ -154,7 +154,7 @@ test st r k = + Just b -> case Backend.verifyKeyContent b of + Nothing -> return True + Just verifier -> verifier k (key2file k) +- get = getViaTmp (RemoteVerify r) k $ \dest -> ++ get = getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> + Remote.retrieveKeyFile r k Nothing dest nullMeterUpdate + store = Remote.storeKey r k Nothing nullMeterUpdate + remove = Remote.removeKey r k +@@ -168,10 +168,10 @@ testUnavailable st r k = + , check (`notElem` [Right True, Right False]) "checkPresent" $ + Remote.checkPresent r k + , check (== Right False) "retrieveKeyFile" $ +- getViaTmp (RemoteVerify r) k $ \dest -> ++ getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> + Remote.retrieveKeyFile r k Nothing dest nullMeterUpdate + , check (== Right False) "retrieveKeyFileCheap" $ +- getViaTmp (RemoteVerify r) k $ \dest -> unVerified $ ++ getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> unVerified $ + Remote.retrieveKeyFileCheap r k Nothing dest + ] + where +--- git-annex-6.20170101.orig/Command/TransferKey.hs ++++ git-annex-6.20170101/Command/TransferKey.hs +@@ -60,7 +60,7 @@ toPerform key file remote = go Upload fi + fromPerform :: Key -> AssociatedFile -> Remote -> CommandPerform + fromPerform key file remote = go Upload file $ + download (uuid remote) key file forwardRetry $ \p -> +- getViaTmp (RemoteVerify remote) key $ ++ getViaTmp (retrievalSecurityPolicy remote) (RemoteVerify remote) key $ + \t -> Remote.retrieveKeyFile remote key file t p + + go :: Direction -> AssociatedFile -> (NotifyWitness -> Annex Bool) -> CommandPerform +--- git-annex-6.20170101.orig/Command/TransferKeys.hs ++++ git-annex-6.20170101/Command/TransferKeys.hs +@@ -42,7 +42,7 @@ start = do + return ok + | otherwise = notifyTransfer direction file $ + download (Remote.uuid remote) key file forwardRetry $ \p -> +- getViaTmp (RemoteVerify remote) key $ \t -> do ++ getViaTmp (Remote.retrievalSecurityPolicy remote) (RemoteVerify remote) key $ \t -> do + r <- Remote.retrieveKeyFile remote key file t p + -- Make sure we get the current + -- associated files data for the key, +--- git-annex-6.20170101.orig/NEWS ++++ git-annex-6.20170101/NEWS +@@ -1,5 +1,11 @@ + git-annex (6.20170101-1+deb9u2) stretch-security; urgency=high + ++ A security fix has changed git-annex to refuse to download content from ++ some special remotes when the content cannot be verified with a hash check. ++ In particular URL and WORM keys stored on such remotes won't be downloaded. ++ See the documentation of the annex.security.allow-unverified-downloads ++ configuration for how to deal with this if it affects your files. ++ + A security fix has changed git-annex to only support http, https, and ftp + URL schemes by default. You can enable other URL schemes, at your own risk, + using annex.security.allowed-url-schemes. +--- git-annex-6.20170101.orig/P2P/Annex.hs ++++ git-annex-6.20170101/P2P/Annex.hs +@@ -23,6 +23,7 @@ import P2P.Protocol + import P2P.IO + import Logs.Location + import Types.NumCopies ++import Types.Remote (RetrievalSecurityPolicy(..)) + import Utility.Metered + import Utility.Tor + import Annex.UUID +@@ -77,9 +78,12 @@ runLocal runmode runner a = case a of + Right Nothing -> runner (next False) + Left e -> return (Left (show e)) + StoreContent k af o l getb next -> do ++ -- This is the same as the retrievalSecurityPolicy of ++ -- Remote.P2P and Remote.Git. ++ let rsp = RetrievalAllKeysSecure + ok <- flip catchNonAsync (const $ return False) $ + transfer download k af $ \p -> +- getViaTmp AlwaysVerify k $ \tmp -> ++ getViaTmp rsp AlwaysVerify k $ \tmp -> + unVerified $ storefile tmp o l getb p + runner (next ok) + StoreContentTo dest o l getb next -> do +--- git-annex-6.20170101.orig/Remote.hs ++++ git-annex-6.20170101/Remote.hs +@@ -12,6 +12,7 @@ module Remote ( + storeKey, + retrieveKeyFile, + retrieveKeyFileCheap, ++ retrievalSecurityPolicy, + removeKey, + hasKey, + hasKeyCheap, +--- git-annex-6.20170101.orig/Remote/Git.hs ++++ git-annex-6.20170101/Remote/Git.hs +@@ -571,10 +571,11 @@ copyToRemote' r key file meterupdate + ensureInitialized + copier <- mkCopier hardlink params + let verify = Annex.Content.RemoteVerify r ++ let rsp = RetrievalAllKeysSecure + runTransfer (Transfer Download u key) file forwardRetry $ \p -> + let p' = combineMeterUpdate meterupdate p + in Annex.Content.saveState True `after` +- Annex.Content.getViaTmp verify key ++ Annex.Content.getViaTmp rsp verify key + (\dest -> copier object dest p' (liftIO checksuccessio)) + ) + +--- git-annex-6.20170101.orig/Types/GitConfig.hs ++++ git-annex-6.20170101/Types/GitConfig.hs +@@ -74,6 +74,7 @@ data GitConfig = GitConfig + , annexAddUnlocked :: Bool + , annexAllowedUrlSchemes :: S.Set Scheme + , annexAllowedHttpAddresses :: String ++ , annexAllowUnverifiedDownloads :: Bool + , coreSymlinks :: Bool + , coreSharedRepository :: SharedRepository + , gcryptId :: Maybe String +@@ -128,6 +129,8 @@ extractGitConfig r = GitConfig + getmaybe (annex "security.allowed-url-schemes") + , annexAllowedHttpAddresses = fromMaybe "" $ + getmaybe (annex "security.allowed-http-addresses") ++ , annexAllowUnverifiedDownloads = (== Just "ACKTHPPT") $ ++ getmaybe (annex "security.allow-unverified-downloads") + , coreSymlinks = getbool "core.symlinks" True + , coreSharedRepository = getSharedRepository r + , gcryptId = getmaybe "core.gcrypt-id" +--- git-annex-6.20170101.orig/Types/Key.hs ++++ git-annex-6.20170101/Types/Key.hs +@@ -15,6 +15,7 @@ module Types.Key ( + chunkKeyOffset, + isChunkKey, + isKeyPrefix, ++ isVerifiable, + + prop_isomorphic_key_encode, + prop_isomorphic_key_decode +--- git-annex-6.20170101.orig/doc/git-annex.mdwn ++++ git-annex-6.20170101/doc/git-annex.mdwn +@@ -1126,6 +1126,10 @@ Here are all the supported configuration + from remotes. If you trust a remote and don't want the overhead + of these checksums, you can set this to `false`. + ++ Note that even when this is set to `false`, git-annex does verification ++ in some edge cases, where it's likely the case than an ++ object was downloaded incorrectly, or when needed for security. ++ + * `remote.<name>.annexUrl` + + Can be used to specify a different url than the regular `remote.<name>.url` +@@ -1268,6 +1272,43 @@ Here are all the supported configuration + causing it to be downloaded into your repository and transferred to + other remotes, exposing its content. + ++* `annex.security.allow-unverified-downloads`, ++ ++ For security reasons, git-annex refuses to download content from ++ most special remotes when it cannot check a hash to verify ++ that the correct content was downloaded. This particularly impacts ++ downloading the content of URL or WORM keys, which lack hashes. ++ ++ The best way to avoid problems due to this is to migrate files ++ away from such keys, before their content reaches a special remote. ++ See [[git-annex-migrate]](1). ++ ++ When the content is only available from a special remote, you can ++ use this configuration to force git-annex to download it. ++ But you do so at your own risk, and it's very important you read and ++ understand the information below first! ++ ++ Downloading unverified content from encrypted special remotes is ++ prevented, because the special remote could send some other encrypted ++ content than what you expect, causing git-annex to decrypt data that you ++ never checked into git-annex, and risking exposing the decrypted ++ data to any non-encrypted remotes you send content to. ++ ++ Downloading unverified content from (non-encrypted) ++ external special remotes is prevented, because they could follow ++ http redirects to web servers on localhost or on a private network, ++ or in some cases to a file:/// url. ++ ++ If you decide to bypass this security check, the best thing to do is ++ to only set it temporarily while running the command that gets the file. ++ The value to set the config to is "ACKTHPPT". ++ For example: ++ ++ git -c annex.security.allow-unverified-downloads=ACKTHPPT annex get myfile ++ ++ It would be a good idea to check that it downloaded the file you expected, ++ too. ++ + * `annex.secure-erase-command` + + This can be set to a command that should be run whenever git-annex diff -Nru git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch --- git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,331 @@ +From: Joey Hess <i...@joeyh.name> +Date: Mon, 18 Jun 2018 15:38:25 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 3a4e7ae8749102cf318a41f32598eb40ec22e92e +Subject: limit url downloads to whitelisted schemes + +backported from 28720c795ff57a55b48e56d15f9b6bcb977f48d9 + +Security fix! Allowing any schemes, particularly file: and +possibly others like scp: allowed file exfiltration by anyone who had +write access to the git repository, since they could add an annexed file +using such an url, or using an url that redirected to such an url, +and wait for the victim to get it into their repository and send them a copy. + +* Added annex.security.allowed-url-schemes setting, which defaults + to only allowing http and https URLs. Note especially that file:/ + is no longer enabled by default. + +* Removed annex.web-download-command, since its interface does not allow + supporting annex.security.allowed-url-schemes across redirects. + If you used this setting, you may want to instead use annex.web-options + to pass options to curl. + +With annex.web-download-command removed, nearly all url accesses in +git-annex are made via Utility.Url via http-client or curl. http-client +only supports http and https, so no problem there. +(Disabling one and not the other is not implemented.) + +Used curl --proto to limit the allowed url schemes. +wget only supports http https ftp, so does not need any limiting. + +Note that this will cause git annex fsck --from web to mark files using +a disallowed url scheme as not being present in the web. That seems +acceptable; fsck --from web also does that when a web server is not available. + +quvi was not changed; it only supports a hardcoded set of urls, which +are http, not file urls. + +This does not address any external special remotes that might download +an url themselves. Current thinking is all external special remotes will +need to be audited for this problem, although many of them will use +http libraries that only support http and not curl's menagarie. + +The related problem of accessing private localhost and LAN urls is not +addressed by this commit. + +This commit was sponsored by Brett Eisenberg on Patreon. + +--- + +--- git-annex-6.20170101.orig/Annex/Content.hs ++++ git-annex-6.20170101/Annex/Content.hs +@@ -902,24 +902,15 @@ saveState nocommit = doSideAction $ do + + {- Downloads content from any of a list of urls. -} + downloadUrl :: Key -> MeterUpdate -> [Url.URLString] -> FilePath -> Annex Bool +-downloadUrl k p urls file = meteredFile file (Just p) k $ +- go =<< annexWebDownloadCommand <$> Annex.getGitConfig ++downloadUrl k p urls file = meteredFile file (Just p) k go + where +- go Nothing = do ++ go = do + a <- ifM commandProgressDisabled + ( return Url.downloadQuiet + , return Url.download + ) + Url.withUrlOptions $ \uo -> + anyM (\u -> a u file uo) urls +- go (Just basecmd) = anyM (downloadcmd basecmd) urls +- downloadcmd basecmd url = +- progressCommand "sh" [Param "-c", Param $ gencmd url basecmd] +- <&&> liftIO (doesFileExist file) +- gencmd url = massReplace +- [ ("%file", shellEscape file) +- , ("%url", shellEscape url) +- ] + + {- Copies a key's content, when present, to a temp file. + - This is used to speed up some rsyncs. -} +--- git-annex-6.20170101.orig/Annex/Url.hs ++++ git-annex-6.20170101/Annex/Url.hs +@@ -30,6 +30,7 @@ getUrlOptions = mkUrlOptions + <$> getUserAgent + <*> headers + <*> options ++ <*> (annexAllowedUrlSchemes <$> Annex.getGitConfig) + where + headers = do + v <- annexHttpHeadersCommand <$> Annex.getGitConfig +--- git-annex-6.20170101.orig/NEWS ++++ git-annex-6.20170101/NEWS +@@ -1,3 +1,14 @@ ++git-annex (6.20170101-1+deb9u2) stretch-security; urgency=high ++ ++ A security fix has changed git-annex to only support http, https, and ftp ++ URL schemes by default. You can enable other URL schemes, at your own risk, ++ using annex.security.allowed-url-schemes. ++ ++ The annex.web-download-command configuration has been removed, ++ use annex.web-options instead. ++ ++ -- Joey Hess <i...@joeyh.name> Fri, 15 Jun 2018 17:54:23 -0400 ++ + git-annex (6.20170101) unstable; urgency=medium + + XMPP support has been removed from the assistant in this release. +--- git-annex-6.20170101.orig/Types/GitConfig.hs ++++ git-annex-6.20170101/Types/GitConfig.hs +@@ -28,6 +28,9 @@ import Types.RefSpec + import Utility.HumanTime + import Utility.Gpg (GpgCmd, mkGpgCmd) + import Utility.ThreadScheduler (Seconds(..)) ++import Utility.Url (Scheme, mkScheme) ++ ++import qualified Data.Set as S + + {- Main git-annex settings. Each setting corresponds to a git-config key + - such as annex.foo -} +@@ -51,7 +54,6 @@ data GitConfig = GitConfig + , annexWebOptions :: [String] + , annexQuviOptions :: [String] + , annexAriaTorrentOptions :: [String] +- , annexWebDownloadCommand :: Maybe String + , annexCrippledFileSystem :: Bool + , annexLargeFiles :: Maybe String + , annexAddSmallFiles :: Bool +@@ -70,6 +72,8 @@ data GitConfig = GitConfig + , annexPidLock :: Bool + , annexPidLockTimeout :: Seconds + , annexAddUnlocked :: Bool ++ , annexAllowedUrlSchemes :: S.Set Scheme ++ , annexAllowedHttpAddresses :: String + , coreSymlinks :: Bool + , coreSharedRepository :: SharedRepository + , gcryptId :: Maybe String +@@ -98,7 +102,6 @@ extractGitConfig r = GitConfig + , annexWebOptions = getwords (annex "web-options") + , annexQuviOptions = getwords (annex "quvi-options") + , annexAriaTorrentOptions = getwords (annex "aria-torrent-options") +- , annexWebDownloadCommand = getmaybe (annex "web-download-command") + , annexCrippledFileSystem = getbool (annex "crippledfilesystem") False + , annexLargeFiles = getmaybe (annex "largefiles") + , annexAddSmallFiles = getbool (annex "addsmallfiles") True +@@ -120,6 +123,11 @@ extractGitConfig r = GitConfig + , annexPidLockTimeout = Seconds $ fromMaybe 300 $ + getmayberead (annex "pidlocktimeout") + , annexAddUnlocked = getbool (annex "addunlocked") False ++ , annexAllowedUrlSchemes = S.fromList $ map mkScheme $ ++ maybe ["http", "https", "ftp"] words $ ++ getmaybe (annex "security.allowed-url-schemes") ++ , annexAllowedHttpAddresses = fromMaybe "" $ ++ getmaybe (annex "security.allowed-http-addresses") + , coreSymlinks = getbool "core.symlinks" True + , coreSharedRepository = getSharedRepository r + , gcryptId = getmaybe "core.gcrypt-id" +--- git-annex-6.20170101.orig/Utility/Url.hs ++++ git-annex-6.20170101/Utility/Url.hs +@@ -15,6 +15,9 @@ module Utility.Url ( + managerSettings, + URLString, + UserAgent, ++ Scheme, ++ mkScheme, ++ allowedScheme, + UrlOptions, + mkUrlOptions, + check, +@@ -38,6 +41,7 @@ import Network.HTTP.Types + import qualified Data.CaseInsensitive as CI + import qualified Data.ByteString as B + import qualified Data.ByteString.UTF8 as B8 ++import qualified Data.Set as S + import Control.Monad.Trans.Resource + import Network.HTTP.Conduit hiding (closeManager) + +@@ -63,6 +67,15 @@ type Headers = [String] + + type UserAgent = String + ++newtype Scheme = Scheme (CI.CI String) ++ deriving (Eq, Ord) ++ ++mkScheme :: String -> Scheme ++mkScheme = Scheme . CI.mk ++ ++fromScheme :: Scheme -> String ++fromScheme (Scheme s) = CI.original s ++ + data UrlOptions = UrlOptions + { userAgent :: Maybe UserAgent + , reqHeaders :: Headers +@@ -72,15 +85,17 @@ data UrlOptions = UrlOptions + #else + , applyRequest :: forall m. Request m -> Request m + #endif ++ , allowedSchemes :: S.Set Scheme + } + + instance Default UrlOptions + where +- def = UrlOptions Nothing [] [] id ++ def = UrlOptions Nothing [] [] id ++ (S.fromList $ map mkScheme ["http", "https", "ftp"]) + +-mkUrlOptions :: Maybe UserAgent -> Headers -> [CommandParam] -> UrlOptions +-mkUrlOptions defuseragent reqheaders reqparams = +- UrlOptions useragent reqheaders reqparams applyrequest ++mkUrlOptions :: Maybe UserAgent -> Headers -> [CommandParam] -> S.Set Scheme -> UrlOptions ++mkUrlOptions defuseragent reqheaders reqparams allowedschemes = ++ UrlOptions useragent reqheaders reqparams applyrequest allowedschemes + where + applyrequest = \r -> r { requestHeaders = requestHeaders r ++ addedheaders } + addedheaders = uaheader ++ otherheaders +@@ -104,6 +119,28 @@ addUserAgent uo ps = case userAgent uo o + -- --user-agent works for both wget and curl commands + Just ua -> ps ++ [Param "--user-agent", Param ua] + ++checkPolicy :: UrlOptions -> URI -> a -> IO a -> IO a ++checkPolicy uo u onerr a ++ | allowedScheme uo u = a ++ | otherwise = do ++ hPutStrLn stderr $ ++ "Configuration does not allow accessing " ++ show u ++ hFlush stderr ++ return onerr ++ ++curlSchemeParams :: UrlOptions -> [CommandParam] ++curlSchemeParams uo = ++ [ Param "--proto" ++ , Param $ intercalate "," ("-all" : schemelist) ++ ] ++ where ++ schemelist = map fromScheme $ S.toList $ allowedSchemes uo ++ ++allowedScheme :: UrlOptions -> URI -> Bool ++allowedScheme uo u = uscheme `S.member` allowedSchemes uo ++ where ++ uscheme = mkScheme $ takeWhile (/=':') (uriScheme u) ++ + {- Checks that an url exists and could be successfully downloaded, + - also checking that its size, if available, matches a specified size. -} + checkBoth :: URLString -> Maybe Integer -> UrlOptions -> IO Bool +@@ -137,7 +174,7 @@ assumeUrlExists = UrlInfo True Nothing N + - also returning its size and suggested filename if available. -} + getUrlInfo :: URLString -> UrlOptions -> IO UrlInfo + getUrlInfo url uo = case parseURIRelaxed url of +- Just u -> case parseurlconduit (show u) of ++ Just u -> checkPolicy uo u dne' $ case parseurlconduit (show u) of + Just req -> catchJust + -- When http redirects to a protocol which + -- conduit does not support, it will throw +@@ -161,7 +198,8 @@ getUrlInfo url uo = case parseURIRelaxed + | otherwise -> dne + Nothing -> dne + where +- dne = return $ UrlInfo False Nothing Nothing ++ dne = return dne' ++ dne' = UrlInfo False Nothing Nothing + found sz f = return $ UrlInfo True sz f + + curlparams = addUserAgent uo $ +@@ -169,7 +207,7 @@ getUrlInfo url uo = case parseURIRelaxed + , Param "--head" + , Param "-L", Param url + , Param "-w", Param "%{http_code}" +- ] ++ concatMap (\h -> [Param "-H", Param h]) (reqHeaders uo) ++ (reqParams uo) ++ ] ++ concatMap (\h -> [Param "-H", Param h]) (reqHeaders uo) ++ (reqParams uo) ++ curlSchemeParams uo + + extractlencurl s = case lastMaybe $ filter ("Content-Length:" `isPrefixOf`) (lines s) of + Just l -> case lastMaybe $ words l of +@@ -263,9 +301,10 @@ downloadQuiet = download' True + download' :: Bool -> URLString -> FilePath -> UrlOptions -> IO Bool + download' quiet url file uo = do + case parseURIRelaxed url of +- Just u +- | uriScheme u == "file:" -> curl +- | otherwise -> ifM (inPath "wget") (wget , curl) ++ Just u -> checkPolicy uo u False $ ++ if uriScheme u == "file:" ++ then curl ++ else ifM (inPath "wget") (wget , curl) + _ -> return False + where + headerparams = map (\h -> Param $ "--header=" ++ h) (reqHeaders uo) +@@ -276,6 +315,10 @@ download' quiet url file uo = do + - + - When the wget version is new enough, pass options for + - a less cluttered download display. ++ - ++ - wget only supports https, http, and ftp, not file, which are ++ - all always allowed, so its url schemes do not need to be ++ - limited. + -} + #ifndef __ANDROID__ + wgetparams = concat +@@ -296,7 +339,7 @@ download' quiet url file uo = do + -- curl does not create destination file + -- if the url happens to be empty, so pre-create. + writeFile file "" +- go "curl" $ headerparams ++ quietopt "-s" ++ ++ go "curl" $ headerparams ++ quietopt "-s" ++ curlSchemeParams uo ++ + [Param "-f", Param "-L", Param "-C", Param "-", Param "-#", Param "-o"] + + {- Run wget in a temp directory because it has been buggy +--- git-annex-6.20170101.orig/doc/git-annex.mdwn ++++ git-annex-6.20170101/doc/git-annex.mdwn +@@ -1238,13 +1238,20 @@ Here are all the supported configuration + If set, the command is run and each line of its output is used as a HTTP + header. This overrides annex.http-headers. + +-* `annex.web-download-command` ++* `annex.security.allowed-url-schemes` + +- Use to specify a command to run to download a file from the web. +- (The default is to use wget or curl.) ++ List of URL schemes that git-annex is allowed to download content from. ++ The default is "http https ftp". + +- In the command line, %url is replaced with the url to download, +- and %file is replaced with the file that it should be saved to. ++ Think very carefully before changing this; there are security ++ implications. For example, if it's changed to allow "file" URLs, then ++ anyone who can get a commit into your git-annex repository could ++ `git-annex addurl` a pointer to a private file located outside that ++ repository, possibly causing it to be copied into your repository ++ and transferred on to other remotes, exposing its content. ++ ++ Some special remotes support their own domain-specific URL ++ schemes; those are not affected by this configuration setting. + + * `annex.secure-erase-command` + diff -Nru git-annex-6.20170101/debian/patches/series git-annex-6.20170101/debian/patches/series --- git-annex-6.20170101/debian/patches/series 2017-10-26 15:28:29.000000000 +0100 +++ git-annex-6.20170101/debian/patches/series 2018-06-22 16:42:37.000000000 +0100 @@ -1 +1,8 @@ CVE-2017-12976.patch +limit-url-downloads-to-whitelisted-schem.patch +block-url-downloads-by-default.patch +update-test-suite-for-security-fix.patch +add-retrievalsecuritypolicy.patch +enforce-retrievalsecuritypolicy.patch +dont-assume-boto-will-remain-secure.patch +set-ddar-to-retrievalallkeyssecure.patch diff -Nru git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch --- git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,25 @@ +From: Joey Hess <jo...@joeyh.name> +Date: Thu, 21 Jun 2018 16:38:47 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 fd8086fe4a9c4bd64e95fd9152b1800b1c0f6b89 +Subject: set ddar to RetrievalAllKeysSecure + +Based on information from Robie Basak. + +(cherry picked from commit 05ecee0db43919b3cfd8d2e2772022e16a804a04) + +--- + +--- git-annex-6.20170101.orig/Remote/Ddar.hs ++++ git-annex-6.20170101/Remote/Ddar.hs +@@ -56,8 +56,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap +- -- Unsure about this, safe default until Robie answers. +- , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure ++ -- ddar communicates over ssh, not subject to http redirect ++ -- type attacks ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy diff -Nru git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch --- git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,31 @@ +From: Joey Hess <i...@joeyh.name> +Date: Mon, 18 Jun 2018 18:18:06 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 9208ae8c4e1c9c6b1fd4cb004b4cda568ffd9406 +Subject: update test suite for security fix + + +--- + +--- git-annex-6.20170101.orig/Test.hs ++++ git-annex-6.20170101/Test.hs +@@ -1688,12 +1688,18 @@ test_add_subdirs = intmpclonerepo $ do + test_addurl :: Assertion + test_addurl = intmpclonerepo $ do + -- file:// only; this test suite should not hit the network ++ let filecmd c ps = git_annex c ++ ( "-cannex.security.allowed-url-schemes=file" ++ : "-cannex.security.allowed-http-addresses=all" ++ : ps ++ ) + f <- absPath "myurl" + let url = replace "\\" "/" ("file:///" ++ dropDrive f) + writeFile f "foo" +- git_annex "addurl" [url] @? ("addurl failed on " ++ url) ++ not <$> git_annex "addurl" [url] @? "addurl failed to fail on file url" ++ filecmd "addurl" [url] @? ("addurl failed on " ++ url) + let dest = "addurlurldest" +- git_annex "addurl" ["--file", dest, url] @? ("addurl failed on " ++ url ++ " with --file") ++ filecmd "addurl" ["--file", dest, url] @? ("addurl failed on " ++ url ++ " with --file") + doesFileExist dest @? (dest ++ " missing after addurl --file") + + -- This is equivilant to running git-annex, but it's all run in-process
signature.asc
Description: PGP signature