Re: Public key pinning in guix?

[Maxime Devos]

Philip McGrath schreef op zo 09-01-2022 om 08:57 [-0500]:
> The part of the deprecation of HPKP that seems most relevant is that 
> some number of servers---I suspect it may be a large number---are 
> configured under the assumption that no one relies on their using any 
> particular public key. For example, Certbot in its default configuration 
> will rotate to a new public key every time it gets a new certificate, 
> i.e. every two months (30 days before expiration). There is a 
> `--reuse-key` flag, but I don't get the impression that it's widely used 
> unless the server knows clients will rely on the key remaining the same. 
> In fact, I've heard some people argue against reusing keys, as a way to 
> limit the window for damage that could be done by a compromised private 
> key. I'm not trying to take a position on which is the best way to 
> manage a web server, just to point out that I think some servers will 
> change keys very often.

Yes, indeed.  I was wondering if the tools for Let's Encrypt generate
new keys or whether they generate new certificates.  That said, we
could ask the upstreams if they want to do --reuse-key.

The number of package updates in a period where the public key remains
the same seems to be:

  * very low for Minetest packages, so I wouldn't pin the public key
for content.minetest.net.  However, we could do another type of
pin: we could pin the Let's Encrypt _root certificate_ for
content.minetest.net.  Presumably, content.minetest.net will
keep using Let's Encrypt for years.

It's somewhat weaker than public key pinning, as it assumes
Let's Encrypt is reliable, but it stops _other_ potentially
compromised/corrupted CA's from interfering.

  * high for Emacs, Python and R packages, so the the cost of pinning
seems low w.r.t. the number of updates, so I believe pinning
would work well here.

I don't have numbers though.

> If we have some reason to believe that, say, "hackage.haskell.org" will 
> have a stable public key for a reasonably long time, then I'm all for 
> this! And I'm not vehemently against it, anyway: there are mitigations 
> to the potential downsides for end users. But, if we don't know the 
> server's policy, I can imagine even just the seven domains in your 
> original email producing more than one break per month, and I don't know 
> how we'd distinguish a real attack from a routine failure. It's just a 
> hypothesis, though: if anyone has more concrete data, I'd be interested 
> to hear it.

Yes, distinguishing an actual MITM from a genuine ‘the public key has
been changed’ seems rather non-trivial.  However, in the script for
updating the list of pins, we could implement heuristics like:

  * Contact the site from multiple countries (e.g. using tor).
If in different countries there are different public keys,
something is wrong.

False-positives possible when using certain load balancing
schemes, so for some sites this heuristic shouldn't be run.

  * If the root certificate in the chain changed, things are probably
wrong.

False-positives are possible when the site changes CA roots,
but that's presumably very rare.  In these cases, we could send
a mail to Minetest, Hackage, etc. people asking if this CA root
change is correct.

  * Automatically send a message to the IRC channel associated with
the website, telling that the public key changed, and ask that
if they see a different public key, that they report it to
guix-devel@gnu.org or #guix on Libera Chat.

This one should only be done if upstream doesn't mind us sending
these messages of course.

These heuristics aren't perfectly but I believe they have few false
positives and false negatives in practice (here positive=MITM).
It would probably be a good idea to look at how TLS compromises in the
wild look like.

Anyway, seems like there are some possibilities in this area, but
personally I'll table it for later.  Feel free to implement though
if interested.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Public key pinning in guix?

[Philip McGrath]

Hi,

On 1/9/22 06:54, Maxime Devos wrote:

Hi,

Philip McGrath schreef op za 08-01-2022 om 11:37 [-0500]:

This sounds like HTTP Public Key Pinning (HPKP).[1] AIUI, HTTP Public
Key Pinning was deprecated, and support has been removed from major
browser engines by January 2020.[2][3][4] While it seemed like a good
idea for reasons like the ones you list, apparently it not only proved
very difficult for site administrators to configure, with severe
consequences for mistakes, it also enabled potential ransomware attacks
and other bad stuff.[6]

I never followed this feature closely and don't have a strongly-held
opinion on the merits, but, if the "web platform" has deprecated this
feature---more concretely, if it is Considered Harmful by sysadmins and
servers are configured with the expectation that no one does this any
more---I don't think it would improve reliability for Guix to
unilaterally revive HPKP.


It does instead sound like HPKP -- however, what I proposed is in some
sense the inverse of HPKP:

Instead of a webserver telling the client to pin a certain key, the
client has pinned a certain key in advance.  So pinning is Guix'
responsibility, not the web server's.

What I propose is more close to ‘certificate pinning’ (actually
public key pinning), see e.g.
.
Even then, it's a bit different: the certificate of the server must
be correct according to both the root CAs in $SSL_CERT_DIR
AND the pin list.


I agree that the specifics of what you proposed are significantly 
different than HPKP as implemented by browsers, and also that public key 
pinning can be quite useful for securing communication with a known server.



That said, let's not use pins when doing "guix pull",
"guix perform-download" or "guix substitute" because "guix pull"
is rather essential and the guix used as the daemon is rarely
updated -- temporarily breaking "guix refresh", "guix download"
or "guix import" is much less a problem.


Yes, it seems much lower-risk to have potential breakage of the 
contributor-oriented commands than the user-facing ones. Also, for many 
of the user-facing commands, we'll ultimately verify the content of what 
we download, whether by hash or channel introductions.



  * Does the fact that web browsers deprecated HPKP matter?

I don't think so. E.g. [5] says that

‘However, this exposes as part of the Open Web Platform considerations
that are external to it: specifically, the choice and selection of CAs
is a product-level security decision made by browsers or by OS vendor,
and the choice and use of sub-CAs, cross-signing, and other aspects of
the PKI hierarchy are made independently by CAs.’

I think that "guix download/refresh/import" qualifies as ‘product level’,
or ‘browser’ here, and that Guix qualifies as OS vendor.

I don't think that the bit about sub-CAs, cross-signing, etc. is relevant
here: we pin public keys, not CAs, and the public key pin can be adjusted
whenever the website decided to use another public key -- albeit with
a (hopefully brief?) period where it is temporarily inaccessible to
"guix download/refresh/import".


The part of the deprecation of HPKP that seems most relevant is that 
some number of servers---I suspect it may be a large number---are 
configured under the assumption that no one relies on their using any 
particular public key. For example, Certbot in its default configuration 
will rotate to a new public key every time it gets a new certificate, 
i.e. every two months (30 days before expiration). There is a 
`--reuse-key` flag, but I don't get the impression that it's widely used 
unless the server knows clients will rely on the key remaining the same. 
In fact, I've heard some people argue against reusing keys, as a way to 
limit the window for damage that could be done by a compromised private 
key. I'm not trying to take a position on which is the best way to 
manage a web server, just to point out that I think some servers will 
change keys very often.


If we have some reason to believe that, say, "hackage.haskell.org" will 
have a stable public key for a reasonably long time, then I'm all for 
this! And I'm not vehemently against it, anyway: there are mitigations 
to the potential downsides for end users. But, if we don't know the 
server's policy, I can imagine even just the seven domains in your 
original email producing more than one break per month, and I don't know 
how we'd distinguish a real attack from a routine failure. It's just a 
hypothesis, though: if anyone has more concrete data, I'd be interested 
to hear it.


-Philip

Re: Public key pinning in guix?

[Maxime Devos]

Hi,

Philip McGrath schreef op za 08-01-2022 om 11:37 [-0500]:
> This sounds like HTTP Public Key Pinning (HPKP).[1] AIUI, HTTP Public 
> Key Pinning was deprecated, and support has been removed from major 
> browser engines by January 2020.[2][3][4] While it seemed like a good 
> idea for reasons like the ones you list, apparently it not only proved 
> very difficult for site administrators to configure, with severe 
> consequences for mistakes, it also enabled potential ransomware attacks 
> and other bad stuff.[6]
> 
> I never followed this feature closely and don't have a strongly-held 
> opinion on the merits, but, if the "web platform" has deprecated this 
> feature---more concretely, if it is Considered Harmful by sysadmins and 
> servers are configured with the expectation that no one does this any 
> more---I don't think it would improve reliability for Guix to 
> unilaterally revive HPKP.

It does instead sound like HPKP -- however, what I proposed is in some
sense the inverse of HPKP:

Instead of a webserver telling the client to pin a certain key, the
client has pinned a certain key in advance.  So pinning is Guix'
responsibility, not the web server's.

What I propose is more close to ‘certificate pinning’ (actually
public key pinning), see e.g.
<https://blogs.fsfe.org/jens.lechtenboerger/2014/03/10/certificate-pinning-with-gnutls-in-the-mess-of-ssltls/>.
Even then, it's a bit different: the certificate of the server must
be correct according to both the root CAs in $SSL_CERT_DIR
AND the pin list.

The sources you referenced listed a few potential problems with HPKP.
Most don't apply to what I propose, and when they do apply, it's at
most a denial of server that can easily be fixed:

 * Using HPKP for Evil:

   Suppose the attacker tries to perform a RansomPKP attack.
   Then they need need to either:

   1. Modify the pin list in guix to list their public key.
  If an attacker has commit acces, then I think we have
  bigger problems.

   or

   2. Do a MITM (or compromise the web server) to some people
  on #guix that have commit access.  Tell people on #guix
  that the pinned public key is outdated.  People on #guix
  confirm, and commit the new pinned public key.

  The attacker says to #guix -- pay me $large number, or else.
  Response #guix -- let's simply revert the pin and inform
  relevant authorities.

  Attacker: ️.

  Note that, to do a MITM, the attacker would have to
  compromise/corrupt a CA. Public key pinning makes no difference
  to the difficulty of compromising the web server. So this attack
  would be at least as hard as when guix doesn't do public key pinning.

 * Supercookies: the key pins are fixed per commit and the server
   cannot tell guix to pin something, so there's no state here,
   so no supercookies.

 * HPKP Suicide:

   Web servers don't set a pin, guix does.  And guix can decide to
   change a pin whenever it wants.  At most, guix might forget to
   change the pin to the new public key, but that's easily rectified
   when noticed (e.g., if multiple people on #guix notice the same
   public key and the committer notices the same public key).

   That said, let's not use pins when doing "guix pull",
   "guix perform-download" or "guix substitute" because "guix pull"
   is rather essential and the guix used as the daemon is rarely
   updated -- temporarily breaking "guix refresh", "guix download"
   or "guix import" is much less a problem.

 * Things about subdomains on [6]: the domain name match must be
   exact; a pin for a domain does not imply a pin for subdomains.
   From the perspective of guix, domains and subdomains would be
   unrelated.

 * Does the fact that web browsers deprecated HPKP matter?

   I don't think so. E.g. [5] says that

   ‘However, this exposes as part of the Open Web Platform considerations
   that are external to it: specifically, the choice and selection of CAs
   is a product-level security decision made by browsers or by OS vendor,
   and the choice and use of sub-CAs, cross-signing, and other aspects of
   the PKI hierarchy are made independently by CAs.’

   I think that "guix download/refresh/import" qualifies as ‘product level’,
   or ‘browser’ here, and that Guix qualifies as OS vendor.

   I don't think that the bit about sub-CAs, cross-signing, etc. is relevant
   here: we pin public keys, not CAs, and the public key pin can be adjusted
   whenever the website decided to use another public key -- albeit with
   a (hopefully brief?) period where it is temporarily inaccessible to
   "guix download/refresh/import".

   Also, they suggest using Expect-CT instead. But Expect-CT is less useful:
   it's rather complicated, and it only prevents forged certificates from
   being unnoticed -- it doesn't actually prevent them from being

Re: Public key pinning in guix?

[Philip McGrath]

Hi,

On 1/7/22 16:24, Maxime Devos wrote:

The purpose is to resist a compromise of the CA system. More
concretely, if you now do "guix refresh -u minetest-moreores"
then a MITM that compromised a CA cannot secretly replace
minetest-moreores with a mod that mines bitcoin for the MITM,
or something.

Possibly also useful for "guix download", "guix import", "guix lint",
"guix build --with-latest=...".

A downside is that whenever content.minetest.net changes public keys,
the pinned public key in Guix needs to be updated. How often does this
happen? I wouldn't now. This could be partially automated with
a "./pre-inst-env guix update-the-pinned-keys" script, and there could
be an "GUIX_IGNORE_KEY_PINNING=yes" environment variable as escape
hatch.

WDYT, worth the trouble or not?



This sounds like HTTP Public Key Pinning (HPKP).[1] AIUI, HTTP Public 
Key Pinning was deprecated, and support has been removed from major 
browser engines by January 2020.[2][3][4] While it seemed like a good 
idea for reasons like the ones you list, apparently it not only proved 
very difficult for site administrators to configure, with severe 
consequences for mistakes, it also enabled potential ransomware attacks 
and other bad stuff.[6]


I never followed this feature closely and don't have a strongly-held 
opinion on the merits, but, if the "web platform" has deprecated this 
feature---more concretely, if it is Considered Harmful by sysadmins and 
servers are configured with the expectation that no one does this any 
more---I don't think it would improve reliability for Guix to 
unilaterally revive HPKP.


-Philip

[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Public_Key_Pinning
[2]: https://scotthelme.co.uk/hpkp-is-no-more/
[3]: 
http://web.archive.org/web/20200618234723/https://www.fxsitecompat.dev/en-CA/docs/2019/http-public-key-pinning-is-no-longer-supported/

[4]: https://chromestatus.com/feature/5903385005916160
[5]: 
https://groups.google.com/a/chromium.org/g/blink-dev/c/he9tr7p3rZ8/m/eNMwKPmUBAAJ

[6]: https://scotthelme.co.uk/using-security-features-to-do-bad-things/

Public key pinning in guix?

[Maxime Devos]

Hi guix-devel,

I coded up the following recently:

[start snip]
;; TODO does this check all the right fields?
(define %pinned-public-keys
  '(("content.minetest.net" . #u8(188 216 200 89 188 149 240 145 93 189
114 207 239 50 157 141 57 196 11 102))
("ftp.gnu.org" . #u8(100 133 126 118 117 115 141 72 253 200 108 158
64 47 85 199 90 0 253 179 181))
("www.stackage.org" . #u8(101 7 34 114 166 42 66 55 116 60 42 253
85 30 134 236 217 108 67 119))
("hackage.haskell.org" . #u8(80 123 226 229 92 27 203 99 130 198 72
113 250 28 247 58 254 19 104 79))
("fastapi.metacpan.org" . #u8(86 7 157 232 133 127 13 58 144 86 37
162 91 239 203 138 150 84 56 67))
("cran.r-project.org" . #u8(60 80 123 29 14 43 131 116 105 126 126
58 154 231 6 150 216 158 70 213))
("web.cvs.savannah.gnu.org" . #u8(66 134 131 20 232 136 162 102 201
229 202 93 21 161 26 192 176 3 61 38

(use-modules (gnutls) (rnrs bytevectors))
(define old (@@ (gnutls) x509-certificate-matches-hostname?))
(set! (@@ (gnutls) x509-certificate-matches-hostname?)
  (lambda (certificate domain)
(and (old certificate domain)
 ;;  If the domain name is known to Guix, verify the public key
 ;; -- the CA could be corrupt or compromised!
 (let ((pinned-key (assoc-ref %pinned-public-keys domain)))
   (or (pk 'd domain pinned-key (not pinned-key))
   (bytevector=? pinned-key (x509-certificate-key-id
certificate)))
[end snip]

I put it in (guix import minetest); it should probably be integrated
into (guix build download) instead.

The purpose is to resist a compromise of the CA system. More
concretely, if you now do "guix refresh -u minetest-moreores"
then a MITM that compromised a CA cannot secretly replace
minetest-moreores with a mod that mines bitcoin for the MITM,
or something.

Possibly also useful for "guix download", "guix import", "guix lint",
"guix build --with-latest=...".

A downside is that whenever content.minetest.net changes public keys,
the pinned public key in Guix needs to be updated. How often does this
happen? I wouldn't now. This could be partially automated with
a "./pre-inst-env guix update-the-pinned-keys" script, and there could
be an "GUIX_IGNORE_KEY_PINNING=yes" environment variable as escape
hatch.

WDYT, worth the trouble or not?


signature.asc
Description: This is a digitally signed message part