Hi! On Tue, 26 May 2015 at 15:34 Apollon Oikonomopoulos <[email protected]> wrote:
> Ciao Guido! > > On 14:34 Tue 26 May , Guido Trotter wrote: > > Thanks for the very detailed report. > > > > Is there anything that you see we could change/improve in Ganeti for > > this, or we need to live with the fact that the bug caused this issue > > and a renewal makes it go away? > > Let me recap (mostly to make sure that I've gotten node-security > correctly). We have the following facts regarding the use of SSL > certificates in Ganeti: > > a) server.pem is self-signed, shared between all nodes and used by > noded's server. > > b) client.pem is also self-signed, generated on each node during node > join or renew-crypto and is used by the master as a client > certificate to connect to the node daemons. It is also used by a > prospective master to initiate a master-failover by connecting to > other nodes' node daemons. > > c) Both, client and server side involve Python code: noded is pure > Python and RPC on the master is currently invoked from Python job > code. noded's server implementation relies explicitly on OpenSSL > (via pyopenssl), while the RPC client code uses pycurl with whatever > underlying TLS library curl uses on the given platform; for Debian > this is GnuTLS. > > d) Certificate generation is always using OpenSSL. server.pem is > (re-)generated on the master, while client.pem is generated by and > on each node separately. This means that client.pem and server.pem > are generated by potentially different OpenSSL versions. > > e) SSL client authentication on the noded is *not* performed using the > X.509 CA model; instead, the client certificate fingerprint is > compared against a list of known-good master(-candidate) certificate > fingerprint. > > ยน excluding RAPI clients. > > The problem arises from the fact that server.pem and client.pem are two > completely unrelated certificates, i.e. one is not signed by the other. > We (implicitly) rely on the fact that they bear the same Subject and > Issuer DN to force the client SSL implementation to send its certificate > over to the noded for authentication. However, the certificate > subject/issuer is a typed value and GnuTLS expects both, the type *and* > the value to match. > > Unfortunately, we have reached a point where OpenSSL seems to have > changed defaults: wheezy's OpenSSL (1.0.1e) generates certificates with > PrintableString subjects, while jessie's OpenSSL (1.0.1k) generates > certificates with UTF8String subjects. Furthermore, we seem to have no > control whatsoever on the encoding used, at least on PyOpenSSL's level. > > So, in a mixed-distribution cluster, failover from a wheezy master to a > jessie master will not be possible, unless server.pem is first replaced > with one generated on a jessie node. Note that gnt-cluster renew-crypto > will likely not work in that case, because the master will still > generate a server.pem with a different OpenSSL version than most other > nodes in the cluster. Renewal will only work if all nodes have been > upgraded to the same OpenSSL version and were just using "old" > certificates. > > What can we do about it? The way I see it, there are two solutions: > > i) Implement a proper CA, have all client certificates signed by that > CA and use that same CA for client authentication, together with the > fingerprint whitelist. Signing the certificate would make sure that > the Subject DN sent by the server would always match the Issuer DN > of client.pem, as per RFC 5280. This is obviously a high development > cost solution, and partly redundant because we don't need to rely on > a CA to perform authentication. > > ii) Collect *all node certificates* and use all of them as "trusted > CAs". As discussed above, when we set server.pem as the "client CA" > (HttpBase._CreateSocket), we do so exploiting the fact that it > shares the same CN with all clients. Instead of collecting just the > fingerprints in ssconf_master_candidate_certificates, we can > collect the certificates themselves and load them all as trusted > CAs using Context.set_client_ca_list()[1]. This will make sure that > the client will send their own certificates regardless of how the > Subject DN is encoded, and will also allow things like setting a > different CN for each node. This can work in parallel with the > fingerprint whitelist, but maybe later we can completely drop the > latter and rely on normal SSL verification rules (not sure about > the security implications, it's just a suggestion). > > [1] > https://pythonhosted.org/pyOpenSSL/api/ssl.html#OpenSSL.SSL.Context.set_client_ca_list > > I think solution ii) above is the easiest and most straight-forward to > implement. It is also more correct from an SSL standpoint than what we > have today. Helga, any opinions on this? > Besides i) and ii), there is another option, if I understand correctly: iii) When creating client.pem, read the server.pem first enforce the same encoding on client.pem that server.pem has. I think we should do iii) first as a quick fix, but do either i) or ii) to solve the problem properly. I personally would love to implement i) as it seems to be the proper way to use SSL, but my suggestion for that did not receive much enthusiasm for getting resources (aka time) to implement it when I proposed that in the first place. Option ii) sounds most feasible wrt to time an resources (although also here, it would take a while if I am supposed to implement it myself). I think I even had that idea in mind when I proposed the node security doc, but realized that it would vastly complicate renewing server.pem as all client.pem would need to be renewed or at least resigned as well. Guido, any thoughts on this? Cheers, Helga > > Cheers, > Apollon >
