I am trying to implement functionality similar to what ssh does with authorized_keys. I have a list of valid public keys (really public key fingerprints in my case) and I want to verify whether the peer has one of the valid keys.
My attempt to do this is to use SSL_CTX_set_cert_verify_callback to set a callback which does the following: 1) Use X509_get_pubkey() to get the EVP_PKEY for the public key from 'ctx->cert' from the X509_STORE_CTX 'ctx' passed to the verify callback 2) Use i2d_PUBKEY() on that EVP_PKEY to get the DER encoded public key 3) SHA256() the DER encoded public key to get the fingerprint and compare it to the valid fingerprints, then set ctx->error to X509_V_OK and return 1 if the fingerprint matches a valid fingerprint or set ctx->error to X509_V_ERR_CERT_REJECTED and return 0 otherwise. This seems to work. Obviously it causes things like certificate chains, whether the certificate has a valid signature, whether the certificate has expired, etc. to be ignored. I'm only interested in whether the public key the peer will use for authentication is on the valid list. What I would prefer is to not use certificates at all and use only public keys, but I'm not aware of any way to do that. Is there a way to use public keys without certificates? If not, am I doing this in a sensible and secure way or what should I be doing instead?
