We are building a peer-to-peer system that uses SSL for connection privacy and performs authentication outside of SSL. The system creates self-signed certificates as needed on every node. Since we use those self-signed certficates in the authentication I have to force peer cert validation as the only means to forcing SSL to send client cert to the server. So I set the following flags on both ends of the connection: SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT | SSL_VERIFY_CLIENT_ONCE
Here is the code which creates the self-signed certificates: // Generates a self-signed x509 certificate static X509 * generate_x509(EVP_PKEY * pkey) { X509 * x509 = X509_new(); if(!x509) { throw CryptoException("Unable to create X509 structure"); } X509_set_version(x509, 2); ASN1_INTEGER_set(X509_get_serialNumber(x509), 1); X509_gmtime_adj(X509_get_notBefore(x509), 0); X509_gmtime_adj(X509_get_notAfter(x509), 31536000L); // one year X509_set_pubkey(x509, pkey); X509_NAME * name = X509_get_subject_name(x509); X509_NAME_add_entry_by_txt(name, "C", MBSTRING_ASC, (unsigned char *)"CA", -1, -1, 0); X509_NAME_add_entry_by_txt(name, "O", MBSTRING_ASC, (unsigned char *)"Amazon", -1, -1, 0); X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC, (unsigned char *)"localhost", -1, -1, 0); X509_set_issuer_name(x509, name); /* Inform openSSL this certificate can be used as CA */ add_ext(x509, NID_basic_constraints, "critical,CA:TRUE"); //add_ext(x509, NID_key_usage, "critical,keyCertSign,cRLSign"); if(!X509_sign(x509, pkey, EVP_sha1())) { X509_free(x509); throw CryptoException("Error signing certificate"); } return x509; } PROBLEM: the server fails SSL connection with SSL_accept: tlsv1 alert unknown ca The funny part I cannot even find where in source code that error (code SSL_R_TLSV1_ALERT_UNKNOWN_CA) gets thrown. I am looking through openssl-1.0.1e/ssl/s3_srvr.c What am I missing? Thank you for your time