I've been struggling with what looks like crazy behaviour with SSL today. I'm probably doing something really wrong, but I can't for the life of me work it out. The code is trying to use the api as trivially as possible, but I can't see anything obviously wrong with it either(?).
Basically I can open an SSL connection fine to a host, but if I try to open another one to a different host, it fails, always. If i swap the site i'm connecting to, then the roles are swapped. I can connect any number of times to any given site without problems.
Its using the NSS libs that came with mozilla 1.0.0, but one of our developers is having a problem with some other version, probably 1.3.x.
The courier imap server at my end spits out the following error when it fails (it uses openssl, with some simple locally generated key).
Jun 19 13:24:06 zZedZone imapd-ssl: starttls: accept: error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02
I get the error code SSL_ERROR_HANDSHAKE_FAILURE_ALERT when it fails.
[output snipped]
I've done a quick peruse of some of the documentation and examples, but nothing jumps out as being wrong with the test code, although surely something is.
Thanks, Michael Zucchi
PS i hope attachments are ok.
I haven't reviewed all the data you provided in detail, but I have a hunch I know what's going on.
You control both of the two servers with which you're testing, right? You probably generated the server certs being used with both servers, using an openSSL or mod_ssl test program/script, right?
My guess is that both servers have certs with the same exact issuer name and serial number, but different public keys. That's invalid. (Am I right?)
Here are some suggestions:
1. Reissue new certs for both servers. Use unique serial numbers that you've never used before and never will use again. Never reuse serial numbers, ever. You could try using the unix time() value as a serial number (assuming you issue them at least 1 second apart).
2. Change your bad cert handler to get the value from PR_GetError first thing, and then display it as a signed decimal number. Also, you should only override certain specific errors, not all errors. For example, you shouldn't override an error that indicates that NSS couldn't find the public key in the certificate!
3. Don't use PR_ErrorToString for SSL errors. Just print the number in decimal. You can look it up on this page http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html
4. Use NSS from moz 1.3.1, or later.
12345678901234567890123456789012345678901234567890123456789012345678901234567890
