Arg, I'm still getting failures even though the pubkey computed by openssl is not negative and padding is not required. Any other ideas? I'm desperate. A year's supply of Guinness for the person who can help here :)
-----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Edward Chan Sent: Tuesday, April 17, 2007 11:15 AM To: openssl-users@openssl.org Subject: RE: BIGNUM library The problem with CryptoAPI is that it doesn't give you direct access to the shared secret. But I suspect it is wrong since the encryption/decryption fails (I encrypt something, and decrypt it, to make sure it is the same as the original). I agree that a format difference is unlikely. That's why I suspect it may be related to padding. I checked the binary output from BN_bn2bin() and it does seem to be ASN.1 formatted. At least it matches the data from a call to BN_to_ASN1_INTEGER(). But like Dr. Steve said, it is not likely a format difference. The endian-ness is a problem. MS likes it in little-endian. That's why I have to reverse the bytes of the pubkey before passing it to CAPI. So what I do is, BN_bn2bin(dh->pub_key, buf); ReverseBytes(buf, size); // MS CAPI requires this in little-endian MS CAPI also seems to require that the pubkey always be 128 bytes. So here, I pad with zeros before reversing the byte order. This works almost all the time this works. But I do still get very infrequent failures. But more often, I get failures even when OpenSSL is giving me exactly 128 bytes. I think the sign seems to be a good place to investigate. I'll let you know what I find. Thanks! -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dr. Stephen Henson Sent: Tuesday, April 17, 2007 11:03 AM To: openssl-users@openssl.org Subject: Re: BIGNUM library On Tue, Apr 17, 2007, Edward Chan wrote: > Right, I see what you're saying. I realize I'm not actually passing a > BIGNUM. Let me clarify, this is what I'm doing. > > First I call DH_new() to get a new DH object. > > Then I fill in the p & g members of the DH obj with pre-generated prime > and generator. > > Then I call DH_generate_key() which generates the pub_key and priv_key > pair. > > I then do a BN_bn2bin() on the pub_key member to get a byte array > representation of the BIGNUM. > > This is then passed to CryptoAPI. What is the binary format returned by > BN_bn2bin()? Is this ASN.1 formatted? It seems to work most of the > time. But occasionally fails. Some times OpenSSL returns me an pubkey > < 128 bytes, whereas CryptoAPI seems to always expect 128 (and > similarly, always returns me 128 bytes). So in cases where OpenSSL > gives me < 128, I front pad with zeros. And this also seems to work > most of the time. But very rarely, I'll still get a failure. But even > if OpenSSL gives me back 128 bytes, I still sporatically get failures. > So I suspect there is some incompatibility with the binary > representations of the BIGNUM. > A format difference would be unlikely to cause problems only occasionally. BN_bn2bin() outputs in big endian format with no padding ignoring any sign. I suggest you log the key pairs of both sides in the case where you get a failure and the shared secrets from both implementations. Steve. -- Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage OpenSSL project core developer and freelance consultant. Funding needed! Details on homepage. Homepage: http://www.drh-consultancy.demon.co.uk ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List openssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
