Hi,
I have a customer with a Kerberos V4 application who is trying to decide if they can substitute their existing Kerberos V4 DES encryption capability with SSL's DES encryption support. When calling DES_CBC_CKSUM() from the Kerberos library, the checksum output buffer and the function's return value read :
OUTBUF= 1234ABCD... // char *, 8 Byte checksum
RetVal = ABCD // unsigned long, Low order(right half) 4 bytes of 8byte checksum
When calling SSL's DES_CBC_CKSUM(), the values read:
OUTBUF= 1234ABCD... // char *, 8 Byte checksum
RetVal = DCBA // unsigned long, Low order(right half) 4 bytes of 8byte checksum
SSL's RetVal produces a checksum error in the KerberosV4 application on return because it is in the oposite byte order than what Kerberos expects.
The Kerberos and SSL macros used to initialize the return value from the OUTBUF follow respectively:
Kerberos:
#define GET_HALF_BLOCK(lr, ip) \
(lr) = ((unsigned int)(*(ip)++)) << 24; \
(lr) |= ((unsigned int)(*(ip)++)) << 16; \
(lr) |= ((unsigned int)(*(ip)++)) << 8; \
(lr) |= (unsigned int)(*(ip)++)
SSL:
#define c2l(c,l) (l =((DES_LONG)(*((c)++))) , \
l|=((DES_LONG)(*((c)++)))<< 8L, \
l|=((DES_LONG)(*((c)++)))<<16L, \
l|=((DES_LONG)(*((c)++)))<<24L)
We do not understand why SSL is swapping the bytes when initializing the output longword return value with the c2l macro in the DES_CBC_CKSUM() function. I have a test program that shows the differences.
As you can see from the output below, the output is reversed between ssl and krb.
$ run test_cbc_cksum
set key 0
ssl 0x738af841, 0x80 0x16 0xd6 0x6b 0x41 0xf8 0x8a 0x73
krb 0x41f88a7
I then added a new macro, ksg_c2l, and had it do the same order of shifting as the get_half_block.
They output is now in the correct order.
$ run test_cbc_cksum2
set key 0
ssl 0x738af841, 0x80 0x16 0xd6 0x6b 0x41 0xf8 0x8a 0x73
krb 0x41f88a73
c2l 0x738af841
ksg_c2l 0x41f88a73
I have included the program with this mail message, and here are the environmental details
we were operating in:
OpenSSL 0.9.6b with Security patches.
OpenVMS Alpha V7.2
TCP/IP V5.0A
DEC C V6.2
Kerberos V4 and 5
Thanks,
Kevin
Kevin Greaney SSL for OpenVMS Team
Hewlett Packard Company OpenVMS Engineering Group
110 Spitbrook Road
Nashua, NH 03062
(603) 884-5099
<<test_cbc_cksum.c>>
test_cbc_cksum.c
Description: test_cbc_cksum.c
