More details on how the 'quirk' came into being.
We have provided our end users the ability to pass into PKCS7 Encryption any
cipher that was allowed. We do not recommend or dissuade them from using any
cipher that is available. While it may not be a good idea, our end users (large
numbers) may have used the unwise cipher choice and we would like to avoid
regressing their implementation.
It seems that several ciphers are prevented from being used by PKCS7
encryption, via the settings in obj_dat.h as it is processed via the following
code path:
we call PKCS7_encrypt() passing in an evp_cipher_st
crypto/pkcs7/pk7_smime.c -- PKCS7_encrypt() hits:
if(!PKCS7_set_cipher(p7, cipher)) {
PKCS7err(PKCS7_F_PKCS7_ENCRYPT,PKCS7_R_ERROR_SETTING_CIPHER);
goto err;
}
crypto/pkcs7/pk7_lib.c -- PKCS7_set_cipher() hits:
/* Check cipher OID exists and has data in it*/
i = EVP_CIPHER_type(cipher);
if(i == NID_undef) {
PKCS7err(PKCS7_F_PKCS7_SET_CIPHER,PKCS7_R_CIPHER_HAS_NO_OBJECT_IDENTIFIER);
return(0);
}
crypto/evp/evp_lib.c -- EVP_CIPHER_type() hits:
default:
/* Check it has an OID and it is valid */
otmp = OBJ_nid2obj(nid);
crypto/objects/obj_dat.c -- OBJ_nid2obj() hits:
return((ASN1_OBJECT *)&(nid_objs[n]));
we return to:
crypto/evp/evp_lib.c -- EVP_CIPHER_type() and hit:
if(!otmp || !otmp->data) nid = NID_undef;
ASN1_OBJECT_free(otmp);
return nid;
And for any Ciphers where otmp->data is NULL, then the PKCS7 Encryption does
not proceed.
So, looking at crypto/objects/obj_dat.h, we can see which DES ciphers will or
won't work:
{"DES-ECB","des-ecb",NID_des_ecb,5,&(lvalues[187]),0},
{"DES-CFB","des-cfb",NID_des_cfb64,5,&(lvalues[192]),0},
{"DES-CBC","des-cbc",NID_des_cbc,5,&(lvalues[197]),0},
{"DES-EDE","des-ede",NID_des_ede_ecb,5,&(lvalues[202]),0},
{"DES-EDE3","des-ede3",NID_des_ede3_ecb,0,NULL},
{"DES-EDE-CBC","des-ede-cbc",NID_des_ede_cbc,0,NULL},
{"DES-EDE3-CBC","des-ede3-cbc",NID_des_ede3_cbc,8,&(lvalues[236]),0},
{"DES-OFB","des-ofb",NID_des_ofb64,5,&(lvalues[244]),0},
{"DES-EDE-CFB","des-ede-cfb",NID_des_ede_cfb64,0,NULL},
{"DES-EDE3-CFB","des-ede3-cfb",NID_des_ede3_cfb64,0,NULL},
{"DES-EDE-OFB","des-ede-ofb",NID_des_ede_ofb64,0,NULL},
{"DES-EDE3-OFB","des-ede3-ofb",NID_des_ede3_ofb64,0,NULL},
So, it seems the ability to prevent a cipher from being used in PKCS7 is
available and is being used in some cases. The settings while inconsistent, are
deterministic for this process. Can we define the ability to use des-ecb and
des-ede is as a Bug or a Feature? If it is a bug, then what is the impact of
fixing it (in our case it is large, since we will have to keep the capability
in place to decrypt everything first, then remove the capability and rebuild
the encryption definition and re-encrypt). If it is a Feature, then the 9.8m
(and other releases?) release regressed this Feature and should have a fix
applied.
More details on how
the 'quirk' came into being.
We have provided our end users the ability to pass
into PKCS7 Encryption any cipher that was allowed. We do not recommend or
dissuade them from using any cipher that is available. While it may not be a
good idea, our end users (large numbers) may have used the unwise cipher choice
and we would like to avoid regressing their implementation.
It seems that several ciphers are prevented from
being used by PKCS7 encryption, via the settings in obj_dat.h as it is processed
via the following code path:
we call PKCS7_encrypt() passing in an evp_cipher_st
crypto/pkcs7/pk7_smime.c -- PKCS7_encrypt()
hits:
if(!PKCS7_set_cipher(p7, cipher))
{
PKCS7err(PKCS7_F_PKCS7_ENCRYPT,PKCS7_R_ERROR_SETTING_CIPHER);
goto
err;
}
crypto/pkcs7/pk7_lib.c -- PKCS7_set_cipher()
hits:
/* Check cipher OID exists and has data in it*/
i =
EVP_CIPHER_type(cipher);
if(i == NID_undef)
{
PKCS7err(PKCS7_F_PKCS7_SET_CIPHER,PKCS7_R_CIPHER_HAS_NO_OBJECT_IDENTIFIER);
return(0);
}
crypto/evp/evp_lib.c -- EVP_CIPHER_type()
hits:
default:
/* Check it has an OID and it
is valid */
otmp = OBJ_nid2obj(nid);
crypto/objects/obj_dat.c -- OBJ_nid2obj()
hits:
return((ASN1_OBJECT
*)&(nid_objs[n]));
we return to:
crypto/evp/evp_lib.c --
EVP_CIPHER_type() and hit:
if(!otmp || !otmp->data) nid =
NID_undef;
ASN1_OBJECT_free(otmp);
return
nid;
And for any Ciphers where otmp->data is NULL, then
the PKCS7 Encryption does not proceed.
So, looking at crypto/objects/obj_dat.h, we can see
which DES ciphers will or won't work:
{"DES-ECB","des-ecb",NID_des_ecb,5,&(lvalues[187]),0},
{"DES-CFB","des-cfb",NID_des_cfb64,5,&(lvalues[192]),0},
{"DES-CBC","des-cbc",NID_des_cbc,5,&(lvalues[197]),0},
{"DES-EDE","des-ede",NID_des_ede_ecb,5,&(lvalues[202]),0},
{"DES-EDE3","des-ede3",NID_des_ede3_ecb,0,NULL},
{"DES-EDE-CBC","des-ede-cbc",NID_des_ede_cbc,0,NULL},
{"DES-EDE3-CBC","des-ede3-cbc",NID_des_ede3_cbc,8,&(lvalues[236]),0},
{"DES-OFB","des-ofb",NID_des_ofb64,5,&(lvalues[244]),0},
{"DES-EDE-CFB","des-ede-cfb",NID_des_ede_cfb64,0,NULL},
{"DES-EDE3-CFB","des-ede3-cfb",NID_des_ede3_cfb64,0,NULL},
{"DES-EDE-OFB","des-ede-ofb",NID_des_ede_ofb64,0,NULL},
{"DES-EDE3-OFB","des-ede3-ofb",NID_des_ede3_ofb64,0,NULL},
So, it seems the ability to prevent a cipher from
being used in PKCS7 is available and is being used in some cases. The settings
while inconsistent, are deterministic for this process. Can we define the
ability to use des-ecb and des-ede is as a Bug or a Feature? If it is a
bug, then what is the impact of fixing it (in our case it is large, since we
will have to keep the capability in place to decrypt everything first, then
remove the capability and rebuild the encryption definition and re-encrypt). If
it is a Feature, then the 9.8m (and other releases?) release regressed this
Feature and should have a fix
applied.
