> From: owner-openssl-users On Behalf Of [email protected] > Sent: Monday, November 25, 2013 19:31
> An RSA cipher algorithm is not a block cipher. It treats the entire > message as a very big integer. The better way is to use the RSA > cipher algorithm to transmit some symmetric cipher key for AES or > another block cipher algorithm and use the block cipher to encrypt > and decrypt very big files using little memory resources. > Yes, PKCS#7/CMS/SMIME uses a symmetric cipher (can be either block in stream mode like CBC or true stream like RC4, but the point is it's not publickey=RSA) for the data and transmits the symmetric key encrypted by the recipient's RSA key in a RecipientInfo structure. You can't directly RSA encrypt or sign a value larger than the key minus some overhead, and openssl (currently?) won't use an RSA key larger than 16Kbits=2Kbytes, and even if it did the time required goes as roughly the square of the keysize and soon becomes completely impractical and after that impossible. The openssl implementation of PKCS7_encrypt optionally streams, and if so can (as you say) handle large data with little memory. (Although I found the non-streaming option *silently* truncates near the 32-bit limit, even on 64-bit, without giving an error!) But the implementation of decryption doesn't stream. It tries to read the entire message into a memory structure first and then decrypt, so it fails if the message is too large. And worse, it doesn't simply fail when memory is exhausted, it fails if the message (in DER) exceeds an arbitrary limit near 1.5G even on a 64-bit system with 8G of memory. As I explained in detail in my earlier posts below. Giving the layering of openssl it looks like it would be pretty difficult to make PKCS7_decrypt stream, so address space is in principle a limit, but on a 64-bit system that limit is probably larger than all the data in the universe. Physical memory is also a *performance* limit, but to some extent can be fixed by buying more memory. > - Steve Hathaway > > > > Hi Dave, > > > > Thanks a lot for this good description. > > Hopefully I'll get time this week to try it out. > > > > Best regards, > > Marco > > > > From: [email protected] > > [mailto:[email protected]] On Behalf Of Dave Thompson > > Sent: Freitag, 22. November 2013 22:42 > > To: [email protected] > > Subject: RE: Malloc failure when decrypting files larger 1.48 GB with > > openssl smime > > > > openssl uses a mostly generic scheme to deal with numerous objects > > that are encoded in 'DER', the most common representation of ASN.1, > > including X.509 certificates and CSRs for them, and CRLs and OCSP data, > > publickey and privatekey objects and PKCS#8 and PKCS#12 key stores, > > and PKCS#7/CMS messages (which SMIME uses and thus is your case). > > For each type two basic operations are to convert the internal data > > (in C structures and buffers) to external DER 'i2d' and the reverse 'd2i'; > > these may write to or read from a memory buffer, file, or BIO. > > (Technically openssl can read and sometimes write 'BER' which is a > > slightly different ASN.1 encoding, but the most important and mostly > > oldest objects like certs are DER so the 'd' has stuck.) > > > > In your case ('smime -decrypt' calls SMIME_read_PKCS7, which indirectly > > calls ASN1_item_d2i_bio to read the message body from the input file > > through a base-64 'filter' BIO and convert the result to internal form, > > and that is where the BUF_MEM_grow limit is hit. My suggestion is > > to use the d2i-from-memory form instead, which should bypass this. > > > > > > From: > > [email protected]<mailto:[email protected]> > > [mailto:[email protected]] On Behalf Of Marco Köllner > > Sent: Friday, November 22, 2013 03:02 > > To: [email protected]<mailto:[email protected]> > > Subject: *** Spam *** RE: Malloc failure when decrypting files larger 1.48 > > GB with openssl smime > > > > Hi Dave, > > > > Thank you for this description. It helps me to understand what's happen. > > So I'll check your suggestion and come back with results as soon as > > possible. Actually I'm working on other issues. > > > > Maybe you could still help me for one thing; What do you mean with "d2i"? > > > > Please apologize, I'm not familiar with this acronym. > > > > Best regards, > > Marco > > > > From: > > [email protected]<mailto:[email protected]> > > [mailto:[email protected]] On Behalf Of Dave Thompson > > Sent: Freitag, 22. November 2013 00:38 > > To: [email protected]<mailto:[email protected]> > > Subject: RE: Malloc failure when decrypting files larger 1.48 GB with > > openssl smime > > > > There's a hardcoded limit of just under 1.5G (0x5ffffffc) on growing > > BUF_MEM, which is used to hold 'streamed' input DER in this case > > and others (but no other openssl ASN.1 data would be this big). > > Apparently this is to allow the data to be b64-ed (without linebreaks?) > > and measured in an int, which can be and usually is (signed) 32bit. > > This seems unnecessarily strict since many BUF_MEM's including this one > > won't be b64-ed, and those that are may not need to be measured in int. > > > > I guess you can get round this by a simple app that reads the DER > > into memory (converting from PEM or SMIME in the process if desired), > > then d2i's from that buffer which is already large enough. But I don't > > have a 64-bit build setup to try it. > > > > For me at least, using the ShiningLight win64 build of 1.0.1c, > > -encrypt without -stream truncates the data without any error > > indication at a bit less than 2GB (0x777c2000). Watch out for that. > > > > > > From: > > [email protected]<mailto:[email protected]> > > [mailto:[email protected]] On Behalf Of Marco Köllner > > Sent: Tuesday, November 19, 2013 05:14 > > To: [email protected]<mailto:[email protected]> > > Subject: *** Spam *** Malloc failure when decrypting files larger 1.48 GB > > with openssl smime > > > > > > Hi, > > > > > > > > For one of our customer I'm trying to test RSA public key encryption for > > file transfer. > > > > The common setup is windos7 & 8 as 64bit system & OpenSSL 0.9.8k (but > > 0.9.8y & 1.0.1e shows the same problem). > > > > > > > > Now I see a problem by decrypting large files with OpenSSL. > > > > > > > > Mainly we use openssl smime for encryption and decryption since it works > > fine for our purpose. > > > > > > > > - openssl.exe smime -encrypt -outform DER -des3 -binary -in LargeFile.dat > > -out LargeFile.dat.en Receip.cer > > > > - openssl.exe smime -decrypt -inform DER -in LargeFile.dat.en -binary > > -inkey PrivateKey.der -passin file:digest.key > > > > > > > > While encryption works, decryption seems to have problems with encrypted > > files larger 1.48 GB. Right after the start I got the following messages: > > > > > > > > Error reading S/MIME message > > > > 11992:error:07069041:memory buffer > routines:BUF_MEM_grow_clean:malloc > > failure:.\crypto\buffer\buffer.c:169: > > > > 11992:error:0D06B041:asn1 encoding routines:ASN1_D2I_READ_BIO:malloc > > failure:.\crypto\asn1\a_d2i_fp.c:242: > > > > The used key has 4096 bits. > > > > I've seen on the News page in section "Major changes between OpenSSL > > 0.9.7g and OpenSSL 0.9.8" that files large 2GB might be supported now. > > Could you help me to solve the problem? Possibly smime is not the correct > > module. > > > > Thanks in advance. > > > > Best regards, > > Marco > > > > > > > > Marco Köllner > > Application Developer > > fon +49 (0) 7062.9565-17 > > > > TMN Systemberatung GmbH > > Unsere Blogs<http://blogs.tmn-net.com/> | Unsere > > Website<http://www.tmn-systemberatung.de/> > > > > > > > > [cid:[email protected]] > > > > > > > > TMN Systemberatung GmbH | Schwabstr. 7 | 74360 Ilsfeld | Germany | fon > +49 > > (0) 7062.9565-0 | fax +49 (0) 7062.9565-19 > > Eingetragen am Amtsgericht Stuttgart, HRB 106526 | Geschäftsführer: Armin > > Neudert, Wolf-Michael Niethammer > > > > Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte > > Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail > > irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und > > vernichten Sie diese E-Mail. Das unerlaubte Kopieren und die unbefugte > > Weitergabe dieser E-Mail sind nicht gestattet. This e-mail may contain > > confidential and/or privileged information. If you are not the intended > > recipient or have received this e-mail in error please notify the sender > > immediately and delete this e-mail. Any unauthorized copying, disclosure > > or distribution of the material in this e-mail is strictly forbidden. > > > > > > > ________________________________________________________________ > ______ > OpenSSL Project http://www.openssl.org > User Support Mailing List [email protected] > Automated List Manager [email protected] ______________________________________________________________________ OpenSSL Project http://www.openssl.org User Support Mailing List [email protected] Automated List Manager [email protected]
