Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure
Hello, On 01.08.2018 14:49, Michael Wojcik wrote: >> On 30.07.2018 20:12, Michael Wojcik wrote: >> >>> FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, >>> but RLIMIT_DATA as >>> applying only to malloc. (That is, mmap'd pages do not count against the >>> data limit.) >> >> mmap() , by defninition, populates the process' virtual address space, and >> if that >> is limited in size, artificially or not, then the call is going to fail, >> eventually. > > That's irrelevant to the statement you quoted, which was about the SUS > process-limit mechanism > (setrusage et al.), not the process address space. I may have skipped a few steps and made too broad a statement, so here are the details and corrections. malloc(3) is a pure C library call, whereas mmap(2) is a proper system call on those, POSIX-conforming systems I have seen, anyway. To a userland process, there is nothing more fundamental than the system call interface. If it requires resources to do its work, it needs to go through this interface and ask "the system" first. The kernel as an implementation of "the system", where process-level resource control actually happens, does not care about what userland code such as the C library does unless that actually decides to call the system. Except for SAS OSes, processes generally live in their own virtual address space, and that is where "the process's data segment" is located that RLIMIT_DATA refers to. s/brk(2) are system calls to manage the end of the process's data segment AKA program break, while mmap() , as much broader a concept, does not have this limited view on the process's address space and is therefore also more powerful. Now, with Linux since 4.7 for example, RLIMIT_DATA also controls mmap() . What's more, glibc malloc() on Linux is actually implemented in terms of mmap() these days, although usage of s/brk() isn't phased out completely (apparently, the runtime linker still likes those). Since s/brk() and mmap() are the only means available to a Linux userland process to manage those parts of its address space that it is designed to manage, controlling a process's RLIMIT_DATA value ultimately controls all of its ability to manage its address space, data segment or otherwise. >> The pure streaming approach may be appropriate for file descriptors that are >> not >> seekable like sockets, pipes, tty ends etcpp., whereas with egular files, >> random >> access schemes using either pread(v)(2) or lseek(2) in combination with >> read(v)(2) can be employed. > > Or regular files could also be processed sequentially. What's the advantage > of making > seekable sources a special case? Making sure that the message under investigation is consistent w.r.t. the standards according to which it was supposedly constructed, without having to consume resources that aren't strictly necessary to make such a decision. In other words, it's about system performance /and/ making sure to get the logic right at the same time. There is simply no need to slurp in the whole file just to get at its end (if so required), when mechanisms for random access are readily available. Please remember our messages can get fairly large. System performance and tight integration are equally important to us as the security aspects. > In any case, the OpenSSL apps are a convenience and a set of samples. My original impression was that those tools represented some kind of reference implementation of the libraries. Clearly, I was wrong ;-) Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure
On 30.07.2018 20:12, Michael Wojcik wrote: >> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf Of >> Jordan Brown >> Sent: Monday, July 30, 2018 10:46 […] > FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, but > RLIMIT_DATA as > applying only to malloc. (That is, mmap'd pages do not count against the data > limit.) mmap() , by defninition, populates the process' virtual address space, and if that is limited in size, artificially or not, then the call is going to fail, eventually. >> If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object will be >> difficult at best. > > Agreed. And I'm not endorsing the mmap approach for this problem anyway - I'd > use a streaming > approach, so I'm not limited by address space. Let's look at the following message that was produced by symmetrically encrypting 757 plaintext octets using the Camellia cipher in CBC mode with a 256 bit key derived from a passphrase: $ cat ciphertext.pem | openssl asn1parse -i -inform PEM 0:d=0 hl=4 l= 978 cons: SEQUENCE 4:d=1 hl=2 l= 9 prim: OBJECT:pkcs7-envelopedData 15:d=1 hl=4 l= 963 cons: cont [ 0 ] 19:d=2 hl=4 l= 959 cons: SEQUENCE 23:d=3 hl=2 l= 1 prim:INTEGER :03 26:d=3 hl=3 l= 133 cons:SET 29:d=4 hl=3 l= 130 cons: cont [ 3 ] 32:d=5 hl=2 l= 1 prim: INTEGER :00 35:d=5 hl=2 l= 27 cons: cont [ 0 ] 37:d=6 hl=2 l= 9 prim: OBJECT:PBKDF2 48:d=6 hl=2 l= 14 cons: SEQUENCE 50:d=7 hl=2 l= 8 prim:OCTET STRING [HEX DUMP]:948BAC4CEDB23DE2 60:d=7 hl=2 l= 2 prim:INTEGER :0800 64:d=5 hl=2 l= 46 cons: SEQUENCE 66:d=6 hl=2 l= 11 prim: OBJECT:id-alg-PWRI-KEK 79:d=6 hl=2 l= 31 cons: SEQUENCE 81:d=7 hl=2 l= 11 prim:OBJECT:camellia-256-cbc 94:d=7 hl=2 l= 16 prim:OCTET STRING [HEX DUMP]:D7A2F88C99A1881C1B8B6AA9E2BDD002 112:d=5 hl=2 l= 48 prim: OCTET STRING [HEX DUMP]:445771F0EA6BAA64CAF35BFC2DA546845C… 162:d=3 hl=4 l= 816 cons:SEQUENCE 166:d=4 hl=2 l= 9 prim: OBJECT:pkcs7-data 177:d=4 hl=2 l= 31 cons: SEQUENCE 179:d=5 hl=2 l= 11 prim: OBJECT:camellia-256-cbc 192:d=5 hl=2 l= 16 prim: OCTET STRING [HEX DUMP]:4F8DAFF8EE165FD78C35A644735CD082 210:d=4 hl=4 l= 768 prim: cont [ 0 ] This structure, if held in a regular file, can be processed quite non-linearly, and without mmap()'ing its entire contents. The only parts that are going to grow as the plaintext grows are the ciphertext (index 192 above) and, to a lesser extend, the ones that depend on key sizes in the recipientInfos and the length components. Once the overall structure of the message is understood, sequential processing of the ciphertext should pose no problem, whichever way implemented. The pure streaming approach may be appropriate for file descriptors that are not seekable like sockets, pipes, tty ends etcpp., whereas with egular files, random access schemes using either pread(v)(2) or lseek(2) in combination with read(v)(2) can be employed. Portability is certainly an issue, but isn't this what the configure scripts are for to figure out? I also do not quite get why CMS should not lend itself to "large" data, given the above. It would seem that the whole point of the definite TLV structures is to be able to learn the type and size requirements of the data to come in the stream /before/ processing it, allowing the "processor" to take appropriate measures, and not necessarily in RAM alone. Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure
On 28.07.2018 22:27, Salz, Rich via openssl-users wrote: >>It would appear that both commands fail due to them being unable to > allocate more memory to slurp the rest of the input file's contents into. > Is this intentional behaviour? > > It is a known issue. What's the reason for using malloc(3) in the first place? Is this a limitation of the library or just openssl cms ? For the latter, if the argument to -in can be determined to resolve to a file descriptor of a regular file, the file's contents can be /very/ conveniently mmap(2)'ed into the process' address space, ignoring possible limits. Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] openssl cms -decrypt failing due to malloc(3) failure
Hello all, Assume that we have $ uname -srvmpio Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux $ openssl version OpenSSL 1.0.2g 1 Mar 2016 $ printenv SHELL /bin/bash $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63575 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 63575 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited $ dd if=/dev/zero of=plaintext.in bs=1024 count=$((1024 * 1024 * 2)) 2097152+0 records in 2097152+0 records out 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.7645 s, 277 MB/s $ echo -n 'uno dos tres cuatro' | openssl cms -encrypt -aes-256-cbc -pwri_password fd:0 -in plaintext.in -binary -out ciphertext.der -outform DER $ ls -lAF total 4054804 -rw-rw-r-- 1 ubuntu ubuntu 2004623580 Jul 28 20:09 ciphertext.der -rw-rw-r-- 1 ubuntu ubuntu 2147483648 Jul 28 19:55 plaintext.in then we get $ openssl asn1parse -in ciphertext.der -inform DER -i 14050799902:error:07064041:memory buffer routines:BUF_MEM_grow:malloc failure:buffer.c:113: or $ echo -n 'uno dos tres cuatro' | openssl cms -decrypt -pwri_password fd:0 -in ciphertext.der -inform DER -out plaintext.out Error reading S/MIME message 139871963694744:error:07069041:memory buffer routines:BUF_MEM_grow_clean:malloc failure:buffer.c:150: 139871963694744:error:0D06B041:asn1 encoding routines:ASN1_D2I_READ_BIO:malloc failure:a_d2i_fp.c:239: It would appear that both commands fail due to them being unable to allocate more memory to slurp the rest of the input file's contents into. Is this intentional behaviour? Both commands work when the plaintext file is half the size, i.e. 1 GiB, BTW. Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] openssl asn1parse -length
On 24.07.2018 11:41, Jakob Bohm wrote: > Best option is to download the documents that specify the DER > (or BER) ASN.1 Encoding, which is the X.690 (2015) ITU-T > "recommendation" which was a freely downloadable PDF last time > I checked. […] > For example, the one you show below is thus: > 0x30 (TAG for SEQUENCE) > Some length value large enough to hold what follows, see X.690 > 0x06 (TAG for OBJECT id) > Any definite encoding of length = 9 bytes(127 possibilities) > 0x2A (The bytes of pkcs7-envelopedData=1.2.840.113549.1.7.3) > 0x86 .840 > 0x48 > 0x86 .113549 > 0xF7 > 0x0D > 0x01 .1 > 0x07 .7 > 0x03 .3 > Rest not needed for identification of a pkcs7-envelopedData file. > > Note that same length values can be encoded in more than one way > if the file is in BER format, as is often the case with PKCS#7 > files. Thanks heaps for your input. Much appreciated! Cheers, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] openssl asn1parse -length
Hello all, I have been trying to find a way to ascertain that the contents of a file is a DER-encoded ASN.1 structure such as $ openssl version OpenSSL 1.0.2g 1 Mar 2016 $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -i 0:d=0 hl=4 l= 978 cons: SEQUENCE 4:d=1 hl=2 l= 9 prim: OBJECT:pkcs7-envelopedData 15:d=1 hl=4 l= 963 cons: cont [ 0 ] 19:d=2 hl=4 l= 959 cons: SEQUENCE 23:d=3 hl=2 l= 1 prim:INTEGER :03 26:d=3 hl=3 l= 133 cons:SET 29:d=4 hl=3 l= 130 cons: cont [ 3 ] 32:d=5 hl=2 l= 1 prim: INTEGER :00 35:d=5 hl=2 l= 27 cons: cont [ 0 ] 37:d=6 hl=2 l= 9 prim: OBJECT:PBKDF2 48:d=6 hl=2 l= 14 cons: SEQUENCE 50:d=7 hl=2 l= 8 prim:OCTET STRING [HEX DUMP]:64C8DCE92BE6CF80 60:d=7 hl=2 l= 2 prim:INTEGER :0800 64:d=5 hl=2 l= 46 cons: SEQUENCE 66:d=6 hl=2 l= 11 prim: OBJECT:id-alg-PWRI-KEK 79:d=6 hl=2 l= 31 cons: SEQUENCE 81:d=7 hl=2 l= 11 prim:OBJECT:camellia-256-cbc 94:d=7 hl=2 l= 16 prim:OCTET STRING [HEX DUMP]:DC131C842F099909DF465439C1B06038 112:d=5 hl=2 l= 48 prim: OCTET STRING [HEX DUMP]:7BEFFB307D05C8242A040B371EEA3C6F59F082C415057BF5A71F67437B92668CEED9C46B0F57B4E4A077B1651892D9D5 162:d=3 hl=4 l= 816 cons:SEQUENCE 166:d=4 hl=2 l= 9 prim: OBJECT:pkcs7-data 177:d=4 hl=2 l= 31 cons: SEQUENCE 179:d=5 hl=2 l= 11 prim: OBJECT:camellia-256-cbc 192:d=5 hl=2 l= 16 prim: OCTET STRING [HEX DUMP]:995169EEF15C876E5F1A92DAF6A129D7 210:d=4 hl=4 l= 768 prim: cont [ 0 ] Since the files to test are rather large, I'd be content with being able to have only the first couple of bytes inspected. It would appear that the -length option allows to do just that. However, whatever argument specified, I get this: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 4 Error in encoding 140548547200664:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:157: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 16 Error in encoding 140076397213336:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:147: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 32 Error in encoding 139879438956184:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:147: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 64 Error in encoding 139887577974424:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:147: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 128 Error in encoding 140008118994584:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:147: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 256 Error in encoding 140518349809304:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:147: $ openssl asn1parse -in ciphertext.der -inform DER -offset 0 -length 512 Error in encoding 140042967262872:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:147: etcpp. The files to test are expected to be at least 512 bytes in size. What's the expected behaviour of the -length option, BTW? Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Authenticated encryption in CMS with OpenSSL
On 20.07.2018 15:42, Salz, Rich via openssl-users wrote: > Sorry I was not clear. > > This has not been implemented. In recent releases, we added a check to > disallow AEAD ciphers, > rather than failing (perhaps SILENTLY) later on. Yeah, the checks happen in crypto/evp/evp_lib.c:EVP_CIPHER_param_to_asn1() . I understand that AEAD cipher support for CMS would have to be written from scratch, then. Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
Re: [openssl-users] Authenticated encryption in CMS with OpenSSL
On 20.07.2018 15:19, Salz, Rich via openssl-users wrote: > The ciphers are available, but the code to use things like AES-GCM never > actually worked. Or if it claimed to work, it was actually broken. I take this to mean there has actually code been written already to that effect. Has it made its way into the repo, and if so, which branch/tree could it be in? Thanks, Christian -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
[openssl-users] Authenticated encryption in CMS with OpenSSL
Hello all, While investigating if and how OpenSSL in several versions could be made to support authenticated encryption in CMS [1], I noticed that, e.g., AES in CCM and GCM modes disappeared completely in newer versions from the command line tools. That is, while, e.g., > openssl version OpenSSL 1.0.2g 1 Mar 2016 > openssl enc -ciphers 2>&1 | grep -E -i -- '-(ccm|gcm)' -aes-128-ccm -aes-128-cfb -aes-128-cfb1 -aes-128-gcm -aes-128-ofb -aes-128-xts -aes-192-cbc -aes-192-ccm -aes-192-cfb -aes-192-ecb -aes-192-gcm -aes-192-ofb -aes-256-ccm -aes-256-cfb -aes-256-cfb1 -aes-256-gcm -aes-256-ofb -aes-256-xts -gost89-cnt-id-aes128-CCM -id-aes128-GCM -id-aes128-wrap-id-aes192-CCM -id-aes192-GCM -id-aes192-wrap-id-aes256-CCM -id-aes256-GCM provides the modes, > openssl version OpenSSL 1.1.0h 27 Mar 2018 > openssl enc -ciphers | grep -E -i -- '-(ccm|gcm)' does not. The respective algorithms, such as EVP_aes_256_gcm() , appear to be available in both versions' libraries, though. Would someone perhaps involved in the release process be able to explain the reasoning behind dropping the authenticated encryption modes from the command line tools? Are there plans to extend OpenSSL's current support for CMS [2] to newer CMS versions? Or is there even a connection between the two, preventing the latter? Thanks, Christian [1] https://tools.ietf.org/html/rfc5083 [2] https://tools.ietf.org/html/rfc3369 -- *Christian Böhme* Developer System Integration CLOUD *CLOUD & HEAT Technologies GmbH* Königsbrücker Str. 96 (Halle 15) | 01099 Dresden Tel: +49 351 479 3670 - 100 Fax: +49 351 479 3670 - 110 E-Mail: christian.boe...@cloudandheat.com <mailto:christian.boe...@cloudandheat.com> Web: https://www.cloudandheat.com <https://www.cloudandheat.com> Handelsregister: Amtsgericht Dresden Registernummer: HRB 30549 USt.-Ident.-Nr.: DE281093504 Geschäftsführer: Nicolas Röhrs signature.asc Description: OpenPGP digital signature -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
