Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Viktor Dukhovni]

On Thu, Aug 02, 2018 at 01:53:42PM +0200, Christian Böhme wrote:

> > In any case, the OpenSSL apps are a convenience and a set of samples.
> 
> My original impression was that those tools represented some kind of reference
> implementation of the libraries.  Clearly, I was wrong ;-)

Well, OpenSSL's cms(1) is not a reference implementation of the CMS
standard.

It is an implementation of CMS via the OpenSSL APIs, and its source
code is a useful resource in understanding how to use those APIs.

IIRC the requirement to extract the complete CMS message into memory
is not just an artefact of the CLI design.  Rather, I seem to recall
that presently the CMS library needs the whole message in memory
in order to process it.  If so, a streaming implementation would
need to extend the CMS implementation in libcrypto to support that
mode of operation.

-- 
Viktor.
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Christian Böhme]

Hello,

On 01.08.2018 14:49, Michael Wojcik wrote:
>> On 30.07.2018 20:12, Michael Wojcik wrote:
>>
>>> FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, 
>>> but RLIMIT_DATA as
>>> applying only to malloc. (That is, mmap'd pages do not count against the 
>>> data limit.)
>>
>> mmap() , by defninition, populates the process' virtual address space, and 
>> if  that
>> is limited in size, artificially or not, then the call is going to fail, 
>> eventually.
> 
> That's irrelevant to the statement you quoted, which was about the SUS 
> process-limit mechanism
> (setrusage et al.), not the process address space.

I may have skipped a few steps and made too broad a statement, so here are the 
details
and corrections.

malloc(3)  is a pure C library call, whereas  mmap(2)  is a proper system call 
on those,
POSIX-conforming systems I have seen, anyway.

To a userland process, there is nothing more fundamental than the system call 
interface.
If it requires resources to do its work, it needs to go through this interface 
and ask
"the system" first.  The kernel as an implementation of "the system", where 
process-level
resource control actually happens, does not care about what userland code such 
as the
C library does unless that actually decides to call the system.

Except for SAS OSes, processes generally live in their own virtual address 
space, and
that is where "the process's data segment" is located that  RLIMIT_DATA  refers 
to.
s/brk(2)  are system calls to manage the end of the process's data segment AKA 
program
break, while  mmap() , as much broader a concept, does not have this limited 
view on the
process's address space and is therefore also more powerful.

Now, with Linux since 4.7 for example,  RLIMIT_DATA  also controls  mmap() .  
What's more,
glibc  malloc()  on Linux is actually implemented in terms of  mmap()  these 
days, although
usage of  s/brk()  isn't phased out completely (apparently, the runtime linker 
still likes
those).  Since  s/brk()  and  mmap()  are the only means available to a Linux 
userland
process to manage those parts of its address space that it is designed to 
manage,
controlling a process's  RLIMIT_DATA  value ultimately controls all of its 
ability
to manage its address space, data segment or otherwise.

>> The pure streaming approach may be appropriate for file descriptors that are 
>> not
>> seekable like sockets, pipes, tty ends etcpp., whereas with egular files, 
>> random
>> access schemes using either  pread(v)(2)  or  lseek(2)  in combination with
>> read(v)(2)  can be employed.
> 
> Or regular files could also be processed sequentially. What's the advantage 
> of making
> seekable sources a special case?

Making sure that the message under investigation is consistent w.r.t. the 
standards
according to which it was supposedly constructed, without having to consume 
resources
that aren't strictly necessary to make such a decision.  In other words, it's 
about
system performance /and/ making sure to get the logic right at the same time.

There is simply no need to slurp in the whole file just to get at its end (if so
required), when mechanisms for random access are readily available.  Please 
remember
our messages can get fairly large.  System performance and tight integration are
equally important to us as the security aspects.

> In any case, the OpenSSL apps are a convenience and a set of samples.

My original impression was that those tools represented some kind of reference
implementation of the libraries.  Clearly, I was wrong ;-)


Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boe...@cloudandheat.com 

Web: https://www.cloudandheat.com 

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs




signature.asc
Description: OpenPGP digital signature
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Jakob Bohm]

On 01/08/2018 15:42, Viktor Dukhovni wrote:



On Aug 1, 2018, at 9:31 AM, Michael Wojcik  
wrote:

CMS with an AEAD mode (such as AES128-GCM) ought to avoid the 
integrity-protection issue for the encrypted content, but not for the other 
parts of the message, I assume. (I'm no CMS expert so I may be missing 
something there.) And, of course, both sender and recipient would have to 
support that algorithm.

Not if you make it streaming.  A streaming implementing will emit almost
the entirety of the decrypted message before checking integrity at the
end and finding out that some part of it (already output) was wrong.


Which is entirely fine if all you do with the stream output before
integrity checking is to store it somewhere larger than process RAM,
such as in a (temporary) disk file (Or perform some other operation
which is safe with garbage input).

Consider the (logically equivalent) fact that most algorithms inside
OpenSSL stream their output to memory because it is rarely possible
to hold an entire message in CPU registers.

But I agree that blindly switching to AEAD modes does nothing to help
the "problem" of allowing a different level of the software stack to
see decrypted output before the integrity check has been completed.

OpenSSL should be an open toolkit, not a bondage-and-discipline
programming environment like NaCl.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Viktor Dukhovni]

> On Aug 1, 2018, at 9:31 AM, Michael Wojcik  
> wrote:
> 
> CMS with an AEAD mode (such as AES128-GCM) ought to avoid the 
> integrity-protection issue for the encrypted content, but not for the other 
> parts of the message, I assume. (I'm no CMS expert so I may be missing 
> something there.) And, of course, both sender and recipient would have to 
> support that algorithm.

Not if you make it streaming.  A streaming implementing will emit almost
the entirety of the decrypted message before checking integrity at the
end and finding out that some part of it (already output) was wrong.

-- 
Viktor.

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf
> Of Viktor Dukhovni
> Sent: Wednesday, August 01, 2018 06:56
>
> On Tue, Jul 31, 2018 at 06:14:18PM +0200, Jakob Bohm wrote:
>
> > Actually, the CMS format itself is clearly designed for streamed decoding.
>
> It is not, because there is no integrity protection until you reach
> the end of the message.  In a packetized format designed for
> streaming, each chunk and their sequencing is integrity protected,
> streaming extractors are only exposed to (tamper-evident) truncation
> attacks.

And thus falling foul of Moxie Marlinspike's Cryptographic Doom Principle: If 
you don't verify integrity first, sooner or later you'll be in trouble.

While CMS has been updated, its roots are long - PKCS#7 is 20 years old, after 
all, and RFC 5652 is nearing the end of its first decade. Back then, deferring 
the integrity check to the end wasn't seen as a problem. Today we know better - 
which is why many people prefer AEAD modes.

CMS with an AEAD mode (such as AES128-GCM) ought to avoid the 
integrity-protection issue for the encrypted content, but not for the other 
parts of the message, I assume. (I'm no CMS expert so I may be missing 
something there.) And, of course, both sender and recipient would have to 
support that algorithm.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Viktor Dukhovni]

On Tue, Jul 31, 2018 at 06:14:18PM +0200, Jakob Bohm wrote:

> > CMS works fine for small messages, and could even be used to construct
> > the integrity-protected chunks in a higher-level protocol.  CMS is
> > not appropriate for multi-gigabyte or terabyte, ... datasets.
>
> Actually, the CMS format itself is clearly designed for streamed decoding.

It is not, because there is no integrity protection until you reach
the end of the message.  In a packetized format designed for
streaming, each chunk and their sequencing is integrity protected,
streaming extractors are only exposed to (tamper-evident) truncation
attacks.

-- 
Viktor.
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf
> Of Christian Böhme
> Sent: Tuesday, July 31, 2018 10:16
>
> On 30.07.2018 20:12, Michael Wojcik wrote:
>
> > FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, 
> > but RLIMIT_DATA as
> > applying only to malloc. (That is, mmap'd pages do not count against the 
> > data limit.)
>
> mmap() , by defninition, populates the process' virtual address space, and if 
>  that
> is limited in size, artificially or not, then the call is going to fail, 
> eventually.

That's irrelevant to the statement you quoted, which was about the SUS 
process-limit mechanism (setrusage et al.), not the process address space.

> > Agreed. And I'm not endorsing the mmap approach for this problem anyway - 
> > I'd use a streaming
> > approach, so I'm not limited by address space.
>
> This structure, if held in a regular file, can be processed quite 
> non-linearly,
> and without  mmap()'ing  its entire contents.

Indeed. I still don't see any compelling reason to mmap it at all.

> The pure streaming approach may be appropriate for file descriptors that are 
> not
> seekable like sockets, pipes, tty ends etcpp., whereas with egular files, 
> random
> access schemes using either  pread(v)(2)  or  lseek(2)  in combination with
> read(v)(2)  can be employed.

Or regular files could also be processed sequentially. What's the advantage of 
making seekable sources a special case?

In any case, the OpenSSL apps are a convenience and a set of samples. You can 
always write your own version of the cms app.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Christian Böhme]

On 30.07.2018 20:12, Michael Wojcik wrote:

>> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf Of 
>> Jordan Brown
>> Sent: Monday, July 30, 2018 10:46
[…]
> FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, but 
> RLIMIT_DATA as
> applying only to malloc. (That is, mmap'd pages do not count against the data 
> limit.)

mmap() , by defninition, populates the process' virtual address space, and if 
that
is limited in size, artificially or not, then the call is going to fail, 
eventually.

>> If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object will be 
>> difficult at best.
> 
> Agreed. And I'm not endorsing the mmap approach for this problem anyway - I'd 
> use a streaming
> approach, so I'm not limited by address space.

Let's look at the following message that was produced by symmetrically 
encrypting
757 plaintext octets using the Camellia cipher in CBC mode with a 256 bit key 
derived
from a passphrase:

$ cat ciphertext.pem | openssl asn1parse -i -inform PEM
0:d=0  hl=4 l= 978 cons: SEQUENCE
4:d=1  hl=2 l=   9 prim:  OBJECT:pkcs7-envelopedData
   15:d=1  hl=4 l= 963 cons:  cont [ 0 ]
   19:d=2  hl=4 l= 959 cons:   SEQUENCE
   23:d=3  hl=2 l=   1 prim:INTEGER   :03
   26:d=3  hl=3 l= 133 cons:SET
   29:d=4  hl=3 l= 130 cons: cont [ 3 ]
   32:d=5  hl=2 l=   1 prim:  INTEGER   :00
   35:d=5  hl=2 l=  27 cons:  cont [ 0 ]
   37:d=6  hl=2 l=   9 prim:   OBJECT:PBKDF2
   48:d=6  hl=2 l=  14 cons:   SEQUENCE
   50:d=7  hl=2 l=   8 prim:OCTET STRING  [HEX 
DUMP]:948BAC4CEDB23DE2
   60:d=7  hl=2 l=   2 prim:INTEGER   :0800
   64:d=5  hl=2 l=  46 cons:  SEQUENCE
   66:d=6  hl=2 l=  11 prim:   OBJECT:id-alg-PWRI-KEK
   79:d=6  hl=2 l=  31 cons:   SEQUENCE
   81:d=7  hl=2 l=  11 prim:OBJECT:camellia-256-cbc
   94:d=7  hl=2 l=  16 prim:OCTET STRING  [HEX 
DUMP]:D7A2F88C99A1881C1B8B6AA9E2BDD002
  112:d=5  hl=2 l=  48 prim:  OCTET STRING  [HEX 
DUMP]:445771F0EA6BAA64CAF35BFC2DA546845C…
  162:d=3  hl=4 l= 816 cons:SEQUENCE
  166:d=4  hl=2 l=   9 prim: OBJECT:pkcs7-data
  177:d=4  hl=2 l=  31 cons: SEQUENCE
  179:d=5  hl=2 l=  11 prim:  OBJECT:camellia-256-cbc
  192:d=5  hl=2 l=  16 prim:  OCTET STRING  [HEX 
DUMP]:4F8DAFF8EE165FD78C35A644735CD082
  210:d=4  hl=4 l= 768 prim: cont [ 0 ]

This structure, if held in a regular file, can be processed quite non-linearly,
and without  mmap()'ing  its entire contents.  The only parts that are going to
grow as the plaintext grows are the ciphertext (index 192 above) and, to a 
lesser
extend, the ones that depend on key sizes in the  recipientInfos  and the length
components.  Once the overall structure of the message is understood, sequential
processing of the ciphertext should pose no problem, whichever way implemented.

The pure streaming approach may be appropriate for file descriptors that are not
seekable like sockets, pipes, tty ends etcpp., whereas with egular files, random
access schemes using either  pread(v)(2)  or  lseek(2)  in combination with
read(v)(2)  can be employed.  Portability is certainly an issue, but isn't
this what the  configure  scripts are for to figure out?

I also do not quite get why CMS should not lend itself to "large" data, given 
the
above.  It would seem that the whole point of the definite TLV structures is to 
be
able to learn the type and size requirements of the data to come in the stream
/before/ processing it, allowing the "processor" to take appropriate measures,
and not necessarily in RAM alone.


Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boe...@cloudandheat.com 

Web: https://www.cloudandheat.com 

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs




signature.asc
Description: OpenPGP digital signature
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Jakob Bohm]

On 30/07/2018 20:07, Viktor Dukhovni wrote:



On Jul 30, 2018, at 12:46 PM, Jordan Brown  wrote:

If you can't malloc the space, you probably can't mmap it either.  I have never 
heard of a malloc implementation that has artificial limits; if it's failing 
it's because it can't find that much contiguous virtual address space, and mmap 
won't be able to find it either.

If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object will be 
difficult at best.

Getting out of the weeds, the core issue is that CMS message input processing
doesn't stream.  The entire CMS message has to fit into memory.  A different
data format is required for streaming large payloads.  The data would need
to be chunked with integrity protection and protection applied to each
chunk (packet) and appropriate sequence number integrity in place to
prevent reordering, insertion or deletion of chunks.

CMS works fine for small messages, and could even be used to construct
the integrity-protected chunks in a higher-level protocol.  CMS is
not appropriate for multi-gigabyte or terabyte, ... datasets.


Actually, the CMS format itself is clearly designed for streamed decoding.

For example, it requires the AlgorithmIdentifier of the hash algorithm(s)
to precede the signed data, so a streaming implementation can set up the
input hashing before knowing the full specification of the signature
algorithm(s).

A streaming encoder will often need to use the indefinite BER encoding of
some of the outer length fields to cope with unknown input length and
variably sized fields after the data.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf Of 
> Jordan Brown
> Sent: Monday, July 30, 2018 10:46

> I have never heard of a malloc implementation that has artificial limits;

Er... setrlimit(RLIMIT_DATA). For OSes that claim Single UNIX Specification 
compliance.

>if it's failing it's because it can't find that much contiguous virtual 
>address space, and mmap won't be able to
> find it either.

FWIW, SUS Issue 5 defines RLIMIT_AS as applying to both malloc and mmap, but 
RLIMIT_DATA as applying only to malloc. (That is, mmap'd pages do not count 
against the data limit.)

> If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object will be 
> difficult at best.

Agreed. And I'm not endorsing the mmap approach for this problem anyway - I'd 
use a streaming approach, so I'm not limited by address space.

--
Michael Wojcik
Distinguished Engineer, Micro Focus



-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Viktor Dukhovni]

> On Jul 30, 2018, at 12:46 PM, Jordan Brown  
> wrote:
> 
> If you can't malloc the space, you probably can't mmap it either.  I have 
> never heard of a malloc implementation that has artificial limits; if it's 
> failing it's because it can't find that much contiguous virtual address 
> space, and mmap won't be able to find it either.
> 
> If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object will be 
> difficult at best.

Getting out of the weeds, the core issue is that CMS message input processing
doesn't stream.  The entire CMS message has to fit into memory.  A different
data format is required for streaming large payloads.  The data would need
to be chunked with integrity protection and protection applied to each
chunk (packet) and appropriate sequence number integrity in place to
prevent reordering, insertion or deletion of chunks.

CMS works fine for small messages, and could even be used to construct
the integrity-protected chunks in a higher-level protocol.  CMS is
not appropriate for multi-gigabyte or terabyte, ... datasets.

-- 
Viktor.

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Jordan Brown]

On 7/30/2018 1:57 AM, Christian Böhme wrote:
> What's the reason for using malloc(3) in the first place? Is this a
> limitation of the library or just openssl cms ?
>
> For the latter, if the argument to -in can be determined to resolve to
> a file descriptor of a regular file, the file's contents can be /very/
> conveniently mmap(2)'ed into the process' address space, ignoring
> possible limits.
>
If you can't malloc the space, you probably can't mmap it either.  I
have never heard of a malloc implementation that has artificial limits;
if it's failing it's because it can't find that much contiguous virtual
address space, and mmap won't be able to find it either.

If you're a 32-bit process, then malloc'ing or mmap'ing a 2GB object
will be difficult at best.

-- 
Jordan Brown, Oracle Solaris

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Michael Wojcik]

> From: openssl-users [mailto:openssl-users-boun...@openssl.org] On Behalf
> Of Christian Böhme
> Sent: Monday, July 30, 2018 02:57
>
> For the latter, if the argument to  -in  can be determined to resolve to a 
> file
> descriptor of a regular file, the file's contents can be /very/ conveniently
> mmap(2)'ed
> into the process' address space, ignoring possible limits.

Not portably, it can't. There are operating systems other than Linux and UNIX, 
and OpenSSL supports a number of them.

--
Michael Wojcik
Distinguished Engineer, Micro Focus


-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Salz, Rich via openssl-users]

>What's the reason for using malloc(3) in the first place?  Is this a 
> limitation
of the library or just  openssl cms ?
  
It is a limitation of the CMS command.  You might look at the -stream option.  
If you need more then that, well, a PR is also welcomed. 

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Christian Böhme]

On 28.07.2018 22:27, Salz, Rich via openssl-users wrote:

>>It would appear that both commands fail due to them being unable to
> allocate more memory to slurp the rest of the input file's contents into.
> Is this intentional behaviour?
>   
> It is a known issue.

What's the reason for using malloc(3) in the first place?  Is this a limitation
of the library or just  openssl cms ?

For the latter, if the argument to  -in  can be determined to resolve to a file
descriptor of a regular file, the file's contents can be /very/ conveniently 
mmap(2)'ed
into the process' address space, ignoring possible limits.


Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boe...@cloudandheat.com 

Web: https://www.cloudandheat.com 

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs




signature.asc
Description: OpenPGP digital signature
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Salz, Rich via openssl-users]

>It would appear that both commands fail due to them being unable to
allocate more memory to slurp the rest of the input file's contents into.
Is this intentional behaviour?
  
It is a known issue.

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

[openssl-users] openssl cms -decrypt failing due to malloc(3) failure

[Christian Böhme]

Hello all,

Assume that we have

$ uname -srvmpio
Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux

$ openssl version
OpenSSL 1.0.2g  1 Mar 2016

$ printenv SHELL
/bin/bash

$ ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 63575
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 63575
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

$ dd if=/dev/zero of=plaintext.in bs=1024 count=$((1024 * 1024 * 2))
2097152+0 records in
2097152+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 7.7645 s, 277 MB/s

$ echo -n 'uno dos tres cuatro' | openssl cms -encrypt -aes-256-cbc 
-pwri_password fd:0 -in plaintext.in -binary -out ciphertext.der -outform DER

$ ls -lAF
total 4054804
-rw-rw-r-- 1 ubuntu ubuntu 2004623580 Jul 28 20:09 ciphertext.der
-rw-rw-r-- 1 ubuntu ubuntu 2147483648 Jul 28 19:55 plaintext.in

then we get

$ openssl asn1parse -in ciphertext.der -inform DER -i
14050799902:error:07064041:memory buffer routines:BUF_MEM_grow:malloc 
failure:buffer.c:113:

or

$ echo -n 'uno dos tres cuatro' | openssl cms -decrypt -pwri_password fd:0 -in 
ciphertext.der -inform DER -out plaintext.out
Error reading S/MIME message
139871963694744:error:07069041:memory buffer routines:BUF_MEM_grow_clean:malloc 
failure:buffer.c:150:
139871963694744:error:0D06B041:asn1 encoding routines:ASN1_D2I_READ_BIO:malloc 
failure:a_d2i_fp.c:239:

It would appear that both commands fail due to them being unable to
allocate more memory to slurp the rest of the input file's contents into.
Is this intentional behaviour?

Both commands work when the plaintext file is half the size, i.e. 1 GiB, BTW.


Thanks,
Christian

-- 
*Christian Böhme*

Developer System Integration

CLOUD

*CLOUD & HEAT Technologies GmbH*
Königsbrücker Str. 96 (Halle 15) | 01099 Dresden
Tel: +49 351 479 3670 - 100
Fax: +49 351 479 3670 - 110
E-Mail: christian.boe...@cloudandheat.com 

Web: https://www.cloudandheat.com 

Handelsregister: Amtsgericht Dresden
Registernummer: HRB 30549
USt.-Ident.-Nr.: DE281093504
Geschäftsführer: Nicolas Röhrs




signature.asc
Description: OpenPGP digital signature
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users