Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Mike Larkin]

On Sun, Apr 16, 2017 at 09:17:44AM +, Paul Chakravarti wrote:
> Hello,
> 
> I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try to
> download a large file using SSL I consistenetly get the following error:
> 
> > SSL read error: read failed: error:06FFF064:digital envelope
> routines:CRYPTO_internal:bad decrypt
> 
> This occasionally (but not always) correlates with the following message in
> the vmd log:
> 
> > vionet queue notify - no space, dropping packet
> 
> Strangely non-SSL and smaller SSL downloads seem to work ok (see below).
> 
> Originally spotted this using installer but can recreate from shell.
> 
> Any ideas?
> 

The diff I just committed should fix this as well as the previously reported
github clone issue. The diff is also already in snaps (as of this morning).

Please update and let me know if you still see this problem.

Thanks for reporting this.

-ml

> # cat /etc/vm.conf                                                           
>                                                        
> vm vm0 {
>   disable
>   memory 512M
>   disk /home/vm/vm0.img
>   kernel /bsd.rd
>   interface { switch uplink }
> }
> 
> switch uplink {
>   interface bridge0
>   add vether0
> }
> 
> # vmctl start vm0 
> vmctl: started vm 11 successfully, tty /dev/ttyp6
> # vmctl status
>ID   PID VCPUS  MAXMEM  CURMEM TTYOWNER NAME
>11 85026 1512M   97.3M   ttyp6 root vm0
> # cu -l /dev/ttyp6
> Connected to /dev/ttyp6 (speed 9600)
> 
> (I)nstall, (U)pgrade, (A)utoinstall or (S)hell? s
> # dhclient vio0                                                             
>  
> DHCPDISCOVER on vio0 - interval 1
> DHCPOFFER from 10.0.0.1 (fe:e1:ba:d3:55:34)
> DHCPREQUEST on vio0 to 255.255.255.255
> DHCPACK from 10.0.0.1 (fe:e1:ba:d3:55:34)
> bound to 10.0.0.105 -- renewal in 21600 seconds.
> #
> # ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
> 440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
> # ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/SHA256 | fgrep
> '(bsd)'
> SHA256 (bsd) =
> 440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
> # ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/base61.tgz |
> sha256 
> 5c467ea369b5632d3b057283857d1998fb3dcd26179365291f16c70785a65282
> # ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/SHA256 | fgrep
> '(base61.tgz)'
> SHA256 (base61.tgz) =
> 5c467ea369b5632d3b057283857d1998fb3dcd26179365291f16c70785a65282
> #
> # ftp -Vo- https://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
> ftp: SSL read error: read failed: error:06FFF064:digital envelope
> routines:CRYPTO_internal:bad decrypt
> 27ad92f2aaf0279dd125ed54d0b7fbf330a3ecbe2e919b4d2d0ed1d07dccc087
> # ftp -Vo- https://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/base61.tgz |
> sha256
> ftp: SSL read error: read failed: error:06FFF064:digital envelope
> routines:CRYPTO_internal:bad decrypt
> d79f6fd884a839d1fc62dc1b5d40de21f97fd5a50b28319a7b25dd8cd82da887
> 
> [On host]
> 
> # top -d1 all
> load averages:  1.14,  1.16,  1.16x230 10:06:31
> 68 processes: 67 idle, 1 on processor  up 2 days, 11:19
> CPU0 states:  0.2% user,  0.0% nice,  0.5% system,  0.2% interrupt, 99.1%
> idle
> CPU1 states:  4.6% user,  0.0% nice,  8.3% system,  0.0% interrupt, 87.2%
> idle
> CPU2 states:  1.2% user,  0.0% nice,  2.2% system,  0.0% interrupt, 96.6%
> idle
> CPU3 states:  0.8% user,  0.0% nice,  1.1% system,  0.0% interrupt, 98.1%
> idle
> Memory: Real: 470M/1376M act/tot Free: 6261M Cache: 652M Swap: 0K/3562M
> 
>   PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU COMMAND
> 85026 _vmd  280  514M   14M idle  thrslee   1:19 13.53% vmd
> 55104 paulc  20  399M  301M sleep/3   poll  4:09  7.86% firefox
>  1136 paulc  20 1180K 9156K sleep/3   poll  0:23  0.05% i3bar
> 91148 paulc  20   14M   50M sleep/2   select0:24  0.00% Xorg
> 48836 paulc 100  752K 1988K sleep/2   nanosle   0:05  0.00% i3status
> 24227 paulc  20 1032K 2820K sleep/2   select0:04  0.00% sshd
> 66378 paulc  20 1564K   10M idle  poll  0:02  0.00% i3
> 67867 paulc  20 5032K   13M idle  select0:02  0.00% urxvt
> 22018 _syslogd   20  904K 1544K sleep/2   kqread0:02  0.00% syslogd
> 1 root  100  380K  416K idle  wait  0:01  0.00% init
> 43749 _pflogd40  668K  428K sleep/1   bpf   0:01  0.00% pflogd
> 27702 _ntp   2  -20  888K 2344K sleep/2   poll  0:01  0.00% ntpd
> 49491 paulc  20 4972K   13M idle  select0:01  0.00% urxvt
> 76489 _vmd   20 1176K 1672K idle  kqread0:00  0.00% vmd
>  6009 root   20  620K  528K idle  poll  0:00  0.00% dhclient
> 39926 paulc  20 4912K   12M idle  select0:00  0.00% urxvt
>  3807 paulc 180  604K  732K idle  pause 0:00  0.00% ksh
> 76917 root   20  220K  

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Tinker]

On 2017-04-18 21:16, Stuart Henderson wrote:

On 2017-04-18, Jan Lambertz  wrote:
I had similar issues, mostly with crypto things in vmd. Can this 
happen

because we get out of entropy?


Run "ftp -o- https://www.gov.uk/ > /dev/random" from time to time,
you won't run out of entropy ever :)

(OpenBSD doesn't do the "run out of entropy" thing anyway).


I have no evidence yet, but i will test things tomorrow.


Mike knows about the problem and has an idea where to look. I think, at
this point, if he needs more information he'll let us know.


Is there even any indication that this is not just either due to broken 
CPU/RAM, or the TCP networking stack letting through broken packets due 
to collissions in its 16-bit checksums (i.e. a packet breaks but still 
matches the checksum and hence is received by httpd+libssl)?

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Paul Chakravarti]

>On 2017-04-17, David Coppa  wrote:
>> On Sun, Apr 16, 2017 at 11:17 AM, Paul Chakravarti  wrote:
>>> Hello,
>>>
>>> I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try to
>>> download a large file using SSL I consistenetly get the following error:
>>>
 SSL read error: read failed: error:06FFF064:digital envelope
>>> routines:CRYPTO_internal:bad decrypt
>>>
>>> This occasionally (but not always) correlates with the following message in
>>> the vmd log:
>>>
 vionet queue notify - no space, dropping packet
>>>
>>> Strangely non-SSL and smaller SSL downloads seem to work ok (see below).
>>>
>>> Originally spotted this using installer but can recreate from shell.
>>>
>>> Any ideas?
>>
>> See http://marc.info/?l=openbsd-misc=148858752003261
>>
>> It's a known problem.
>
>I've seen corruption with non-SSL network transfers too. It's just more
>obvious with SSL because in that case the session gets killed, whereas
>otherwise the corrupt input is silently accepsilently accepted.
>

It does seem more prevalent with SSL transfers - the SHA256s of the files 
transferred vis http are correct (over several transfers) while there is always 
an always an error on the https transfers from the same site.

Interestingly the problem only seems to come up on 'fast' connections - 
possibly something CPU related (cpu load exacerbated by SSL?). I'm still not 
sure why the TCP layer doesn't sort out the dropped packets though.

# ftp -Vo- https://ftp.openbsd.org/pub/OpenBSD/6.1/amd64/bsd | sha256  
440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
# 
# 
# ftp -Vo- https://mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
ftp: SSL read error: read failed: error:06FFF064:digital envelope 
routines:CRYPTO_internal:bad decrypt

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Stuart Henderson]

On 2017-04-18, Jan Lambertz  wrote:
> I had similar issues, mostly with crypto things in vmd. Can this happen
> because we get out of entropy?

Run "ftp -o- https://www.gov.uk/ > /dev/random" from time to time,
you won't run out of entropy ever :)

(OpenBSD doesn't do the "run out of entropy" thing anyway).

> I have no evidence yet, but i will test things tomorrow.

Mike knows about the problem and has an idea where to look. I think, at
this point, if he needs more information he'll let us know.

[vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Jan Lambertz]

I had similar issues, mostly with crypto things in vmd. Can this happen
because we get out of entropy? I have no evidence yet, but i will test
things tomorrow.

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Mike Larkin]

On Mon, Apr 17, 2017 at 10:29:31AM +, Paul Chakravarti wrote:
> >On 2017-04-17, David Coppa  wrote:
> >> On Sun, Apr 16, 2017 at 11:17 AM, Paul Chakravarti  
> >> wrote:
> >>> Hello,
> >>>
> >>> I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try 
> >>> to
> >>> download a large file using SSL I consistenetly get the following error:
> >>>
>  SSL read error: read failed: error:06FFF064:digital envelope
> >>> routines:CRYPTO_internal:bad decrypt
> >>>
> >>> This occasionally (but not always) correlates with the following message 
> >>> in
> >>> the vmd log:
> >>>
>  vionet queue notify - no space, dropping packet
> >>>
> >>> Strangely non-SSL and smaller SSL downloads seem to work ok (see below).
> >>>
> >>> Originally spotted this using installer but can recreate from shell.
> >>>
> >>> Any ideas?
> >>
> >> See http://marc.info/?l=openbsd-misc=148858752003261
> >>
> >> It's a known problem.
> >
> >I've seen corruption with non-SSL network transfers too. It's just more
> >obvious with SSL because in that case the session gets killed, whereas
> >otherwise the corrupt input is silently accepsilently accepted.
> >
> 
> It does seem more prevalent with SSL transfers - the SHA256s of the files 
> transferred vis http are correct (over several transfers) while there is 
> always an always an error on the https transfers from the same site.
> 
> Interestingly the problem only seems to come up on 'fast' connections - 
> possibly something CPU related (cpu load exacerbated by SSL?). I'm still not 
> sure why the TCP layer doesn't sort out the dropped packets though.
> 
> # ftp -Vo- https://ftp.openbsd.org/pub/OpenBSD/6.1/amd64/bsd | sha256 
>  
> 440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
> # 
> # 
> # ftp -Vo- https://mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
> ftp: SSL read error: read failed: error:06FFF064:digital envelope 
> routines:CRYPTO_internal:bad decrypt

I think I know what's going on, I just haven't had time to sort through it yet.
I don't think it's related to the network stack, FWIW.

-ml

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Paul Chakravarti]

>On 2017-04-17, David Coppa  wrote:
>> On Sun, Apr 16, 2017 at 11:17 AM, Paul Chakravarti  wrote:
>>> Hello,
>>>
>>> I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try to
>>> download a large file using SSL I consistenetly get the following error:
>>>
 SSL read error: read failed: error:06FFF064:digital envelope
>>> routines:CRYPTO_internal:bad decrypt
>>>
>>> This occasionally (but not always) correlates with the following message in
>>> the vmd log:
>>>
 vionet queue notify - no space, dropping packet
>>>
>>> Strangely non-SSL and smaller SSL downloads seem to work ok (see below).
>>>
>>> Originally spotted this using installer but can recreate from shell.
>>>
>>> Any ideas?
>>
>> See http://marc.info/?l=openbsd-misc=148858752003261
>>
>> It's a known problem.
>
>I've seen corruption with non-SSL network transfers too. It's just more
>obvious with SSL because in that case the session gets killed, whereas
>otherwise the corrupt input is silently accepsilently accepted.
>

It does seem more prevalent with SSL transfers - the SHA256s of the files 
transferred vis http are correct (over several transfers) while there is always 
an always an error on the https transfers from the same site.

Interestingly the problem only seems to come up on 'fast' connections - 
possibly something CPU related (cpu load exacerbated by SSL?). I'm still not 
sure why the TCP layer doesn't sort out the dropped packets though.

# ftp -Vo- https://ftp.openbsd.org/pub/OpenBSD/6.1/amd64/bsd | sha256  
440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
# 
# 
# ftp -Vo- https://mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
ftp: SSL read error: read failed: error:06FFF064:digital envelope 
routines:CRYPTO_internal:bad decrypt

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Stuart Henderson]

On 2017-04-17, David Coppa  wrote:
> On Sun, Apr 16, 2017 at 11:17 AM, Paul Chakravarti  wrote:
>> Hello,
>>
>> I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try to
>> download a large file using SSL I consistenetly get the following error:
>>
>>> SSL read error: read failed: error:06FFF064:digital envelope
>> routines:CRYPTO_internal:bad decrypt
>>
>> This occasionally (but not always) correlates with the following message in
>> the vmd log:
>>
>>> vionet queue notify - no space, dropping packet
>>
>> Strangely non-SSL and smaller SSL downloads seem to work ok (see below).
>>
>> Originally spotted this using installer but can recreate from shell.
>>
>> Any ideas?
>
> See http://marc.info/?l=openbsd-misc=148858752003261
>
> It's a known problem.

I've seen corruption with non-SSL network transfers too. It's just more
obvious with SSL because in that case the session gets killed, whereas
otherwise the corrupt input is silently accepted.

Re: [vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[David Coppa]

On Sun, Apr 16, 2017 at 11:17 AM, Paul Chakravarti  wrote:
> Hello,
>
> I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try to
> download a large file using SSL I consistenetly get the following error:
>
>> SSL read error: read failed: error:06FFF064:digital envelope
> routines:CRYPTO_internal:bad decrypt
>
> This occasionally (but not always) correlates with the following message in
> the vmd log:
>
>> vionet queue notify - no space, dropping packet
>
> Strangely non-SSL and smaller SSL downloads seem to work ok (see below).
>
> Originally spotted this using installer but can recreate from shell.
>
> Any ideas?

See http://marc.info/?l=openbsd-misc=148858752003261

It's a known problem.

Ciao!
David

[vmm] SSL read error: read failed: error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt

[Paul Chakravarti]

Hello,

I am trying out vmm on 6.1 and can setup/boot vm etc. however when I try to
download a large file using SSL I consistenetly get the following error:

> SSL read error: read failed: error:06FFF064:digital envelope
routines:CRYPTO_internal:bad decrypt

This occasionally (but not always) correlates with the following message in
the vmd log:

> vionet queue notify - no space, dropping packet

Strangely non-SSL and smaller SSL downloads seem to work ok (see below).

Originally spotted this using installer but can recreate from shell.

Any ideas?

# cat /etc/vm.conf                                                           
                                                       
vm vm0 {
  disable
  memory 512M
  disk /home/vm/vm0.img
  kernel /bsd.rd
  interface { switch uplink }
}

switch uplink {
  interface bridge0
  add vether0
}

# vmctl start vm0 
vmctl: started vm 11 successfully, tty /dev/ttyp6
# vmctl status
   ID   PID VCPUS  MAXMEM  CURMEM TTYOWNER NAME
   11 85026 1512M   97.3M   ttyp6 root vm0
# cu -l /dev/ttyp6
Connected to /dev/ttyp6 (speed 9600)

(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? s
# dhclient vio0                                                             
 
DHCPDISCOVER on vio0 - interval 1
DHCPOFFER from 10.0.0.1 (fe:e1:ba:d3:55:34)
DHCPREQUEST on vio0 to 255.255.255.255
DHCPACK from 10.0.0.1 (fe:e1:ba:d3:55:34)
bound to 10.0.0.105 -- renewal in 21600 seconds.
#
# ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
# ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/SHA256 | fgrep
'(bsd)'
SHA256 (bsd) =
440311305f27f0efcfcc88116299a21cb3f890fb91ee611c2a79cc9163e8fceb
# ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/base61.tgz |
sha256 
5c467ea369b5632d3b057283857d1998fb3dcd26179365291f16c70785a65282
# ftp -Vo- http://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/SHA256 | fgrep
'(base61.tgz)'
SHA256 (base61.tgz) =
5c467ea369b5632d3b057283857d1998fb3dcd26179365291f16c70785a65282
#
# ftp -Vo- https://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/bsd | sha256
ftp: SSL read error: read failed: error:06FFF064:digital envelope
routines:CRYPTO_internal:bad decrypt
27ad92f2aaf0279dd125ed54d0b7fbf330a3ecbe2e919b4d2d0ed1d07dccc087
# ftp -Vo- https://www.mirrorservice.org/pub/OpenBSD/6.1/amd64/base61.tgz |
sha256
ftp: SSL read error: read failed: error:06FFF064:digital envelope
routines:CRYPTO_internal:bad decrypt
d79f6fd884a839d1fc62dc1b5d40de21f97fd5a50b28319a7b25dd8cd82da887

[On host]

# top -d1 all
load averages:  1.14,  1.16,  1.16x230 10:06:31
68 processes: 67 idle, 1 on processor  up 2 days, 11:19
CPU0 states:  0.2% user,  0.0% nice,  0.5% system,  0.2% interrupt, 99.1%
idle
CPU1 states:  4.6% user,  0.0% nice,  8.3% system,  0.0% interrupt, 87.2%
idle
CPU2 states:  1.2% user,  0.0% nice,  2.2% system,  0.0% interrupt, 96.6%
idle
CPU3 states:  0.8% user,  0.0% nice,  1.1% system,  0.0% interrupt, 98.1%
idle
Memory: Real: 470M/1376M act/tot Free: 6261M Cache: 652M Swap: 0K/3562M

  PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU COMMAND
85026 _vmd  280  514M   14M idle  thrslee   1:19 13.53% vmd
55104 paulc  20  399M  301M sleep/3   poll  4:09  7.86% firefox
 1136 paulc  20 1180K 9156K sleep/3   poll  0:23  0.05% i3bar
91148 paulc  20   14M   50M sleep/2   select0:24  0.00% Xorg
48836 paulc 100  752K 1988K sleep/2   nanosle   0:05  0.00% i3status
24227 paulc  20 1032K 2820K sleep/2   select0:04  0.00% sshd
66378 paulc  20 1564K   10M idle  poll  0:02  0.00% i3
67867 paulc  20 5032K   13M idle  select0:02  0.00% urxvt
22018 _syslogd   20  904K 1544K sleep/2   kqread0:02  0.00% syslogd
1 root  100  380K  416K idle  wait  0:01  0.00% init
43749 _pflogd40  668K  428K sleep/1   bpf   0:01  0.00% pflogd
27702 _ntp   2  -20  888K 2344K sleep/2   poll  0:01  0.00% ntpd
49491 paulc  20 4972K   13M idle  select0:01  0.00% urxvt
76489 _vmd   20 1176K 1672K idle  kqread0:00  0.00% vmd
 6009 root   20  620K  528K idle  poll  0:00  0.00% dhclient
39926 paulc  20 4912K   12M idle  select0:00  0.00% urxvt
 3807 paulc 180  604K  732K idle  pause 0:00  0.00% ksh
76917 root   20  220K  780K sleep/1   kqread0:00  0.00% apmd
33176 _smtpd 20 1280K 3444K idle  kqread0:00  0.00% smtpd
51016 root   30  736K  796K idle  ttyin 0:00  0.00% ksh
86730 _smtpd 20 1368K 3492K idle  kqread0:00  0.00% smtpd
 5062 root   20 1556K 2128K idle  kqread0:00  0.00% smtpd
59926 root   20  868K 1368K idle  select0:00  0.00% sshd
46705 root  180  668K  756K idle  pause 0:00  0.00% ksh
92059 _smtpd 20 1420K 3616K idle  kqread