Bug#923479: [Pkg-openssl-devel] Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Sebastian Andrzej Siewior]

On 2021-01-20 11:50:05 [+0100], Julien Cristau wrote:
> > Can this be reassigned to openssl in the meantime? I think it's a goal to 
> > have
> > all of Debian built with largefile support, anyways.
> 
> Yes.

meh. Is this stable or sid?

> Cheers,
> Julien

Sebastian

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Julien Cristau]

Control: reassign -1 src:openssl
Control: retitle -1 openssl: please enable LFS on 32-bit archs
Control: tag -1 + lfs
Control: affects -1 + ca-certificates

On Tue, Mar 10, 2020 at 10:37:46AM -0600, Dan Nicholson wrote:
> On Wed, 5 Feb 2020 11:40:32 -0600 Michael Shuler 
> wrote:
> > On 2/5/20 11:12 AM, Dan Nicholson wrote:
> > > 2. Build openssl with LFS as noted above.
> >
> > Noted, thanks. This is why I am starting to believe this is a bug with
> > openssl, not ca-certificates.
> 
> Can this be reassigned to openssl in the meantime? I think it's a goal to have
> all of Debian built with largefile support, anyways.

Yes.

Cheers,
Julien

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Dan Nicholson]

On Wed, 5 Feb 2020 11:40:32 -0600 Michael Shuler 
wrote:
> On 2/5/20 11:12 AM, Dan Nicholson wrote:
> > 2. Build openssl with LFS as noted above.
>
> Noted, thanks. This is why I am starting to believe this is a bug with
> openssl, not ca-certificates.

Can this be reassigned to openssl in the meantime? I think it's a goal to
have all of Debian built with largefile support, anyways.

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Michael Shuler]

On 2/5/20 11:12 AM, Dan Nicholson wrote:

I recently ran into this same issue and dug into it for a while. The
real problem stems from
https://sourceware.org/bugzilla/show_bug.cgi?id=23960. The issue is
that glibc 2.28 changed readdir to always use getdents64. This causes
problems when you're running a 32 bit guest in QEMU on a 64 bit host.
So, this would also affect running an i386 guest on an x86_64 host,
for example.


Thanks for the details. This appears more to me now that this is a lower 
level issue than the symptomatic behavior we see in processing 
certificates. (ie. this is not a bug with ca-certificates, but with openssl)





So, I think there are a few options on what to do.

1. Change update-ca-certificates back to using c_rehash. Presumably
perl is built with LFS and it's readdir wrapper DTRT.


c_rehash could be removed at any time, so I don't think this is a good 
option.



2. Build openssl with LFS as noted above.


Noted, thanks. This is why I am starting to believe this is a bug with 
openssl, not ca-certificates.



3. Wait for a fix to be hashed out between the glibc, kernel and qemu folks.


..which I suppose is indeed the eventual lower level issue to resolve, 
below openssl.


--
Kind regards,
Michael

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Dan Nicholson]

It posted https://salsa.debian.org/debian/openssl/merge_requests/4 to
build openssl with largefile support.

--
Dan Nicholson  |  +1.206.437.0833  |  Endless

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Dan Nicholson]

I recently ran into this same issue and dug into it for a while. The
real problem stems from
https://sourceware.org/bugzilla/show_bug.cgi?id=23960. The issue is
that glibc 2.28 changed readdir to always use getdents64. This causes
problems when you're running a 32 bit guest in QEMU on a 64 bit host.
So, this would also affect running an i386 guest on an x86_64 host,
for example.

The issue is that rehash calls OPENSSL_DIR_read on the certificate
directory. This is actually just a wrapper around readdir. However, if
you compile openssl (really libcrypto) with large file support, then
glibc actually uses readdir64, which will use a dirent64 structure
that doesn't have the overflow issues that the regular dirent
structure has in this situation.

After building openssl locally with future=+lfs in
DEB_BUILD_MAINT_OPTIONS, I was able to get the right behavior. Here's
a comparison (certs is just a local directory with one of the symlinks
from /etc/ssl/certs copied into it).

Current behavior:

root@talkbox:~# /qemu-arm-static -strace /usr/bin/openssl rehash
certs/ 2>&1 | grep getdents64
20411 getdents64(3,-13637592,32768,-559685376,-13637592,-622472) = 128

With LFS libcrypto:

root@talkbox:~#
LD_LIBRARY_PATH=libssl1.1-patched/usr/lib/arm-linux-gnueabihf
/qemu-arm-static -strace /usr/bin/openssl rehash certs/ 2>&1 | grep
getdents64
20402 getdents64(3,-13637592,32768,0,32,-13637624) = 128
20402 getdents64(3,-13637592,32768,128,32,-13637624) = 0

And just the normal rehash output:

root@talkbox:~# openssl rehash -v certs/
Doing certs/
root@talkbox:~#
LD_LIBRARY_PATH=libssl1.1-patched/usr/lib/arm-linux-gnueabihf/ openssl
rehash -v certs/
Doing certs/
link thawte_Primary_Root_CA.pem -> 2e4eed3c.0

So, I think there are a few options on what to do.

1. Change update-ca-certificates back to using c_rehash. Presumably
perl is built with LFS and it's readdir wrapper DTRT.

2. Build openssl with LFS as noted above.

3. Wait for a fix to be hashed out between the glibc, kernel and qemu folks.

--
Dan Nicholson  |  +1.206.437.0833  |  Endless

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[James Pooton]

> You could try to debug by directly running `sudo openssl rehash -v
> /etc/ssl/certs` on either version of ca-certificates (new one has new
> CAs added, old ones removed, and a couple other bug fixes, but openssl
> behavior should be the same).

Well, this is very odd….  

First with ca-certificates (20170717) in place and running `openssl rehash -v
/etc/ssl/cert` as root.  It’s successful and spits out:

Doing /etc/ssl/certs
rehash: warning: skipping ca-certificates.crt,it does not contain exactly 
one certificate or CRL
link D-TRUST_Root_Class_3_CA_2_EV_2009.pem -> d4dae3dd.0
link Certplus_Root_CA_G2.pem -> 451b5485.0
link Amazon_Root_CA_1.pem -> ce5e74ef.0
…..

Then with ca-certificates (20190110) in place and running `openssl rehash -v
/etc/ssl/cert` as root.  It fails only outputting:

Doing /etc/ssl/certs

Try it several more times, get the same single line output as above.  So I was 
curious that it didn’t even give the warning about ca-certificates.crt at 
least.  So I moved that with `mv ca-certificates.crt ..` which curiously spit 
out:

qemu: Unsupported syscall: 382
qemu: Unsupported syscall: 382

But now running `openssl rehash -v /etc/ssl/cert` as root again was 
successful(!) with:

Doing /etc/ssl/certs
link D-TRUST_Root_Class_3_CA_2_EV_2009.pem -> d4dae3dd.0
link Amazon_Root_CA_1.pem -> ce5e74ef.0
link Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem -> 
3bde41ac.0
link Certum_Trusted_Network_CA_2.pem -> 40193066.0
…..


So then, for fun I take the “suspect" ca-certificates.crt from (20190110) and 
toss it in to replace the one in (20170717) and try running rehash.  It works 
fine ?!

Ok, so then for a final test, I take take a fresh install of ca-certificates 
(20190110) and do the following:

root@e67c226047ba:/# openssl rehash -v /etc/ssl/certs
Doing /etc/ssl/certs

root@e67c226047ba:/# openssl rehash -v /etc/ssl/certs
Doing /etc/ssl/certs

root@e67c226047ba:/# openssl rehash -v /etc/ssl/certs
Doing /etc/ssl/certs

root@e67c226047ba:/# touch /etc/ssl/certs/ca-certificates.crt 

root@e67c226047ba:/# openssl rehash -v /etc/ssl/certs
Doing /etc/ssl/certs
rehash: warning: skipping ca-certificates.crt,it does not contain exactly 
one certificate or CRL
link D-TRUST_Root_Class_3_CA_2_EV_2009.pem -> d4dae3dd.0
link Amazon_Root_CA_1.pem -> ce5e74ef.0
link Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem -> 
3bde41ac.0
link Certum_Trusted_Network_CA_2.pem -> 40193066.0
…..

Seems crazy….  but touching that file allows rehash to work under QEMU.

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Michael Shuler]

On 2/28/19 7:37 PM, James Pooton wrote:
> So installing ca-certificates (20170717) with the latest openssl
> (1.1.1a-1), does produce the hashes in /etc/ssl/certs when doing an ARM
> 32bit build via QEMU.
> 
> One interesting thing is that the 382 syscalls were still present in the
> build, so that may be a red herring:
> 
> ...
>     Preparing to unpack .../ca-certificates_20170717_all.deb ...
>     Unpacking ca-certificates (20170717) ...
>     Setting up ca-certificates (20170717) ...
>     Updating certificates in /etc/ssl/certs...
>     qemu: Unsupported syscall: 382
>     148 added, 0 removed; done.
>     Processing triggers for ca-certificates (20170717) ...
>     Updating certificates in /etc/ssl/certs...
>     qemu: Unsupported syscall: 382
>     0 added, 0 removed; done.
> ...

Interesting, thanks. So c_rehash works fine-ish, which we were pretty
sure of, and the same behavior with the syscall errors.

/usr/bin/c_rehash is perl with a few calls out to openssl within.

You could try to debug by directly running `sudo openssl rehash -v
/etc/ssl/certs` on either version of ca-certificates (new one has new
CAs added, old ones removed, and a couple other bug fixes, but openssl
behavior should be the same).

> The only other thing I noticed (which certainly may not be related) is
> that a a few of the CA cert filename must have some crazy UTF8
> characters that get encoded (“NetLock_Arany”,
> “RKTRUST_Elektronik_Sertifika_Hizmet_Sa”, “k_Sertifika_Hizmet_Sa”).
>  Just seemed odd, and potentially something that could trip things up.

Yep, there are a few CAs with UTF8 labels, so the files are written out
that way. I was included in a conversation maybe 2 years ago with
someone that did some UTF8 filename research through the package
repositories, but there haven't been any issues raised.

An example of the options would be the existing UTF8:
"NetLock Arany (Class Gold) Főtanúsítvány"
or:
"NetLock Arany (Class Gold) F..tan..s..tv..ny"

F..who? :) I think the native UTF8 is a better option. It's probably a
bug if some file widget does not handle UTF8 files correctly.

-- 
Kind regards,
Michael

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[James Pooton]

> 20170717 was the previous version in testing (20180409 introduced the
> `openssl rehash` usage and was kept from testing for an important bug).
> 
> 20170717 used the perl c_rehash utility. If you need it, here it is!
> 
> http://snapshot.debian.org/package/ca-certificates/20170717/#ca-certificates_20170717
>  
> 
> 

Thanks, that site is a keeper. :) 

So installing ca-certificates (20170717) with the latest openssl (1.1.1a-1), 
does produce the hashes in /etc/ssl/certs when doing an ARM 32bit build via 
QEMU.   I used the following dockerfile to make the test image:


FROM arm32v7/debian:buster-slim
ENV DEBIAN_FRONTEND noninteractive

COPY ca-certificates_20170717_all.deb /tmp/ca-certificates_20170717_all.deb

RUN apt-get update && \
apt-get install -y openssl && \
dpkg -i /tmp/ca-certificates_20170717_all.deb && \
apt-get install -y curl 

CMD [ "curl", "-sSL", "https://www.google.com;]


A container from it ran fine, pulling back Google.

One interesting thing is that the 382 syscalls were still present in the build, 
so that may be a red herring:

...
Preparing to unpack .../ca-certificates_20170717_all.deb ...
Unpacking ca-certificates (20170717) ...
Setting up ca-certificates (20170717) ...
Updating certificates in /etc/ssl/certs...
qemu: Unsupported syscall: 382
148 added, 0 removed; done.
Processing triggers for ca-certificates (20170717) ...
Updating certificates in /etc/ssl/certs...
qemu: Unsupported syscall: 382
0 added, 0 removed; done.
...

The only other thing I noticed (which certainly may not be related) is that a a 
few of the CA cert filename must have some crazy UTF8 characters that get 
encoded (“NetLock_Arany”, “RKTRUST_Elektronik_Sertifika_Hizmet_Sa”, 
“k_Sertifika_Hizmet_Sa”).  Just seemed odd, and potentially something that 
could trip things up.

Thanks again...

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Michael Shuler]

On 2/28/19 4:30 PM, James Pooton wrote:
> 
>> What version of the openssl package is installed?
> 
> Currently we’ve got the following versions getting installed:
> 
> openssl: Installed: 1.1.1a-1 Candidate: 1.1.1a-1 Version table: ***
> 1.1.1a-1 500 500 http://deb.debian.org/debian buster/main armhf
> Packages 100 /var/lib/dpkg/status
> 
> ca-certificates: Installed: 20190110 Candidate: 20190110 Version
> table: *** 20190110 500 500 http://deb.debian.org/debian buster/main
> armhf Packages 100 /var/lib/dpkg/status
> 
> 
>> This does sound like a potential issue with `openssl rehash`. Your 
>> workaround looks OK for the moment, but the problem is that the
>> openssl devs would like to remove the deprecated `c_rehash`
>> utility. At this point in the release cycle, it's possible c_rehash
>> may stick around for Buster, but no guarantees from me on that one
>> - that's up to the openssl folks.
> 
> Yep..  I actually read about that in the issue where that was
> changed.  Just wasn’t able to get ‘openssl rehash’ to work myself
> either, so fell back to the deprecated call for now.
> 
> 
>>> qemu: Unsupported syscall: 382
>> 
>> I can't find what syscall 382 should be, it seems to be higher than
>> all the numbers I can find, the higest number is 294.
> 
> I’m with you on that.  It’s odd, but that’s the syscall number I’ve
> seen on Docker for Mac, Windows, and Linux using QEMU.  Linux also
> kicks out some 384 but it didn’t appear related.
> 
>> Can you give combination of ca-certificates / openssl that works /
>> fails?
> 
> I actually was looking to see if I could just apt install the
> previous packages to verify things, but they don’t seem to be options
> in the repo any more according to apt-cache.  But I’ll see if I can’t
> find the previous packages that were being install when things were
> building fine.  I’ll let you know.

20170717 was the previous version in testing (20180409 introduced the
`openssl rehash` usage and was kept from testing for an important bug).

20170717 used the perl c_rehash utility. If you need it, here it is!

http://snapshot.debian.org/package/ca-certificates/20170717/#ca-certificates_20170717

-- 
Kind regards,
Michael

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[James Pooton]

> What version of the openssl package is installed?

Currently we’ve got the following versions getting installed:

openssl:
  Installed: 1.1.1a-1
  Candidate: 1.1.1a-1
  Version table:
 *** 1.1.1a-1 500
500 http://deb.debian.org/debian buster/main armhf Packages
100 /var/lib/dpkg/status

ca-certificates:
  Installed: 20190110
  Candidate: 20190110
  Version table:
 *** 20190110 500
500 http://deb.debian.org/debian buster/main armhf Packages
100 /var/lib/dpkg/status


> This does sound like a potential issue with `openssl rehash`. Your
> workaround looks OK for the moment, but the problem is that the openssl
> devs would like to remove the deprecated `c_rehash` utility. At this
> point in the release cycle, it's possible c_rehash may stick around for
> Buster, but no guarantees from me on that one - that's up to the openssl
> folks.

Yep..  I actually read about that in the issue where that was changed.  Just 
wasn’t able to get ‘openssl rehash’ to work myself either, so fell back to the 
deprecated call for now. 


>>qemu: Unsupported syscall: 382
> 
> I can't find what syscall 382 should be, it seems to be higher
> than all the numbers I can find, the higest number is 294.

I’m with you on that.  It’s odd, but that’s the syscall number I’ve seen on 
Docker for Mac, Windows, and Linux using QEMU.  Linux also kicks out some 384 
but it didn’t appear related. 

> Can you give combination of ca-certificates / openssl that works
> / fails?

I actually was looking to see if I could just apt install the previous packages 
to verify things, but they don’t seem to be options in the repo any more 
according to apt-cache.  But I’ll see if I can’t find the previous packages 
that were being install when things were building fine.  I’ll let you know. 

Thanks! 

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Kurt Roeckx]

On Thu, Feb 28, 2019 at 11:50:52AM -0700, James Pooton wrote:
> qemu: Unsupported syscall: 382

I can't find what syscall 382 should be, it seems to be higher
than all the numbers I can find, the higest number is 294.

Can you give combination of ca-certificates / openssl that works
/ fails?


Kurt

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[Michael Shuler]

What version of the openssl package is installed?

This does sound like a potential issue with `openssl rehash`. Your
workaround looks OK for the moment, but the problem is that the openssl
devs would like to remove the deprecated `c_rehash` utility. At this
point in the release cycle, it's possible c_rehash may stick around for
Buster, but no guarantees from me on that one - that's up to the openssl
folks.

-- 
Kind regards,
Michael

Bug#923479: update-ca-certificates fails to create hashes under 32-bit ARM (qemu)

[James Pooton]

Package: ca-certificates
Version: 20190110

This is a little fringe and likely crosses projects, as it seems to be specific 
to QEMU emulating 32bit ARM, and may involve the “openssl rehash” function from 
openssl. But it did appear as a result of the 20190110 ca-certificates update, 
and affects anything (like CURL) that depend on the CA cert hashes created by 
via update-ca-certificates, so I’ll start by posting here.  

Basically changes in ca-certificates 20190110 seem to have broken our 32 bit 
ARM Buster Docker image builds (which are done with QEMU).  Essentially, 
installing ca-certificates in the image (which triggers 
update-ca-certificates), fails to create the hashes used by programs like CURL, 
resulting in failure of any subsequent CURL calls.  Running the previously used 
c_rehash command (manually) however, does create the hashes.

For example the following fails running under Docker for Mac/Windows (which has 
QEMU baked in) or Docker Linux x86 with QEMU added, but is successful when run 
on 32bit ARM Docker Linux directly:

docker run -it --rm arm32v7/debian:buster  \
  /bin/bash -c \
'export DEBIAN_FRONTEND=noninteractive && \
apt-get update &&  apt-get install -y ca-certificates curl && \
curl -sSL https://www.google.com'

QEMU Docker hosts fail with:

….
Setting up ca-certificates (20190110) ...
Updating certificates in /etc/ssl/certs...
qemu: Unsupported syscall: 382
128 added, 0 removed; done.
Setting up libkrb5support0:armhf (1.17-1) ...
Setting up libk5crypto3:armhf (1.17-1) ...
Setting up libkrb5-3:armhf (1.17-1) ...
Setting up libgssapi-krb5-2:armhf (1.17-1) ...
Setting up libcurl4:armhf (7.64.0-1) ...
Setting up curl (7.64.0-1) ...
Processing triggers for libc-bin (2.28-5) ...
Processing triggers for ca-certificates (20190110) ...
Updating certificates in /etc/ssl/certs...
qemu: Unsupported syscall: 382
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


However, adding the previously used c_rehash function does create the hashes 
and is successful for QEMU Docker hosts:

docker run -it --rm arm32v7/debian:buster  \
  /bin/bash -c \
'export DEBIAN_FRONTEN=noninteractive && \
apt-get update &&  apt-get install -y ca-certificates curl && \
c_rehash &&  \
curl -sSL https://www.google.com'

We’re using that work-around for 32-bit ARM image builds currently, but just a 
heads up that something seems to have changed that affects the hash creation 
under QEMU for ARM 32-bit.  It was working prior to the 20190110 update.

-James