Just this week, I saw a similar error from shellinabox. I was replacing
the default self-signed certificates with letsencrypt certificates.
It appears, from your messages, that shellinabox is unable to create or
unable to read the self-signed certificates.
The first thing to do is look at the /var/lib/shellinabox directory.
What is the ownership? Is there a certificate.pem? For example:
# ls -ld /var/lib/shellinabox/
drwxr-xr-x 2 shellinabox shellinabox 4096 Sep 25 20:00 /var/lib/shellinabox
# ls -l /var/lib/shellinabox/certificate.pem
-rw------- 1 shellinabox shellinabox 2811 Sep 6 18:04
/var/lib/shellinabox/certificate.pem
The most likely culprit for your messages is that shellinabox cannot
write or read the certificate. Let us know what you find.