Re: [strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Noel Kuntze 

Hi,

Have you tried ipsec stroke rereadsecrets? (Btw, better switch to swanctl)

Kind regards
Noel

Am 06.10.21 um 16:54 schrieb Philip Veale:

So about a week about, one of the CAs in the chain Let'sEncrypt use (DST Root 
CA X3) expired. This shouldn't have been a problem for most clients, as it was 
cross signed with a CA that had not expired (ISRG Root X1) which most modern 
clients and devices should trust, though some older ones may not which was 
(AIUI) why they kept the DST Root CA X3 in there too.

I use a Let'sEncrypt certificate with StrongSWAN and for years it has mostly 
just worked (mostly being, certbot very usefully renewed the certificate 
dutifully every so many days, but it didn't get ipsec to re-read this so I'd 
have to manually punt it about 4 times a year. I could have fixed that but 
never bothered, anyway, I digress...)

Recently I found the StrongSWAN client on my Android 10 phone wouldn't connect, 
and at first I thought I just needed to punt it again to re-read the cert, but 
then I realised it was reporting it had expired even after I'd manually run 
certbot. And thus I discovered that something that the Android Client is 
looking at doesn't trust the cert the server was offering as one CA was expired 
even though another was still valid. This I don't entirely understand as the 
Let'sEncrypt people seem pretty adamant it should all still work.

I don't entirely understand why new certificates issued today are still being 
signed with a cert that expired last week, though.

Anyway, moving on. I figured the best way to solve this was to request a cert 
from LE that was only signed by ISRG Root X1 and NOT by DST Root CA X3 as well, 
which is not the default behaviour but can be achieved by passing a switch to 
certbot to ask it to do it that way.

My system was running Debian Switch and I wanted to continue to use certbot, 
and I didn't want to pollute the system with certbot's suggested tool snap, 
which imports a little bit of Ubuntu stuff.

Looking at repositories' versions of the certbot package, it was clear I had to 
upgrade from Stretch to Bullseye, via Buster in between.

So, done that, everything all up to date, got new cert, all should be well now, 
except it fails, client and server both reporting basically the same thing;

Oct  6 15:20:30 VPN-Server charon: 10[IKE] no private key found for 
'vpn.my-hostname.net '
Oct  6 15:20:30 VPN-Server charon: 10[ENC] generating IKE_AUTH response 1 [ 
N(AUTH_FAILED) ]

I've not modified ipsec.secrets, it's still intact and contains the correct 
reference to the privkey.pem file, which pans out as running the below command 
brings up the expected result;

# ipsec listcerts

List of X.509 End Entity Certificates

   subject:  "CN=vpn.my-hostname.net "
   issuer:   "C=US, O=Let's Encrypt, CN=R3"
   validity:  not before Oct 06 14:03:36 2021, ok
              not after  Jan 04 13:03:35 2022, ok (expires in 89 days)
   serial:    [redacted]
   altNames: vpn.my-hostname.net 
   flags:     serverAuth clientAuth
   OCSP URIs: http://r3.o.lencr.org 
   certificatePolicies:
              2.23.140.1.2.1
              1.3.6.1.4.1.44947.1.1.1
              CPS: http://cps.letsencrypt.org 
[other lines trimmed]


all looks correct to me. All certs present and correct by my reckoning, config 
unaltered from previous working state before certificate trouble started within 
the last week.

I say unaltered, I've obviously gone up TWO Debian release versions which might 
have some bearing on it, but I can't see how and the logs and pointing the 
finger at a certificate issue which seems more likely.

Only other thing I can thing of is at some point in the past I had manually imported a 
cacert from Let'sEncrypt onto the system such that "ipsec listcacerts" produced 
some output, they are gone now so this produces nothing.

Not sure how they'd be needed, though


Can anyone spot or think of what I've missed, or has anyone else been through 
similar recently due to what's happened with Let'sEncrypt ?


OpenPGP_signature
Description: OpenPGP digital signature

Re: [strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Simon Deziel 

On 2021-10-06 2:27 p.m., Philip Veale wrote:

On Wed, 6 Oct 2021 at 17:24, Simon Deziel  wrote:


On 2021-10-06 12:22 p.m., Simon Deziel wrote:

On 2021-10-06 12:08 p.m., Philip Veale wrote:

Oct  6 16:43:55 VPN-Server charon: 00[LIB]   opening
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed: Permission
denied

Debian Stretch didn't have AppArmor but it's been enabled by default in
Debian since Buster. So yeah, the dist-upgrade kinda broke things.

Thanks to Simon Deziel in this old thread from years ago;
https://lists.strongswan.org/pipermail/users/2017-February/010537.html


I've not quite yet figured out how I want to fix it (there are a few
options) but at least I know why it does not work.



At first glance, I'd add "#include " to charon's
profile. Would you mind testing this for me (as root):


Oops, here's the corrected version:

cat < EOF >> /etc/apparmor.d/local/usr.lib.ipsec.charon
#include 
EOF
apparmor_parser -rTW /etc/apparmor.d/usr.lib.ipsec.charon
systemctl restart strongswan-starter




I added it using vim instead but Yes, that's worked perfectly, thank you.
System is now fully operational :)


Thanks for testing and reporting back, I'll submit a PR to Debian soon.

Regards,
Simon

Re: [strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Philip Veale 
On Wed, 6 Oct 2021 at 17:24, Simon Deziel  wrote:

> On 2021-10-06 12:22 p.m., Simon Deziel wrote:
> > On 2021-10-06 12:08 p.m., Philip Veale wrote:
> >> Oct  6 16:43:55 VPN-Server charon: 00[LIB]   opening
> >> '/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed: Permission
> >> denied
> >>
> >> Debian Stretch didn't have AppArmor but it's been enabled by default in
> >> Debian since Buster. So yeah, the dist-upgrade kinda broke things.
> >>
> >> Thanks to Simon Deziel in this old thread from years ago;
> >> https://lists.strongswan.org/pipermail/users/2017-February/010537.html
> >>
> >>
> >> I've not quite yet figured out how I want to fix it (there are a few
> >> options) but at least I know why it does not work.
> >
> >
> > At first glance, I'd add "#include " to charon's
> > profile. Would you mind testing this for me (as root):
>
> Oops, here's the corrected version:
>
> cat < EOF >> /etc/apparmor.d/local/usr.lib.ipsec.charon
> #include 
> EOF
> apparmor_parser -rTW /etc/apparmor.d/usr.lib.ipsec.charon
> systemctl restart strongswan-starter
>


I added it using vim instead but Yes, that's worked perfectly, thank you.
System is now fully operational :)

Re: [strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Simon Deziel 

On 2021-10-06 12:22 p.m., Simon Deziel wrote:

On 2021-10-06 12:08 p.m., Philip Veale wrote:

I hadn't tried that, but tried, didn't change anything. I noticed things
specifically related to StrongSWAN aren't working since the update to
Bullseye and swanctl is not a recognised command. StrongSWAN is installed
via apt, version 5.9.1-1

swanctl doesn't exist as a command and there is no service called
strongswan anymore. I'm not sure how weird that is.


swanctl lives in a different package. The strongswan unit got renamed to 
strongswan-starter.



Just been trawling more logs and spotted something else which should be a
massive clue;

Oct  6 16:43:55 VPN-Server charon: 00[CFG] loading secrets from
'/etc/ipsec.secrets'
Oct  6 16:43:55 VPN-Server charon: 00[LIB]   opening
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed: Permission
denied
Oct  6 16:43:55 VPN-Server charon: 00[LIB] building CRED_PRIVATE_KEY - 
RSA

failed, tried 11 builders
Oct  6 16:43:55 VPN-Server charon: 00[CFG]   loading private key from
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed


So yeah it looks like it's a simple permissions issue, I'm guessing the
dist upgrade has changed the user the service runs as and that uid 
doesn't

have read access to the privkey. I should have thought of that. For some
reason I thought it just ran as root.

Oh..so no, it does run as root, but It's AppArmor, interfering with 
Charon

apparently - the PEM files are created by certbot with symlinks (from
'live' to 'archive') as it rotates through and creates new ones, keeping
the old, the newest versions are always symlinked.

Debian Stretch didn't have AppArmor but it's been enabled by default in
Debian since Buster. So yeah, the dist-upgrade kinda broke things.

Thanks to Simon Deziel in this old thread from years ago;
https://lists.strongswan.org/pipermail/users/2017-February/010537.html


I've not quite yet figured out how I want to fix it (there are a few
options) but at least I know why it does not work.



At first glance, I'd add "#include " to charon's 
profile. Would you mind testing this for me (as root):


Oops, here's the corrected version:

cat < EOF >> /etc/apparmor.d/local/usr.lib.ipsec.charon
#include 
EOF
apparmor_parser -rTW /etc/apparmor.d/usr.lib.ipsec.charon
systemctl restart strongswan-starter

Simon

Re: [strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Simon Deziel 

On 2021-10-06 12:08 p.m., Philip Veale wrote:

I hadn't tried that, but tried, didn't change anything. I noticed things
specifically related to StrongSWAN aren't working since the update to
Bullseye and swanctl is not a recognised command. StrongSWAN is installed
via apt, version 5.9.1-1

swanctl doesn't exist as a command and there is no service called
strongswan anymore. I'm not sure how weird that is.


swanctl lives in a different package. The strongswan unit got renamed to 
strongswan-starter.



Just been trawling more logs and spotted something else which should be a
massive clue;

Oct  6 16:43:55 VPN-Server charon: 00[CFG] loading secrets from
'/etc/ipsec.secrets'
Oct  6 16:43:55 VPN-Server charon: 00[LIB]   opening
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed: Permission
denied
Oct  6 16:43:55 VPN-Server charon: 00[LIB] building CRED_PRIVATE_KEY - RSA
failed, tried 11 builders
Oct  6 16:43:55 VPN-Server charon: 00[CFG]   loading private key from
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed


So yeah it looks like it's a simple permissions issue, I'm guessing the
dist upgrade has changed the user the service runs as and that uid doesn't
have read access to the privkey. I should have thought of that. For some
reason I thought it just ran as root.

Oh..so no, it does run as root, but It's AppArmor, interfering with Charon
apparently - the PEM files are created by certbot with symlinks (from
'live' to 'archive') as it rotates through and creates new ones, keeping
the old, the newest versions are always symlinked.

Debian Stretch didn't have AppArmor but it's been enabled by default in
Debian since Buster. So yeah, the dist-upgrade kinda broke things.

Thanks to Simon Deziel in this old thread from years ago;
https://lists.strongswan.org/pipermail/users/2017-February/010537.html


I've not quite yet figured out how I want to fix it (there are a few
options) but at least I know why it does not work.



At first glance, I'd add "#include " to charon's 
profile. Would you mind testing this for me (as root):


cat < EOF >> /etc/apparmor.d/local/usr.lib.ipsec.charon
echo "#include "
EOF
apparmor_parser -rTW /etc/apparmor.d/usr.lib.ipsec.charon
systemctl restart strongswan-starter


And report back, please?

Simon

Re: [strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Philip Veale 
I hadn't tried that, but tried, didn't change anything. I noticed things
specifically related to StrongSWAN aren't working since the update to
Bullseye and swanctl is not a recognised command. StrongSWAN is installed
via apt, version 5.9.1-1

swanctl doesn't exist as a command and there is no service called
strongswan anymore. I'm not sure how weird that is.


Just been trawling more logs and spotted something else which should be a
massive clue;

Oct  6 16:43:55 VPN-Server charon: 00[CFG] loading secrets from
'/etc/ipsec.secrets'
Oct  6 16:43:55 VPN-Server charon: 00[LIB]   opening
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed: Permission
denied
Oct  6 16:43:55 VPN-Server charon: 00[LIB] building CRED_PRIVATE_KEY - RSA
failed, tried 11 builders
Oct  6 16:43:55 VPN-Server charon: 00[CFG]   loading private key from
'/etc/letsencrypt/live/vpn.my-hostname/privkey.pem' failed


So yeah it looks like it's a simple permissions issue, I'm guessing the
dist upgrade has changed the user the service runs as and that uid doesn't
have read access to the privkey. I should have thought of that. For some
reason I thought it just ran as root.

Oh..so no, it does run as root, but It's AppArmor, interfering with Charon
apparently - the PEM files are created by certbot with symlinks (from
'live' to 'archive') as it rotates through and creates new ones, keeping
the old, the newest versions are always symlinked.

Debian Stretch didn't have AppArmor but it's been enabled by default in
Debian since Buster. So yeah, the dist-upgrade kinda broke things.

Thanks to Simon Deziel in this old thread from years ago;
https://lists.strongswan.org/pipermail/users/2017-February/010537.html


I've not quite yet figured out how I want to fix it (there are a few
options) but at least I know why it does not work.



On Wed, 6 Oct 2021 at 16:02, Noel Kuntze
 wrote:

> Hi,
>
>>
> Have you tried ipsec stroke rereadsecrets? (Btw, better switch to swanctl)
>
>>
> Kind regards
>
>> Noel
>
>>
> Am 06.10.21 um 16:54 schrieb Philip Veale:
>
>> > So about a week about, one of the CAs in the chain Let'sEncrypt use
> (DST Root CA X3) expired. This shouldn't have been a problem for most
> clients, as it was cross signed with a CA that had not expired (ISRG Root
> X1) which most modern clients and devices should trust, though some older
> ones may not which was (AIUI) why they kept the DST Root CA X3 in there too.
>
>> >
>
>> > I use a Let'sEncrypt certificate with StrongSWAN and for years it has
> mostly just worked (mostly being, certbot very usefully renewed the
> certificate dutifully every so many days, but it didn't get ipsec to
> re-read this so I'd have to manually punt it about 4 times a year. I could
> have fixed that but never bothered, anyway, I digress...)
>
>> >
>
>> > Recently I found the StrongSWAN client on my Android 10 phone wouldn't
> connect, and at first I thought I just needed to punt it again to re-read
> the cert, but then I realised it was reporting it had expired even after
> I'd manually run certbot. And thus I discovered that something that the
> Android Client is looking at doesn't trust the cert the server was offering
> as one CA was expired even though another was still valid. This I don't
> entirely understand as the Let'sEncrypt people seem pretty adamant it
> should all still work.
>
>> >
>
>> > I don't entirely understand why new certificates issued today are still
> being signed with a cert that expired last week, though.
>
>> >
>
>> > Anyway, moving on. I figured the best way to solve this was to request
> a cert from LE that was only signed by ISRG Root X1 and NOT by DST Root CA
> X3 as well, which is not the default behaviour but can be achieved by
> passing a switch to certbot to ask it to do it that way.
>
>> >
>
>> > My system was running Debian Switch and I wanted to continue to use
> certbot, and I didn't want to pollute the system with certbot's suggested
> tool snap, which imports a little bit of Ubuntu stuff.
>
>> >
>
>> > Looking at repositories' versions of the certbot package, it was clear
> I had to upgrade from Stretch to Bullseye, via Buster in between.
>
>> >
>
>> > So, done that, everything all up to date, got new cert, all should be
> well now, except it fails, client and server both reporting basically the
> same thing;
>
>> >
>
>> > Oct  6 15:20:30 VPN-Server charon: 10[IKE] no private key found for '
> vpn.my-hostname.net'
>
>> > Oct  6 15:20:30 VPN-Server charon: 10[ENC] generating IKE_AUTH response
> 1 [ N(AUTH_FAILED) ]
>
>> >
>
>> > I've not modified ipsec.secrets, it's still intact and contains the
> correct reference to the privkey.pem file, which pans out as running the
> below command brings up the expected result;
>
>> >
>
>> > # ipsec listcerts
>
>> >
>
>> > List of X.509 End Entity Certificates
>
>> >
>
>> >subject:  "CN=vpn.my-hostname.net"
>
>> >issuer:   "C=US, O=Let's Encrypt, CN=R3"
>
>> >validity:  not before Oct 06 14:03:36 2021, ok
>
>> >   n

[strongSwan] Let's Encrypt CA Expiry & related StrongSWAN trouble

2021-10-06 Thread Philip Veale 
So about a week about, one of the CAs in the chain Let'sEncrypt use (DST
Root CA X3) expired. This shouldn't have been a problem for most clients,
as it was cross signed with a CA that had not expired (ISRG Root X1) which
most modern clients and devices should trust, though some older ones may
not which was (AIUI) why they kept the DST Root CA X3 in there too.

I use a Let'sEncrypt certificate with StrongSWAN and for years it has
mostly just worked (mostly being, certbot very usefully renewed the
certificate dutifully every so many days, but it didn't get ipsec to
re-read this so I'd have to manually punt it about 4 times a year. I could
have fixed that but never bothered, anyway, I digress...)

Recently I found the StrongSWAN client on my Android 10 phone wouldn't
connect, and at first I thought I just needed to punt it again to re-read
the cert, but then I realised it was reporting it had expired even after
I'd manually run certbot. And thus I discovered that something that the
Android Client is looking at doesn't trust the cert the server was offering
as one CA was expired even though another was still valid. This I don't
entirely understand as the Let'sEncrypt people seem pretty adamant it
should all still work.

I don't entirely understand why new certificates issued today are still
being signed with a cert that expired last week, though.

Anyway, moving on. I figured the best way to solve this was to request a
cert from LE that was only signed by ISRG Root X1 and NOT by DST Root CA X3
as well, which is not the default behaviour but can be achieved by passing
a switch to certbot to ask it to do it that way.

My system was running Debian Switch and I wanted to continue to use
certbot, and I didn't want to pollute the system with certbot's suggested
tool snap, which imports a little bit of Ubuntu stuff.

Looking at repositories' versions of the certbot package, it was clear I
had to upgrade from Stretch to Bullseye, via Buster in between.

So, done that, everything all up to date, got new cert, all should be well
now, except it fails, client and server both reporting basically the same
thing;

Oct  6 15:20:30 VPN-Server charon: 10[IKE] no private key found for '
vpn.my-hostname.net'
Oct  6 15:20:30 VPN-Server charon: 10[ENC] generating IKE_AUTH response 1 [
N(AUTH_FAILED) ]

I've not modified ipsec.secrets, it's still intact and contains the correct
reference to the privkey.pem file, which pans out as running the below
command brings up the expected result;

# ipsec listcerts

List of X.509 End Entity Certificates

  subject:  "CN=vpn.my-hostname.net"
  issuer:   "C=US, O=Let's Encrypt, CN=R3"
  validity:  not before Oct 06 14:03:36 2021, ok
 not after  Jan 04 13:03:35 2022, ok (expires in 89 days)
  serial:[redacted]
  altNames:   vpn.my-hostname.net
  flags: serverAuth clientAuth
  OCSP URIs: http://r3.o.lencr.org
  certificatePolicies:
 2.23.140.1.2.1
 1.3.6.1.4.1.44947.1.1.1
 CPS: http://cps.letsencrypt.org
[other lines trimmed]


all looks correct to me. All certs present and correct by my reckoning,
config unaltered from previous working state before certificate trouble
started within the last week.

I say unaltered, I've obviously gone up TWO Debian release versions which
might have some bearing on it, but I can't see how and the logs and
pointing the finger at a certificate issue which seems more likely.

Only other thing I can thing of is at some point in the past I had manually
imported a cacert from Let'sEncrypt onto the system such that "ipsec
listcacerts" produced some output, they are gone now so this produces
nothing.

Not sure how they'd be needed, though


Can anyone spot or think of what I've missed, or has anyone else been
through similar recently due to what's happened with Let'sEncrypt ?