Re: Automatic Certificate Switching Idea

[Daniel Schneller]

> 
> That's perfect! Your feedback and possible trouble in doing this will
> also definitely help!
> 

Oh, if experience tells me one thing, no matter how “straightforward” this may 
look, there _will_ be trouble ;-)

Cheers
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431

Re: Automatic Certificate Switching Idea

[Willy Tarreau]

On Fri, May 12, 2017 at 06:42:20PM +0200, Daniel Schneller wrote:
> > That said, given that we can already look up a cert based on a name,
> > maybe in fact we could load all of them and just try to find a more
> > recent one if the first one reported by the SNI is outdated. I don't
> > know if that solves everything there.
> 
> 
> It actually might. In the end it would be something like a map, with the
> key being the domain, and the value a list of pointers to the actual
> certificates, sorted by remaining validity, having shortest first.

That's already what is done in the SNI trees, except that the validity
date is not considered, the first one matching is retrieved.

> I think it would benefit Let's Encrypt and similar scenarios. I would
> still require reloads to pick up newly added certificates. But as renewed
> certificates overlap their predecessors' validity period, dropping them
> into a directory and just doing a reload maybe once a day would work.
> Clients would still get the older one, until it finally expired, but that
> should not matter, as we are not talking about revocations where
> switching to a new cert is wanted quickly.

Using the old one "until it expires" is what really causes me a problem
(and I understand that in your case that's what you need). There are
several reasons for prefering the latest one instead :
  - it might provide stronger algorithms

  - it might use a CA which is not being blacklisted (remember that people
started to complain about haproxy.org causing them some warnings because
the CA was considered unsafe)

  - it was issued in the past (minutes, hours, days) so is likely already
valid regardless of any small time shift. Using the old one one minute
past its validity date will be a big problem.

  - the change will be effective at the moment of reload, meaning that any
surprize like an incomplete chain, incorrect OCSP, key size incompatible
with certain browsers, will be identified at an expected moment and when
it's not too late to fix it. By using the oldest one as long as possible,
it would break at any time in the middle of the night and would do it
once you cannot roll back.

And that's the point. Users praise haproxy's reliability but in fact it's
not (just) the code's reliability (git log --grep BUG shames us), but the
fact that it has always been designed to be used by humans, who make
mistakes and who want to spot them very quickly and to fix them before
they become a big trouble. Config warnings/errors, checks for suspicious
constructs and logs are directly involved here. And we do know that our
users occasionally fail and we must help them recover, and even possibly
cover their mistakes before the boss or the customer has any chance to
notice.

So creating something designed to fail by default in their back without
prior notice and without the ability to quickly stop before anyone notices
is contrary to the philosophy here.

That doesn't mean that what you need must not be implemented, it means
that under no circumstance it should be the default nor happen to be
enabled by default. Thus I think that at minima if we ever go in that
direction, the default behaviour must be the expected one (ie: use the
most recent valid cert), and maybe there could be an option to prefer
the old one instead and to apply a date margin (eg: avoid using this
one if there's less than a day left).

(...)
> PS: This is an interesting discussion, and I am happy to continue
> it, if anyone feels the same.

I would not be surprized if we get some followups in either direction.
Over the mid term, more and more people will be affected by related
situations and the whole aspect of cert renewal will eventually become
hot. But I strongly doubt we'll do anything for this in 1.8, though
collecting views, ideas and constraints can be useful to try to serve
everyone the best later.

> As I said, I will try to solve this via
> provisioning scripts in the meantime, so there is no time pressure.

That's perfect! Your feedback and possible trouble in doing this will
also definitely help!

thanks,
Willy

Re: Automatic Certificate Switching Idea

[Daniel Schneller]

Willy,

thanks for your elaborate reply! See my remarks below.

> possible impacts nor complexity (but I don't want to have the complete MS
> Office suite merged in, just Word, Excel and PowerPoint :-)).


:-D

>  - renewed certs can and will sometimes provide extra alt names, so
>they are not always 100% equivalent.
> […]

> That said, given that we can already look up a cert based on a name,
> maybe in fact we could load all of them and just try to find a more
> recent one if the first one reported by the SNI is outdated. I don't
> know if that solves everything there.


It actually might. In the end it would be something like a map, with the
key being the domain, and the value a list of pointers to the actual
certificates, sorted by remaining validity, having shortest first.

> In any case, this will not provide any benefit regarding let's encrypt
> or such solutions, because the next cert would have to be known in
> advance and loaded already, so reloads will have to be performed to
> take it into account. So I think that the approach making it possible
> to feed them over the CLI would still be mor interesting (and possibly
> complementary).

I think it would benefit Let’s Encrypt and similar scenarios. I would
still require reloads to pick up newly added certificates. But as renewed
certificates overlap their predecessors’ validity period, dropping them
into a directory and just doing a reload maybe once a day would work.
Clients would still get the older one, until it finally expired, but that
should not matter, as we are not talking about revocations where
switching to a new cert is wanted quickly.

> Daniel I'm pretty sure that most users
> would prefer the approach consisting in picking the most recent
> valid cert instead of the last one as you'd like. I don't really
> know if it's common to issue a cert with a "not-before" date in the
> future. And that might be the whole point in the end.


Well, I was just thinking about the not-after date. In general, from a
client perspective it shouldn’t matter to get an older one, until it
really expires. And the case where you have a new certificate
already, and you want it handed out to clients ASAP is already taken
care of today — just replace the file and reload :-)
Unless I misunderstood what you meant when referring to the
“not-before” date.

Daniel

PS: This is an interesting discussion, and I am happy to continue
it, if anyone feels the same. As I said, I will try to solve this via
provisioning scripts in the meantime, so there is no time pressure.


-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431

Re: Automatic Certificate Switching Idea

[Willy Tarreau]

Hi,

On Tue, May 09, 2017 at 07:04:01PM +0200, Daniel Schneller wrote:
> Hi!
> 
> > On 9. May. 2017, at 00:30, Lukas Tribus  wrote:
> > 
> > [...]
> > I'm opposed to heavy feature-bloating for provisioning use-cases, that
> > can quite easily fixed where the fix belongs - the provisioning layer.
> 
> You are right, that this can be handled outside / in the provisioning layer.
> And I have no problem implementing it there, if it is considered too narrow a
> niche feature. However, I was curious to see, if this is something that other
> people also need constantly -- sometimes you believe you are in a specific
> bubble, but aren't. But from the amount of feedback the original post
> generated, I think I know my anser already ;-)

In fact I'm less opposed than Lukas here given that I have no idea of the
possible impacts nor complexity (but I don't want to have the complete MS
Office suite merged in, just Word, Excel and PowerPoint :-)).

I'd tend to say that since we're progressively evolving in a more dynamic
world where users want the ability to perform *some* updates without
reloading, the day we realize that 90% of the haproxy reloads are only
caused by cert updates, we need to think about a way to address this. I
remember that Thierry started to look at how to feed a cert from the CLI
but apparently it was everything but obvious.

Loading multiple certs could be nice in theory, but there are a few
shortcomings to keep in mind :
  - for embedded users you don't want haproxy's date check to become
strict because it's frequent that such devices have a totally
wrong date. Or at least you want to ensure that you always keep
the most recent cert and never kill any outdated one.

  - renewed certs can and will sometimes provide extra alt names, so
they are not always 100% equivalent.

  - renewed certs will also change the key size once in a while, and
sometimes the algorithm. Technically speaking it might cause
difficulties to change this on the fly, or at least some
verifications have to be performed at load time.

  - I think that most of the crt-list config is per-certificate file
and not per-name. That might also make certain things more
complicated to configure

That said, given that we can already look up a cert based on a name,
maybe in fact we could load all of them and just try to find a more
recent one if the first one reported by the SNI is outdated. I don't
know if that solves everything there.

In any case, this will not provide any benefit regarding let's encrypt
or such solutions, because the next cert would have to be known in
advance and loaded already, so reloads will have to be performed to
take it into account. So I think that the approach making it possible
to feed them over the CLI would still be mor interesting (and possibly
complementary).

It could be interesting to study what it would require to implement
a "strict-date" option or something like this per certificate to
enable checking of their validity during the pick-up. Still, one
point has to be kept in mind. Daniel I'm pretty sure that most users
would prefer the approach consisting in picking the most recent
valid cert instead of the last one as you'd like. I don't really
know if it's common to issue a cert with a "not-before" date in the
future. And that might be the whole point in the end.

Hoping this helps,
Willy

Re: Automatic Certificate Switching Idea

[Daniel Schneller]

Hi!

> On 9. May. 2017, at 00:30, Lukas Tribus  wrote:
> 
> [...]
> I'm opposed to heavy feature-bloating for provisioning use-cases, that
> can quite easily fixed where the fix belongs - the provisioning layer.

You are right, that this can be handled outside / in the provisioning layer. 
And I have no problem implementing it there, if it is considered too narrow a 
niche feature. However, I was curious to see, if this is something that other 
people also need constantly — sometimes you believe you are in a specific 
bubble, but aren’t. But from the amount of feedback the original post 
generated, I think I know my anser already ;-)

Also, if it was something that could be implemented in a 10-liner (I know, I 
exaggerate :-)) it might still have been a useful addition.

I’ll see what I can get whipped up externally. Depending on how well I can get 
it separated from our specific setup, I might then release it into the wild for 
the select few who might find it useful :)

Daniel

-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431

Re: Automatic Certificate Switching Idea

[Lukas Tribus]

Hello,


Am 30.04.2017 um 22:16 schrieb Daniel Schneller:
> Hi!
>
> Yes, you got it right. I have no idea if there are technical limitations in 
> the SSL library or other parts of the code that would make several 
> certificate/key pairs for the same domain infeasible. 
>
> If there were hard restrictions, it could certainly be done "externally" with 
> a set of clever scripts and haproxy reloads, but IMO it would be a less 
> brittle solution if it were built right in. Also, it wouldn't require 
> separate scripts per platform and init system (SysV, systemd, …). 
>
> Daniel

Similar to the "Certificate order" thread, this is a very specific
requirement that only a handful of people require and it is imho best
achieved with external configuration tools. If we implement everyone of
those use-cases we are gonna transform haproxy into Microsoft Office.

I'm opposed to heavy feature-bloating for provisioning use-cases, that
can quite easily fixed where the fix belongs - the provisioning layer.


Those are just my two cents though ...


cheers,
lukas

Re: Automatic Certificate Switching Idea

[Daniel Schneller]

Hi!

Yes, you got it right. I have no idea if there are technical limitations in the 
SSL library or other parts of the code that would make several certificate/key 
pairs for the same domain infeasible. 

If there were hard restrictions, it could certainly be done "externally" with a 
set of clever scripts and haproxy reloads, but IMO it would be a less brittle 
solution if it were built right in. Also, it wouldn't require separate scripts 
per platform and init system (SysV, systemd, …). 

Daniel


> On 30. Apr 2017, at 11:50, Aleksandar Lazic  wrote:
> 
> HI.
> 
> Am 28-04-2017 09:26, schrieb Daniel Schneller:
> 
>> Hello!
>> I am managing a few haproxy instances that each manage a good number of 
>> domains and do the TLS termination on behalf of what you might call "hosted" 
>> sites.
>> Most of the clients connecting to these haproxys implement certificate 
>> pinning and verify that the certificate presented by the server is on a 
>> white list for their respective domains.
>> We have alerts on upcoming expirations with a few weeks advance notice, so 
>> that we can tell our customers to get a renewal done with their CA and 
>> provide it to us. Then clients (mostly mobile apps) get be updated, built 
>> and released to include both the current and the renewed certificates for a 
>> while. Once the current cert has actually expired, it will be removed from 
>> the white list with the next update.
>> To give the end users the longest possible opportunity to download and 
>> install the updated client, we perform the certificate replacement on 
>> haproxy very close to the actual expiration point in time.
>> With an increasing number of domains and certificates, and the tendency 
>> toward shorter certificate life times, some cert is about to expire all the 
>> time, making this a rather regular task.
>> So I was wondering if there was a better way to achieve the client-friendly 
>> "last minute" replacements without having to manually care about the exact 
>> timing and hopefully never making a mistake.
>> If haproxy could load multiple certificates for the same domain (similar to 
>> what it currently already does for wildcard and more specific domain 
>> certificates), and would additionally consider their expiration dates,  
>> serving the one with the least remaining validity as long as it was still 
>> valid, but then automatically switch to an available replacement once the 
>> expiration is reached, we could just schedule regular (maybe daily) reloads 
>> (to let haproxy read any new files in) and just drop any renewed 
>> certificate/key files into the appropriate directory as soon as you got them.
>> I would welcome feedback on this idea, if only to be pointed at the obvious 
>> and glaring shortcomings it may have :D
> 
> This sounds to me a very interesting use case especially with the let's 
> encrypt certificates.
> 
> To reflect what I have understand I will try to explain your request from my 
> point of view.
> 
> mysecurewww.mysecuredomain.xxx will expire 01.02.2017
> This certificate pair will created at 01.10.2016
> 
> Now you will create another certificate pair on 07.01.2017 with the same name 
> and add it to haproxy, right?
> 
> This ends up with 2 certificate pairs for the same domain but with different 
> expire time.
> Is this possible with the ssl lib?
> 
> Is this the scenario you have described?
> 
>> Cheers,
>> Daniel
> 
> Regards
> Aleks
> 
>> --
>> Daniel Schneller
>> Principal Cloud Engineer
>> CenterDevice GmbH  | Hochstraße 11
>> | 42697 Solingen
>> tel: +49 1754155711| Deutschland
>> daniel.schnel...@centerdevice.de   | www.centerdevice.de
>> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
>> Michael Rosbach, Handelsregister-Nr.: HRB 18655,
>> HR-Gericht: Bonn, USt-IdNr.: DE-815299431

Re: Automatic Certificate Switching Idea

[Aleksandar Lazic]

HI.

Am 28-04-2017 09:26, schrieb Daniel Schneller:


Hello!

I am managing a few haproxy instances that each manage a good number of 
domains and do the TLS termination on behalf of what you might call 
"hosted" sites.


Most of the clients connecting to these haproxys implement certificate 
pinning and verify that the certificate presented by the server is on a 
white list for their respective domains.


We have alerts on upcoming expirations with a few weeks advance notice, 
so that we can tell our customers to get a renewal done with their CA 
and provide it to us. Then clients (mostly mobile apps) get be updated, 
built and released to include both the current and the renewed 
certificates for a while. Once the current cert has actually expired, 
it will be removed from the white list with the next update.


To give the end users the longest possible opportunity to download and 
install the updated client, we perform the certificate replacement on 
haproxy very close to the actual expiration point in time.


With an increasing number of domains and certificates, and the tendency 
toward shorter certificate life times, some cert is about to expire all 
the time, making this a rather regular task.


So I was wondering if there was a better way to achieve the 
client-friendly "last minute" replacements without having to manually 
care about the exact timing and hopefully never making a mistake.


If haproxy could load multiple certificates for the same domain 
(similar to what it currently already does for wildcard and more 
specific domain certificates), and would additionally consider their 
expiration dates,  serving the one with the least remaining validity as 
long as it was still valid, but then automatically switch to an 
available replacement once the expiration is reached, we could just 
schedule regular (maybe daily) reloads (to let haproxy read any new 
files in) and just drop any renewed certificate/key files into the 
appropriate directory as soon as you got them.


I would welcome feedback on this idea, if only to be pointed at the 
obvious and glaring shortcomings it may have :D


This sounds to me a very interesting use case especially with the let's 
encrypt certificates.


To reflect what I have understand I will try to explain your request 
from my point of view.


mysecurewww.mysecuredomain.xxx will expire 01.02.2017
This certificate pair will created at 01.10.2016

Now you will create another certificate pair on 07.01.2017 with the same 
name and add it to haproxy, right?


This ends up with 2 certificate pairs for the same domain but with 
different expire time.

Is this possible with the ssl lib?

Is this the scenario you have described?


Cheers,
Daniel


Regards
Aleks


--
Daniel Schneller
Principal Cloud Engineer

CenterDevice GmbH  | Hochstraße 11
| 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431

Automatic Certificate Switching Idea

[Daniel Schneller]

Hello!

I am managing a few haproxy instances that each manage a good number of domains 
and do the TLS termination on behalf of what you might call “hosted” sites.

Most of the clients connecting to these haproxys implement certificate pinning 
and verify that the certificate presented by the server is on a white list for 
their respective domains.

We have alerts on upcoming expirations with a few weeks advance notice, so that 
we can tell our customers to get a renewal done with their CA and provide it to 
us. Then clients (mostly mobile apps) get be updated, built and released to 
include both the current and the renewed certificates for a while. Once the 
current cert has actually expired, it will be removed from the white list with 
the next update.

To give the end users the longest possible opportunity to download and install 
the updated client, we perform the certificate replacement on haproxy very 
close to the actual expiration point in time.

With an increasing number of domains and certificates, and the tendency toward 
shorter certificate life times, some cert is about to expire all the time, 
making this a rather regular task.

So I was wondering if there was a better way to achieve the client-friendly 
“last minute” replacements without having to manually care about the exact 
timing and hopefully never making a mistake.

If haproxy could load multiple certificates for the same domain (similar to 
what it currently already does for wildcard and more specific domain 
certificates), and would additionally consider their expiration dates,  serving 
the one with the least remaining validity as long as it was still valid, but 
then automatically switch to an available replacement once the expiration is 
reached, we could just schedule regular (maybe daily) reloads (to let haproxy 
read any new files in) and just drop any renewed certificate/key files into the 
appropriate directory as soon as you got them. 

I would welcome feedback on this idea, if only to be pointed at the obvious and 
glaring shortcomings it may have :D

Cheers,
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431