Re: SSL OCSP Stapling

[Karel Sedláček]

I'm all for the idea of OCSP stapling at some point, but if this is
indeed still the current state of the world, then stapling continues
to be broken and probably should take lower priority to things that
are not broken, e.g. client cert info passing, or less broken, e.g.
OCSP checking of client certs. Here's some food for thought, though:

Mozilla talking about implementing a multiple certificate extension to
OCSP stapling
https://bugzilla.mozilla.org/show_bug.cgi?id=611836
Summary: Check out the first two IETF threads linked to in the
discussion, and see the conclusions; these guys think that a
client-side pre-fetch/caching mechanism for OCSP responses might be
the way to go for now, with stapling to come later, when it is needed.
Architecture-wise, I tend to agree with this, but it can nicely be
combined with (a future-non-broken) OCSP stapling at the request of
the client (which is already how it works), for the desired
latency/roundtrip reduction.

On Wed, Nov 7, 2012 at 1:42 PM, Hervé COMMOWICK
 wrote:
> As of now, on client side, it is only working on IE9 (not before not
> after) and Opera, not so common...
>
> Look this : http://www.imperialviolet.org/2012/02/05/crlsets.html for
> Google's thoughts
> Short : "On this basis, we're currently planning on disabling online
> revocation checks in a future version of Chrome. (There is a class of
> higher-security certificate, called an EV certificate, where we haven't
> made a decision about what to do yet.)"
>
> And this : https://bugzilla.mozilla.org/show_bug.cgi?id=360420#c10 for
> Mozilla's thoughts.
> Short : "it's busted by design. It can only carry a single response and
> hardly any sites have only one OCSP certificate in their chain these
> days. So it doesn't eliminate the OCSP lookup delay, which it's primary
> attraction.
>
> Hervé C.
>
>
> On 11/06/2012 11:02 PM, Willy Tarreau wrote:
>> Hi Lukas,
>>
>> On Tue, Nov 06, 2012 at 04:57:59PM +0100, Lukas Tribus wrote:
>>>
>>> Don't know if it helps without some knowledge of the nginx source code, but
>>> here [1] you can find the patches applied to nginx to introduce ocsp 
>>> support.
>>
>> Thanks for the pointer. Anyway as you suspect, source code alone doesn't
>> tell much about the real benefits to expect from this feature, nor how
>> it's supposed to be used (especially by clients).
>>
>>> Its doesn't seem to be trivial to implement though, because you also need to
>>> run (at regular intervals) an OCSP query towards the CA's OCSP server...
>>
>> Amusingly, running a task at regular intervals is the easiest part to do,
>> it's just like health checks. We could decide to dedicate such a task per
>> stapling-enabled bind line and it would not be much of an issue. The overhead
>> would not even be measurable if we were working at insane refresh rates.
>>
>> What's unclear to me is how many clients do support this nowadays, how many
>> servers do, whether or not users are willing to allow outgoing connections
>> to fetch such cert statuses, whether or not non-stapling aware clients would
>> be impacted by the feature (eg: increased handshake size due to advertised
>> extension and data to everyone) etc...
>>
>> I think we need to take more time to study this in details, but until
>> someone comes with a detailed description of what this will bring to
>> his site, I'm not sure anyone will spend more time on this :-/
>>
>> Regards,
>> Willy
>>
>>
>
> --
> Hervé COMMOWICK
> Ingénieur systèmes et réseaux.
>
> http://www.rezulteo.com
> by Lizeo Online Media Group 
> 42 quai Rambaud - 69002 Lyon (France) ⎮ ☎ +33 (0)4 63 05 95 30
>

Re: FW: SSL OCSP Stapling

[Karel Sedláček]

On Tue, Nov 6, 2012 at 11:08 PM, Willy Tarreau  wrote:
>
> > I would say the periodic-request aspect of it is pretty trivial; you add a
> > timer to the event loop that expires in some configurable amount of time,
> > e.g. a bit before the last OCSP response expires, and you cache the result
> > until it expires or a more recent result overwrites it. Given that the
> > overhead of making a single OCSP request for the cert inside HAProxy is very
> > low, you can easily do this every few minutes with no perceivable overhead.
> > Obviously some logic re: failing requests and retrying has to be 
> > implemented,
> > which amounts to nothing more than a formulation for how much time to wait
> > until retrying again.
>
> I confirm that this part it clearly nothing.
>
> > The user should also be able to configure whether to
> > deliver an expired OCSP response or none at all in the case that an upstream
> > OCSP response cannot be received by the time the currently cached response
> > expires.
>
> That's one of the points of attention, I agree.
>
> > A single timer and single cache slot are used for each certificate chain. 
> > The
> > timer is reset with a new value when:
> > - a request fails; in this case we need
> >   to use our retry/backoff algorithm to decide how long to wait before
> >   retrying;
> > - a request succeeds; in this case we need to use our expires algorithm,
> >   which can be parameterized over the expiration time of the OCSP response, 
> > to
> >   decide how long to wait before trying to get a fresh response.
>
> Hmmm OK it's per certificate... Obviously in fact. So that probably means
> some funny mechanisms to connect to various places depending on the cert
> chain (eg: for those connecting via proxies, etc...).
>
> > One thing to keep in mind is that OCSP stapling in many libraries has (or
> > had, at one point) buggy or nonexistent support for OCSP payloads containing
> > multiple certificates,
>
> That's a very useful and interesting piece of information.
>
> > and a bit of research should be done prior to
> > implementation to discover the current state of the world in this regard.
>
> I agree!
>
> > I believe the official word at one point was that OCSP stapling of chains
> > should be accomplished by including the entire chain in the OCSP request,
> > delivering that compound OCSP response via the TLS Certificate Status 
> > Request
> > extension.
>
> And do you know how large this could be for average web sites ? Maybe
> there is a cross-over point where doing so has a more negative impact
> than letting the client check by itself ?

There might be such a point, but arguably one could let the user
decide when it is reached simply by enabling/disabling OCSP stapling.
Note also that it's pretty easy to imagine that each certificate could
have its own OCSP stapling options, which override whatever the
specified global defaults are. Running OCSP against a 3-cert chain, I
get a DER-encoded response that is 1866 bytes. In a typical
configuration this represents a negligible amount of caching and
bandwidth overhead.

>
>
> Thanks for your comments and suggestions!
> Willy
>

Re: Client certificate info in HTTP headers

[Karel Sedláček]

On Wed, Nov 7, 2012 at 8:07 AM, Willy Tarreau  wrote:
> Hi Karel,
>
> On Wed, Nov 07, 2012 at 05:55:15AM +0100, Karel Sedlá??ek wrote:
>> I was hoping that feature branch would start with the code you mentioned.
>> For my use case, just piping the data into the backend is more than
>> sufficient, and I'd be happy to flesh out the number of items that are
>> being parsed. In addition to the already-extant X-Forwarded-For, I'd
>> recommend the following:
>>
>> - X-SSL-Protocol
>> - X-SSL-Cipher
>> - X-Client-Cert-Verify
>> - X-Client-Cert
>> - X-Client-Cert-Serial
>> - X-Client-Cert-Subject-DN
>> - X-Client-Cert-Issuer-DN
>
> From what I've seen on the net and at various customers, we cannot fix
> the header names, we need to have them user-configurable. Also, some
> users want some of these info, others want some other info. For example
> I know at least one place where the whole PEM cert is desired (we don't
> have it right now), because of encoding issues in the ASCII representation
> of the subject DN.

Right; my intention with including the header names wasn't to suggest
that they can't be configurable, but because they were c&p'ed out of a
file that I had been working on to fill this functional role. They
should certainly be configurable.

>
> What I'd like to do is to be able to define the equivalent of a new reqadd
> directive which supports fetch names. It could probably be done in the
> http-request rulesets, which can be set both in frontends and backends.
> It could also be done somewhere else. The idea would be to reuse the
> string construction we have for log-format (which allows to mix text and
> tags and adjust the format, quoted vs non-quoted).
>
> For instance it could end up with something approximately like this :
>
>http-request add X-Client-Cert-Subject-DN %{ssl_c_s_dn}
> or :
>http-request add X-Client-Date %{ssl_c_notbefore}-%{ssl_c_notafter}
>
> Among the points I'm thinking about, maybe http-request is not well
> suited because it's called early and we'd prefer something that is
> called late, just before use_backend (for the frontend) and just
> before use-server (for the backend). That would avoid issues with
> conditions based on these headers in the request, and will also allow
> us to clear them. We must find a syntax which avoids as much as
> possible conflicts with regex so that we can reuse it later when we
> want to introduce this in reqrep, redirects, etc... That's why I think
> that %{} is not bad at all.

I'm ok with %{}; other common syntaxes include #var $var #{} and ${},
particularly in the case you allow variables to be operated on, e.g.
${var / 2}.

>
>> An additional X-Compromise-Attempt or X-Illegal-Headers would be a nice
>> option for backends that want to know when clients try to pass these
>> reserved headers from outside.
>
> These ones are obvious to do by configuration and do not need any code,
> they're just an ACL. You could even imagine to replace each of these
> headers with X-Illegal-$header-name for logging purposes.

Ditto for this regarding configurability. And, glad to hear it.

>
>> I can follow this up with a bit of example
>> code to grab each of these out of the SSL connection object.
>
> I did this exercise several weeks ago and found that if we systematically
> add each header that at least one user is interested in, you have so large
> requests that they don't fit in buffers anymore. And that's not counting
> the varying header names ! So clearly we need to make them configurable.

Ditto again; the headers (i.e. their corresponding properties) that I
listed are an overlapping set of data. Probably you wouldn't need the
whole client cert and each of its individual parts at the same time.
If you're doing certificate-based auth, you probably either just want
the subject CN (and configure HAProxy to reject unverified client
certs) or the CN and verified status. If you want to verify the cert
as well, until HAProxy supports doing OCSP client cert verification,
you could use either the whole cert (preparing the request yourself),
or, if it's your own infrastructure issuing certs, set up a service to
reject serial numbers, and have HAProxy pass the serial along. Because
there are many possible use cases, there should be many possible
options, but only a small subset of them is likely to be used at any
given time.

>
> Some tricks about encoding are important too. If some headers contain
> commas, we're supposed to quote-encode them. Idem for spaces at begin
> or end of values. For example, if the following subject is sent as-is
> in a header :
>
>C=US, ST=Maryland, L=Pasadena, O=Brent Baccala, OU=FreeSoft, 
> CN=www.freesoft.org/emailAddress=bacc...@freesoft.org
>
> Then the header technically contains 6 comma-delimited values, which
> intermediaries are free to split into 6 different headers if needed,
> or even to remove spaces around commas. So this value must have quotes
> around it. But I'm sure that not all software support quote

Re: Client certificate info in HTTP headers

[Karel Sedláček]

I was hoping that feature branch would start with the code you mentioned.
For my use case, just piping the data into the backend is more than
sufficient, and I'd be happy to flesh out the number of items that are
being parsed. In addition to the already-extant X-Forwarded-For, I'd
recommend the following:

- X-SSL-Protocol
- X-SSL-Cipher
- X-Client-Cert-Verify
- X-Client-Cert
- X-Client-Cert-Serial
- X-Client-Cert-Subject-DN
- X-Client-Cert-Issuer-DN

An additional X-Compromise-Attempt or X-Illegal-Headers would be a nice
option for backends that want to know when clients try to pass these
reserved headers from outside. I can follow this up with a bit of example
code to grab each of these out of the SSL connection object.

k

On Tue, Nov 6, 2012 at 11:16 PM, Willy Tarreau  wrote:

> Hi Karel,
>
> On Tue, Nov 06, 2012 at 02:47:33PM +0100, Karel Sedlá??ek wrote:
> > I understand that there's some prototypical functionality for analogies
> to
> > nginx $ssl_client_s_dn et al. I'd be quite happy to get my hands on this,
> > as I've been having issues with way too many components of my stack re:
> > properly extracting certificate info, supporting newer cipher suites, or
> > even exposing the certificate info in their TLS bindings/implementations.
> > For better or worse, nginx is also not a solution, as its current
> > proxy_pass method fully buffers the request, killing websocket. The other
> > solution is that I write a very small streaming proxy to unwrap the TLS
> and
> > inject the relevant headers, but haproxy already does 99% of this, and
> that
> > level of code duplication seems like madness.
>
> We've already experimented with this at Exceliance. We've not produced code
> that makes us happy. We developped the fetches not for ACLs but in order to
> extract fields to put into logs and HTTP headers. What we need to do now is
> to add the configuration language to map ones to the others. We've made
> some
> attempts but were not much satisfied with them. To be honnest, we're
> currently
> stabilizing the code for dev13 first and we'll get more focused on this
> just
> after.
>
> The code is not much an issue, we need to decide how we'd like users to
> configure this in a flexible way to satisfy all needs. This can be
> discussed
> right here on the list if you want, after all it's a development list and
> it's where the users are (ie those who decide in the end).
>
> > Code beauty is not a major
> > concern to me at this point, and I'd be happy to contribute back any
> > changes I make--could we get a feature branch for this in github?
>
> Feel free to do so if you want but better settle on how this should be
> configured and used first, it would avoid painful issues at the end.
>
> Willy
>
>

Client certificate info in HTTP headers

[Karel Sedláček]

I understand that there's some prototypical functionality for analogies to
nginx $ssl_client_s_dn et al. I'd be quite happy to get my hands on this,
as I've been having issues with way too many components of my stack re:
properly extracting certificate info, supporting newer cipher suites, or
even exposing the certificate info in their TLS bindings/implementations.
For better or worse, nginx is also not a solution, as its current
proxy_pass method fully buffers the request, killing websocket. The other
solution is that I write a very small streaming proxy to unwrap the TLS and
inject the relevant headers, but haproxy already does 99% of this, and that
level of code duplication seems like madness. Code beauty is not a major
concern to me at this point, and I'd be happy to contribute back any
changes I make--could we get a feature branch for this in github?

k

Re: SSL OCSP Stapling

[Karel Sedláček]

OCSP stapling exists to reduce the latency and client burden involved in
establishing an OCSP-verified TLS connection. This is achieved by a TLS
extension: "Certificate Status Request".

Basically, what this means is that backend services that want to prove
validity of their certificate fetch (and optionally cache for the validity
period) OCSP responses from their CA's OCSP responder, and bundle them as
part of the TLS handshake.

k

> On Tue, Oct 30, 2012 at 03:26:21PM +0100, Baptiste wrote:
> > Hi,
> >
> > I discussed about it a few weeks ago with @emericbr @exceliance, but
> > he was a bit doubtful about it.
> > As far as I'm concerned, I think this would be a nice new feature.
> >
> > so let's wait for Willy's response.
>
> well, after having checked the RFC on this, I must confess that what it
> provides and the way it's supposed to work are still cryptic to me :-/
>
> If someone could explain in a simple way (assuming that something in TLS
> can be explained that way), and provide some real world use case, it would
> be nice.
>
> Regards,
> Willy