Re: Incident Report – Certificates issued without proper domain validation

[Yuhong Bao]

In this case, Nest's 404 page happens not to include the original URL in the 
HTML so they are not affected, but you see what I mean now.

From: Ryan Sleevi 
Sent: Wednesday, January 11, 2017 6:41:46 PM
To: Yuhong Bao
Cc: Richard Wang; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan 
Sleevi
Subject: Re: Incident Report – Certificates issued without proper domain 
validation

Hi Yuhong,

Perhaps it be best if you create a separate thread for your question -
it's not really clear at all how it relates to the topic at hand.

On Wed, Jan 11, 2017 at 6:08 PM, Yuhong Bao  wrote:
> That is what the current certificate by Google Internet Authority says.
> What I am referring to is that before Google bought Nest they used GoDaddy as 
> the CA.
> 
> From: Richard Wang 
> Sent: Wednesday, January 11, 2017 5:01:08 PM
> To: Yuhong Bao; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan 
> Sleevi
> Subject: RE: Incident Report – Certificates issued without proper domain 
> validation
>
> The nest.com certificate subject is:
> CN = www.nest.com
> O = Google Inc
> L = Mountain View
> S = California
> C = US
>
> This means this website owned by Google Inc. Right?
>
>
> Best Regards,
>
> Richard
>
> -Original Message-
> From: dev-security-policy
> [mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
> Behalf Of Yuhong Bao
> Sent: Thursday, January 12, 2017 6:12 AM
> To: Wayne Thayer ;
> dev-security-policy@lists.mozilla.org; Ryan Sleevi 
> Subject: Re: Incident Report - Certificates issued without proper domain
> validation
>
> I wonder if nest.com is now considered high-risk now. They recently switched
> from GoDaddy to Google Internet Authority.
> 
> From: dev-security-policy
>  on
> behalf of Wayne Thayer 
> Sent: Tuesday, January 10, 2017 7:02:28 PM
> To: dev-security-policy@lists.mozilla.org
> Subject: Incident Report - Certificates issued without proper domain
> validation
>
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
> domain validation processing system. The bug that caused the issue was fixed
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
> determine the scope of the problem, and identified 8850 certificates that
> were issued without proper domain validation as a result of the bug. The
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
> n...@godaddy.com and two GoDaddy employees. Due to
> holiday vacations and the fact that the issue was not reported properly per
> our CPS, we did not become aware of the issue until one of the employees
> opened the email on Friday Jan 6th and promptly alerted management. The
> issue was originally reported to Microsoft by one of their own customers and
> was described as only affecting certificate requests when the DNS A record
> of the domain was set to 127.0.0.1. An investigation was initiated
> immediately and within a few hours we determined that the problem was
> broader in scope. The root cause of the problem was fixed via a code change
> at approximately 10 PM MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on
> July 29th, 2016 as part of a routine code change intended to improve our
> certificate issuance process. The bug is related to our use of practical
> demonstration of control to validate authority to receive a certificate for
> a given fully-qualified domain name. In the problematic case, we provide a
> random code to a customer and ask them to place it in a specific location on
> their website. Our system automatically checks for the presence of that code
> via an HTTP and/or HTTPS request to the website. If the code is found, the
> domain control check is completed successfully.  Prior to the bug, the
> library used to query the website and check for the code was configured to
> return a failure if the HTTP status code was not 200 (success). A
> configuration change to the library caused it to return results even when
> the HTTP status code was not 200. Since many web servers are configured to
> include the URL of the req  uest in the body of a 404 (not found) response,
> and the URL also contained the random code, any web server configured this
> way caused domain control verification to complete successfully.
> We are currently unaware of any malicious exploitation of this bug to
> procure a certificate for a domain that was not authorized. The customer who
> discovered the bug 

Re: Incident Report – Certificates issued without proper domain validation

[Ryan Sleevi]

Hi Yuhong,

Perhaps it be best if you create a separate thread for your question -
it's not really clear at all how it relates to the topic at hand.

On Wed, Jan 11, 2017 at 6:08 PM, Yuhong Bao  wrote:
> That is what the current certificate by Google Internet Authority says.
> What I am referring to is that before Google bought Nest they used GoDaddy as 
> the CA.
> 
> From: Richard Wang 
> Sent: Wednesday, January 11, 2017 5:01:08 PM
> To: Yuhong Bao; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan 
> Sleevi
> Subject: RE: Incident Report – Certificates issued without proper domain 
> validation
>
> The nest.com certificate subject is:
> CN = www.nest.com
> O = Google Inc
> L = Mountain View
> S = California
> C = US
>
> This means this website owned by Google Inc. Right?
>
>
> Best Regards,
>
> Richard
>
> -Original Message-
> From: dev-security-policy
> [mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
> Behalf Of Yuhong Bao
> Sent: Thursday, January 12, 2017 6:12 AM
> To: Wayne Thayer ;
> dev-security-policy@lists.mozilla.org; Ryan Sleevi 
> Subject: Re: Incident Report - Certificates issued without proper domain
> validation
>
> I wonder if nest.com is now considered high-risk now. They recently switched
> from GoDaddy to Google Internet Authority.
> 
> From: dev-security-policy
>  on
> behalf of Wayne Thayer 
> Sent: Tuesday, January 10, 2017 7:02:28 PM
> To: dev-security-policy@lists.mozilla.org
> Subject: Incident Report - Certificates issued without proper domain
> validation
>
> Summary:
> On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
> domain validation processing system. The bug that caused the issue was fixed
> late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
> determine the scope of the problem, and identified 8850 certificates that
> were issued without proper domain validation as a result of the bug. The
> impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
> will also be logged to the Google Pilot CT log.
> Detailed Description:
> On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
> n...@godaddy.com and two GoDaddy employees. Due to
> holiday vacations and the fact that the issue was not reported properly per
> our CPS, we did not become aware of the issue until one of the employees
> opened the email on Friday Jan 6th and promptly alerted management. The
> issue was originally reported to Microsoft by one of their own customers and
> was described as only affecting certificate requests when the DNS A record
> of the domain was set to 127.0.0.1. An investigation was initiated
> immediately and within a few hours we determined that the problem was
> broader in scope. The root cause of the problem was fixed via a code change
> at approximately 10 PM MST on Friday, Jan 6th.
> On Saturday, January 7th, we determined that the bug was first introduced on
> July 29th, 2016 as part of a routine code change intended to improve our
> certificate issuance process. The bug is related to our use of practical
> demonstration of control to validate authority to receive a certificate for
> a given fully-qualified domain name. In the problematic case, we provide a
> random code to a customer and ask them to place it in a specific location on
> their website. Our system automatically checks for the presence of that code
> via an HTTP and/or HTTPS request to the website. If the code is found, the
> domain control check is completed successfully.  Prior to the bug, the
> library used to query the website and check for the code was configured to
> return a failure if the HTTP status code was not 200 (success). A
> configuration change to the library caused it to return results even when
> the HTTP status code was not 200. Since many web servers are configured to
> include the URL of the req  uest in the body of a 404 (not found) response,
> and the URL also contained the random code, any web server configured this
> way caused domain control verification to complete successfully.
> We are currently unaware of any malicious exploitation of this bug to
> procure a certificate for a domain that was not authorized. The customer who
> discovered the bug revoked the certificate they obtained, and subsequent
> certificates issued as the result of requests used for testing by Microsoft
> and GoDaddy have been revoked. Further, any certificate requests made for
> domains we flag as high-risk were also subjected to manual review (rather
> than being issued purely based on an invalid domain authorization).
> We have re-verified domain control on every certificate issued using this
> method of 

Re: Incident Report – Certificates issued without proper domain validation

[Yuhong Bao]

That is what the current certificate by Google Internet Authority says.
What I am referring to is that before Google bought Nest they used GoDaddy as 
the CA.

From: Richard Wang 
Sent: Wednesday, January 11, 2017 5:01:08 PM
To: Yuhong Bao; Wayne Thayer; dev-security-policy@lists.mozilla.org; Ryan Sleevi
Subject: RE: Incident Report – Certificates issued without proper domain 
validation

The nest.com certificate subject is:
CN = www.nest.com
O = Google Inc
L = Mountain View
S = California
C = US

This means this website owned by Google Inc. Right?


Best Regards,

Richard

-Original Message-
From: dev-security-policy
[mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
Behalf Of Yuhong Bao
Sent: Thursday, January 12, 2017 6:12 AM
To: Wayne Thayer ;
dev-security-policy@lists.mozilla.org; Ryan Sleevi 
Subject: Re: Incident Report - Certificates issued without proper domain
validation

I wonder if nest.com is now considered high-risk now. They recently switched
from GoDaddy to Google Internet Authority.

From: dev-security-policy
 on
behalf of Wayne Thayer 
Sent: Tuesday, January 10, 2017 7:02:28 PM
To: dev-security-policy@lists.mozilla.org
Subject: Incident Report - Certificates issued without proper domain
validation

Summary:
On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
domain validation processing system. The bug that caused the issue was fixed
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
determine the scope of the problem, and identified 8850 certificates that
were issued without proper domain validation as a result of the bug. The
impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
will also be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
n...@godaddy.com and two GoDaddy employees. Due to
holiday vacations and the fact that the issue was not reported properly per
our CPS, we did not become aware of the issue until one of the employees
opened the email on Friday Jan 6th and promptly alerted management. The
issue was originally reported to Microsoft by one of their own customers and
was described as only affecting certificate requests when the DNS A record
of the domain was set to 127.0.0.1. An investigation was initiated
immediately and within a few hours we determined that the problem was
broader in scope. The root cause of the problem was fixed via a code change
at approximately 10 PM MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on
July 29th, 2016 as part of a routine code change intended to improve our
certificate issuance process. The bug is related to our use of practical
demonstration of control to validate authority to receive a certificate for
a given fully-qualified domain name. In the problematic case, we provide a
random code to a customer and ask them to place it in a specific location on
their website. Our system automatically checks for the presence of that code
via an HTTP and/or HTTPS request to the website. If the code is found, the
domain control check is completed successfully.  Prior to the bug, the
library used to query the website and check for the code was configured to
return a failure if the HTTP status code was not 200 (success). A
configuration change to the library caused it to return results even when
the HTTP status code was not 200. Since many web servers are configured to
include the URL of the req  uest in the body of a 404 (not found) response,
and the URL also contained the random code, any web server configured this
way caused domain control verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to
procure a certificate for a domain that was not authorized. The customer who
discovered the bug revoked the certificate they obtained, and subsequent
certificates issued as the result of requests used for testing by Microsoft
and GoDaddy have been revoked. Further, any certificate requests made for
domains we flag as high-risk were also subjected to manual review (rather
than being issued purely based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this
method of validation in the period from when the bug was introduced until it
was fixed. A list of 8850 potentially unverified certificates (representing
less than 2% of the total issued during the period) was compiled at 10 PM
PST on Monday Jan 9th. As mentioned above, potentially impacted certificates
will be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT
log. Additional code changes were 

RE: Incident Report – Certificates issued without proper domain validation

[Richard Wang]

The nest.com certificate subject is:
CN = www.nest.com
O = Google Inc
L = Mountain View
S = California
C = US

This means this website owned by Google Inc. Right?


Best Regards,

Richard

-Original Message-
From: dev-security-policy
[mailto:dev-security-policy-bounces+richard=wosign@lists.mozilla.org] On
Behalf Of Yuhong Bao
Sent: Thursday, January 12, 2017 6:12 AM
To: Wayne Thayer ;
dev-security-policy@lists.mozilla.org; Ryan Sleevi 
Subject: Re: Incident Report - Certificates issued without proper domain
validation

I wonder if nest.com is now considered high-risk now. They recently switched
from GoDaddy to Google Internet Authority.

From: dev-security-policy
 on
behalf of Wayne Thayer 
Sent: Tuesday, January 10, 2017 7:02:28 PM
To: dev-security-policy@lists.mozilla.org
Subject: Incident Report - Certificates issued without proper domain
validation

Summary:
On Friday, January 6th, 2017, GoDaddy became aware of a bug affecting our
domain validation processing system. The bug that caused the issue was fixed
late Friday. At 10 PM PST on Monday, Jan 9th we completed our review to
determine the scope of the problem, and identified 8850 certificates that
were issued without proper domain validation as a result of the bug. The
impacted certificates will be revoked by 10 PM PST on Tuesday, Jan 10th, and
will also be logged to the Google Pilot CT log.
Detailed Description:
On Tuesday, Jan 3rd, 2017, one of our resellers (Microsoft) sent an email to
n...@godaddy.com and two GoDaddy employees. Due to
holiday vacations and the fact that the issue was not reported properly per
our CPS, we did not become aware of the issue until one of the employees
opened the email on Friday Jan 6th and promptly alerted management. The
issue was originally reported to Microsoft by one of their own customers and
was described as only affecting certificate requests when the DNS A record
of the domain was set to 127.0.0.1. An investigation was initiated
immediately and within a few hours we determined that the problem was
broader in scope. The root cause of the problem was fixed via a code change
at approximately 10 PM MST on Friday, Jan 6th.
On Saturday, January 7th, we determined that the bug was first introduced on
July 29th, 2016 as part of a routine code change intended to improve our
certificate issuance process. The bug is related to our use of practical
demonstration of control to validate authority to receive a certificate for
a given fully-qualified domain name. In the problematic case, we provide a
random code to a customer and ask them to place it in a specific location on
their website. Our system automatically checks for the presence of that code
via an HTTP and/or HTTPS request to the website. If the code is found, the
domain control check is completed successfully.  Prior to the bug, the
library used to query the website and check for the code was configured to
return a failure if the HTTP status code was not 200 (success). A
configuration change to the library caused it to return results even when
the HTTP status code was not 200. Since many web servers are configured to
include the URL of the req  uest in the body of a 404 (not found) response,
and the URL also contained the random code, any web server configured this
way caused domain control verification to complete successfully.
We are currently unaware of any malicious exploitation of this bug to
procure a certificate for a domain that was not authorized. The customer who
discovered the bug revoked the certificate they obtained, and subsequent
certificates issued as the result of requests used for testing by Microsoft
and GoDaddy have been revoked. Further, any certificate requests made for
domains we flag as high-risk were also subjected to manual review (rather
than being issued purely based on an invalid domain authorization).
We have re-verified domain control on every certificate issued using this
method of validation in the period from when the bug was introduced until it
was fixed. A list of 8850 potentially unverified certificates (representing
less than 2% of the total issued during the period) was compiled at 10 PM
PST on Monday Jan 9th. As mentioned above, potentially impacted certificates
will be revoked by 10 PM PST on Tuesday Jan 10th and logged to a Google CT
log. Additional code changes were deployed on Monday Jan 9th and Tuesday
10th to prevent the re-issuance of certificates using cached and potentially
unverified domain validation information. However, prior to identifying and
shutting down this path, an additional 101 certificates were reissued using
such cached and potentially unverified domain validation information,
resulting in an overall total of 8951 certificates that were issued without
proper domain validation as a result of the 

RE: Incident Report – Certificates issued without proper domain validation

[Wayne Thayer]

Responding to Ryan, Nick and Gerv:

> What's unclear is what steps GoDaddy has taken to remedy this.
> 
> For example:
> 1) Disabling domain control demonstrations through the use of a file on a
> server
> 2) Switching to /.well-known/pkivalidation
> 3) Ensuring that the random value is not part of the HTTP[S] request
> 
> etc
> 
> Could you speak further to how GoDaddy has resolved this problem? My
> hope is that it doesn't involve "Only look for 200 responses" =)

Our process for verifying domain control via a change to the website worked by 
generating a random alphanumeric code. The code was then placed in a file in 
the root directory of the website. The structure of the filename is 
.html. Our system queries for that URL and looks for the code in the 
response.

Our initial response as reported yesterday was to fix the bug introduced in 
July. Based on internal discussions and comments here, as of 12 midnight PST 
last night (1/11) we stopped using this method of file based domain control 
validation.

> Has GoDaddy been following ACME
> https://datatracker.ietf.org/wg/acme/charter/ development, either with a
> view to eventually implementing ACME, or just to learn the same lessons
> about automating domain validation ?

We are aware of the work being done on ACME. We’ll continue to watch its 
development and evolution. Our current focus is on implementing the new domain 
validation processes published by the CAB Forum.

> Perhaps the most surprising thing the ACME WG discovered was that due to
> a common misconfiguration customers sharing a bulk host can often answer
> HTTPS requests for other people's sites that haven't for whatever reason
> enabled SSL yet. GoDaddy's validation method as described would be
> vulnerable to this problem. Can you say what, if anything, GoDaddy does to
> avoid being tricked into issuing a certificate on this basis ?

Here is what we were doing to prevent this: When performing the domain control 
check over HTTPS, we validate the certificate – including verifying that the 
domain name of the site matches one in the cert and that the cert chains to a 
root in the Java root store – and the check fails if the certificate does not 
validate.

> As you will know, the method being used by GoDaddy here corresponds
> broadly to method 3.2.2.4.6 from ballot 169 - "Agreed-Upon Change to
> Website". (Although this method is not currently in the Baseline
> Requirements due to it being part of ballot 182 and having a related IPR
> disclosure, at least one root store operator has suggested they are going to
> require strict adherence to the methods listed in that ballot by 1st March.)
> https://cabforum.org/2016/08/05/ballot-169-revised-validation-
> requirements/
> 
> One of the sentences in 3.2.2.4.6 is the following:
> 
> "The entire Required Website Content MUST NOT appear in the request
> used to retrieve the file or web page"
> 
> This sentence is there precisely because the problem which hit GoDaddy was
> anticipated when the Validation WG was discussing the possible problems
> with this validation method.
> 
> Has GoDaddy already, or is GoDaddy planning to, update its implementation
> to conform to that requirement?

GoDaddy was on track to implement the new requirements prior to March 1, but we 
hope to be able to implement them even sooner than that date.

> > We are currently unaware of
> > any malicious exploitation of this bug to procure a certificate for a
> > domain that was not authorized.
> 
> Does that mean "we have revalidated all the domains", or does it mean "no-
> one has actively reported to us that someone else is using a certificate for a
> domain name the reporter owns"?

As soon as we learned of this issue, we went through every certificate that was 
validated with the HTML method utilized during this period and attempted to 
verify. If it could not be immediately verified, we revoked the certificate.

> > The customer who discovered the bug
> > revoked the certificate they obtained, and subsequent certificates
> > issued as the result of requests used for testing by Microsoft and
> > GoDaddy have been revoked.
> 
> I would hope and assume that such testing was done using domains owned
> by Microsoft and/or GoDaddy, or someone else whose permission you had
> gained?

Yes, testing was done using domains registered by Microsoft and GoDaddy 
employees.

> > authorization). We have re-verified domain control on every
> > certificate issued using this method of validation in the period from
> > when the bug was introduced until it was fixed.
> 
> How was that possible for all domains, as surely some domain owners will
> have taken the necessary file down?

When we learned of this issue, we re-validated every affected certificate. If 
we were unable to properly validate, we revoked the certificate. That is how we 
got the total of 8,951 revoked certificates.

> > A list of 8850
> > potentially unverified certificates (representing less than 2% of the
> > total 

Re: Incident Report – Certificates issued without proper domain validation

[Nick Lamb]

On Wednesday, 11 January 2017 18:31:32 UTC, Ryan Sleevi  wrote:

> Because you're not required to setup the webserver for
> smtp.example.com.

Ah yes, silly me

> I'm not saying they'd be right in arguing so

I suppose that from a husbanding of resources point of view it makes sense to 
wait and see what happens to the remnants of ballot 169 before Mozilla jumps up 
and down about this.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Jakob Bohm]

On 11/01/2017 19:17, Patrick Figel wrote:

On 11/01/2017 19:06, Nick Lamb wrote:

For those who haven't (unlike Patrick) sat down and read the ACME
specification, ACME http-01 won't get tripped here because the
checked content of the URL is very much not the random string (it's a
JWS signature over a data structure containing that random string,
thereby proving it was made by whoever the ACME server is talking
to). But yes, doing something that _looks_ superficially like the
ACME style of validation without such subtlety will trip you up.


Thanks, that's a better way of phrasing what I was trying to say. ;-)


In this very particular case, where the affected validation was
specific to web servers, it seems extremely likely that almost all of
the legitimate certificates (which may be, and we hope is, all of
them) were subsequently put into use on a web server.

Why go to the bother of setting up a web server on say,
smtp.example.com, only to get yourself a certificate, and then turn
off the web server and use the certificate for SMTP? It's not
impossible, but it would be very much the exception.


I don't know this specifically for GoDaddy, but many commercial CAs I've
dealt with in the past typically only validate the "registered domain"
portion (e.g. example.com) of the FQDN and then give you certificates
for any subdomain (e.g. smtp.example.com) under that domain. I think the
approach used by Let's Encrypt, where each FQDN has to be validated
individually, is not all that common otherwise.





For your information, here are the ones I have encountered:

AlphaSSL (Globalsign sub) and one other (a Symantec sub, as far as I
recall) verify control of the corresponding hostmaster e-mail address
for the second level domain (hostmas...@example.com).

Google (for non-certificate purposes) used to verify a url that just
had to return a string that said "google-site-verification: URL" where
URL was the file name part of the Url, this may or may not have been
foolable.  This was done per exact domain.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Ryan Sleevi]

On Wed, Jan 11, 2017 at 10:06 AM, Nick Lamb  wrote:
> Why go to the bother of setting up a web server on say, smtp.example.com, 
> only to get yourself a certificate, and then turn off the web server and use 
> the certificate for SMTP? It's not impossible, but it would be very much the 
> exception.

Because you're not required to setup the webserver for
smtp.example.com. It's sufficient to setup the webserver for
example.com to authorize the name, by creatively interpreting the
Method 7 (prior to Ballot 169) and applying the logic from Method 4 to
suggest it's OK to prune the domain (despite Method 6 not allowing
this).

I'm not saying they'd be right in arguing so, but they wouldn't be the
only CA who applied such an interpretation.
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Paul Wouters]

On Wed, 11 Jan 2017, Patrick Figel wrote:


On 11/01/2017 04:08, Ryan Sleevi wrote:

Could you speak further to how GoDaddy has resolved this problem? My
hope is that it doesn't involve "Only look for 200 responses" =)


In case anyone is wondering why this is problematic, during the Ballot
169 review process, Peter Bowen ran a check against the top 10,000 Alexa
domains and noted that more than 400 sites returned a HTTP 200 response
for a request to
http://www.$DOMAIN/.well-known/pki-validation/4c079484040e32529577b6a5aade31c5af6fe0c7
[1]. A number of those included the URL in the response body, which
would presumably be good enough for GoDaddy's domain validation process
if they indeed only check for a HTTP 200 response.

[1]: https://cabforum.org/pipermail/public/2016-April/007506.html


Are you saying that for an unknown amount of time (years?) someone could
have faked the domain validation check, and once it was publicly pointed
out so everyone could do this, it took one registrar 10 months to fix,
during which 8800 domains could have been falsely obtained and been used
in targetted attacks? Have other registrars made any statement on
whether they were or were not vulnerable to this attack?

Is there a way to find out if this has actually happened for any domain?
I would expect this would show up as "validated" certificates that were
logged in CT but that were never deployed on the real public TLS servers.
Is anyone monitoring that? I assume that for the "big players" who do
self-monitoring, were not affected? *crosses fingers*

Paul
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Patrick Figel]

On 11/01/2017 04:08, Ryan Sleevi wrote:
> Could you speak further to how GoDaddy has resolved this problem? My
> hope is that it doesn't involve "Only look for 200 responses" =)

In case anyone is wondering why this is problematic, during the Ballot
169 review process, Peter Bowen ran a check against the top 10,000 Alexa
domains and noted that more than 400 sites returned a HTTP 200 response
for a request to
http://www.$DOMAIN/.well-known/pki-validation/4c079484040e32529577b6a5aade31c5af6fe0c7
[1]. A number of those included the URL in the response body, which
would presumably be good enough for GoDaddy's domain validation process
if they indeed only check for a HTTP 200 response.

[1]: https://cabforum.org/pipermail/public/2016-April/007506.html
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Compliance with 7.1.4.2.1 (internal names revocation)

[Rick Andrews]

We conducted a search of our databases in September 2016, in which we examined 
every CN and SAN in every certificate still valid at the time. Each CN and SAN 
was examined to see if it contained no dot or an invalid DNS suffix; if so, the 
certificate was classified as an internal server cert and revoked. For all 
remaining CNs and SANs, those were checked against our internal list of TLDs 
built from information provided by ICANN and IANA. That list had a status value 
associated with each TLD, and our mistake was in excluding TLDs with certain 
status values.

Our scans conducted this week discovered three additional certificates that had 
not been revoked as of October 2016. These, and the certificate discovered by 
Nick, have now been revoked. Here are the links to those certificates:

https://crt.sh/?sha256=A642406A2BDF92DF8C9FB9322A81736506DDED79A20A7CD33CBEFD2AD2581167
https://crt.sh/?sha256=12B3CCC45D66B9CB2206DEF1C5A24B062CCC938694C92A0806D1D34845C0FC19
https://crt.sh/?sha256=E90AFAE4998D2B8103058ADF35810D87CCE5E98A0E1D691D2A558A6A4E115BAC

Thanks again to Nick for discovering this and pointing it out.

___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Gervase Markham]

Hi Wayne,

As others have said, thanks for bringing this to our attention.

On 11/01/17 03:02, Wayne Thayer wrote:
> results even when the HTTP status code was not 200. Since many web
> servers are configured to include the URL of the request in the body
> of a 404 (not found) response, and the URL also contained the random
> code, 

As you will know, the method being used by GoDaddy here corresponds
broadly to method 3.2.2.4.6 from ballot 169 - "Agreed-Upon Change to
Website". (Although this method is not currently in the Baseline
Requirements due to it being part of ballot 182 and having a related IPR
disclosure, at least one root store operator has suggested they are
going to require strict adherence to the methods listed in that ballot
by 1st March.)
https://cabforum.org/2016/08/05/ballot-169-revised-validation-requirements/

One of the sentences in 3.2.2.4.6 is the following:

"The entire Required Website Content MUST NOT appear in the request used
to retrieve the file or web page"

This sentence is there precisely because the problem which hit GoDaddy
was anticipated when the Validation WG was discussing the possible
problems with this validation method.

Has GoDaddy already, or is GoDaddy planning to, update its
implementation to conform to that requirement?

> We are currently unaware of
> any malicious exploitation of this bug to procure a certificate for a
> domain that was not authorized. 

Does that mean "we have revalidated all the domains", or does it mean
"no-one has actively reported to us that someone else is using a
certificate for a domain name the reporter owns"?

> The customer who discovered the bug
> revoked the certificate they obtained, and subsequent certificates
> issued as the result of requests used for testing by Microsoft and
> GoDaddy have been revoked.

I would hope and assume that such testing was done using domains owned
by Microsoft and/or GoDaddy, or someone else whose permission you had
gained?

> authorization). We have re-verified domain control on every
> certificate issued using this method of validation in the period from
> when the bug was introduced until it was fixed.

How was that possible for all domains, as surely some domain owners will
have taken the necessary file down?

> A list of 8850
> potentially unverified certificates (representing less than 2% of the
> total issued during the period) was compiled at 10 PM PST on Monday
> Jan 9th. 

How were you able to create that list? Do you store the HTTP status code
and content returned from the website, and just searched for non-200
codes? Or some other way?

Gerv
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Re: Incident Report – Certificates issued without proper domain validation

[Nick Lamb]

As Ryan said, thanks for informing m.d.s.policy about this issue. I am 
interested in the same general area as Ryan but I will ask my question 
separately, feel free to answer them together.

Has GoDaddy been following ACME https://datatracker.ietf.org/wg/acme/charter/ 
development, either with a view to eventually implementing ACME, or just to 
learn the same lessons about automating domain validation ?

Perhaps the most surprising thing the ACME WG discovered was that due to a 
common misconfiguration customers sharing a bulk host can often answer HTTPS 
requests for other people's sites that haven't for whatever reason enabled SSL 
yet. GoDaddy's validation method as described would be vulnerable to this 
problem. Can you say what, if anything, GoDaddy does to avoid being tricked 
into issuing a certificate on this basis ?
___
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy