Resent without a signature....


Hi everyone,



This email is intended to gather public and browser feedback on how we are 
handling the transitioning Verizon's customers to DigiCert and share with 
everyone the plan for when all non-DigiCert hosted sub CAs will be fully 
compliant with the BRs and network security guidelines. Primarily, this letter 
addresses when all issuing CAs previously held by Verizon will be covered by an 
unqualified audit and how we are responding to the sub CAs that issued SHA1 
certificates. We are looking forward to the browser and public feedback on how 
to handle the non-compliant cross-signs and insight on how the public views our 
transition progress and planned completion dates.



Background

Prior to presenting the plan for remediation, I thought I'd share a bit about 
our progress with migrating Verizon customers.  About a year ago in July, 
DigiCert closed an agreement with Verizon where DigiCert took ownership of 
three publicly-trusted Verizon root certificates. These certificates are the 
Verizon Global Root CA, the Baltimore CyberTrust Root CA, and the Verizon Root 
CA. There were many reasons the acquisition made sense, including acquisition 
of a root that had cross-signed DigiCert for many years. The ubiquity of the 
Verizon roots is wide-spread, which meant a lot of CAs like to cross-sign with 
them. Another significant reason for the acquisition was an attempt to improve 
the CA industry. After watching the issuance of wildcard EV certs, 
non-compliant subordinate CAs, and certs with poor profiles, we made a 
conscious decision to purchase these roots with the intention of migrating them 
to more complaint system. We felt that helping these CAs get to the point of
  regular audits and best practices would raise the quality of the entire 
industry.



Prior to the acquisition, we were made aware of several potential 
non-compliances by Verizon's customers. We worked closely with Verizon to clean 
up many of the Sub CAs, including revoking CAs that would never be compliant 
with the requirements and attempting to technically constrain others.  Sub CAs 
were revoked because they either didn't have an audit, were unresponsive to 
communication, or couldn't control their issuance. Verizon was a great partner 
and took a very proactive approach in removing CAs that were not actively 
working towards compliance. Unfortunately, the age of the roots and large 
number of cross-signs led to a lot of missing paperwork and certain issues in 
identifying which CAs were covered by audits. Prior to closing we believed 
there were approximately five technically constrained sub CAs and around 20 
unconstrained sub CAs. Turns out there were actually more than 200, each with 
various levels of compliance. Most of these Sub CAs operated under the Baltimo
 re Cybertrust root, which was created in 2000.



To their credit, Verizon revoked 48 of the issuing CAs prior to DigiCert's 
acquisition of the roots. Unfortunately, this left around 150 for our small 
team to work through. Although the endeavor was daunting, Ben Wilson and others 
worked to gather audit reports, email customers about compliance dates, monitor 
certificates issued, and revoke non-compliant customers. Verizon was very good 
at assisting us in these efforts. This information is now recorded in the 
Mozilla database and continuously updated as the information changes.



Transition Process

After our operational acquisition of the roots (which occurred the last part of 
September, 2015), we identified 15 expired issuing CAs, 70 revoked sub CAs, 131 
audited sub CAs, and 36 where the status was unknown. Since then, we've revoked 
25 issuing CAs and assisted others with technical constraints, exodus from the 
Omniroot cross-signing program, or obtaining audits. Of the existing sub CAs, 
about half remain operational and are either audited or technically 
constrained. The other half are either winding down or have been revoked. All 
revoked certificates were disclosed to Mozilla via Salesforce and should be 
part of OneCRL.



Issuing CAs

There are only a handful of non-audited sub CAs remaining (see 
https://crt.sh/mozilla-disclosures#disclosureincomplete). We have a plan for 
each of them that we'd like to share with you. We welcome both browser and 
public feedback. This is, of course, simply a proposal on how to finish the 
transition and provide transparency into where we are at. We are certainly 
willing to modify the plan as needed to further online security and meet the 
requirements of the root store operators.



The seven companies listed below are responsible for the remaining unaudited 
sub CAs:



1.       ABB has three unaudited issuing CAs. ABB didn't undergo an audit 
earlier this year on the assumption that their sub CAs were technically 
constrained. Unfortunately, the constraints weren't properly implemented, a 
fact that escaped notice until Rob Stradling's excellent tool exposed a 
deficiency a few months ago in how Verizon issuance process. Although Verizon 
restricted domain names and IPv4 IP addresses, the CAs could still issue for 
the IPv6 space.  Despite promptly replacing the certificates after discovering 
the gap, we haven't revoked the issuing CAs. Right now, ABB is actively working 
to transition its servers to the new, properly-constrained certificates. We 
expect the transition to complete by the end of December 2016. We are revoking 
the issuing CAs as the migration completes and expect all of the ABB 
non-compliant issuing CAs to be revoked on January 5th.  No new certs are being 
issued from the faulty issuing CAs.



2.     Similar to ABB, Verizon created two technically constrained issuing CAs 
for Bechtel. Like ABB, Verizon failed to restrict IPv6 issuance for each of the 
issuing CAs. Bechtel is actively migrating the remaining 785 certificates to 
properly constrained issuing CAs. The migration process will complete at the 
end of December, after which the non-conforming issuing CAs will be revoked. 
They are included in our January 5th revocation plan. This will also revoke the 
three issuing CAs created further down in the chain.



3.     Dell also thought they were technically constrained. Similar to ABB and 
Bechtel, the restriction was inadequate. Fortunately, Dell isn't using the 
issuing CAs and doesn't require migration. We've previously revoked four 
issuing certificates (Bug 1279066) but missed one during the process. We plan 
to immediately revoke the remaining issuing CA and all CAs subordinate to the 
issuing CA.



4.     While Certipost is hosted by Verizon's internal infrastructure, the 
Verizon WebTrust audit letter did not specifically identify the Certipost CAs 
as audited. DigiCert has been in the process of revoking these CAs but there is 
some confusion about the effects of the revocation. Apparently the certificates 
are used for air traffic control in Europe and there is a fear that revoking 
the intermediate may cause planes to fall out of the sky. After trying to sort 
out this mess and giving Certipost time to transition, we've decided to revoke 
the issuing certificates the last week of November.  We believe there are over 
400 outstanding certificates that will be affected.



5.     Postecom refused an audit and decided to exit operation of a CA. We have 
not yet revoked their issuing CAs in order to give them time to migrate to a 
different CA infrastructure. They asked that we keep the issuing CAs active 
until December 31, 2016 to complete the migration. Although they have worked on 
migrating certs, Postecom still has 1185 certificates to migrate in the next 
two months. The issuing CAs are scheduled for revocation on January 5th.



6.     Vodafone completed a Webtrust audit this year. Unfortunately, we found 
out that the audit has qualifications.  The audit report hasn't issued yet but 
should later in November. Once the audit report issues, we will provide a copy 
of the report for evaluation. We've been told that the qualifications are 
related to the network security guidelines. They probably also include the SHA1 
issuance, although we haven't been told that directly. Vodafone is currently 
designing and building a new CA infrastructure that will house all of their 
operations. To ensure compliance, we are requiring the new infrastructure to 
undergo a point-in-time audit in December. We are also requiring a full audit 
on their issuance processes at the start of next year. A full, non-qualified 
audit report is expected by March 31, 2016. Although this is well past the 
compliance date, we are sympathetic to Vodafone as they have been actively 
working towards compliance. They also have 4,580 public facing SS
 L certs makes revocation tricky. They issue a lot of client certificates from 
these sub CAs where revocation could have a potentially devastating effect on 
telecommunications. Because of the large impact of revocation and how hard 
Vodafone is working to migrate, we'd like permission for them to comply by 
March 31, 2017, assuming serious issues are not uncovered by the audit report 
and that the December and January audits pass unqualified.



7.       The Belgium government is our biggest challenge in migrating Verizon 
customers. With over 20 issuing CAs, Belgium has the largest outstanding 
non-compliant infrastructure. The operators have also claimed that revoking 
their issuing CAs is illegal (in Belgium). The government is using the issuing 
CA for creating personal identification (e-ID) cards throughout the country. 
The Belgium government has dictated that they set the rules, not us. Although 
the Belgium government does not have an audit yet, Verizon has represented that 
the issuing CAs are hosted in the Verizon infrastructure and are potentially 
covered by the Verizon audit. We've asked Verizon to provide an updated audit 
report showing coverage of the Belgium issuing CAs by December 1, 2016. If the 
report is not delivered by December 1, 2016, we plan to immediately revoke the 
issuing CAs.  If, for whatever reason, we are unable to revoke the issuing CAs 
at that time, we would  certainly not object to the brows
 ers distrusting the issuing CAs issued to Belgium.



Resolving these remaining seven issues will, bring all Verizon's sub CAs into 
compliance with the audit requirements by March 31, 2017. As of January 2017, 
all sub CAs operated under the Verizon umbrella will have a non-qualified audit 
with the exception of Vodafone, which will have a non-qualified point-in-time 
audit by then.



SHA-1

Although by March 31, 2017, all former Verizon customers will comply with the 
audit portion of the BRs, experience has recently shown that strict compliance 
of profile requirements is more difficult to enforce. Such is the case of the 
SHA1 certificates issued by Nets Norway, Siemens, and Vodafone. Nets Norway and 
Siemens were both ETSI audited while Vodafone opted for a Webtrust audit.



Shortly after the Verizon transaction, we sent a statement to each CA 
announcing the acquisition and our expectation of strict compliance with the 
BRs and network security requirements. On January 27, 2016, we sent another 
email to all CAs that reiterated the need for strict adherence to standards and 
requirement for CAs to update their infrastructure as needed to maintain an 
unqualified audit. The specifically stated that issuance of internal name 
certificates and SHA1 certificates is strictly forbidden. Nets Norway, 
Vodafone, and Siemens all received a copy of this email.



As all the sub CAs should have been aware of the requirements in 2015, were 
reminded this year, and had audits of their infrastructure, we are very 
disappointed by the recently discovered SHA-1 issuance. We've contacted each of 
the sub CAs asking for an explanation on what happened and why their 
infrastructure wasn't sufficient to prevent SHA1 issuance.



Of the SHA1 cert issuers, Vodafone is the only sub CA that responded by 
providing a timeline for compliance, information about how they are migrating 
to new CA (which we already knew), and a plan going forward to undergo an audit 
and make sure technical constraints are active to prevent additional SHA1 
certificates. Siemens replied that they are revoking the SHA1 certificates, 
that issuance was a mistake, and that they are taking action to prevent 
additional mis-issuance. However, the specifics of their plan are vague (other 
than providing operational training to prevent reoccurrence). For both Vodafone 
and Siemens, this is their first offense in mis-issuing a SHA1 certificate, and 
we'd ask for leniency. We do recognize that this is a serious concern and can 
revoke the issuing CAs if the action is decided necessary. For full disclosure, 
Siemens is transitioning away from the Verizon root certificate to another CA. 
We are only maintained the validity of their cross-sign in a g
 ood faith effort to help them complete the transition. We were planning to 
revoke the issuing CAs early next year (January 5th) once the transition is 
completed.



The final CA is Nets Norway. Unlike the other two issuing CAs, Nets Norway 
issued a SHA1 certificate earlier this year. Although Nets Norway promptly 
revoked the certificate and provided verbal assurances that SHA1 issuance would 
be restricted, the company obviously failed to take adequate measures. We 
informed Nets Norway that we are revoking their issuing CA later this month. We 
haven't revoked them yet because we wanted to complete the incident report 
first and allow public comment before finalizing the decision.



Our investigation of Verizon's, Postecom's, and Telecom Italia's issuance of a 
SHA1 certificate is ongoing. As mentioned above, Postecom is scheduled for 
revocation. Our plan for now is to keep the current revocation schedule for 
Postecom as the SHA1 issuance occurred twice during January.  We haven't formed 
a plan for Verizon or Telecom Italia yet and are waiting to hear back from 
their representatives. Verizon issued a single certificate back in January. 
Telecom Italia issued three in early January and February. Each of these 
incidents were only recently discovered by DigCert. We've since added an active 
monitor to crt.sh to alert our compliance team if a SHA1 certificate issues.



Conclusion

Hopefully this helps explain the status of the Verizon transition and the 
recent SHA1 incidents. We are still collecting information from the three 
issuing sub CAs about what happened, but I hope this is enough to provide 
transparency and make a decision on how to proceed.



Despite the difficulties in bringing these customers into compliance, we still 
think the Verizon transaction was a good move from a compliance perspective. 
Helping these sub CAs identify problems and improve their processes has been an 
enlightening experience and given us insight on some of the difficulties other 
CAs may face in complying with the BRs. The process has been interesting 
exercise in helping others make remediation plans and trying to accelerate what 
feels like a slow-moving project. Despite the long nights, headaches, and 
frustrations, I think we've been successful in our original goal of improving 
the security ecosystem. I'm also very pleased that the end of the transition is 
almost here!



We appreciate your feedback and look forward to your comments.



Sincerely,

Jeremy











_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

Reply via email to