All,
Thanks again to all who contributed your expertise and resources in investigating and commenting on the e-Tugra breach, as described in a blog post by Ian Carroll <https://ian.sh/etugra> and in subsequent discussions here on the Mozilla dev-security-policy list <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ>, and on the CCADB Public list <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ>, and in Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>. Summary We have determined that the combination of concerns about e-Tugra, which we previously summarized <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/C-HrP1SEq1A/m/b5ccAO5fAwAJ>, indicates that the risk of keeping the current e-Tugra root certificates in Mozilla’s root store outweighs the benefits to our users. Adequate change-control processes were not in place, and non-CA systems with connections to CA systems were not being adequately protected. The security holes that were initially reported were concerning in themselves, and alone they might not have led us to our decision, but these were compounded by e-Tugra’s inadequate responses, as recorded in Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>. Incident Response As stated on the Maintenance and Enforcement <https://wiki.mozilla.org/CA/Maintenance_and_Enforcement#Potential_Problems.2C_Prevention.2C_Response> wiki page, we want to see that: - A CA takes responsibility for their own actions. - Incidents are taken with an appropriate level of seriousness. - Incidents are handled with urgency. - Root cause analysis is performed. - Any questions posed, by anyone, are answered quickly and in detail. - A reasonably-detailed incident report <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report> is made public on what happened, why, and how things have changed to make sure it won’t happen again. e-Tugra’s responses fell short of these expectations. Incident Report Our guidance on incident reports <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report> (that has recently moved to the CCADB website <https://www.ccadb.org/cas/incident-report#incident-reports>) states: “The incident report should explain how systems or processes failed, how the mis-issuance or incident was made possible, and why the problem was not detected earlier. In addition to the timeline of responding to and resolving the incident, the incident report should explain how the CA Owner’s systems or processes will be made more robust, and how other CAs may learn from the incident.” e-Tugra’s incident report ( https://bugzilla.mozilla.org/attachment.cgi?id=9330825) was not reasonably detailed, downplayed the seriousness of the incident, and did not provide details on remediation steps. For instance, we requested a timeline of the events that occurred before November 2022, but e-Tugra only responded with “Oct 10, 2022: An Internal application was published in Internet Zone.” The incident report did not contain a comprehensive root cause analysis, and e-Tugra did not explain how things have changed to ensure incidents like these won’t happen again. We expected more detailed information and a stronger sense of responsibility from e-Tugra. Urgency e-Tugra’s slow and inconsistent responses during the course of the investigation did not portray appropriate urgency. e-Tugra’s representatives initially said that there were processes in place to conduct penetration testing and vulnerability assessments of its systems. They said that penetration testing would start on November 18, 2022. They also said that information about the penetration test would be shared as soon as it became available. On December 12, we requested that information be provided by January 6, 2023, but it wasn’t until February 2, 2023, (after prompting) that e-Tugra admitted that they were still getting started with the penetration testing company. On February 28 we reminded e-Tugra to provide weekly updates. The first information on penetration testing was not made available until April 2, 2023, and then the executive summary provided only a few brief statements. The executive summary from a follow-up report also lacked sufficient detail. e-Tugra indicates that everything will be provided to its auditors during its next audit. However, this highlights not only the lack of urgency with which this has been viewed by e-Tugra, but also the lack of information that we have to objectively assess whether meaningful change has taken or will take place at e-Tugra. Proficiency e-Tugra’s statements during the course of the investigation were inconsistent and in some cases contradictory. e-Tugra representatives downplayed the severity of the incident by responding, “Only three documents related to some users' (non SSL) agreements are accessed by the reporter” and “No sensitive information disclosure revealed in this issue”. They also said, “no sensitive information is logged in the logging system”. However, these statements were refuted by the fact that the contents of 568,647 emails and 251,230 documents were accessible, including access to password resets and confirmation codes. e-Tugra representatives first said there was “no connection” with CA systems and then said that there was a "one-way flow". Then they said that their customer portal would allow reissuance for already-validated domain validation certificates, but that the accessed application had no impact on the process. This was contradictory because the password-reset function could have been compromised with information in the log to change a user’s password. Root Removal The responses from e-Tugra’s representatives demonstrate that this CA lacks the expertise needed to operate a sufficiently secure environment for a publicly trusted CA. As such, we will remove the following root certificates in our next batch of changes to Mozilla’s root store: E-Tugra Global Root CA RSA v3 SHA-1 Fingerprint: E9A85D2214521C5BAA0AB4BE246A238AC9BAE2A9 SHA-256 Fingerprint: EF66B0B10A3CDB9F2E3648C76BD2AF18EAD2BFE6F117655E28C4060DA1A3F4C2 E-Tugra Global Root CA ECC v3 SHA-1 Fingerprint: 8A2FAF5753B1B0E6A104EC5B6A69716DF61CE284 SHA-256 Fingerprint: 873F4685FA7F563625252E6D36BCD7F16FC24951F264E47E1B954F4908CDCA13 Note: We will also be removing the expired “e-Tugra Certification Authority” root certificate, as previously planned. Follow-up e-Tugra may continue to be a reseller to another CA operator with root certificates in Mozilla’s root store, as long as that other CA operator ensures correct systems, procedures, and processes in relation to reselling certificates within their CA hierarchy, and all domain validation and certificate issuance is performed by that other CA whose root certificates are included in Mozilla’s root store. e-Tugra may not become an externally-operated subordinate CA <https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy#84-externally-operated-subordinate-cas> chaining up to a root certificate in Mozilla’s root store, and in order for e-Tugra representatives to apply to have new root certificates included in Mozilla’s root store in the future, they would need to: - Provide sufficiently detailed artifacts and testing to demonstrate that they have implemented an infrastructure with processes in place to ensure that systems are rigorously secured and thoroughly hardened. - Create new root certificates within the properly configured/secured infrastructure. - Re-apply for inclusion in Mozilla’s root store, and go through Mozilla’s full root inclusion process <https://wiki.mozilla.org/CA/Application_Process>. Thanks, Ben and Kathleen On Mon, Jun 5, 2023 at 11:36 AM Ben Wilson <bwil...@mozilla.com> wrote: > Dear Mozilla Community, > > This email relates to the e-Tugra breach that was described in a blog > post by Ian Carroll <https://ian.sh/etugra> and subsequent discussions > here > <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ> > and in CCADB Public > <https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/yqALPG5PC4s/m/sIkv6eLAAAAJ> > and Bugzilla <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>. We > are grateful for the involvement of various individuals and > organizations, particularly Ian Carroll and Google Chrome, who have > contributed their expertise and resources while investigating this breach. > > We are now opening this discussion to help determine whether the e-Tugra > root certificates should be removed from Mozilla’s Root Store. We will > greatly appreciate your thoughtful and constructive feedback on this. > > Below are some questions for us to consider in this discussion. > > What were the main concerns raised by the community during the discussions > that took place in Bugzilla Bug #1801345 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1801345>? (this is not a > complete list; details may be found in the bug) > > - > > Mr. Carroll indicated that he was able to log in and conduct > reconnaissance on e-Tugra’s email and document storage systems, gaining > access to customer PII. > - > > There were security holes in e-Tugra’s internal systems that > existed because access to internal resources was not adequately secured, > namely, default passwords on some administrative tools had not been > changed, and internal applications were left exposed without appropriate > access control mechanisms. > - > > Statements by e-Tugra about the lack of impact were refuted by the > fact that the contents of 568,647 emails and 251,230 documents were > accessible, including access to password resets and confirmation codes. > - > > e-Tugra indicated in their incident report that they would conduct > additional vulnerability scanning and penetration testing before the end of > 2022. They also said that they were preparing a detailed report of the > problem and would provide more information. > - > > None of this information was provided until April 2023. > - > > The explanations in subsequent reports and executive summaries of > penetration testing did not fully answer the questions that had been > asked. > - > > A detailed analysis was not provided, no root cause analysis was ever > provided, and the output of the penetration test (i.e., reports) did not > offer any detail addressing concerns raised in the incident report. (The > penetration test reports only provided brief statements and insufficient > information to evaluate the security posture of e-Tugra.) > > Does the benefit of having e-Tugra root certificates in Mozilla’s Root > Store outweigh the risk? > > - > > The breach that was found by Mr. Carroll may indicate that the CA > operators may not have the level of expertise necessary to securely operate > a publicly trusted CA. > - > > The slow response by e-Tugra may also be an indication that they may > not be prepared to respond to concerns and incidents in the required timely > manner. > > How would the distrust of e-Tugra affect our users? > > - > > Since the e-Tugra root certificates are not currently being used, > there would not be any impact to our users. > - > > e-Tugra is not currently issuing TLS certificates within their own CA > hierarchy. Instead, e-Tugra has been acting as a reseller for the SSL.com > CA, which means that all domain validation and certificate issuance is > being performed by systems operated by the SSL.com CA, so SSL.com is the > party currently responsible for issuance/security procedures. > > If distrusted, could e-Tugra continue to be a reseller for a publicly > trusted CA? > > - > > Yes, as long as the publicly trusted CA ensures correct > systems/procedures/processes in relation to reselling certificates within > their CA hierarchy. > > If distrusted, could e-Tugra re-apply to have their root certificates > included in Mozilla’s Root Store? > > - > > If we decide to remove the current e-Tugra roots, then in order for > e-Tugra to apply to have their root certificates included in Mozilla’s Root > Store again, they would need to: > - > > Implement an infrastructure that is properly secure and tested, > with sufficiently detailed artifacts to show that the systems are > properly > configured. > - > > Then create new root certificates within the properly secured > infrastructure. > - > > Then re-apply for inclusion in Mozilla’s root store. > - > > As always, the onus is on the CA operator to demonstrate their ability > to operate a publicly trusted CA hierarchy. > > How are other root stores addressing this issue? > > - > > Google Chrome announced > > <https://groups.google.com/a/ccadb.org/g/public/c/SXAeHT04TFc/m/TIh1ANmHAwAJ> > that they will remove the “E-Tugra Global Root CA ECC v3” and “E-Tugra > Global Root CA RSA v3” roots from the Chrome Root Store. > - > > Apple stated > > <https://groups.google.com/a/ccadb.org/g/public/c/SXAeHT04TFc/m/B_snEIWZAwAJ> > that the e-Tugra roots are not in the Apple Root Store. > > Given that SSL.com has been the operator of the issuing CA, e-Tugra’s > breach has not necessitated that we take any urgent safeguarding actions. > However, e-Tugra’s problems give us reason to question whether the e-Tugra > CA should continue to be in our root store, based on the level of expertise > required to securely operate a publicly trusted CA infrastructure. > Additionally, e-Tugra’s inadequate responses to the problem and resulting > requests for information, give us reason to question e-Tugra’s commitment > to the important role that CAs have in ensuring security on the web. > > We appreciate your engagement and dedication, and we kindly request that > all participants adhere to our Community Participation Guidelines > <https://www.mozilla.org/en-US/about/governance/policies/participation/>, > fostering a respectful and constructive environment for dialogue. > > Thanks, > > Ben and Kathleen > > PS: For your reference, below are excerpts from > https://wiki.mozilla.org/CA/Maintenance_and_Enforcement > > CA behavior > > - > > Did the CA notice the error and take appropriate action in a timely > manner? > - > > Was the threat or error properly contained? > - > > Did the CA notice the hacker intrusion and take appropriate action in > a timely manner? > - > > Were the CA's defense mechanisms current and sufficient? > - > > Was the intrusion appropriately contained? > - > > Was the CA's behavior reckless? > - > > Did the CA try to cover up the mistake/threat rather than follow > Mozilla's Security Policy? > > In incident response, Mozilla is looking for the following factors: > > - > > A CA takes responsibility for their own actions. > - > > Incidents are taken with an appropriate level of seriousness. > - > > Incidents are handled with urgency. > - > > Root cause analysis is performed. > - > > Any questions posed, by anyone, are answered quickly and in detail. > - > > A reasonably-detailed incident report > <https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report> > is made public on what happened, why, and how things have changed to make > sure it won’t happen again > > > -- You received this message because you are subscribed to the Google Groups "dev-security-policy@mozilla.org" group. To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsubscr...@mozilla.org. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaYePvZFz3evfd0zPd1LvB%3DiWaW%2B%3DsQ4RgSb_5jAKBx9Kw%40mail.gmail.com.
