I apologize for the length, but I didn't find places where I could remove things that were not IMO material to Mozilla's evaluation of Entrust as a root, or constructive advice that I genuinely wish Entrust to incorporate into their plans. Thoughts on Entrust’s revised report
I'm not going to quote and cite individual missteps extensively, but if I'm misrepresenting something I'm happy to dig up the things that I used as inputs, and make corrections. Instead, I'm trying to summarize some themes that continue to concern me even after Entrust's updated report and responses. In summary, while I think Entrust was wise to reuse the Navex guide to Compliance Program assessment, I don't feel they have done so in a way that sufficiently allays concerns about their willingness and indeed ability to operate within the BRs and MRSP, consistently and in good faith. Looking critically at how Entrust has described its actions and decisions shows, in my opinion, not only a company that has been unable to meet the expectations to which CAs must be held, but indeed a company that either doesn't understand or doesn't accept what those expectations are. Apportionment of Risk and Responsibility Entrust continues to be mistaken about their responsibilities in the WebPKI and MRSP. They say that the subscriber cannot be held responsible for operational limitations on certificate deployment. On the contrary, only the subscriber can be responsible for their own operations, within the CA/root-program/Subscriber/relying-party dynamic of WebPKI. Only the subscriber can ultimately make changes to their own operations, and they will bear the cost. Entrust as a security vendor can take on some responsibility for enabling better Subscriber operations, but their degree of success with that should not bear positively or negatively on the evaluation of Entrust as a CA. They also describe being in "tension" between WebPKI and critical systems but omit that this tension is entirely of their own continuous creation, versus reducing, since 2020, the cases in which they encourage or permit subscribers to use web PKI certificates in circumstances incompatible with the operational properties of the web PKI. Entrust's action items for delayed revocation often have elements of encouraging subscribers to adopt automation. I think that is a fine thing for them to do, but IMO it should not be considered to be a remediation for a delayed revocation incident, because it's not up to subscribers to decide on delayed revocation. To treat it as remediation is to say that subscriber choices to invest (or not) in improvements to their certificate management operations are allowed to prevent a CA from upholding the requirements. The message needs to be "you might want to improve your operations so you don't have an outage the next time we need to revoke one of your certificates, which we will do on time", not "please improve your operations so that we are able to revoke the certificate we chose to issue you". The Improvement Plan says that they will "work with our subscribers to ensure awareness and minimize delayed revocation requests". This language echoes the 2020 commitments, but does not give any meaningful description of a future state they’re seeking. I don't care how many delayed revocation requests they get. That's a function of whether they choose to issue and reissue Web PKI certificates into environments that they know "cannot" actually tolerate immediate revocation in spite of the subscriber's legal commitment to the contrary. Entrust can't solve this problem by reducing the number of times they get asked for exceptions. They have to ensure that they have appropriately narrow exception criteria regardless of how many requests they get . Are we to believe that if Entrust gets more requests in the future, they will permit more exceptions? It's hard to understand why that is relevant to their improvement except that it might cause fewer customers to be disappointed or frustrated; that would be an improvement to Entrust's experience as a vendor, but not material to how Entrust or its Subscribers interact with the web PKI. When Entrust decides that it must delay revocation due to intolerable impact on the web ecosystem (paging the Definitions & Glossary WG), proper remediation actions would include an enforced timeline for the subscriber being able to tolerate prompt revocation, possibly by being moved off of web PKI certificates. After that timeline, delayed revocation should no longer be permitted; if Entrust doesn't want to be in that situation, then they should not re-issue to that subscriber. As recently as June 21, presumably with the knowledge that Entrust was admitting that its practices had been deficient in the updated report, a representative of Entrust was still defending their previous decisions to delay revocation. "If the strict enforcement of rules begins to take priority over the facilitation of safe and smooth internet transactions, it brings discredit on the entire ecosystem. Entrust has been trying to avoid that, by showing a nuanced understanding of the complex issues faced by subscribers." I disagree with most of that, but that hardly matters. It's a sign that Entrust has not actually accepted that their behaviour was inappropriate. Notably, there is no commitment from Entrust to avoid future issuance to systems that will not tolerate prompt revocation. They have stated, "for the record", their position that any CA should be free to issue certificates to subscribers who they know will not be able to tolerate the BRs being upheld, in terms of prompt revocation. This would give CAs license to knowingly create situations where they will need "exceptional" delays to revocation. That must be held to be incompatible with the CA then delaying revocation due to the situation they helped create, or else we find ourselves in a situation that risks rendering meaningless the BR's timelines and subscriber commitments. (It would also put conformant, good-faith CAs at a commercial disadvantage relative to those who give more weight to customer convenience. While I do not think that financial considerations for CAs should hold much weight in policy making here, incentives are real and we should be careful about creating incentives for undesirable behaviour.) Entrust claims that they "generally recommend that subscribers have a backup CA", but I can't find that recommendation anywhere in Entrust's public materials. And again, Entrust opposes making this an industry standard, though IMO it would be a valuable partial mitigation against misuse of web PKI certificates in contexts which cannot tolerate web PKI revocation rules and timelines. The "Policy Updates" section of the improvement plan says that they are "considering ways to increase visibility of the CA’s right to revoke certificates on short notice beyond our contract language", but to be frank their Subscriber Agreement itself really does not make that clear. Customers would have to read the BRs directly to get that information. (I consulted https://www.entrust.com/sites/default/files/documentation/licensingandagreements/sept-2020-entrust-site-launch/ecs-subscription-agreement-sep-10-2020.pdf .) Insufficient Detail Regarding Leadership and Process Changes Entrust has told us that they've made many organizational changes, including of "leadership" and "staffing". Unfortunately, they have not really detailed what was wrong with the old leadership and staffing, and how the new leadership and staffing plans will improve on them, in spite of very specific requests for that information from the community. This is a basic and essential element of leading effective change in any context. They have described operational error, but haven't explained why a leadership change this time will be more successful than the promises made in 2020. I recognize that talking about errors by company leaders can be awkward, but if Entrust doesn't want to tell us why we should believe that the new leadership is better, then they probably shouldn't expect us to count the change in their favour. (Entrust has stated that this context for the leadership changes was in the June 7 report, and seemingly on that basis refused to restate it when asked, but if so it is very well-hidden.) While I have certainly expressed my frustration with Entrust very directly at times, I would personally be happy to help with such an assessment in a blameless, learning-oriented process. On the topic of leadership, I want to emphasize that I am not concerned here with individual performance, be it by individuals or leaders, and I do not feel that the Mozilla community should be either. A problem of this magnitude and duration, and reluctance to address in a deep and transparent way, must be a systematic problem and not an individual one. Entrust has created a system that in turn created forces that directed people to act in certain ways. When someone is agreeing to a delayed revocation without writing down the reason, they are doing that for one of two reasons: either they are making an error out of misunderstanding or incompetence (rare, very easy to resolve), or they are doing what they believe Entrust wants them to do (almost universally the case, difficult to remedy). Incentives are very powerful, and if Entrust's employees believe that Entrust wants them to put customer convenience ahead of the integrity of the web PKI, then those employees will do so. If a member of CA business unit staff was not escalating to senior leadership their difficulties with staffing, or investment in linting, or seeing customers successfully adapt their operations to be tolerant of prompt revocation, that's because they believed that practically Entrust did not want them to escalate that way. It's very common for organizations to loudly say "we want to uphold some value", while communicating in a much stronger way that they wish to compromise on that value. That stronger communication to the individuals who ultimately implement the company's operations comes through organizational actions and even explicit policies (such as related to performance evaluation or budgeting). A change in leadership is only effective to the extent that it changes the context in which individual humans and their tools will make decisions and take action. There has been disappointingly little evidence that Entrust actually understands what led (at a truly "root cause" level) to the sustained, unacceptable behaviour. Similarly, they promise to "review procedures" and "synchronize with compliance team" but we haven't established that they even know what the procedures should be. Surely, these procedures have been reviewed before, and that has not remedied the situation. In a recent bug, Entrust said that while they were going to revoke the certificates, they didn't agree with the analysis of the root programs. How will they be analyzing similar situations in the future, given that disagreement? The updated report says that "Entrust has the technical capability to meet the 24-hour and 5-day revocation requirements", but does not describe what they think is required of that capability, and importantly does not describe non-technical capability that might be required to perform such revocations (such as executive willingness to lose a customer). Even very recently we have seen confusion in Entrust's determination of certificates affected by an incident, and mismatched lists provided. If I were in their situation and wished to demonstrate a better alignment with root program and community expectations, I would do this: go back over the cpsURI cert list and say how each would be decided today. I might have to reach out to customers to get more information about how the certificates are used and what the operational context is, but it would let me demonstrate the concrete differences in decision-making that these changes cause. This would also give me a basis to work from for informing subscribers who were granted delayed revocation this time that they would not be in the future. IMO, Entrust doing this would be a valuable contribution to the CA community's understanding of what "exceptional" cases are, and possibly help move towards helpful clarifications of the wording. Entrust has said that between 2020 and 2024, they believed that their "existing policies and procedures would be sufficient to support [their] commitments". This is an extremely worrying perspective to take, because it means that the people who were operating the relevant systems thought that everything was OK for four years; as we know from cpsUri and friends, there was no material improvement in handling revocation delay requests. Materially, according to Entrust's explanation, over those four years on the heels of a major incident about excessive delays in revocation, Entrust never wrote down a policy on how to adjudicate delay requests. In 2020, Entrust had been a public CA for twenty years. They had participated in CABF since its inception. They were told by root programs, and agreed, that they were making incorrect decisions about delays, and committed to doing better, but never wrote a policy. I'm honestly disappointed that auditors don't require that policy to be concrete and published, but even beyond that: imagine operating a security and compliance company for twenty years and then getting rinsed because of a failure to decide something correctly, and after that still not writing down how to make the decisions in the future, or not capturing customer communication related to something as critical as revocation and certificate correctness. It's very hard not to interpret that as Entrust being flawed in a fundamental way that needs more than moving the org around. I would honestly rather that Entrust was lying about not having a policy, and instead choosing not to disclose it because of content that would not reflect well on them. Similarly, I would prefer to believe that they simply chose to be untruthful about the customer justifications for delays not being captured (and in fact we know that some were, because they were later provided as cherry-picked examples). The alternative is to believe some really unfortunate things about a CA whose certificates, by their own account, are used in very important systems. Relationship to Mozilla Root Program A concern that has been raised repeatedly and ignored by Entrust is their different behaviour with respect to the Mozilla and Chrome root programs. We have seen them act promptly upon Chrome's team entering a discussion, when that team is saying the same thing that Mozilla has been saying previously. This continues to be unaddressed in the updated report, even though it was called out explicitly in response to the initial report, and repeatedly in incident bugs. In spite of frequent reminders of the Mozilla incident response requirements, they continued to not meet those requirements even after recommitting on June 1, 2024. They have not provided the required per-subscriber detail: first claiming that it wasn’t captured, then sharing some unattributed and hiding behind confidentiality that they control. AFAIK they have not gone back to subscribers to get that information and capture it properly. There has been no elaboration on expected harm, just vague industry categorizations and claims of subscriber slowness. This refusal to cooperate fulsomely with the Mozilla incident response policy makes oversight by the root program and community more difficult, because it becomes impossible to identify cases in which Entrust's analysis of the risk tradeoff is out of alignment with that of the community. (To be sure, either side or both might be "wrong" here! But we have to see the gap in order to even start figuring that out.) There are a lot of CAs and very few full-time staff allocated to Mozilla's program, which makes it very important that the community be able to assist in that analysis. Even when I was overseeing the program, we needed community assistance, and that was with many fewer CAs, with Chrome's direct assistance with the shared program, and to be frank without an actor behaving in as challenging a manner as Entrust has been. This lack of transparency also makes it impossible for other CAs to learn from or teach Entrust about how to better handle similar situations in the future. Similarly, they say that the redaction of a portion of an email to customers (said portion detailing revocation procedures), was "absolutely not" them trying to "conceal" anything about revocation. But of course they did literally conceal things, and they have not responded when asked to explain what their other motivation for redaction might be. In short I do not believe that Entrust has operated in good faith through these incidents, with respect to Mozilla and its policies. If Entrust's behaviour became the norm for trusted certificate authorities, I think it would be disastrous for the program and the web. Conclusion I genuinely worry that it would be irresponsible to trust future Entrust-issued certificates without concrete evidence that improvements have been made, and not just planned. They have not even detailed the plans or success criteria for these initiatives, generally, and there is no way for root programs or the community to evaluate whether the efforts are even pointed in the right direction without much more detail. If a new CA using a cross-signed root had Entrust's track record and applied to add their own root to the Mozilla root store today, I believe very strongly that we would first require them to demonstrate improved competence and alignment. Entrust has no more inherent right to operate a CA business than such a newcomer, and I think that we should be very careful not to overweight the duration of Entrust's inclusion as a root, or the size of its business, as justifying even more patience or risk tolerance. To permit Entrust to continue to issue certificates as it has been to date would be to subject the web PKI and Mozilla's users to substantial risk. Mike On Fri, 21 Jun 2024 at 14:59, 'Bruce Morton' via dev-security-policy@mozilla.org <dev-security-policy@mozilla.org> wrote: > Attached is a letter from Bhagwat Swaroop, President of Entrust Digital > Security Solutions, along with an updated response to address questions > from the community. > > Thanks, Bruce. > > On Tuesday, June 18, 2024 at 1:35:48 PM UTC-4 Amir Omidi (aaomidi) wrote: > >> I am not going to say with certainty that Entrust is definitely putting >> Chrome over Mozilla. However, I hope they know that most Linux systems out >> there use the Mozilla root store directly. >> On Tuesday, June 18, 2024 at 1:12:19 PM UTC-4 Mike Shaver wrote: >> >>> On Tue, Jun 18, 2024 at 12:49 PM Walt <walter...@gmail.com> wrote: >>> >>>> I'd just like to point out that we now have a situation where Entrust >>>> is in the position of seemingly valuing the opinion of other Root Programs >>>> over Mozilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1890898#c42 >>>> >>>> In Comment #37, it was hinted at (and made slightly more explicit in >>>> #39) that the opinion of the Mozilla RP is that the attempt to >>>> re-characterize these certs was not going to be looked kindly upon, and >>>> only once a Google RP member explicitly said that it was the Google RP >>>> opinion that the certs remained mis-issued was any movement made on >>>> re-confirming the mis-issuance and taking action to revoke them. >>>> >>>> Also, if we're in a position where Entrust is finally able to commit to >>>> revoking certs within a 5 day period (setting aside that these certs >>>> technically need a delayed revocation bug as the mis-issuance was known as >>>> far back as 2024-04-10), why are other incidents not able to be resolved in >>>> this amount of time? Is it because Google showed up? >>>> >>> >>> We’ve seen this behaviour in other incidents as well, I believe >>> including the cpsURI one that has turned into a magnet for evidence of poor >>> operation and lack of transparency and responsiveness. I remarked on it in >>> my initial snarky reply to the Entrust Report, in fact. >>> >>> From a realpolitik perspective their behaviour could indeed be rational, >>> especially when the only tool root programs have is distrust. Firefox would >>> suffer substantial market disadvantage if it stopped trusting Entrust >>> certificates when other browsers didn’t. I think people generally >>> underestimate how much Mozilla would be willing to take near-term pain to >>> protect users, but it’s also possible that I am overestimating it. >>> >>> Related to that, I think Chrome’s root program representatives have >>> generally been more willing to take a concrete position quickly, so Mozilla >>> might be waiting for more explanation when Chrome decides that there’s no >>> explanation that could suffice, or similar. The root programs tend to be in >>> agreement more often than not (virtually always with Chrome and Mozilla, I >>> would say, excepting some slightly different root store populations), so it >>> may be somewhat irrelevant whose opinion spurs motion. >>> >>> Realpolitik analysis aside, I do agree that Entrust has created the >>> impression that they care much more about Chrome’s opinion than Mozilla’s, >>> which IMO might not be the best posture to take given that Mozilla and its >>> community are the locus for the processing and evaluation of the incidents >>> in question. >>> >>> Mike >>> >>> >>> >>> -- > You received this message because you are subscribed to the Google Groups " > dev-security-policy@mozilla.org" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to dev-security-policy+unsubscr...@mozilla.org. > To view this discussion on the web visit > https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/f3cebe9b-fa25-4b11-ba3d-b7f3f6e0f719n%40mozilla.org > <https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/f3cebe9b-fa25-4b11-ba3d-b7f3f6e0f719n%40mozilla.org?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "dev-security-policy@mozilla.org" group. To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-policy+unsubscr...@mozilla.org. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CADQzZqsdYBfc%2BvbUQu7fMXFnnXa_gs_x2MatkxSB0yjjBpLo7A%40mail.gmail.com.
