I appreciate that this matter already has been discussed and accept Jeremy’s explanation and retraction, and I have no interest in reopening that conversation. Nonetheless, based on this thread it feels to me that we would be wise to articulate our motivations.
About five years ago, Sectigo was in a bad place as a public CA. The company’s commitment to quality of execution, accurate issuance, responsibility to the WebPKI, and continuous improvement was abysmal. This manifested itself in a series of severe errors that should not have occurred accompanied by terrible management of the resulting Bugzilla incidents. I sincerely believe that had Sectigo continued down that path, our roots would be distrusted today <https://bugzilla.mozilla.org/show_bug.cgi?id=1714628#c11>. Fortunately, a few thought leaders inside the company had this epiphany, made their voices heard, and ultimately convinced those around them. This led to clear policy changes inside Sectigo and began a process that in the long term required widespread technical, cultural, procedural, and leadership reform. We forged new (for us at least) philosophies about information, quality control, operational efficiency, certificate morphology, policy governance, internal communication, external communication, automation, and the proper role of humans in our processes. There is pretty much no aspect of our public CA operation that we did not scrutinize and renovate in the past five years. While we don’t think we’re perfect by a long shot, we know beyond doubt that we have come a great distance. It's been a lengthy, hard journey that has required deep thinking, unflinching self-examination, guts, persistence, and lots of sweat. So we appreciate that it can be difficult for a CA to commit to the kind of improvement that, unfortunately, many of us still clearly need. Sectigo would like to help. Looking at the behaviors of the WebPKI CA community, we perceive a common tendency for CAs not to understand the importance of self-improvement, or know how to go about it, or even recognize the incredible privilege they have as stewards of public trust. We see CAs treating the short-term and short-sighted desires of their corporate owners and local governments as more important than five-and-a-half billion internet users around the globe. We routinely see CAs placing their Subscribers’ convenience over the good of the WebPKI. We see these things, and we have chosen to act. We are trying to show CAs that self-improvement not only is possible but is an asset to the strength of your operation. We are trying to spur improvement through encouragement and information sharing and where necessary strong rhetoric. We try to provide concrete coaching while challenging our fellows to become their better selves. This is a learning process for us. A sad fact is that many CAs appear quite intractable and unreceptive. But we also see wins, instances where some CA will experience an authentic “light bulb” moment or a change of heart. So we know it’s possible to connect in that way. At least on occasion we see the idea of “CAs helping CAs” actually work. Doubtless we have made and will continue to make errors along the way, as we try to figure out how to be sherpas to other CAs who don’t have the benefit of our last five years’ experience. Surely some will never appreciate it, or value it, or care in any way. Or for that matter improve. But some already have. This makes us think that, however preliminary our efforts and inconsistent the results, there is something here worth pursuing. On Friday, February 7, 2025 at 12:34:00 PM UTC-5 Jeremy Rowley wrote: > Thanks Mike! Sorry again about the Sectio comment. My own interest > (because my name is on them) in closing these bugs is pretty obvious.... > Not an excuse by any means, but I know it is the main reason I reacted so > poorly. > > Overall, I think a lot of the CAs are confused about what they need to do > to close the bug. I think with the delayed revocation bugs in particular, > there's a lack of clarity on what the community needs to close the bug as > resolved considering it was a conscious decision by the CA to delay > revocation. Although I do not represent a CA, I cannot think of what the CA > can say or do to ensure that delayed revocation doesn't happen again, > especially when you can't get browsers to say "Delayed revocation is never > acceptable". Is there a bug with intentional delayed revocation where the > action items are ones that were good technical controls to ensure it didn't > happen again? I'm struggling to think of any that give real good reasons on > how delayed revocation won't happen again outside of just promising "No > more delayed revocation". > > https://bugzilla.mozilla.org/show_bug.cgi?id=1945389 - No comments so far >> > > >> To me this doesn't mean "there is nothing to do with this bug" but more > likely "in the 5 days since it opened, nobody has bothered to ask anything > ahead of the full incident report". These days I'm trying to focus my > limited root-meddling time on inadequate action items, so I generally don't > bother with preliminary incident reports because I would have to guess what > the flaws in the future action items would be. > > Yeah - I agree this one needs more time and attention. It isn't old and > should not be closed. > > >> https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 - Entrust was >> distrusted so no need to keep it open >> <https://bugzilla.mozilla.org/show_bug.cgi?id=1942879> >> > > >> The purpose of a delayed revocation incident is not only to provide > root programs with visibility into the practices and reliability of CAs. It > is also to ensure that all (ahem, remaining) CAs have the opportunity to > learn from the incident such that they might avoid a similar incident in > the future. Entrust being distrusted does not necessarily mean that there > is no value in further discussion of the incident, and indeed I think > generally many incidents are pretty poorly reasoned and explained in those > matters. > > Makes sense. Is Entrust going to keep responding now that they have sold > the business to Sectigo? I realise Entrust still owns the roots which means > we need them to reply to the comments going forward. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 - Issue identified >> and the delay was unintentional >> > > >> Note that in this case the issue was "malware filtering blocked CPR", > which was sufficiently distinct from "spam filter blocked CPR" that > multiple CAs did not extrapolate from the latter to the former when > monitoring previous incidents. This makes me feel that perhaps more > attention could be paid by the root programs to explaining how they expect > CAs to address classes of issue, rather than just the most > narrowly-interpreted case specifically implicated in an incident. Closing > the bug ahead of that clarity coming from root programs or, failing that, > peer CAs or other community members, seems like a missed opportunity to > avoid future incidents like "we block things that have a non-ASCII sender > name" or whatever the next fine speciation would be. > > I think so too. Be good to get clarity on how to address different > situations. IMO, an intentional delay of revocation is more egregious than > something like this where a tool blocked the CPR. I think it's far more > acceptable where a technical solution caused the issue instead of a human > doing something wrong as it shows the CA has technology in place to control > security and (hopefully) automate a lot of the process. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1916478 - Delay appears >> unintentional and was an operational issue >> > > >> This is a great example of where the work hasn't been done to make the > incident useful for other CAs. It contains the old canards about "improving > procedures" and "training staff", but doesn't detail exactly what was wrong > with the previous policies or training, or—more importantly—what caused the > training or policies to have those defects in the first place. Without > that, all a CA can take away is "have procedures and train staff" which, I > still permit myself to hope, they all understand at that level of detail > already. > > Agreed on this one. Unintentional delays need technical controls to > prevent them from reoccuring. This one doesn't have technical controls. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 - Acknowleded that >> it was a mistake to delay yet Tim keeps pushing >> > > >> I don't see Tim continuing to push in this bug at all. The last comment > was some time ago, and the discussion seems to mostly have been between > Entrust representatives and Wayne. > > You're right. This one is one I would close though. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1889062 - Acknowledges that >> it was against the BRs to delay >> > > >> This one is definitely not done, given that the subscriber rationale > was given as "these agencies take too long to replace stuff and they're > very important in some way", and there was nothing specific in the action > items or otherwise to indicate how GDCA will navigate that tension in the > future other than "encouragement" or "make a policy (of unspecified > contents)". > > Okay - what action item should they have? > > https://bugzilla.mozilla.org/show_bug.cgi?id=1888882 - Comment 28 pretty >> much states that will ensure awareness of teh 5 day requirement >> > > >> Yep, for sure *this* time, "customer education" will be sufficient. :P > "When users apply for or obtain certificates, they will be clearly informed > of the revocation scenarios and time limits" is the sort of thing that's > hard for me to take seriously when the BRs already required that the > Subscriber make a legally binding commitment to that effect. > > I think these are hard as I know customers cite the previous Mozilla > policy as allowing delayed revocation. Getting rid of that language will be > a significant boon. > > https://bugzilla.mozilla.org/show_bug.cgi?id=1887888 - This one never >> answered the question of what they will do going forward. >> > > >> Indeed. Seems like closing in that situation would send an unfortunate > message that you can just wait it out and the awkward questions will go > away. > > Yeah - I definitely agree with this one. > > On Fri, Feb 7, 2025 at 8:41 AM Mike Shaver <[email protected]> wrote: > >> Hi Jeremy, >> >> I don't agree with your listed reasoning for these bugs all being >> suitable for closure, though I do genuinely appreciate you taking the time >> to enumerate them like this. Some responses to selected entries below! >> >> On Thu, Feb 6, 2025 at 4:58 PM Jeremy Rowley <[email protected]> wrote: >> >>> Perhaps it would be better to look at each delayed revocation bug: >>> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1945389 - No comments so >>> far >>> >> >> To me this doesn't mean "there is nothing to do with this bug" but more >> likely "in the 5 days since it opened, nobody has bothered to ask anything >> ahead of the full incident report". These days I'm trying to focus my >> limited root-meddling time on inadequate action items, so I generally don't >> bother with preliminary incident reports because I would have to guess what >> the flaws in the future action items would be. >> >> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1943528 - Entrust was >>> distrusted so no need to keep it open >>> <https://bugzilla.mozilla.org/show_bug.cgi?id=1942879> >>> >> >> The purpose of a delayed revocation incident is not only to provide root >> programs with visibility into the practices and reliability of CAs. It is >> also to ensure that all (ahem, remaining) CAs have the opportunity to learn >> from the incident such that they might avoid a similar incident in the >> future. Entrust being distrusted does not necessarily mean that there is no >> value in further discussion of the incident, and indeed I think generally >> many incidents are pretty poorly reasoned and explained in those matters. >> >> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1942879 - Issue identified >>> and the delay was unintentional >>> >> >> Note that in this case the issue was "malware filtering blocked CPR", >> which was sufficiently distinct from "spam filter blocked CPR" that >> multiple CAs did not extrapolate from the latter to the former when >> monitoring previous incidents. This makes me feel that perhaps more >> attention could be paid by the root programs to explaining how they expect >> CAs to address classes of issue, rather than just the most >> narrowly-interpreted case specifically implicated in an incident. Closing >> the bug ahead of that clarity coming from root programs or, failing that, >> peer CAs or other community members, seems like a missed opportunity to >> avoid future incidents like "we block things that have a non-ASCII sender >> name" or whatever the next fine speciation would be. >> >> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1916478 - Delay appears >>> unintentional and was an operational issue >>> >> >> This is a great example of where the work hasn't been done to make the >> incident useful for other CAs. It contains the old canards about "improving >> procedures" and "training staff", but doesn't detail exactly what was wrong >> with the previous policies or training, or—more importantly—what caused the >> training or policies to have those defects in the first place. Without >> that, all a CA can take away is "have procedures and train staff" which, I >> still permit myself to hope, they all understand at that level of detail >> already. >> >> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1898848 - Acknowleded that >>> it was a mistake to delay yet Tim keeps pushing >>> >> >> I don't see Tim continuing to push in this bug at all. The last comment >> was some time ago, and the discussion seems to mostly have been between >> Entrust representatives and Wayne. >> >> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1889062 - Acknowledges >>> that it was against the BRs to delay >>> >> >> This one is definitely not done, given that the subscriber rationale was >> given as "these agencies take too long to replace stuff and they're very >> important in some way", and there was nothing specific in the action items >> or otherwise to indicate how GDCA will navigate that tension in the future >> other than "encouragement" or "make a policy (of unspecified contents)". >> >> >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1888882 - Comment 28 >>> pretty much states that will ensure awareness of teh 5 day requirement >>> >> >> Yep, for sure *this* time, "customer education" will be sufficient. :P >> "When users apply for or obtain certificates, they will be clearly informed >> of the revocation scenarios and time limits" is the sort of thing that's >> hard for me to take seriously when the BRs already required that the >> Subscriber make a legally binding commitment to that effect. >> >> Also, that report describes an action item (maybe? a commitment at least) >> that was to be completed a month ago, and for which no status has been >> given: >> >> > This action plan will take a long time, but we will still accelerate >> the progress to encourage as many of our existing customers with larger >> certificate volumes to use the automated certificate deployment system as >> much as possible by 2025. >> >> Unless the goal is just to complete the *encouragement* by 2025, in which >> case it's a nothing-burger again. >> >> https://bugzilla.mozilla.org/show_bug.cgi?id=1887888 - This one never >>> answered the question of what they will do going forward. >>> >> >> Indeed. Seems like closing in that situation would send an unfortunate >> message that you can just wait it out and the awkward questions will go >> away. >> >> >>> The value to Sectigo and the Sectigo podcast is pretty obvious, >>> >> >> This is bullshit, unwelcome on the list, and quite surprising. I know you >> apologized but I still can't let it just go by when I'm replying to the >> message. >> >> Mike >> >> -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/aa6ee7a9-1049-4a3d-8aa4-f6850be1cdbbn%40mozilla.org.
