I'd just like to add or reinforce a few points based on my approach to managing open incident bugs:
* I have leaned heavily to the side of leaving bugs open if there is the potential for additional questions, and always if there are any incomplete remediations. This means that bugs do tend to stay open longer than may be considered ideal, but it ensures that the follow-up is as complete as possible. * This has a somewhat natural result, given that closing bugs is not the #1 priority, of bugs being left open when they really are ready to be closed. Thus any metric based on if/when a bug is closed is not a good representation of the CA's diligence. * I am happy to review any bug that a CA feels is stuck in that state - just set the needs-info flag for me in the bug. I suspect the same goes for Ryan and Ben. * I agree that the attention paid to incident bugs by CAs is a meaningful indicator of their professionalism, if not their trustworthiness. Bug hygiene should matter to CAs. - Wayne On Tue, May 19, 2020 at 11:40 AM Ryan Sleevi via dev-security-policy < dev-security-policy@lists.mozilla.org> wrote: > On Tue, May 19, 2020 at 2:22 PM Matthias van de Meent > <matthias.vandeme...@cofano.nl> wrote: > > I agree that for any one bug, this metadata is not anything to make > > decisions over, but when looking over e.g. the last 3 years, you can > > start making more informed guesses on the metadata only. E.g. when you > > find that a CA has consistently had 4-8 compliance issues open with > > only sporadic updates, although this doesn't tell anything about this > > CA in and of itself, it does give a (basic) sense of concern. But, as > > I've heard, the metadata is even less precise than I'd previously > > expected, and thus this road to knowledge is yet to be built. > > Exactly, for better or worse. > > I'm not sure, even in our idealized world, we'd necessarily /want/ > Bugzilla to be this. Historically, the approach has been to > systematize patterns (e.g. https://wiki.mozilla.org/CA:Symantec_Issues > , https://wiki.mozilla.org/CA:PROCERT_Issues , > https://wiki.mozilla.org/CA:WoSign_Issues ), to try and provide that > information distilled in a way that's useful to decision making and > policy stakeholders. > > It's difficult to see that purely encapsulated in the bug metadata, > because there's issues where something seemingly small can be quite > eye opening, and something seemingly major can be actually quite > benign. The dimension of severity is subjective across a number of > dimensions, and so there's no quick and pert way of summarizing that. > > I'm not trying to defend the status quo, to be sure. There's a lot I > would love to see improved here, it's just not the highest priority in > the overall set of issues. For example, I'd rather see things like ALV > for intermediates, improvements to auditor qualifications, > improvements to audits themselves, structural improvements to the BRs > (for which I have many in-flight pull requests), etc. Similarly, I'm > concerned about patterns of CAs simply not responding in a timely > fashion ( e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1563579 ), > and would love to improve the tooling to detect and alert on this. Or, > for that matter, better tooling to help us digest and correlate > responses across CAs, or identify when an issue is a duplicate (e.g. > https://bugzilla.mozilla.org/show_bug.cgi?id=1638898 ) or not yet > revoked (e.g. http://misissued.com ). I have a long list of things > that could be improved, for sure, but alas, time and resource limited > :) > > > To continue on your metaphor: I do not expect Tier 1, but maybe more > > like somewhere between Tier 2 en Tier 3? A biweekly, monthly or any > > regular check for open compliance issues waiting for a reply from > > Mozilla would already improve the (observed) situation tremendously. > > I don't disagree on room for improvements, for sure, but I'm still > going to push back on a Mozilla obligation here, because I don't think > that's reasonable. The interesting thing about this particular > situation is that if CAs consistently adhered to their existing > expectations, it does tend to end up resolved quicker. The ball is > placed consistently in the CA's court to drive this to resolution, and > they're expected to be continually doing that. This is, in part, due > to how Bugzilla searching, filtering, and alerting works. This again > goes back to where there's room for improvement here, if you're > passionate, and help is welcome :) > > This goes back to workflow. If CAs followed the defined workflow, "it > works". You're right that CAs aren't following the defined workflow, > and so it's not working as ideally as possible, and that makes some of > the data you want more difficult. Should Mozilla start removing CAs > that don't adhere to that work flow? In some extreme cases (e.g. see > above), it's probably justified, if not outright necessary. In other > cases, that may be seen as extreme. The important thing here, for > setting expectations, is that the CA is expected to be continually > driving things. That they aren't is an issue, and it's one I'm fully > supportive of filing incidents against CAs to improve. > _______________________________________________ > dev-security-policy mailing list > dev-security-policy@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-security-policy > _______________________________________________ dev-security-policy mailing list dev-security-policy@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-security-policy
