* Could you highlight a bit more your proposal here? My understanding is
that, despite the Handelsregister ("Commercial Register") being available at a
country level, it's further subdivided into a list of couunty or region - e.g.
the Amtsgericht Herne ("Local Court Herne").
* It sounds like you're still preparing to allow for manual/human input,
and simply consistency checking. Is there a reason to not use an
allowlist-based approach, in which your Registration Agents may only select
from an approved list of County/Region/Locality managed by your Compliance Team?
* That, of course, still allows for human error. Using the excellent
example of the Handelsregister, perhaps you could describe a bit more the flow
a Validation Specialist would go through. Are they expected to examine a faxed
hardcopy? Or do they go to handelsregister.de<http://handelsregister.de> and
look up via the registration code?
* I ask, because it strikes me that this could be an example where a CA
could further improve automation. For example, it's not difficult to imagine
that a locally-developed extension could know the webpages used for validation
of the information, and extract the salient info, when that information is not
easily encoded in a URL. For those not familiar, Handelsregister encodes the
parameters via form POST, a fairly common approach for these company registers,
and thus makes it difficult to store a canonical resource URL for, say, a
server-to-server retrieval. This would help you quickly and systematically
identify the relevant jurisdiction and court, and in a way that doesn't involve
human error.
I did not know that about Handelsregister. So that’s good info. Right now, the
validation staff selects Handelsregister as the source, the system retrieves
the information, the staff then selects the jurisdiction information and enters
the registration information. Germany is locked in as the country of
verification (because Handelsregister is the source), but the staff enters the
locality/state type information as the system doesn’t know which region is
correct.
The idea is that everywhere we can, the process should automatically fill in
jurisdiction information for the validation staff so no typing is required.
This is being done in three parts:
1. Immediate (aka Stop the Hurt): The first step is to put the GeoCode check
in place to ensure that no matter what there will be valid non-mispelled
information in the certificate. There will still be user-typed information
during the phase since this phase is Aug 18 2019. The system will work exactly
as it does now except that the JOI information will run through the GeoCode
system to verify that yes, this information isn’t wrong. If wrong, the system
won’t allow the cert to be approved. At this point, no new issues should
occur, but I won’t be satisfied as its way too manual – and the registration
number is still a manual entry. That needs to change.
2. Intermediate (aka Neuter the Staff): During this process we plan to
eliminate typing of sources. Instead, the sources will be picklists based on
jurisdiction. This means that if you select Germany and the company type is an
LLC, you get a list of available sources. Fool proof-ish. There’s still a
copy/paste or manual entry of the registration number. For those sources that
do provide an API, we can tie into the API, retrieve the documentation, and
populate the information. We want to do that as well, provided it doesn’t
throw off phase 3. Since the intermediate solution is also a stop-gap to the
final solution, we want it to be a substantial improvement but one that doesn’t
impede our final destination.
3. The refactor (aka Documents r Us): This is still very much being specc’ed
but we’re currently thinking we want to evolve the system to a document system.
Right now the system works on checklists. For JOI, you enter the JOI part,
select a document (or two) that you’ll to verify JOI and then transfer
information to the system from the document. The revamp moves it to where you
have the document and specify on the document which parts of the document apply
to the organization. For example, you specify on the document that a number is
a registration number or that a name is an org name, highlighting the info.
With auto-detection of the fields (just based on key words), you end up with a
pretty dang automated system. The validation staff is there to review for
accuracy and highlight things that might be missed. Hence, no typing or
specifying any information. It’s all directly from the source.
Naming conventions also not approved yet. Since the engineers watch this forum,
they’ll probably throw things at me when they see the code names.
* I'm curious how well that approach generalizes, and/or what challenges
may exist. I totally understand that for registries which solely use hard
copies, this is a far more difficult task than it needs to be, and thus an
element of human review. However, depending on how prevalent the hardcopy vs
online copy is, we might be able to pursue automation for more, and thus
increase the stringency for the exceptions that do involve physical copies.
Right now we get the hard copies and turn them into a PDF to store in the audit
system for review during internal and external audits. During validation, all
documentation must be present and reviewed. Using OCR better, we can always at
least copy and paste information instead of typing it.
The more interesting part (in my opinion) is how to find and address these
certs. Right now, every time we have an issue or whenever a guideline changes
we write a lot of code, pull a lot of certs, and spend a lot of time reviewing.
Instead of doing this every time, we're going to develop a tool that will run
automatically every time we change a validation rule to find everything else
that will fail the new update rules. IN essence, building unit tests on the
data. What I like about this approach is it ends up building a system that lets
us see how all the rule changes interplay since sometimes they may intercept in
weird ways. It'll also let us easier measure impact of changes on the system.
Anyway, I like the idea. Thought I'd share it here to get feedback and
suggestions for improvement. Still in spec phase, but I can share more info as
it gets developed.
* This sounds like a great idea, and would love to know more details here.
For example, what's the process now for identifying these
jurisdictionOfIncorporation issues? How would it improve or change with this
system?
The process right now is we right a script based on things we can think of that
might be wrong (abbreviated states, the word “some” in the state field, etc).
We usually pull a sampling of a couple thousand certs and review those to see
if we can find anything wrong that can help identify other patterns. We’re in
the middle of doing that for the JOI issues. What would be WAY better is if we
had rule sets for validation information (similar to cablint) that checked
validation information and how it is stored in our system and made these rule
sets run on the complete data every time we change something in validation.
Right now, we build quick and dirty checks that run one time when we have an
incident. That’s not great as it’s a lot of stuff we can’t reuse. What we
should do is build something (that crossing my fingers we can open source and
share) that will be a library of checks on validation information. Sure, it’ll
take a lot of configuration to work with how other CAs store data, but one
thing we’ve seen problems with is that changes in one system lead to
un-expected potential non-compliances in others. Having something that works
cross-functionally throughout the system helps.
A better example in some-state. We scanned for values not listed as states and
cities that have “some”, “any”, “none”, etc. That only finds a limited set of
the problem, and obviously missed the JOI information (not part of the same
data set. Going forward, I want a rule set that says, is this a state? If so,
then check this source to see if it’s a real state. Then check this to see if
it also exists in the country specified. Then check to see if the locality
specified exists in the state. Then see if there is a red flag from a map that
says the org doesn’t exist. (The map check is coming – not there yet….) Instead
of finding small one off problems people report, find them on a global scale
with a rule we run every time something in the CAB forum, Mozilla policy, or
our own system changes.
* You describe it as "validation rule" changes - and I'm not sure if you're
talking about the BRs (i.e. "we validated this org at time X") or something
else. I'm not sure whether you're adding additional data, or formalizing checks
on existing data. More details here could definitely help try and generalize
it, and might be able to formalize it as a best practice. Alternatively, even
if we can't formalize it as a requirement, it may be able to use as the basis
when evaluating potential impact or cost of changes (to policy or the BRs) in
the future. That is, "any CA that has implemented (system you describe) should
be able to provide quantifiable data about the impact of (proposed change X).
If CAs cannot do so (because they did not implement the change), their feedback
and concerns will not be considered."
Validation rule meaning our own system, the CAB forum, mozilla policy.
Basically, anything that could call into question the integrity of some data
piece within our system. The point is to catch all changes that may happen
proactively, not just when someone pings me with a problem. The requirement I
think we’re trying to meet is “never have the same problem again, even if a
rule changes” because the system will take that one problem, log it as a unit
test, and run that unit test ever time we change the internal rule set to
detect all data that violates that rule as modified. Illustrative example:
Assume we decide we want all states abbreviated. Note this would contradict
the rule in the EV guidelines that requires JOI states to be written out. Right
now, this contradiction could pass undetected by a lot of CA systems I think.
However, if you have a rule set that can be enforced globally across the entire
data set, you end up instantly detecting that no valid EV cert could ever
issue. Danger! Anyway, the value of this is pretty huge internally IMO. And for
compliance, it’ll make our job easier. No more 3% audits trying to catch
mistakes.
_______________________________________________
dev-security-policy mailing list
dev-security-policy@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy
