On 15/12/2025 6:55, John Curran via NANOG wrote:
A masterclass in owning a mistake and handling it properly.
Regards,
Hank
On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG <[email protected]> wrote:
On Fri, 12 Dec 2025, John Curran via NANOG wrote:
Short version – ARIN failed here (as you noted in your post). We’ve published a
public incident report that lays out what happened, the impact, and what we’re
changing: https://www.arin.net/announcements/20251212/
This is a pretty epic failure considering ARIN's purpose is the assignment of
unique Internet numbers (and the necessary record keeping to facilitate that
function).
Jon –
I agree completely. This was a failure of ARIN in the performance of its core
mission, and one that resulted in customer impact. The community is entitled to
full transparency in understanding how this occurred and the steps we have
taken to prevent any similar incident in the future.
Analysts have performed this particular allocation process thousands of times
previously without issue, but in this case a resource analyst made an error
that resulted in the reallocation of a previously assigned NRPM 4.10 address
block. Clearly, while that was the trigger for this incident, the real fault
here is that ARIN should not have any processes that are predicated on perfect
human performance. Prior to this incident, it was my belief that this was
already the case and that assigned resource blocks could not be impacted by
analyst error.
That is not the case for the manual processes used in the management of NRPM
4.10 address blocks. As a result, we have corrected the process to require a
second set of eyes before any change is committed. Longer term, I prefer to
fully automate this process, but until that can be implemented we will continue
with the manual process, as amended with a mandatory supervisor confirmation
step, as a reasonable and appropriate mitigation.
I have experience running several major ISPs and am fully aware that operators
rely on ARIN for flawless performance. Even a single customer impact is not
acceptable, which is why we issued the report to the community detailing the
incident and its resolution. To the extent that there is any need for
additional clarity, please don’t hesitate to ask – either here on the list (or
to me directly as you prefer.)
Thanks,
/John
John Curran
President and CEO
American Registry for Internet Numbers
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/[email protected]/message/2F2TMDVELVVFUX6MT6A3566CEOMKLMN4/
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/[email protected]/message/U33VXD6K4YROJH7T7AQHQYYOCMZWMJVQ/
