Brian –
I actually just replied to Jon’s email with very much the same sentiment – this
was not a “simple error”;
it was an incident that revealed ARIN has operated with manual processes for
management of 4.10
address space that lacked appropriate controls to catch human error. That’s a
very serious problem,
and we have promptly remedied it with appropriate review and cross-validation.
We will be reviewing this incident and our broader registry change management
practices with the ARIN
Board of Trustees, and I will note your suggestions regarding an independent
audit [1] and looking to more
formal, documentation-driven models (i.e. aerospace) for consideration.
Thanks,
/John
John Curran
President and CEO
American Registry for Internet Numbers
[1] Note that ARIN does periodically engage an independent firm to conduct a
full review and audit of the
Registration Services Department’s processes and procedures
<https://www.arin.net/about/corporate/rsd_audits/>
but that is focused on ensuring that registry changes are made consistent
with the policies described in the
Number Resource Policy Manual (NRPM) – as opposed to your suggestion which
implies an audit that is
done against best practices in change management.
> On Dec 14, 2025, at 11:12 PM, Brian Russo via NANOG <[email protected]>
> wrote:
>
> Sounds like you’re treating this as a simple error when the root cause is a
> lack of change management practices.
>
> Having a supervisor approve something is meaningless if there is no actual
> cross-validation of the change and associated data sources.
>
> Frankly I agree with others that you are likely not treating this seriously
> enough. I’ll reiterate what others have said - your whole literal purpose
> is to manage these assignments. You failed at your fundamental mission.
>
> I would recommend an independent audit and review of practices, as well as
> seeking inspiration from documentation-driven aerospace maintenance
> practices.
>
> - bri
>
> On Sun, Dec 14, 2025 at 16:08 John Sweeting via NANOG <[email protected]>
> wrote:
>
>> See inline (ARIN)
>>
>> John S.
>>
>> From: Jon Lewis via NANOG <[email protected]>
>> Date: Sunday, December 14, 2025 at 6:55 PM
>> To: John Curran via NANOG <[email protected]>
>> Cc: John Curran <[email protected]>, Jon Lewis <[email protected]>
>> Subject: Re: Accidental ARIN Reallocation
>>
>> On Fri, 12 Dec 2025, John Curran via NANOG wrote:
>>
>>> Short version – ARIN failed here (as you noted in your post). We’ve
>>> published a public incident report that lays out what happened, the
>>> impact, and what we’re changing:
>>> https://www.arin.net/announcements/20251212/
>>
>> This is a pretty epic failure considering ARIN's purpose is the assignment
>> of unique Internet numbers (and the necessary record keeping to facilitate
>> that function).
>>
>> I assume 23.128.0.0/10 records are maintained partially in the "off line
>> Excel file" because your primary system lacks necessary features to
>> differentiate free space / sparse allocations in 23.128.0.0/10 from
>> "regular" free space? I assume the sparse allocation strategy here is to
>> allow a member to come back for more 4.10 space and ideally just extend an
>> original /24 to a /23, perhaps getting the next /24 eventually, and
>> finally swap the adjacent /23 and /24 for the covering /22?
>>
>> (ARIN) correct, and the community developed policy requires sparse
>> allocation. From the policy
>> "A /24 will be allocated. ARIN should use sparse allocation when possible
>> within that /10 block. “
>>
>> Things I'm curious about that are not mentioned in the incident report:
>>
>> 1) When the analyst looked in the e-black-book and selected 23.150.164.0
>> (23.150.164.0/22, I presume) to be used to satisfy the request they
>> were working on, did they fail to see that this /22 was already in use
>> for a sparse assignment and 23.150.164.0/24 was already assigned, or
>> when 23.150.164.0/24 was originally assigned to AS397031, was the
>> e-black-book not properly updated to reflect the sparse assignment of
>> 23.150.164.0/22?
>>
>> (ARIN) The e-black-book (spreadsheet) was not properly updated which led
>> to the analyst selecting that block.
>>
>> 2) When assigning IP space, is it customary for the analyst to check
>> current/recent snapshots of the global BGP tables to see if the space
>> selected for assignment is currently or has recently been advertised?
>> It's not guaranteed that assigned space will be in the DFZ, but if it
>> is, that's a pretty good indicator that something is going wrong with
>> the current assignment of the space.
>>
>> (ARIN) Yes the steps call for the analyst to check for routing, this step
>> was missed.
>>
>> 3) Did the block split automatically delete any/all child subnets of
>> 23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24
>> from whois (and that automatically deleted the ROA and rDNS
>> delegation)?
>>
>> (ARIN) The split, in this case, returned 23.150.164.0/24, 23.150.165.0/24
>> and 23.150.166.0/23 so the analyst selected 23.150.164.0/24 for issuance.
>> The next step is to delete it from the ARIN OrgID in order to have it
>> available to allocate to the customer. The fact that it was assigned to
>> OrgID GL-700 versus OrgID ARIN was missed.
>>
>> (ARIN) We have inserted a supervisor mandatory check and approval in the
>> process prior to the delete action being initiated.
>>
>>
>> ----------------------------------------------------------------------
>> Jon Lewis, MCP :) | I route
>> Blue Stream Fiber, Sr. Neteng | therefore you are
>> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
>> _______________________________________________
>> NANOG mailing list
>>
>> https://lists.nanog.org/archives/list/[email protected]/message/X3WPNTJZUNDJMAOOJ5F3JKI4UAT65C3L/
>> _______________________________________________
>> NANOG mailing list
>>
>> https://lists.nanog.org/archives/list/[email protected]/message/6AC5KQD26PVHCN4T5HAFECIZWI67NJX4/
> _______________________________________________
> NANOG mailing list
> https://lists.nanog.org/archives/list/[email protected]/message/65OO7CNHXJ2MUBWRGVLRLQF5HXASJ44E/
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/[email protected]/message/5MEDPCFLWBRA3BNXBHEGLZUMRFUFS5KZ/
