An unfortunate occurrence. Respect to ARIN for owning it completely, and being transparent enough to share it with us.
Aaron > On Dec 12, 2025, at 11:28 AM, John Curran via NANOG <[email protected]> > wrote: > > Chase (and the NANOG operator community) – > > Apologies for not replying earlier, but I wanted to make sure we understood > exactly what went wrong and had that written up as an incident report (i.e., > rather than responding piecemeal and without full clarity). > > Short version – ARIN failed here (as you noted in your post). We’ve published > a public incident report that lays out what happened, the impact, and what > we’re changing: https://www.arin.net/announcements/20251212/ > > This came down to some remaining manual handling around NRPM 4.10 space, > which uses a sparse allocation model and wasn’t yet fully integrated into our > automated inventory. That gap allowed your already in-use IPv4 /24 block to > be mistaken as available and reissued to another customer. When it was > removed and reissued, your associated ROA was removed as well, along with > reverse DNS services, etc. > > We had plans to automate our NRPM 4.10 inventory management (largely for > efficiency reasons), but this incident and subsequent review showed that the > remaining manual steps pose more risk than is reasonable. As a result, we’ve > moved that work well up the priority list for development. In the meantime, > and as detailed in the incident report, we’ve put additional controls in > place – including a mandatory second review on any resource deletion from an > organization – to prevent this from happening again. > > I will also be reviewing our number resource inventory management practices > internally (and with the ARIN Board of Trustees) to ensure there are not any > other similar situations that might pose such a risk. My deepest apologies > for this incident; we are acutely aware that the integrity of Internet number > resources is essential to network operators, and thus it must be inherent to > ARIN’s performance at all times. > > Sincerely, > /John > > John Curran > President and CEO > American Registry for Internet Numbers > > On Dec 9, 2025, at 1:19 PM, Chase via NANOG <[email protected]> wrote: > > Hey NANOG, > > > > After receiving a BGPAlerter notification that one of our subnets > (23.150.164.0/24) had been hijacked, I checked and noticed the prefix in > question was missing RPKI. Assuming I had fat fingered something and > butchered the ROA, I logged into ARIN and found that the prefix was missing > from our resource list entirely, and had been reallocated to another > organization and announced from their network. I created a ticket in ARIN and > called immediately. > > > > They confirmed that our subnet had been accidentally reallocated to another > customer, and that they are currently working on returning it to us. After a > couple hours, they told us the other organization will stop announcing the > prefix, and WHOIS will be returned shortly. > > > > I’m guessing there’s no way to prevent this kind of thing on our side if the > RPKI ROA itself is removed along with the allocation? I’m planning on adding > checks to look for missing ROAs (in addition to invalid/expiring ones), which > I'm guessing would've caught this earlier. > > > > Have any of you had anything like this happen with ARIN or another RIR? I’m > especially curious what might have happened if we’d only noticed and reached > out a few weeks later instead of within a few minutes. > > > > Best, > > Chase Lauer > > GalaxyGate, AS397031 > > https://galaxygate.net > _______________________________________________ > NANOG mailing list > https://lists.nanog.org/archives/list/[email protected]/message/5MCMSACQADNXE65BTK34MQ3PXY4PDETF/ > > _______________________________________________ > NANOG mailing list > https://lists.nanog.org/archives/list/[email protected]/message/FY3SDD72W5OFTJHIPHMB46JBGQFE2G6G/ _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/JJ5IYY6I46PLZJZNXOG2TEJ5JPHZZ5HQ/
