Dear colleagues, On 7 January 2026 at 15:11 UTC, all objects were removed from both our RRDP and rsync RPKI repositories. At 15:20 UTC, the repository state was restored, and all objects were back.
At 15:08 UTC, we did a production deployment of our CA software on the first application node. We deploy one node at a time so that we have zero downtime during deployments. We carried out a database DDL change on the table storing published objects, which took roughly 40 seconds to execute. During those 40 seconds, the online node queried the published objects in order to publish them. This transaction had to wait for the table lock held by the DDL change to be released. When the lock was released, PostgreSQL returned zero rows. We learned that this is a known caveat [1]: after rewrite commits, the table will appear empty to concurrent transactions. We will implement circuit breakers between all phases of our publication process to prevent this situation from happening in the future. [1] - https://www.postgresql.org/docs/18/mvcc-caveats.html Apologies for the inconvenience this has caused. Kind regards, Bart Bakker Specialist Software Engineer RIPE NCC ----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/routing-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
