> > One of the main aspects is that the majority of infrastructure-related > incidents that we’ve been dealing with over the last few months have been > of a sensitive nature. As per WP:DNFTT > <https://en.wikipedia.org/wiki/Wikipedia:Deny_recognition> and our > policies around sensitive information, we don’t publish information on our > response to these outages even if some of the action items are fine to be > public.
Another factor is the need to scale our incident tracking process. We now > coordinate all incidents using Corto > <https://wikitech.wikimedia.org/wiki/Corto>, which creates Phabricator > tickets automatically - defaulting to closed. >From looking at the #wikimedia-incidents <https://phabricator.wikimedia.org/project/board/2143/query/all/> dashboard, even with volunteer NDA access, only 3 tasks remain non-public; the "rest" are public. This means either the vast majority of such tasks are being filed in a way that even community members with appropriate access have very little visibility into the state of incidents on Wikimedia services (which, to remind you, previously used to be an open on-wiki process), or y'all really aren't having that many cases where DENY is being followed and I cannot make up my mind which in this instance is the correct answer. The last factor is our incident review process. To account for the > diversity of incidents and involvement of many teams, we have moved to a > less rigid process where the filling out of the wikitech document and the > scorecard etc don’t feature. This is the main reason for the lack of > updates to the incident status page. > > The process of filling out the wikitech incident reports has been a fairly > arduous process of copying information out of Phabricator tasks and Google > docs which duplicates effort. If the main thrust is "we would like our newer crop of engineers/managers to not use wikitext and instead use an IRC bot" (which, imo, is definitely not a great take to have given that this is Wikimedia), why not just code the bot to transparently automatically copy everything to a wikitext page (similar to SAL)? The reason I say this is because there appears to be no current easy way to chronologically look at incident reports per year in a nice and organized manner, and the current approach being used is very hard to scale if I want to query the question, "So, the day before yesterday, somebody on VPT saw a ton of broken thumbnails, was that an incident or a me problem ?" Regards, Sohom Datta --- Open-source contributor @Wikimedia On Tue, May 12, 2026 at 7:29 AM Hugh Nowlan via Wikitech-l < [email protected]> wrote: > Hello, > > On Sun, 26 Apr 2026 at 21:11, a smart kitten via Wikitech-l < > [email protected]> wrote: > >> Hey all, >> >> Just a question -- do folks know if incident reports are still being >> completed & published on Wikitech (such that they're listed at [0])? >> >> I just thought to ask given that - for me - the Phabricator search >> results at [1] list 9 tasks created since the start of this year that are >> tagged with `#Wikimedia-Incident` (& some folks with an NDA may be able to >> see more tasks than that), but the Wikitech page at [0] currently only >> lists there as having been one incident that's occurred since the start of >> this year [2]. >> > > The short answer is no, they are not being published regularly any more - > there have been a few shifts in incident response that we need to document > and align our processes on. > > There are a few factors at work here: > * One of the main aspects is that the majority of infrastructure-related > incidents that we’ve been dealing with over the last few months have been > of a sensitive nature. As per WP:DNFTT > <https://en.wikipedia.org/wiki/Wikipedia:Deny_recognition> and our > policies around sensitive information, we don’t publish information on our > response to these outages even if some of the action items are fine to be > public. > * Another factor is the need to scale our incident tracking process. We > now coordinate all incidents using Corto > <https://wikitech.wikimedia.org/wiki/Corto>, which creates Phabricator > tickets automatically - defaulting to closed. > * The last factor is our incident review process. To account for the > diversity of incidents and involvement of many teams, we have moved to a > less rigid process where the filling out of the wikitech document and the > scorecard etc don’t feature. This is the main reason for the lack of > updates to the incident status page. > > The process of filling out the wikitech incident reports has been a fairly > arduous process of copying information out of Phabricator tasks and Google > docs which duplicates effort. > For the time being, given the more concerted use of Phabricator tasks in > the era of Corto, we will try to update and publish Phabricator tickets > wherever possible. When an incident is being closed out by the incident > coordinator, they will transfer relevant information from the Google doc > (if needed) and open up the task if it is suitable to do so in order to > provide context on our outages and any interesting technical lessons > learned. We’ll also work towards formalising a process to replace the > existing process officially. > > Cheers, > Hugh > _______________________________________________ > Wikitech-l mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
_______________________________________________ Wikitech-l mailing list -- [email protected] To unsubscribe send an email to [email protected] https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
