Hi, between 5:30 and 7:10 UTC on 2023/09/27 the WDQS servers running in the eqiad datacenter have returned results with more than 10 minutes of lag. This caused bots using MW maxlag [0] to stop functioning properly during that time. The incident started after a failure of the mirroring system between two of our kafka clusters [1], such incident should not have impacted WDQS but it uncovered improper sandboxing of the WDQS updater test setup [2].
Sorry for the inconvenience. -- David Causse Software Engineer, Wikimedia Foundation 0: https://www.mediawiki.org/wiki/Manual:Maxlag_parameter 1: https://wikitech.wikimedia.org/wiki/Incidents/2023-09-27_Kafka-jumbo_mirror-makers 2: https://phabricator.wikimedia.org/T347515
_______________________________________________ Wikidata mailing list -- wikidata@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/wikidata@lists.wikimedia.org/message/Z7X2JMLCHP4F6HS4C4XW4GF4RS7WFJHI/ To unsubscribe send an email to wikidata-le...@lists.wikimedia.org
