[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

jcrespo added a comment. labsdb1009.mgmt (stress on management interface) is down according to icinga for 14 hours (around net maintenance), maybe a loose cable or misconfiguration? Not a huge blocker, but better make sure it is not an intended/known state. TASK DETAIL https://phabricator.

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496721 abandoned by Marostegui: db-eqiad.php: Failover db1066 to db1076 on s2 Reason: Not needed https://gerrit.wikimedia.org/r/496721 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496720 abandoned by Marostegui: mariadb: Failover db1066 to db1076 on s2 Reason: Not needed https://gerrit.wikimedia.org/r/496720 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/setti

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496723 abandoned by Marostegui: mariadb: Promote db1120 as x1 master Reason: Not needed https://gerrit.wikimedia.org/r/496723 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496724 abandoned by Marostegui: db-eqiad.php: Promote db1120 as x1 master Reason: Not needed https://gerrit.wikimedia.org/r/496724 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-03-20T06:09:58Z] Synchronized wmf-config/db-eqiad.php: Repool databases in row A - T187960 (duration: 00m 49s) TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Marostegui added a comment. For the record, read only times on s2: Read only ON: 15:21:10 Read only OFF: 15:27:56 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, Marostegui Cc: T

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-03-19T15:27:56Z] Synchronized wmf-config/db-eqiad.php: Set s2 read only OFF - T187960 (duration: 00m 26s) TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFE

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-03-19T15:21:10Z] Synchronized wmf-config/db-eqiad.php: Set s2 database master on read only - T187960 (duration: 00m 48s) TASK DETAIL https://phabricator.wikimedia.org/T187

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497472 **merged** by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Set s2 on read only https://gerrit.wikimedia.org/r/497472 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-03-19T15:12:24Z] eqiad A7 servers uplink move - T187960 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/setting

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-03-19T14:44:51Z] Synchronized wmf-config/db-eqiad.php: Depool databases in row A - T187960 (duration: 00m 48s) TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497469 **merged** by jenkins-bot: [operations/mediawiki-config@master] db-eqiad.php: Depool hosts in row A https://gerrit.wikimedia.org/r/497469 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.o

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497472 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Set s2 on read only https://gerrit.wikimedia.org/r/497472 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREF

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497469 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Depool hosts in row A https://gerrit.wikimedia.org/r/497469 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PR

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497319 **merged** by Bstorm: [operations/puppet@production] dumps distribution: failing over the web only for network changes https://gerrit.wikimedia.org/r/497319 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://pha

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497420 **merged** by Bstorm: [operations/dns@master] dumps distribution: fail over dumps to labstore1007 https://gerrit.wikimedia.org/r/497420 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497420 had a related patch set uploaded (by Bstorm; owner: Bstorm): [operations/dns@master] dumps distribution: fail over dumps to labstore1007 https://gerrit.wikimedia.org/r/497420 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFEREN

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497328 **merged** by Bstorm: [operations/dns@master] dumps distribution: reduce TTL on labstore1006 for failover https://gerrit.wikimedia.org/r/497328 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wiki

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497328 had a related patch set uploaded (by Bstorm; owner: Bstorm): [operations/dns@master] dumps distribution: reduce TTL on labstore1006 for failover https://gerrit.wikimedia.org/r/497328 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 497319 had a related patch set uploaded (by Bstorm; owner: Bstorm): [operations/puppet@production] dumps distribution: failing over the web only for network changes https://gerrit.wikimedia.org/r/497319 TASK DETAIL https://phabricator.wikimedia.org/T

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496724 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Promote db1120 as x1 master https://gerrit.wikimedia.org/r/496724 TASK DETAIL https://phabricator.wikimedia.org/T187960 EM

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496723 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/puppet@production] mariadb: Promote db1120 as x1 master https://gerrit.wikimedia.org/r/496723 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERE

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496721 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/mediawiki-config@master] db-eqiad.php: Failover db1066 to db1076 on s2 https://gerrit.wikimedia.org/r/496721 TASK DETAIL https://phabricator.wikimedia.org/T18796

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

gerritbot added a comment. Change 496720 had a related patch set uploaded (by Marostegui; owner: Marostegui): [operations/puppet@production] mariadb: Failover db1066 to db1076 on s2 https://gerrit.wikimedia.org/r/496720 TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PRE

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

ayounsi added a comment. Thank you all for the quick replies! In T187960#5000934 , @hashar wrote: > Is that just about re cabling the server from a switch to another? If so I would expect the downtime to be rather minimal. C

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

elukey added a comment. ge-6/0/25 - mc1019 ge-6/0/26 - mc1020 ge-6/0/27 - mc1021 ge-6/0/28 - mc1022 ge-6/0/29 - mc1023 The above ones are holding the eqiad mediawiki object cache, they are extremely important. IIUC this maintenance will not require any host shutdown, th

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

elukey added a comment. About Analytics nodes: - ge-1/0/7 - kafka-jumbo1001 -> Kafka needs to be stopped ~10/15 minutes beforehand to have a graceful shutdown (if possible) - ge-2/0/12 - kafka-jumbo1002 -> same thing, and it is better to have kafka-jumbo1001 up and running (with Kafka

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Marostegui added a comment. Will do! Thanks! TASK DETAIL https://phabricator.wikimedia.org/T187960 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Cmjohnson, Marostegui Cc: Addshore, MMiller_WMF, Catrope, elukey, Marostegui, Stashbot, Paladox, ge

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Addshore added a comment. In T187960#4998831 , @Marostegui wrote: > In T187960#4998807 , @Addshore wrote: > > > From where I am sat setting $wgReadOnly for the few seconds would be t

[Wikidata-bugs] [Maniphest] [Commented On] T187960: Rack/cable/configure asw2-a-eqiad switch stack

Marostegui added a comment. In T187960#4998807 , @Addshore wrote: > From where I am sat setting $wgReadOnly for the few seconds would be the best thing for cognate, and would avoid having to run the maintenance scripts. Thanks fo