[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-09-05T09:05:54Z] rebuilt and synchronized wikiversions files: Promote wikidatawiki to 1.34.0-wmf.21 for T232035 - T220746 TASK

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534585 **merged** by jenkins-bot: [operations/mediawiki-config@master] Wikidata back to 1.34.0-wmf.21 https://gerrit.wikimedia.org/r/534585 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534585 had a related patch set uploaded (by Hashar; owner: Hashar): [operations/mediawiki-config@master] Wikidata back to 1.34.0-wmf.21 https://gerrit.wikimedia.org/r/534585 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

hashar added a comment. Rolled back to 1.34.0-wmf.20. The latency is similar ~ 750ms https://grafana.wikimedia.org/d/AJf0z_7Wz/termbox on codfw datacenter shows some errors. So all indications seem to aim at queries hitting codfw and thus cold caches which cause some queries to be

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-09-05T08:55:26Z] rebuilt and synchronized wikiversions files: Rollback wikidatawiki to 1.34.0-wmf.20 for T232035 TASK DETAIL https://phabricator.wikimedia.org/T232035

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Joe added a comment. So the real issue was: - termbox **correctly** uses the `api-ro.discovery.wmnet` host - the discovery record was **incorrectly** set to active-active - so requests from termbox would just go to the nearest dc, meaning that in codfw it would face super-cold caches

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534583 **merged** by jenkins-bot: [operations/mediawiki-config@master] Rollback wikidata to 1.34.0-wmf.20 https://gerrit.wikimedia.org/r/534583 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

hashar added a comment. Looking at `icinga_contact.raw: "irc"`, `Termbox` over 15 days https://logstash.wikimedia.org/goto/336ca058ad55c63297324fd699fd7b83 That shows alarms every time we deploy. So most probably there is a weird issue with hitting cold caches due to the new wmf version

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534583 had a related patch set uploaded (by Hashar; owner: Hashar): [operations/mediawiki-config@master] Rollback wikidata to 1.34.0-wmf.20 https://gerrit.wikimedia.org/r/534583 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

hashar added a comment. Promoted wikidatawiki again and I ran the service checker again on deploy1001.eqiad.wmnet using: time service-checker-swagger -t 15 termbox.svc.codfw.wmnet http://termbox.svc.codfw.wmnet:3030 All endpoints are healthy Query takes roughly 750ms - 1000

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-09-05T08:21:35Z] rebuilt and synchronized wikiversions files: Promote wikidatawiki to 1.34.0-wmf.21 for T232035 - T220746 TASK

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534578 **merged** by jenkins-bot: [operations/mediawiki-config@master] Promote wikidata to 1.34.0-wmf.21 https://gerrit.wikimedia.org/r/534578 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534578 had a related patch set uploaded (by Hashar; owner: Hashar): [operations/mediawiki-config@master] Promote wikidata to 1.34.0-wmf.21 https://gerrit.wikimedia.org/r/534578 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Tarrow added a comment. I tried the same from codfw's deployment host and saw the same. No obvious difference between api-ro and mwdebug1001 in the time to respond. TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

hashar added a comment. Some debugging thingies, reaching out to https://www.wikidata.org/w/index.php?title=Special:EntityData=json=Q1=103 via mwdebug1001.eqiad.wmnet: time curl -H 'Host: www.wikidata.org'

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-09-05T07:56:13Z] Switching "wikidatawiki" on mwdebug1001 to 1.34.0-wmf.21 by editing /srv/mediawiki/wikiversions.php # T232035 TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Joe added a comment. In T232035#5467309 , @Tarrow wrote: > I think this is probably the same as T229313 . We suspected it might be related to T231011

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

Ladsgroup added a comment. The link to logs: https://logstash.wikimedia.org/goto/3146ff65c2320362e12501bb1fe55789 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ladsgroup Cc: WMDE-leszek,

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534514 **merged** by jenkins-bot: [operations/mediawiki-config@master] Rollback wikidata to 1.34.0-wmf.20 https://gerrit.wikimedia.org/r/534514 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

hashar added a comment. Note that the alarm keeps flapping since we also have: RECOVERY - termbox codfw on termbox.svc.codfw.wmnet is OK: All endpoints are healthy TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T232035: 1.34.0-wmf.21 cause termbox to emit: Test get rendered termbox returned the unexpected status 500

gerritbot added a comment. Change 534514 had a related patch set uploaded (by Hashar; owner: Hashar): [operations/mediawiki-config@master] Rollback wikidata to 1.34.0-wmf.20 https://gerrit.wikimedia.org/r/534514 TASK DETAIL https://phabricator.wikimedia.org/T232035 EMAIL PREFERENCES