[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2022-06-06 Thread Maintenance_bot
Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Maintenance_bot Cc: dcausse, RKemper, Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-lesz

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2022-06-06 Thread gerritbot
gerritbot added a comment. Change 803336 **merged** by Ryan Kemper: [operations/puppet@production] query service: clean up absented resources https://gerrit.wikimedia.org/r/803336 TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimed

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2022-06-06 Thread gerritbot
gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, gerritbot Cc: dcausse, RKemper, Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-leszek, Gehel, La

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2022-06-06 Thread gerritbot
gerritbot added a comment. Change 803336 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] query service: clean up absented resources https://gerrit.wikimedia.org/r/803336 TASK DETAIL https://phabricator.wikimedia.org/T290330 E

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-11-15 Thread Gehel
Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, Gehel Cc: dcausse, RKemper, Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-leszek, Gehel, Ladsgroup, Ak

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-10-19 Thread dcausse
dcausse assigned this task to RKemper. dcausse added a comment. @RKemper is there still something left to do related to this? TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RKemper, dcausse Cc

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread RKemper
RKemper added a comment. We're no longer doing hourly restarts; we should monitor availability over the weekend to make sure that we're capable of handling current load without the crutch of the hourly restarts. We still have the useragent we banned in the last full outage (dailymotion) bann

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-17T02:28:39Z] T290330 [Remove WDQS codfw ~hourly restarts] Successfully rolled out to rest of fleet `sudo cumin 'C:query_service::crontasks' 'sudo run-puppet-agent --force

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-17T02:22:00Z] T290330 [Remove WDQS codfw ~hourly restarts] `wdqs2001` and `wdqs2004` look fine after running `sudo systemctl reset-failed wdqs-restart-hourly-w-random-dela

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread Maintenance_bot
Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-leszek, Mholloway, Gehel, Lads

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-17T01:55:10Z] T290330 [Remove WDQS codfw ~hourly restarts] Testing on arbitrary codfw host: `ryankemper@wdqs2001:~$ sudo run-puppet-agent` TASK DETAIL https://phabricat

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-17T01:47:59Z] T290330 [Remove WDQS codfw ~hourly restarts] `sudo cumin 'C:query_service::crontasks' 'sudo disable-puppet "Stop doing wdqs codfw ~hourly restarts - T290330"

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-16 Thread gerritbot
gerritbot added a comment. Change 720102 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: remove codfw hourly restarts https://gerrit.wikimedia.org/r/720102 TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/s

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-09 Thread gerritbot
gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: Addshore, LSobanski, RhinosF1, Zbyszko, WMDE-leszek, Mholloway, Gehel, Ladsgroup, Aklappe

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-09 Thread gerritbot
gerritbot added a comment. Change 720102 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wdqs: remove codfw hourly restarts https://gerrit.wikimedia.org/r/720102 TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PRE

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-03T19:04:43Z] T290330 `ryankemper@cumin1001:~$ sudo -E cumin 'P{wdqs2*}' 'sudo rm -fv /etc/cron.hourly/restart-blazegraph'` (Cleaned up manually created crons now that we

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Maintenance_bot
Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: RhinosF1, Zbyszko, WMDE-leszek, Mholloway, Gehel, Ladsgroup, Aklapper, Bugr

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-03T17:17:55Z] T290330 Deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/717508 across `wdqs` fleet; codfw wdqs hosts will restart on average once per hour now

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread gerritbot
gerritbot added a comment. Change 717508 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: temp mitigation => restart hourly w random https://gerrit.wikimedia.org/r/717508 TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread gerritbot
gerritbot added a comment. Change 717508 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wdqs: temp mitigation => restart hourly w random https://gerrit.wikimedia.org/r/717508 TASK DETAIL https://phabricator.wikimedia.org/T29

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread gerritbot
gerritbot added a comment. Change 717494 **merged** by Ryan Kemper: [operations/puppet@production] wdqs: temp mitigation => restart hourly w random https://gerrit.wikimedia.org/r/717494 TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Maintenance_bot
Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: RhinosF1, Zbyszko, WMDE-leszek, Mholloway, Gehel, Ladsgroup, Aklapper, Bugreporter, S

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread gerritbot
gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: RhinosF1, Zbyszko, WMDE-leszek, Mholloway, Gehel, Ladsgroup, Aklapper, Bugreporter, Suran

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread gerritbot
gerritbot added a comment. Change 717494 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper): [operations/puppet@production] wdqs: temp mitigation => restart hourly w random https://gerrit.wikimedia.org/r/717494 TASK DETAIL https://phabricator.wikimedia.org/T29

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Bugreporter
Bugreporter added a comment. For higher availability you may want to depool each server before restarting them. TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Bugreporter Cc: RhinosF1, Zbyszk

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Stashbot
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-09-03T16:10:13Z] blazegraph (public cofdfw cluster) will now restart every hour - T290330 TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https:/

[Wikidata-bugs] [Maniphest] T290330: Wikidata Query Service unstable in codfw

2021-09-03 Thread Gehel
Gehel renamed this task from "502 Bad Gateway on WDQS on ulsfo" to "Wikidata Query Service unstable in codfw". TASK DETAIL https://phabricator.wikimedia.org/T290330 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Zbyszko, WMDE-leszek, Mholl