[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 553097 **abandoned** by Majavah: [operations/puppet@production] Update cron with lb and lb-pool params Reason: https://gerrit.wikimedia.org/r/553097 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a comment. itshappening TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ItamarWMDE closed subtask T315423: Revive and merge patch to update maxlag calculation as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, ItamarWMDE Cc: ItamarWMDE, akosiaris,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

hoo closed this task as "Resolved". hoo claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo Cc: ItamarWMDE, akosiaris, Lydia_Pintscher, bking, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-10-20T20:59:37Z] Finished scap: Backport for [[gerrit:845016|Only generate QS maxlag for pooled servers (T315423 T238751 )]]

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 845016 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@wmf/1.40.0-wmf.6] Only generate QS maxlag for pooled servers https://gerrit.wikimedia.org/r/845016 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 844993 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@master] Add an integration test for updateQueryServiceLag https://gerrit.wikimedia.org/r/844993 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 552544 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@master] Only generate QS maxlag for pooled servers https://gerrit.wikimedia.org/r/552544 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 845016 had a related patch set uploaded (by Hoo man; author: Addshore): [mediawiki/extensions/Wikidata.org@wmf/1.40.0-wmf.6] Only generate QS maxlag for pooled servers https://gerrit.wikimedia.org/r/845016 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 844993 had a related patch set uploaded (by Hoo man; author: Hoo man): [mediawiki/extensions/Wikidata.org@master] Add an integration test for updateQueryServiceLag https://gerrit.wikimedia.org/r/844993 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.5; 2022-10-10); removed MW-1.40-notes (1.40.0-wmf.4; 2022-10-03). TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 841165 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@wmf/1.40.0-wmf.5] updateQueryServiceLag: Add lb(-pool) options for forward compatibility https://gerrit.wikimedia.org/r/841165 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-10-11T14:01:45Z] Finished scap: Backport for [[gerrit:841164|updateQueryServiceLag: Add lb(-pool) options for forward compatibility (T315423 T238751

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ReleaseTaggerBot edited projects, added MW-1.40-notes (1.40.0-wmf.4; 2022-10-03); removed MW-1.40-notes (1.40.0-wmf.6; 2022-10-17). TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-10-11T13:56:47Z] Started scap: Backport for [[gerrit:841164|updateQueryServiceLag: Add lb(-pool) options for forward compatibility (T315423 T238751

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-10-11T13:57:07Z] hoo and hoo: Backport for [[gerrit:841164|updateQueryServiceLag: Add lb(-pool) options for forward compatibility (T315423 T238751

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 841164 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@wmf/1.40.0-wmf.4] updateQueryServiceLag: Add lb(-pool) options for forward compatibility https://gerrit.wikimedia.org/r/841164 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 841165 had a related patch set uploaded (by Hoo man; author: Hoo man): [mediawiki/extensions/Wikidata.org@wmf/1.40.0-wmf.5] updateQueryServiceLag: Add lb(-pool) options for forward compatibility https://gerrit.wikimedia.org/r/841165 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 841164 had a related patch set uploaded (by Hoo man; author: Hoo man): [mediawiki/extensions/Wikidata.org@wmf/1.40.0-wmf.4] updateQueryServiceLag: Add lb(-pool) options for forward compatibility https://gerrit.wikimedia.org/r/841164 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ReleaseTaggerBot added a project: MW-1.40-notes (1.40.0-wmf.6; 2022-10-17). TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: ReleaseTaggerBot Cc: ItamarWMDE, akosiaris, Lydia_Pintscher, bking,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 841136 **merged** by jenkins-bot: [mediawiki/extensions/Wikidata.org@master] updateQueryServiceLag: Add lb(-pool) options for forward compatibility https://gerrit.wikimedia.org/r/841136 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 841136 had a related patch set uploaded (by Hoo man; author: Hoo man): [mediawiki/extensions/Wikidata.org@master] updateQueryServiceLag: Add lb(-pool) options for forward compatibility https://gerrit.wikimedia.org/r/841136 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Joe changed the task status from "Open" to "Stalled". Joe removed Joe as the assignee of this task. Joe added a comment. Hi, any news on this front? I'll release this bug as its completion doesn't depend on me right now. When the functionality has been merged, please reassign to me. TASK

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Gehel edited projects, added Wikidata-Query-Service; removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Gehel Cc: ItamarWMDE, akosiaris, Lydia_Pintscher, bking,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Gehel edited projects, added Discovery-Search; removed Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Gehel Cc: ItamarWMDE, akosiaris, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ItamarWMDE added a project: wmde-team-a-tech. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, ItamarWMDE Cc: ItamarWMDE, akosiaris, Lydia_Pintscher, bking, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 552544 **restored** by Itamar Givon: [mediawiki/extensions/Wikidata.org@master] Only generate QS maxlag for pooled servers https://gerrit.wikimedia.org/r/552544 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a comment. In T238751#7988593 , @ItamarWMDE wrote: > @Addshore Does it mean we need to then de-abandon that change, or should we just create a new patch to re-enable these options? Why was it abandoned in the first

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ItamarWMDE added a comment. @Addshore Does it mean we need to then de-abandon that change, or should we just create a new patch to re-enable these options? TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a comment. Right, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikidata.org/+/552544 needs to go in first! (Sorry for missing that @Joe , the extended timeline messed with my head) TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a comment. *looks at the script closer* TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Addshore Cc: akosiaris, Lydia_Pintscher, bking, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 801762 **merged** by Giuseppe Lavagetto: [operations/puppet@production] Revert "maintenance::wikidata: Update cron with lb and lb-pool params" https://gerrit.wikimedia.org/r/801762 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Joe added a comment. Sadly I had to revert, because the `--lb` and the `--lb-pool` commands are not recognized by the script. mwmaint1002:~$ /usr/local/bin/mwscript extensions/Wikidata.org/maintenance/updateQueryServiceLag.php --wiki wikidatawiki --cluster wdqs --prometheus

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 801762 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto): [operations/puppet@production] Revert "maintenance::wikidata: Update cron with lb and lb-pool params" https://gerrit.wikimedia.org/r/801762 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 797077 **merged** by Giuseppe Lavagetto: [operations/puppet@production] maintenance::wikidata: Update cron with lb and lb-pool params https://gerrit.wikimedia.org/r/797077 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a comment. I believe that this will do the right thing! TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Addshore Cc: akosiaris, Lydia_Pintscher, bking, karapayneWMDE,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Joe added a comment. Given the changes we've made to puppet in the meantime, I am now able to feed the right parameters to the script if we want to. The following patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/797077 will cause these corresponding changes in puppet:

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 797077 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto): [operations/puppet@production] maintenance::wikidata: Update cron with lb and lb-pool params https://gerrit.wikimedia.org/r/797077 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a comment. I think @Ladsgroup s point is rather than we potentially do not need to include wdqs delay in maxlag now, as the updater is stable and fast. In T238751#7854615 , @Lydia_Pintscher wrote: > Yes, we've had

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Lydia_Pintscher added a comment. Yes, we've had several occurrences over the last months where editors complained about maxlag being high and it was caused by a server being depooled. (And yes I really believe we need to keep the maxlag coupling.) TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Ladsgroup added a comment. Do we really need this now that everything is on flink and fancy? TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Ladsgroup Cc: akosiaris, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

akosiaris added a comment. In T238751#7851690 , @Addshore wrote: > @Joe (Also pinging @akosiaris as I know joe is out right now). > It seems like the ideal solution of T239392: Applications and scripts need to be able to understand

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore added a subscriber: akosiaris. Addshore added a comment. @Joe (Also pinging @akosiaris as I know joe is out right now). It seems like the ideal solution of T239392: Applications and scripts need to be able to understand the pooled status of servers in our load balancers.

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board. MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T238751 WORKBOARD

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 764830 **merged** by Bking: [operations/puppet@production] Revert "Temp remove codfw from wikidata updateQueryServiceLag check" https://gerrit.wikimedia.org/r/764830 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Lydia_Pintscher added a subtask: T239392: Applications and scripts need to be able to understand the pooled status of servers in our load balancers.. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Lydia_Pintscher added a parent task: T270614: Automatically depool wdqs servers that are "lagged". TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Lydia_Pintscher Cc: Lydia_Pintscher, bking,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Lydia_Pintscher raised the priority of this task from "Low" to "High". Lydia_Pintscher added a comment. Changing the priority to high based on today's discussion in the query service sync. This is becoming more important for editors because maxlag is growing when a server is depooled,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

ItamarWMDE added a project: User-ItamarWMDE. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, ItamarWMDE Cc: karapayneWMDE, Lucas_Werkmeister_WMDE, Ladsgroup, Gehel, Jheald, Joe, Addshore,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

dcausse added a project: Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, dcausse Cc: karapayneWMDE, Lucas_Werkmeister_WMDE, Ladsgroup, Gehel, Jheald, Joe, Addshore,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

RKemper added a project: SRE-OnFire. TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, RKemper Cc: Lucas_Werkmeister_WMDE, Ladsgroup, Gehel, Jheald, Joe, Addshore, Aklapper,

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 764830 had a related patch set uploaded (by Addshore; author: Addshore): [operations/puppet@production] Revert \"Temp remove codfw from wikidata updateQueryServiceLag check\" https://gerrit.wikimedia.org/r/764830 TASK DETAIL

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 764875 **merged** by Ryan Kemper: [operations/puppet@production] Temp remove codfw from wikidata updateQueryServiceLag check https://gerrit.wikimedia.org/r/764875 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

gerritbot added a comment. Change 764875 had a related patch set uploaded (by Addshore; author: Addshore): [operations/puppet@production] Temp remove codfw https://gerrit.wikimedia.org/r/764875 TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Lucas_Werkmeister_WMDE raised the priority of this task from "Lowest" to "Low". Lucas_Werkmeister_WMDE added a comment. Bumping priority as it has come up again. Between 5:43 and 8:35 UTC today, wdqs1005 caught up on triples

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore lowered the priority of this task from "Low" to "Lowest". Addshore added a comment. Lowest as this hasn't come up in quite some time, and blocked on being able to do this "properly". TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T238751: Only generate maxlag from pooled query service servers.

Addshore edited projects, added wdwb-tech-focus; removed Wikidata-Campsite (Wikidata-Campsite-Iteration-∞). TASK DETAIL https://phabricator.wikimedia.org/T238751 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Joe, Addshore Cc: Ladsgroup, Gehel,