[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

gerritbot added a comment. Change 552277 **merged** by jenkins-bot: [wikidata/query/rdf@master] Start dropping requests when load is too high. https://gerrit.wikimedia.org/r/552277 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-23T18:19:57Z] repool wdqs1007, catched up on lag - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-22T09:27:39Z] depool wdqs1007 to allow to catch up on lag - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

gerritbot added a comment. Change 552277 had a related patch set uploaded (by Gehel; owner: Gehel): [wikidata/query/rdf@master] Start dropping requests when load is too high. https://gerrit.wikimedia.org/r/552277 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Addshore added a comment. I looked into some of the requests and made a quick poke to the agent filtering. Didn't tag the task, so here is the link... https://gerrit.wikimedia.org/r/#/c/wikidata/query/deploy/+/552236/ TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Tarrow added a comment. Looks like depooling 1004 resulted in a pretty quick drop in lag. 1.5 hours was enough and it hasn't regrown in greater than 12 hours. Its not a pretty solution but: is there any point in regularly rotating each wdqs instance out of the pool? This might keep it

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-20T21:35:52Z] repool wdqs1004 - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-20T16:03:58Z] depool wdqs1004 to allow catching up on lag - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Gehel added a comment. one additional server has been moved to the public cluster to provide more resources. Let's see if it helps. TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

gerritbot added a comment. Change 551189 **merged** by Gehel: [operations/puppet@production] wdqs: move wdqs1007 from internal to public cluster https://gerrit.wikimedia.org/r/551189 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

gerritbot added a comment. Change 551189 had a related patch set uploaded (by Gehel; owner: Gehel): [operations/puppet@production] wdqs: move wdqs1007 from internal to public cluster https://gerrit.wikimedia.org/r/551189 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-14T20:04:25Z] reloading data on wdqs1004 from wdqs1007 to catch up on lag faster - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Gehel added a comment. Current situation: - `wdqs1004` is depooled to see if it helps it catch up on lag - `wdqs1005` has had it's journal reset as part of T238232 - `wdqs1006` has catch up on lag on its own Currently, the public WDQS

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-14T13:35:22Z] depool wdqs1004 to allow catching up on lag - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Jheald added a comment. One thing that seems odd (to an outsider like me who knows very little about the system) is that some servers seem to be performing so much worse than others. Is there a simple reason for this (eg an entire cluster having problems?), or does this suggest there

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Gehel added a comment. now that all traffic is going to codfw, lag should be down. It will probably start to rise now that the load is increasing. TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] [Commented On] T238229: WDQS is having high update lag for the last week

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-11-14T09:55:32Z] depool wdqs (public) eqiad - high lag - T238229 TASK DETAIL https://phabricator.wikimedia.org/T238229 EMAIL PREFERENCES