[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

jijiki closed this task as "Resolved". jijiki claimed this task. jijiki added a comment. Bluntly closing TASK DETAIL https://phabricator.wikimedia.org/T264821 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jijiki Cc: jijiki, Michael, RhinosF1,

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

LGoto closed subtask T264881: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC as Resolved. Restricted Application added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T264821 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

RLazarus added a subtask: T266040: Large text objects are randomized to cache backends. TASK DETAIL https://phabricator.wikimedia.org/T264821 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RLazarus Cc: Michael, RhinosF1, Joe, LSobanski, Addshore,

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

JMeybohm triaged this task as "Medium" priority. TASK DETAIL https://phabricator.wikimedia.org/T264821 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm Cc: Michael, RhinosF1, Joe, LSobanski, Addshore, Ladsgroup, RLazarus, Marostegui,

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

LSobanski removed a project: DBA. LSobanski added a comment. Removing #DBA as there's nothing specific for us to do right now, do add us back if anything comes up. TASK DETAIL https://phabricator.wikimedia.org/T264821 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Joe added a comment. To clarify a bit - restbase has hourly spikes of requests for the `feed` endpoint, which go back to wikifeeds, which calls both restbase and the action api. From the graphs of calls from wikifeeds it's clear we have hourly peaks happening at :00 in the number of

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Addshore added a comment. 1:57 PM While i was browsing around I also saw spikes in action=query in grafana, but couldn't dive much deeper, but I guess wikifeeds calls that, and some of the wikibase api modules there Amir1 ... 1:58 PM which I guess is

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Ladsgroup added a comment. An update from IRC, it seems the culprit is wikifeeds: https://grafana.wikimedia.org/d/b1jttnFMz/envoy-telemetry-k8s?viewPanel=4=1=1602010785563=1602037491008=thanos=codfw=k8s=wikifeeds=restbase-for-services F32376861: image.png

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Addshore added a comment. 1:23 PM I also notice this? https://grafana.wikimedia.org/d/00202/api-frontend-summary?orgId=1=1602020402068=1602023755950 REST API backend internla requests spike? 1:23 PM also happened a week ago?

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Addshore added a comment. Flagging T263999: Some lua-calls with language specified does not end up in formatterCache up here as it could be related (not a currently prooven link) TASK DETAIL https://phabricator.wikimedia.org/T264821 EMAIL

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Addshore added a comment. One of the places to look that should track all access to this term related storage is https://grafana.wikimedia.org/d/00548/wikibase-sql-term-storage?orgId=1=30s=now-12h=now I don't see any worrying looking spikes or abnormal access patterns there. I

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Marostegui added a comment. The only common query I have found that executes on all hosts is: SELECT /* Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsResolver::selectTermsViaJoin 141.136.76.164 */ wbtl_id, wbtl_type_id, wbxl_language, wbx_text, wbpt_property_id FROM

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Marostegui added a comment. I forgot to paste: root@mwmaint2001:~# crontab -l -uwww-data | grep -w wikidatawiki */3 * * * * echo "$$: Starting dispatcher" >> /var/log/wikidata/dispatchChanges-wikidatawiki.log; /usr/local/bin/mwscript

[Wikidata-bugs] [Maniphest] T264821: Hourly read spikes against s8 resulting in occasional user-visible latency & error spikes

Marostegui added a project: Wikidata. Marostegui added subscribers: Ladsgroup, Addshore. Marostegui moved this task from Triage to In progress on the DBA board. Marostegui added a comment. Adding @Addshore and @Ladsgroup as they have lots of cool dashboards (that I am unable to find) where