[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Clement_Goubert closed this task as "Resolved". Clement_Goubert claimed this task. Clement_Goubert added a comment. I fixed your alert too, which will now alert if p50 on 15 minutes goes over 10 minutes. We can resolve since you don't appear to have any other alerts on cp-jobqueue metrics.

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Lucas_Werkmeister_WMDE added a comment. Alright, thanks for making it make sense! Does that mean we can close this? TASK DETAIL https://phabricator.wikimedia.org/T341054 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lucas_Werkmeister_WMDE Cc:

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

akosiaris added a comment. Fixed the alert too. Took me a bit to figure out how to find it, thanks for posting the link in the task. TASK DETAIL https://phabricator.wikimedia.org/T341054 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: akosiaris

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

akosiaris added a comment. I think I have fixed the graphs now to be correct. They will definitely be more correct than previously where they were doing statistically wrong things (aggregating aggregates) TASK DETAIL https://phabricator.wikimedia.org/T341054 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

akosiaris added a comment. Sorry about that. For what is worth, we are approaching this piecemeal and this is the first instance. There are more changeprop related metrics that are wrongly summaries and not histograms, we will ping you before changing the next few ones. TASK DETAIL

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Clement_Goubert added a comment. In T341054#8987701 , @Lucas_Werkmeister_WMDE wrote: > In T341054#8987695 , @Clement_Goubert wrote: > >> Ah you have an alert on that metric, sorry

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Lucas_Werkmeister_WMDE added a comment. In T341054#8987695 , @Clement_Goubert wrote: > Ah you have an alert on that metric, sorry :( > We switched the metric to an histogram because aggregation was wrong, the job itself is ok. >

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Lucas_Werkmeister_WMDE added a comment. From an end-user perspective, Wikidata edits on English Wikipedia recent changes still seem to arrive as usual, so I don’t think anything’s immediately on fire.

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Clement_Goubert added a subscriber: akosiaris. Clement_Goubert added a comment. Ah you have an alert on that metric, sorry :( We switched the metric to an histogram because aggregation was wrong, the job itself is ok. @akosiaris is fixing the charts. TASK DETAIL

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Lucas_Werkmeister_WMDE added a comment. Also, if I reload the Grafana tab, the graph starts to look quite different, which is very confusing: F37127916: image.png TASK DETAIL https://phabricator.wikimedia.org/T341054 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Lucas_Werkmeister_WMDE added a subscriber: Clement_Goubert. Lucas_Werkmeister_WMDE added a comment. The cutoff in the 1h graph seems to be 45 minutes after the cutoff in the 15min graph, so if we speculate that the cause was 15 minutes before the cutoff in the 15min graph, then that would

[Wikidata-bugs] [Maniphest] T341054: Wikibase DispatchChanges job potentially broken

Lucas_Werkmeister_WMDE created this task. Lucas_Werkmeister_WMDE added projects: Wikidata, wdwb-tech, WMF-JobQueue. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION At 10:04 UTC, the wikidata-monitoring email received an alert about “DispatchChanges Normal job backlog time