[Wikidata-bugs] [Maniphest] T330770: Investigate DispatchChanges Normal job backlog time (mean avg, 15min) alert post datacenter switch

2023-08-02 Thread lmata 
lmata removed a project: observability.

TASK DETAIL
  https://phabricator.wikimedia.org/T330770

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: lmata
Cc: Michael, Aklapper, Lucas_Werkmeister_WMDE, fgiunchedi, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
lmata, ItamarWMDE, Akuckartz, Nandana, colewhite, Lahi, Gq86, herron, 
GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, 
Wikidata-bugs, aude, Mbch331, Chicocvenancio, Volans
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T330770: Investigate DispatchChanges Normal job backlog time (mean avg, 15min) alert post datacenter switch

2023-08-02 Thread lmata 
lmata added a project: Observability-Alerting.

TASK DETAIL
  https://phabricator.wikimedia.org/T330770

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: lmata
Cc: Michael, Aklapper, Lucas_Werkmeister_WMDE, fgiunchedi, 
Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, 
lmata, ItamarWMDE, Akuckartz, Nandana, colewhite, Lahi, Gq86, herron, 
GoranSMilovanovic, Chicocvenancio, QZanden, LawExplorer, Volans, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T330770: Investigate DispatchChanges Normal job backlog time (mean avg, 15min) alert post datacenter switch

2023-04-25 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  > However, on Grafana it was still Alerting until I changed the data source 
 (following a 
recommendation by @fgiunchedi).
  
  FTR, I just applied the same fix to the corresponding panel 
 on the 
Wikidata Alerts dashboard.

TASK DETAIL
  https://phabricator.wikimedia.org/T330770

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Michael, Aklapper, Lucas_Werkmeister_WMDE, fgiunchedi, Astuthiodit_1, 
karapayneWMDE, Invadibot, maantietaja, lmata, ItamarWMDE, Akuckartz, Nandana, 
Lahi, Gq86, herron, GoranSMilovanovic, Chicocvenancio, QZanden, LawExplorer, 
Volans, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T330770: Investigate DispatchChanges Normal job backlog time (mean avg, 15min) alert post datacenter switch

2023-02-28 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  The two emails I received (I assume they’re fine to share):
  F36877228: [FIRING 1] DispatchChanges Normal job backlog time (mean avg, 
15min) alert (MF0FSjJ4z AlertManager , cxserver Wikidata MF0FSjJ4z critical 
wikidata).eml 
  F36877227: [RESOLVED] DispatchChanges Normal job backlog time (mean avg, 
15min) alert (MF0FSjJ4z AlertManager , cxserver Wikidata MF0FSjJ4z critical 
wikidata).eml 
  
  Quoting the first one for convenience:
  
  >  [1] Firing
  > Labels
  > alertname = DispatchChanges Normal job backlog time (mean avg, 15min) alert
  > __alert_rule_uid__ = MF0FSjJ4z
  > __contacts__ = "AlertManager","cxserver"
  > grafana_folder = Wikidata
  > rule_uid = MF0FSjJ4z
  > severity = critical
  > team = wikidata
  > Annotations
  > _alertId__ = 309
  > __dashboardUid__ = TUJ0V-0Zk
  > __orgId__ = 1
  > __panelId__ = 28
  > __value_string__ = [ var='B0' metric='NoData' labels={} value=null ]
  > grafana_state_reason = NoData
  > message = DispatchChanges job backlog is over 10 minutes! Normal values are 
between 0.5s and 1s
  >
  > Source 

TASK DETAIL
  https://phabricator.wikimedia.org/T330770

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Aklapper, Lucas_Werkmeister_WMDE, fgiunchedi, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, lmata, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
herron, GoranSMilovanovic, Chicocvenancio, QZanden, LawExplorer, Volans, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T330770: Investigate DispatchChanges Normal job backlog time (mean avg, 15min) alert post datacenter switch

2023-02-28 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE created this task.
Lucas_Werkmeister_WMDE added projects: observability, Wikidata.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  The alert DispatchChanges Normal job backlog time (mean avg, 15min) 
 became 
Firing at 14:40 UTC, but Resolved at 15:00 UTC (according to my email inbox). 
However, on Grafana it was still Alerting until I changed the data source 
 (following a 
recommendation by @fgiunchedi). It’s currently unclear why it seemed to 
disappear from alerts.w.o when grafana.w.o still had it.

TASK DETAIL
  https://phabricator.wikimedia.org/T330770

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Aklapper, Lucas_Werkmeister_WMDE, fgiunchedi, Astuthiodit_1, karapayneWMDE, 
Invadibot, maantietaja, lmata, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, 
herron, GoranSMilovanovic, Chicocvenancio, QZanden, LawExplorer, Volans, 
_jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org