Krinkle added a comment. |
Added a mitigation section to the task description. Also a summary of the impact of the mitigations so far (based on input from @aaron).
Dashboard: Job Queue Helath |
---|
F9232210: Screen Shot 2017-08-31 at 00.00.31.png F9232209: Screen Shot 2017-08-31 at 00.00.18.png |
Job queue size down from 10M to a steady ~2M. Before the regression it was a steady between 100K -1M. |
Dashboard: Varnish stats |
---|
F9232784: Screen Shot 2017-08-31 at 00.15.34.png |
Purge rate from production Varnish servers reduced by 2-3X, from 75-100K/s to ~30K/s. |
Dashboard: Job Queue Rate for htmlCacheUpdate |
---|
F9232855: Screen Shot 2017-08-31 at 00.21.36.png |
Queue rate of htmlCacheUpdate back to normal. Deduplication/Superseding optimisation is now working. Execution speed has increased. |
TASK DETAIL
EMAIL PREFERENCES
To: Krinkle
Cc: GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, Vali.matei, Avner, Zppix, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb
Cc: GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, Vali.matei, Avner, Zppix, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb
_______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs