We, as in the performance team, are currently working on an automatic alerting system for Telemetry's data. At the moment we can detect sensible changes in the distributions of the collected histograms and we would like to notify the authors of the histograms with an e-mail.
The e-mail will include the name of the histogram, the date of the regression and a plot that shows two histograms: the reference and the regression distribution. The regression histogram shows the distribution of the histogram taken the day it changed, while the reference histogram shows how the distribution should look like based on the previous days. In the future the e-mail will include a link to Telemetry's dashboard which will likely show some more information. Since using 'hg blame' or a similar approach to pinpoint the author of the histogram doesn't always yield the right match (see Bug 1037494), we are likely going to ask the interested authors to manually add their e-mail address to toolkit/components/telemetry/Histograms.json. The question to the histogram authors now is if receiving alerts through e-mails is OK or they would prefer something else. Note that a great deal of care has been taken to ensure that there the number of false positives is very close to 0, i.e. if you receive an alert then the distribution has clearly changed and the effect size is big enough that it can be considered more than significant. So don't expect your mailbox to be flooded with useless alerts. Roberto _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
