On 2015-08-31 14:49:33 -0400 (-0400), Ronald Bradford wrote: [...] > fungi provided https://review.openstack.org/#/c/192253/ which is a > draft spec for a much wider ci-watch effort however I see this a > more specific example. I am seeking feedback from various projects > on interest, concerns and recommendations before creating a spec.
Agreed, I simply brought up Sean's proposed spec since code coverage job results fall within this class of (periodic and post pipeline) result discoverability and analysis problems, so we should make sure that whatever solution we choose for coverage reports specifically is compatible with a broader solution and the spec review is a good place to discuss that. > I see the initial steps to implement this as. > > 1. Agree on future domain url location for reports (e.g. > coverage.openstack.org) Well, we already have a URL location for the reports, and it's logs.openstack.org. They're published there today. What we need is some better indexing and perhaps automated analysis tools linking into there, which could either be hosted from logs.o.o or some other domain name. I just worry that "what URL do we want?" is descending into bikeshed territory, since it's putting the _least_ important decision first. > 2. Add to the merge committing process a step similar to the > existing coverage-log publisher > (project-config/jenkins/jobs/macros.yaml) to agreed location. Why publish the same file in two places? > 3. Provide initially a top level index page at url for published > reports This seems like 99% of the need, and we've been in agreement for at least a couple cycles that we'd love for someone to tackle getting this designed and implemented. > These steps enable a single current occurrence of code coverage > per project. It is not the objective to keep the historical HTML > output. Though we do (for a few months at the moment, but we could work out a way to keep summaries longer if there's interest). > Following this, the next steps are to record coverage for > historical analysis with the following. > > 1. Alter tox.ini [testenv:cover] to produce a text based version > (aka coverage report -m). This is much easier to parse and compare > different versions. I would propose we keep a weekly/daily dated > file of this output, which would be automatically published as per > previous publisher. Things are likely to stall here. We have 291 active repos with a [testenv:cover] section in their tox.ini at the moment. Might be worth considering whether we could extend jenkins/scripts/run-cover.sh in openstack-infra/project-config so that it does this part (assuming that devs are unlikely to want that file most of the time anyway). > 2. Use these dated files to produce basic per project analysis of > lines/percentage coverage over time, publishing this to a set > directory structure (e.g. /analysis/) and linked from the top > level index page. Over time this can be improved to provide simple > graphical output. We already have a statsd server at graphite.openstack.org, so poking values into it might be the simplest way forward. Then we can embed graphs into the resulting dashboard interface from there similar to what the http://status.openstack.org/zuul/ page does (all the graphs on that page are generated real-time from data in graphite). -- Jeremy Stanley __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev