I wonder if we can add graph to community metrics showing ignored tests by language/project/overall. That can be useful to see focus area.
On Tue, May 12, 2020 at 12:28 PM Jan Lukavský <je...@seznam.cz> wrote: > +1, visualizing the number of ignored tests in a graph seems useful. Even > better with some slices (e.g. per runner, module, ...). > On 5/12/20 8:02 PM, Ahmet Altay wrote: > > +1 to generate a report instead of removing these tests. A report like > this could help us with prioritization. It is easier to address issues when > we can quantify how much of a problem it is. > > I am curious what we can do to incentivize reducing the number of > flaky/ignored tests? A report itself might provide incentive, it is > rewarding to see ignored tests numbers go down over time. > > On Mon, May 11, 2020 at 8:30 AM Luke Cwik <lc...@google.com> wrote: > >> Deleting ignored tests does lead us to losing the reason as to why the >> test case was around so I would rather keep it around. I think it would be >> more valuable to generate a report that goes on the website/wiki showing >> stability of the modules (num tests, num passed, num skipped, num failed >> (running averages over the past N runs)). We had discussed doing something >> like this for ValidatesRunner so we could show which runner supports what >> automatically. >> >> On Mon, May 11, 2020 at 12:53 AM Jan Lukavský <je...@seznam.cz> wrote: >> >>> I think that we do have Jira issues for ignored test, there should be no >>> problem with that. The questionable point is that when test gets Ignored, >>> people might consider the problem as "less painful" and postpone the >>> correct solution until ... forever. I'd just like to discuss if people see >>> this as an issue. If yes, should we do something about that, or if no, >>> maybe we can create a rule that test marked as Ignored for long time might >>> be deleted, because apparently is only a dead code. >>> On 5/6/20 6:30 PM, Kenneth Knowles wrote: >>> >>> Good point. >>> >>> The raw numbers are available in the test run output. See >>> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PreCommit_Java_Cron/2718/testReport/ >>> for >>> the "skipped" column. >>> And you get the same on console or Gradle Scan: >>> https://scans.gradle.com/s/ml3jv5xctkrmg/tests?collapse-all >>> This would be good to review periodically for obvious trouble spots. >>> >>> But I think you mean something more detailed. Some report with columns: >>> Test Suite, Test Method, Jira, Date Ignored, Most Recent Update >>> >>> I think we can get most of this from Jira, if we just make sure that >>> each ignored test has a Jira and they are all labeled in a consistent way. >>> That would be the quickest way to get some result, even though it is not >>> perfectly automated and audited. >>> >>> Kenn >>> >>> On Tue, May 5, 2020 at 2:41 PM Jan Lukavský <je...@seznam.cz> wrote: >>> >>>> Hi, >>>> >>>> it seems we are accumulating test cases (see discussion in [1]) that >>>> are >>>> marked as @Ignored (mostly due to flakiness), which is generally >>>> undesirable. Associated JIRAs seem to be open for a long time, and this >>>> might generally cause that we loose code coverage. Would anyone have >>>> idea on how to visualize these Ignored tests better? My first idea >>>> would >>>> be something similar to "Beam dependency check report", but that seems >>>> to be not the best example (which is completely different issue :)). >>>> >>>> Jan >>>> >>>> [1] https://github.com/apache/beam/pull/11614 >>>> >>>>
