> Is there any reason the first portion of the test name is being truncated?
There is, although it is slightly embarrassing: We currently only store the detailed data including full test case name and platform for about a week, for anything older than that the abridged version is the best I could find. The data should still be good, though, since we hopefully don't have two tests with the same name that are both frequently flaky. In particular, the ResourceStatistics refers to the 'MesosContainerizerSlaveRecoveryTest.ResourceStatistics' test tracked in MESOS-5048. On Fri, Oct 12, 2018 at 7:03 PM Benjamin Mahler <bmah...@apache.org> wrote: > Thanks for sending this Benno! I for one would love to see more regular > communication about the state of CI, especially so that I know how I can > help fix tests (right now I don't know which flaky tests are in areas I am > maintaining). > > Is there any reason the first portion of the test name is being truncated? > For example, ResourceStatistics matches several tests: > > $ grep -R ' ResourceStatistics)' src/tests > src/tests/containerizer/xfs_quota_tests.cpp:TEST_F(ROOT_XFS_QuotaTest, > ResourceStatistics) > > src/tests/slave_recovery_tests.cpp:TEST_F(MesosContainerizerSlaveRecoveryTest, > ResourceStatistics) > src/tests/disk_quota_tests.cpp:TEST_F(DiskQuotaTest, ResourceStatistics) > > Did we actually fix the flaky tests or did we disable them? I see only 22 > disabled tests, which is better than I expected, but I hope there's good > tracking on getting these un-disabled again: > > $ grep -R DISABLED src/tests | grep -v DISABLED_ON_WINDOWS | grep -v > NestedQuota | grep -v ChildRole | grep -v NestedRoles | grep -v > environment.cpp | wc -l > 22 > > On Fri, Oct 12, 2018 at 7:38 AM Benno Evers <bev...@mesosphere.com> wrote: > > > Hey all, > > > > as you might know, we've set up an internal CI system that is running > `make > > check` on a variety of different platforms and configurations, 16 in > total. > > > > As we've experienced more and more pain maintaining a green master, I've > > compiled some statistics about which tests are most flaky. I thought > other > > people might also be interested to have a look at that data: > > > > Last Week: > > > > # CI Statistics since 2018-10-05 14:22:35.422882 for branches > > containing 'asf/master' > > Total: 41 failing tests, 28 unique. (avg 0.142361111111 failing tests > > per build) > > > > Top 5 failing tests: > > 6x: [empty] > > 4x: ResourceStatistics > > 2x: CreateDestroyDiskRecovery > > 2x: INTERNET_CURL_InvokeFetchByName > > 2x: RecoverNestedContainer > > > > Last Month: > > > > # CI Statistics since 2018-09-12 14:23:36.272031 for branches > > containing 'asf/master' > > Total: 320 failing tests, 75 unique. (avg 0.285714285714 failing > tests > > per build) > > > > Top 5 failing tests: > > 57x: Used > > 32x: LongLivedDefaultExecutorRestart > > 27x: PythonFramework > > 23x: ROOT_CGROUPS_LaunchNestedContainerSessionsInParallel > > 22x: ResourceStatistics > > > > Last year: > > > > # CI Statistics since 2017-10-12 14:24:31.639792 for branches > > containing 'asf/master' > > Total: 3045 failing tests, 225 unique. (avg 0.184054642166 failing > > tests per build) > > > > Top 5 failing tests: > > 292x: [empty] > > 272x: > ROOT_LOGROTATE_UNPRIVILEGED_USER_RotateWithSwitchUserTrueOrFalse > > 136x: LOGROTATE_RotateInSandbox > > 136x: LOGROTATE_CustomRotateOptions > > 131x: ResourceStatistics > > > > > > I don't really have a point with all of this, but some observations: > > - [empty] means that the `mesos-tests` binary crashed > > - The data also includes "real", i.e. non-flaky test failures, but they > > should not appear in the top 5 lists because we would hopefully either > > revert or fix them before they can accumulate dozens of failures > > - Over the whole year, we seem to be pretty good at fixing the nastiest > > flakes, with only one of the top 5 still appearing in this weeks test > > results > > - Sadly, the fail percentage isn't as different between now and then as > we > > might have hoped. > > > > Hope this was interesting, and best regards, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > > -- Benno Evers Software Engineer, Mesosphere
