[
https://issues.apache.org/jira/browse/HBASE-8018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-8018:
---------------------------------
Attachment: hbase-8018-v2.patch
Incorporated [~enis] comments and added the support to dump skipped test suites
from last N runs(N is a user specified command line argument)
> Add "Flaky Testcase Detector" tool into dev-tools
> -------------------------------------------------
>
> Key: HBASE-8018
> URL: https://issues.apache.org/jira/browse/HBASE-8018
> Project: HBase
> Issue Type: Bug
> Components: util
> Reporter: Jeffrey Zhong
> Assignee: Jeffrey Zhong
> Fix For: 0.98.0
>
> Attachments: hbase-8018.patch, hbase-8018-v1.patch,
> hbase-8018-v2.patch
>
>
> Recently I'm working on some hbase test case failures and I think it's useful
> if we can see a report of all failed test cases from most recent runs. So we
> can easily see how flaky a test case is. I wrote a tool some time back and
> below are some reports against different branches from today's run. You can
> get the tools' source at https://github.com/jeffreyz88/jenkins-tools. If we
> can run it daily and send out an email then we can quickly notice what may
> break after recent check-ins.
> Notes:
> 1) " 873 874 875 876 877 878 879 880 881" are builds which has failed
> test cases in current or previous runs
> 2) 1 means "PASSED", 0 means "NOT RUN AT ALL", -1 means "FAILED"
> HBase-0.95 (from last 10 runs configurable)
> Failed Test Cases 21 22 23 24 25 27
> ========================================================
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
> 0 0 0 0 -1 -1
> org.apache.hadoop.hbase.coprocessor.example.testbulkdeleteprotocol.testbulkdeletecolumn
> -1 0 0 0 0 0
> org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint.org.apache.hadoop.hbase.coprocessor.example.testrowcountendpoint
> -1 0 0 0 0 0
> org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver.org.apache.hadoop.hbase.coprocessor.example.testzookeeperscanpolicyobserver
> -1 0 0 0 0 0
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
> 1 -1 -1 0 1 1
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
> 1 1 1 1 1 -1
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
> 1 1 1 -1 -1 -1
> org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 1
> 1 1 -1 0
> org.apache.hadoop.hbase.replication.testmasterreplication.testcyclicreplication
> 1 1 1 1 1 -1
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
> 1 -1 0 1 -1 0
> org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover
> 0 1 -1 0 -1 0
> org.apache.hadoop.hbase.security.access.testaccesscontroller.org.apache.hadoop.hbase.security.access.testaccesscontroller
> 0 -1 0 0 0 0
> As you can see we have a few test cases don't run successfully at all or
> recently.
> HBase-TRUNK (from last 10 runs)
> Failed Test Cases 3908 3909 3910 3912 3913 3914 3915 3916
> ========================================================
> org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb.org.apache.hadoop.hbase.catalog.testmetamigrationconvertingtopb
> 0 -1 0 0 0 0 -1 -1
> org.apache.hadoop.hbase.client.testadmin.testcloseregionwhenservernameisempty
> 1 1 1 1 1 1 -1 0
> org.apache.hadoop.hbase.client.testscannertimeout.test3686a 1 1 -1
> 0 1 1 1 1
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testofflinesnapshotregionoperationsindependent
> 0 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.client.testsnapshotcloneindependence.testonlinesnapshotregionoperationsindependent
> 1 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.master.testassignmentmanageroncluster.testmoveregion
> 1 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.master.testdistributedlogsplitting.testdelayeddeleteonfailure
> 1 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
> 1 1 1 1 1 -1 0 1
> org.apache.hadoop.hbase.master.testtablelockmanager.testdelete 1 1 -1
> 0 1 1 1 1
> org.apache.hadoop.hbase.procedure.testzkprocedurecontrollers.testsimplezkcohortmembercontroller
> 1 1 1 1 1 1 -1 0
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testexistingznodeblockssplitandwerollback
> 1 1 1 1 1 -1 0 1
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshouldthrowioexceptionifstorefilesizeisemptyandshouldsuccessfullyexecuterollback
> 1 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
> -1 -1 -1 0 -1 0 -1 0
> org.apache.hadoop.hbase.regionserver.wal.testhlog.testlogcleaning 0 -1
> -1 0 1 1 1 1
> org.apache.hadoop.hbase.replication.testmultislavereplication.testmultislavereplication
> 1 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.replication.testreplicationqueuefailover.queuefailover
> 1 1 1 -1 0 -1 0 1
> org.apache.hadoop.hbase.replication.testreplicationqueuefailovercompressed.queuefailover
> 1 1 1 -1 0 -1 0 1
> org.apache.hadoop.hbase.security.access.testaccesscontrolfilter.testqualifieraccess
> -1 0 1 -1 0 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testdegenerateregions 1 -1
> 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testfixassignmentsandnohdfschecking
> 1 -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testfixbytable 1 -1 0 1
> 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testhbckthreadpooling 1 -1
> 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testlingeringreferencefile 1
> -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testmissingfirstregion 1 -1
> 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testmissinglastregion 1 -1
> 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testnotinhdfs 1 -1 0 1
> 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testnotinmetaordeployedhole 1
> -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testoverlapandorphan 1 -1
> 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testregionhole 1 -1 0 1
> 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testregionshouldnotbedeployed 1
> -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testsidelineoverlapregion 1
> -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.util.testhbasefsck.testsplitdaughtersnotinmeta 1
> -1 0 1 1 1 1 1
> More test cases are not in good shape.
> HBase-0.94(from last 10 runs)
> Failed Test Cases 873 874 875 876 877 878 879 880 881
> ========================================================
> org.apache.hadoop.hbase.backup.testhfilearchiving.testcleaningrace -1 0
> 1 1 1 1 1 1 1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testgetregion 1 1
> -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testretrying 1 1
> -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testscanmetafortable
> 1 1 -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.catalog.testmetareadereditor.testtableexists 1
> 1 -1 0 1 1 1 1 1
> org.apache.hadoop.hbase.client.testmetamigrationremovinghtd.org.apache.hadoop.hbase.client.testmetamigrationremovinghtd
> 0 -1 0 0 0 0 0 0 0
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitsnoabort 1
> 1 1 1 1 -1 0 1 1
> org.apache.hadoop.hbase.client.testmultiparallel.testflushcommitswithabort
> 1 1 1 1 1 -1 0 1 1
> org.apache.hadoop.hbase.client.testshell.testrunshelltests 1 1 1
> 1 1 1 -1 0 1
> org.apache.hadoop.hbase.master.testmasterfailover.testmasterfailoverwithmockedritondeadrs
> 1 1 -1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.regionserver.testatomicoperation.testmultirowmutationmultithreads
> 1 1 1 -1 0 1 1 1 1
> org.apache.hadoop.hbase.regionserver.testsplittransactiononcluster.testshutdownfixupwhendaughterhassplit
> 1 -1 0 1 1 1 1 1 1
> As you can see 0.94 is stable and testmetamigrationremovinghtd hasn't been
> run for a while either being removed or due to other failures. Some test
> cases aren't stable but not too bad.
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
