David, Jini I understand your concerns. But the original idea of tiered testing is that tier1 failures are treated as urgent issues and to resolve. [1]
Here is list of test failures for 1000 runs of tier1 tests in Mach5. (I am not able to provide a link here) Please note that all SA tests are excluded on Solaris and MacosX already. 1 compiler/aot/calls/fromAot/AotInvokeSpecial2AotTest.java 2 serviceability/sa/ClhsdbFindPC.java 3 serviceability/sa/TestPrintMdo.java 4 serviceability/sa/ClhsdbJstack.java 5 serviceability/sa/ClhsdbJdis.java 6 compiler/c2/Test8004741.java 7 runtime/handshake/HandshakeWalkSuspendExitTest.java 8 runtime/handshake/HandshakeWalkSuspendExitTest.java 9 compiler/aot/calls/fromAot/AotInvokeVirtual2AotTest.java 10 runtime/handshake/HandshakeWalkExitTest.java 11 runtime/handshake/HandshakeWalkSuspendExitTest.java 12 serviceability/sa/ClhsdbRegionDetailsScanOopsForG1.java 13 serviceability/sa/ClhsdbRegionDetailsScanOopsForG1.java 14 compiler/aot/calls/fromAot/AotInvokeVirtual2AotTest.java The failures in of 'runtime/handshake/' are relatively caused by https://bugs.openjdk.java.net/browse/JDK-8214174 <https://bugs.openjdk.java.net/browse/JDK-8214174> but should be also fixed/excluded. SA tests are also unstable and there are no plans to fix them soon. So it means that we are going to have tier1 tests unstable for a long time. The possible way to make tier1 more stable would be to run only some very basic sanity SA tests in tier1. Might be to develop new sanity test which have some failover for existing SA bugs. Leonid [1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html <http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html> > On Dec 9, 2018, at 7:19 PM, Jini George <jini.geo...@oracle.com> wrote: > > Hi Leonid, > > I agree with David. I am also concerned about us not detecting SA breakages > (which could happen along with hotspot changes) soon enough. (Which was the > primary reason to get these tests in). > > Thank you, > Jini. > > On 12/8/2018 4:48 PM, David Holmes wrote: >> Hi Leonid, >> My concern here, if we care about keeping the SA operational, is that in >> tier3 these tests will not be covered by the jdk/submit testing process. >> David >> On 8/12/2018 3:53 pm, Leonid Mesnik wrote: >>> Hi >>> >>> Could you please review following fix which moves SA tests from tier1 to >>> tier3. There are some bugs which cause intermittent failures of any test. >>> SA tests fail intermittently are not stable enough for tier1. >>> However failures are not very frequent. Also I don't think that putting all >>> test in Problemlist.txt is very good idea because it left SA without any >>> testing at all. >>> So now all SA tests which are included in hotspot_tier3_runtime group. >>> >>> webrev: http://cr.openjdk.java.net/~lmesnik/8215042/webrev.00/ >>> bug: https://bugs.openjdk.java.net/browse/JDK-8215042 >>> >>> Leonid
