Hi Andrey, As far as I know tests annotated by @Ignore can be conveniently tracked on Teamcity but this requires such tests to be included into JUnit 4 test suite (org.junit.runners.Suite).
Above requirement currently isn't met by many Ignite tests which are included in JUnit 3 suites (junit.framework.TestSuite). This means that test cases annotated @Ignore simply are hidden when present in such old-fashioned suites. We are currently working on migrating Junit 3 suites to JUnit 4. If you are interested, work breakdown and progress can be found in JIRA: - https://issues.apache.org/jira/browse/IGNITE-10762 ----- With regards to Assume API, it seems more complicated. I have recently read that tests that are conditionally skipped using this API are reported by JUnit 4 test framework as passed. This is not very convenient. I think I have seen somewhere mentioned that proper reporting of conditionally skipped tests can be achieved only with JUnit 5. But I didn't dig deeper there because we haven't yet decided whether it is worth it for Ignite to move to JUnit 5 (https://issues.apache.org/jira/browse/IGNITE-10180). Given above, I believe that the proper way to mute tests is currently with @Ignore and Assume API. It's not 100% ideal but overall looks better than the old one. regards, Oleg Andrew Mashenkov wrote > Dmitry, > > As far as I understand, @IgniteIgnore and fail("") works in similar way > and > requires test to be muted on TC manually. > And we have to unmute these tests on TC right after fix will be merged to > master, isn't it? > > @Ignore and Assume.that works different way comparing to previous two. > Junit just exclude such tests from run. > So, there is no need to mute\unmute them on TC at all, but... > Does TC treat such tests as muted ones and we can see them on "muted > tests" > page? > This may be useful to track tests state if someone forget to remove > @ignore > while fixing related ticket. > > > On Mon, Dec 17, 2018 at 5:54 PM Anton Vinogradov < > av@ > > wrote: > >> >> In this case, we will not need TeamCity mutes anymore. >> Sounds great! >> >> On Mon, Dec 17, 2018 at 5:23 PM Dmitrii Ryabov < > somefireone@ > > >> wrote: >> >> > Dmitriy, >> > >> > with ignore annotations we will need to check forgotten tests >> > manually. Thats bad, but we will save some machine time in common >> > runs. >> > пн, 17 дек. 2018 г. в 17:06, Dmitriy Pavlov < > dpavlov@ > >: >> > > >> > > Investigation parameters: Can leave as empty >> > > Unmute: Manual (don't use auto because of flaky tests) >> > > Text: https://issues.apache.org/jira/browse/IGNITE-NNNNN >> > > >> > > But anyway, I hope in the nearest future best way to mute test will >> be >> to >> > > ignore it using: >> > > @IgniteIgnore >> > > or @Ignore >> > > or Assume.that() >> > > >> > > In this case, we will not need TeamCity mutes anymore. >> > > >> > > I'm not sure which way of ignoring will be best. >> > > >> > > пн, 17 дек. 2018 г. в 17:02, Anton Vinogradov < > av@ > >: >> > > >> > > > Dmitrii, >> > > > >> > > > My question was mostly about how to mute test properly. >> > > > >> > > > >> > > > On Mon, Dec 17, 2018 at 4:27 PM Dmitrii Ryabov < >> > somefireone@ >> >> > > > wrote: >> > > > >> > > > > Anton, >> > > > > >> > > > > In the "Mute" column you can find link to the test on TeamCity. >> On >> > > > > this page you can see test details like run history and current >> > mutes. >> > > > > Here you should be able to unmute existing mutes. >> > > > > пн, 17 дек. 2018 г. в 16:21, Anton Vinogradov < > av@ > >: >> > > > > > >> > > > > > Great feature! >> > > > > > >> > > > > > Could you please explain how to have a deal with it? >> > > > > > How should I mite/unmute tests for now? >> > > > > > >> > > > > > On Fri, Dec 14, 2018 at 6:50 PM Dmitriy Pavlov < >> > dpavlov@ >> > > >> > > > > wrote: >> > > > > > >> > > > > > > We have IgniteIgnore, so we can use it. It seems behavior is >> the >> > > > same. >> > > > > We >> > > > > > > need just to understand in which case we can get issue ID >> from >> TC >> > > > REST >> > > > > (for >> > > > > > > the Bot). >> > > > > > > >> > > > > > > пт, 14 дек. 2018 г. в 17:51, Vyacheslav Daradur < >> > > daradurvs@ >> > > > >: >> > > > > > > >> > > > > > > > Dmitry Pavlov, >> > > > > > > > >> > > > > > > > As far as I know, we are able to use following construction >> to >> > > > ignore >> > > > > > > test: >> > > > > > > > Assume.assumeTrue("link-to-issue", false); >> > > > > > > > >> > > > > > > > On Fri, Dec 14, 2018 at 5:44 PM Dmitriy Pavlov < >> > > dpavlov@ >> > > > > >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > Thank you, Dmitrii, >> > > > > > > > > >> > > > > > > > > I've started unmuting tests starting from yesterday. >> > > > > > > > > >> > > > > > > > > Now muted test count was decreased from 3540 to 3408 >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > > >> > >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&tab=mutedProblems&branch_IgniteTests24Java8=%3Cdefault%3E >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > Probably we could automate this activity by auto-unmute, >> but >> > > > > hopefully, >> > > > > > > > > JUnit4 & @Ignore opportunity will come earlier. >> > > > > > > > > >> > > > > > > > > Sincerely, >> > > > > > > > > Dmitriy Pavlov >> > > > > > > > > >> > > > > > > > > пт, 14 дек. 2018 г. в 17:24, Dmitrii Ryabov < >> > > > > somefireone@ >> > > > > >: >> > > > > > > > > >> > > > > > > > > > Hello, Igniters! >> > > > > > > > > > >> > > > > > > > > > Our Bot got new functionality - it shows tests muted on >> > > > TeamCity >> > > > > and >> > > > > > > > their >> > > > > > > > > > ticket status [1]. >> > > > > > > > > > >> > > > > > > > > > So, we can see forgotten mutes for resolved tickets >> now. >> > > > > > > > > > >> > > > > > > > > > Igniters with rights to unmute tests on TC - please, >> check >> > this >> > > > > page >> > > > > > > > > > periodically for forgotten mutes. >> > > > > > > > > > >> > > > > > > > > > [1] https://mtcga.gridgain.com/mutes.html >> > > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > Best Regards, Vyacheslav D. >> > > > > > > > >> > > > > > > >> > > > > >> > > > >> > >> > > > -- > Best regards, > Andrey V. Mashenkov -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
