> On 2 Dec 2018, at 17:05, Vincent Massol <[email protected]> wrote:
>
>
>
>> On 2 Dec 2018, at 17:02, Vincent Massol <[email protected]> wrote:
>>
>>
>>
>>> On 27 Nov 2018, at 18:19, Vincent Massol <[email protected]> wrote:
>>>
>>>
>>>
>>>> On 26 Nov 2018, at 00:44, Vincent Massol <[email protected]> wrote:
>>>>
>>>> Hi devs,
>>>>
>>>> I wanted to keep up to date on the list of stuff that we still need to do
>>>> in the domain of Docker, CI & Clover, specifically on the domains I’ve
>>>> been working on. All the items below are items I need to work on (but any
>>>> help is welcome of course).
>>>>
>>>> 1) Clover pipeline job to modify to add docker-based functional tests:
>>>> right now the global TPC is going to decrease because of MenuIT and
>>>> MailIT. This needs to be fixed ASAP before we convert too many tests.
>>>
>>> This should be good now.
>>>
>>> One more idea:
>>>
>>> 1.1) In the XWiki Clover report, compare the failing tests reported by
>>> Clover between old report (reference point) and the new one to gauge if the
>>> TPC computed is meaningful or not.
>>>
>>> 1.2) In the XWiki clover report, provide a diff list of executed tests to
>>> find out problems with some tests not executed which would lead to TPC
>>> being wrong.
>>
>> I’ve almost finished to implement this but there’s a problem… For some
>> reason I don’t understand Clover doesn’t report correctly failing tests:
>>
>> 1) A lot of tests are failing in the clover report but not in the build. For
>> example:
>>
>> In Clover build logs:
>>
>> "
>> [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011
>> s - in IdGeneratorTest
>> [INFO] Running TransformationContextTest
>> “
>>
>> In clover.xml report:
>> https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1
>>
>> ie 2 tests failing in IdGeneratorTest:
>> testGenerateUniqueIdWhenInvalidEmptyPrefix &
>> testGenerateUniqueIdWhenInvalidNonAlphaPrefix
>>
>> 2) Several tests failing in the Clover build logs but not in the clover
>> report. For example:
>>
>> In Clover build logs:
>>
>> “
>> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
>> 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT
>> [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR!
>> “
>>
>> In clover.xml report:
>> https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45
>>
>> ie no failure at all…
>>
>> Thus it’s not possible to trust the clover.xml report… The only solution I
>> can think of would be to compare either with the local Maven build logs or
>> with the Jenkins test report logs. But that means we can’t go back in the
>> past since we don’t have those logs for past clover reports.
>>
>> The good news is that when Clover says that tests failed, they actually
>> didn't fail for a lot of them (However we would need to check if the
>> coverage done by those tests is included but I think they are from what I
>> see: IdGenerator.java has coveredelements="75” & elements=“84”. That could
>> be from tests other than IdGeneratorTest though).
>
> hmm found this in some forum:
>
> “
> Regarding, your original question regarding failing tests. Yes, if test is
> failing Clover won't include its contribution to overall coverage stats. The
> reasoning behind this behavior is that since the test fails, its not
> trustworthy to say it provide test coverage (confidence) to your code. “
>
> https://community.atlassian.com/t5/Clover-questions/clover-xml-does-not-contain-code-coverage-for-failing-test-case/qaq-p/704154
>
> So maybe they’re not counted after all…
Ok seems it’s actually counted in the global report and TPC, see:
https://stackoverflow.com/questions/41500243/does-clover-report-coverage-of-failed-unit-tests-also
"
• global coverage - from entire test run, this will always contain code
coverage from all tests (and not only tests, also from setUp methods etc)
• per-test coverage - code coverage is tracked for every test individually
“
Thanks
-Vincent
>
> Thanks
> -Vincent
>
>>
>> Thanks
>> -Vincent
>>
>>>
>>>> 2) Also fix Clover pipeline job in general which hasn’t passed since the
>>>> 22nd of October
>>>
>>> Just passed today. Need to monitor it. Future failures should be related to
>>> point 6) below. Note that the clover job is now executed on a4 which should
>>> make it a lot more stable.
>>>
>>> Thanks
>>> -Vincent
>>>
>>>> 3) Execute Docker-based tests in main pipeline job for the default config
>>>> as otherwise we might have the platform job succeeding but it’ll miss
>>>> several functional tests! Option 1: execute Browser outside of docker as
>>>> option, Option 2: add more docker-based agents. For the moment, we might
>>>> be able to run the docker tests on agent4.
>>>>
>>>> 4) Screenshot attachment on UI test failures to fix in pipeline job
>>>>
>>>> 5) Flicker recognition on tests to fix in pipeline job
>>>>
>>>> 6) Fix the ClosedChannelExcption/InterruptedException in our CI
>>>>
>>>> 7) Research running Jenkins agents in docker images and make our Docker
>>>> test work (docker on docker). Once this works, set up 2 or 2 agents on a4.
>>>> And then move all other agents to this.
>>>>
>>>> 8) Still need to apply the Clover strategy we agreed on (global TPC needs
>>>> to not go down during releases).
>>>>
>>>> 9) Find strategy to move Docker image creation in platform build
>>>>
>>>> Let me know if you have questions.
>>>>
>>>> Thanks
>>>> -Vincent
>>>>
>>>> PS: I didn’t list fixing tests (such and flicker fixing or WCAG test
>>>> fixing) since this not infra and I wanted to limit this list to infra.
