> On 2 Dec 2018, at 17:02, Vincent Massol <[email protected]> wrote:
>
>
>
>> On 27 Nov 2018, at 18:19, Vincent Massol <[email protected]> wrote:
>>
>>
>>
>>> On 26 Nov 2018, at 00:44, Vincent Massol <[email protected]> wrote:
>>>
>>> Hi devs,
>>>
>>> I wanted to keep up to date on the list of stuff that we still need to do
>>> in the domain of Docker, CI & Clover, specifically on the domains I’ve been
>>> working on. All the items below are items I need to work on (but any help
>>> is welcome of course).
>>>
>>> 1) Clover pipeline job to modify to add docker-based functional tests:
>>> right now the global TPC is going to decrease because of MenuIT and MailIT.
>>> This needs to be fixed ASAP before we convert too many tests.
>>
>> This should be good now.
>>
>> One more idea:
>>
>> 1.1) In the XWiki Clover report, compare the failing tests reported by
>> Clover between old report (reference point) and the new one to gauge if the
>> TPC computed is meaningful or not.
>>
>> 1.2) In the XWiki clover report, provide a diff list of executed tests to
>> find out problems with some tests not executed which would lead to TPC being
>> wrong.
>
> I’ve almost finished to implement this but there’s a problem… For some reason
> I don’t understand Clover doesn’t report correctly failing tests:
>
> 1) A lot of tests are failing in the clover report but not in the build. For
> example:
>
> In Clover build logs:
>
> "
> [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011
> s - in IdGeneratorTest
> [INFO] Running TransformationContextTest
> “
>
> In clover.xml report:
> https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1
>
> ie 2 tests failing in IdGeneratorTest:
> testGenerateUniqueIdWhenInvalidEmptyPrefix &
> testGenerateUniqueIdWhenInvalidNonAlphaPrefix
>
> 2) Several tests failing in the Clover build logs but not in the clover
> report. For example:
>
> In Clover build logs:
>
> “
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT
> [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR!
> “
>
> In clover.xml report:
> https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45
>
> ie no failure at all…
>
> Thus it’s not possible to trust the clover.xml report… The only solution I
> can think of would be to compare either with the local Maven build logs or
> with the Jenkins test report logs. But that means we can’t go back in the
> past since we don’t have those logs for past clover reports.
>
> The good news is that when Clover says that tests failed, they actually
> didn't fail for a lot of them (However we would need to check if the coverage
> done by those tests is included but I think they are from what I see:
> IdGenerator.java has coveredelements="75” & elements=“84”. That could be from
> tests other than IdGeneratorTest though).
3) Some tests fail because of Clover.
For example IncludeMacroTest:
https://gist.github.com/vmassol/30cb78322dc5a4a50ee5d4a41c383a79
We should be able to figure out the problem with his one.
Thanks
-Vincent
>
> Thanks
> -Vincent
>
>>
>>> 2) Also fix Clover pipeline job in general which hasn’t passed since the
>>> 22nd of October
>>
>> Just passed today. Need to monitor it. Future failures should be related to
>> point 6) below. Note that the clover job is now executed on a4 which should
>> make it a lot more stable.
>>
>> Thanks
>> -Vincent
>>
>>> 3) Execute Docker-based tests in main pipeline job for the default config
>>> as otherwise we might have the platform job succeeding but it’ll miss
>>> several functional tests! Option 1: execute Browser outside of docker as
>>> option, Option 2: add more docker-based agents. For the moment, we might be
>>> able to run the docker tests on agent4.
>>>
>>> 4) Screenshot attachment on UI test failures to fix in pipeline job
>>>
>>> 5) Flicker recognition on tests to fix in pipeline job
>>>
>>> 6) Fix the ClosedChannelExcption/InterruptedException in our CI
>>>
>>> 7) Research running Jenkins agents in docker images and make our Docker
>>> test work (docker on docker). Once this works, set up 2 or 2 agents on a4.
>>> And then move all other agents to this.
>>>
>>> 8) Still need to apply the Clover strategy we agreed on (global TPC needs
>>> to not go down during releases).
>>>
>>> 9) Find strategy to move Docker image creation in platform build
>>>
>>> Let me know if you have questions.
>>>
>>> Thanks
>>> -Vincent
>>>
>>> PS: I didn’t list fixing tests (such and flicker fixing or WCAG test
>>> fixing) since this not infra and I wanted to limit this list to infra.
