Self reminder: Just realized that I now need to modify the clover pipeline so that we now run docker-based tests too! This means the clover job can only run on agent4. This also means that I’ll need to setup FF on agent4 for non-docker-based functional tests.
Thanks -Vincent > On 18 Nov 2018, at 17:49, Vincent Massol <[email protected]> wrote: > > Ok I’ve created a 3rd job for unsupported configs: > https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_unsupported/ > > So we now have: > * Job 1: run all tests on latest versions of supported configs > * Job 2: run smoke tests (ie a few tests only) on all supported configs > * Job 3: run smoke tests on unsupported configs (MySQL 8.x, Tomcat 9.x, etc) > > Definitions are all here: > https://github.com/xwiki/xwiki-jenkins-pipeline/blob/06b679bf511261dbd0125a352a5f9074746aff83/src/org/xwiki/jenkins/DockerTests.groovy > > Thanks > -Vincent > > >> On 18 Nov 2018, at 16:57, Vincent Massol <[email protected]> wrote: >> >> >> >>> On 16 Nov 2018, at 22:03, Vincent Massol <[email protected]> wrote: >>> >>> Hi devs, >>> >>> So we now have 2 dockers jobs: >>> - one that executes daily and runs the latest version of the supported >>> configs. It runs all docker tests (right now that’s MenuIT but I’ll start >>> converting some other tests and we need to convert them all). See >>> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker/ >>> - another one that executes weekly and that runs all supported configs but >>> only on smoke tests (right now that MenuIT only). See >>> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/ >> >> And to see which configs are tested and passing or not: >> https://ci.xwiki.org/view/Recommended%20Builds/job/xwiki-platform_docker_all/lastCompletedBuild/testReport/org.xwiki.menu.test.ui/MenuIT/ >> >> Thanks >> -Vincent >> >>> All are passing FTM :) >>> >>> Thanks >>> -Vincent >>> >>>> On 5 Nov 2018, at 17:22, Vincent Massol <[email protected]> wrote: >>>> >>>> Hi devs, >>>> >>>> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: >>>> >>>> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on >>>> the maximum number of configurations, in order to flesh out configs that >>>> don’t start properly. For example XWiki on Tomcat 9.x would fail (since >>>> the Tomcat 9.x docker image uses java9+). The job would not send a mail on >>>> failure but it would update a report page (could even update a page on >>>> xwiki.org directly or if too complex update some page on maven.xwiki.org >>>> somewhere). This job would run not very often but say once per week. Note >>>> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours >>>> which is acceptable. >>>> >>>> 2) Job 2: Execute all functional tests on a subset of supported configs. >>>> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome >>>> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This >>>> job will take a long time to execute. We’ll start with 3-4 configs and >>>> will go to about 10 configs when we add more. The tests will take roughly >>>> 2 hours to execute per config I think. So a total of 20 hours when we have >>>> 10 configs. If we run those once per week it should be fine. >>>> >>>> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add >>>> as part of the platform JenkinsFile. >>>> >>>> WDYT? >>>> >>>> Thanks >>>> -Vincent >>>> >>> >> >
