Hi Thomas, > On 7 Nov 2018, at 08:44, Thomas Mortagne <[email protected]> wrote: > > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote: >> >> Hi devs, >> >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: >> >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the >> maximum number of configurations, in order to flesh out configs that don’t >> start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat >> 9.x docker image uses java9+). The job would not send a mail on failure but >> it would update a report page (could even update a page on xwiki.org >> directly or if too complex update some page on maven.xwiki.org somewhere). >> This job would run not very often but say once per week. Note that one >> config takes 3-4 minutes to run, so 50 configs would take 3 hours which is >> acceptable. > > That's interesting indeed. +1 > >> >> 2) Job 2: Execute all functional tests on a subset of supported configs. For >> example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we >> already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will >> take a long time to execute. We’ll start with 3-4 configs and will go to >> about 10 configs when we add more. The tests will take roughly 2 hours to >> execute per config I think. So a total of 20 hours when we have 10 configs. >> If we run those once per week it should be fine. > > Can't those configs be executed in parallel in different agents ?
Yes this is fully parallelizable. However we have only one agent with docker ATM. Once we migrate more agents then we can have them in parallel. Thanks -Vincent > >> >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as >> part of the platform JenkinsFile. >> >> WDYT? >> >> Thanks >> -Vincent >> > > > -- > Thomas Mortagne
