On Wed, Nov 7, 2018 at 9:20 AM Vincent Massol <[email protected]> wrote: > > Hi Thomas, > > > On 7 Nov 2018, at 08:44, Thomas Mortagne <[email protected]> wrote: > > > > On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote: > >> > >> Hi devs, > >> > >> I’m still not sure but FTM I was thinking of having 2 pipeline jobs: > >> > >> 1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on > >> the maximum number of configurations, in order to flesh out configs that > >> don’t start properly. For example XWiki on Tomcat 9.x would fail (since > >> the Tomcat 9.x docker image uses java9+). The job would not send a mail on > >> failure but it would update a report page (could even update a page on > >> xwiki.org directly or if too complex update some page on maven.xwiki.org > >> somewhere). This job would run not very often but say once per week. Note > >> that one config takes 3-4 minutes to run, so 50 configs would take 3 hours > >> which is acceptable. > > > > That's interesting indeed. +1 > > > >> > >> 2) Job 2: Execute all functional tests on a subset of supported configs. > >> For example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome > >> if we already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This > >> job will take a long time to execute. We’ll start with 3-4 configs and > >> will go to about 10 configs when we add more. The tests will take roughly > >> 2 hours to execute per config I think. So a total of 20 hours when we have > >> 10 configs. If we run those once per week it should be fine. > > > > Can't those configs be executed in parallel in different agents ? > > Yes this is fully parallelizable. However we have only one agent with docker > ATM. Once we migrate more agents then we can have them in parallel.
Indeed forgot about that. Anyway +1 for both > > Thanks > -Vincent > > > > >> > >> Note: Once we have job 1 & 2, we won't need to have the smoke tests I add > >> as part of the platform JenkinsFile. > >> > >> WDYT? > >> > >> Thanks > >> -Vincent > >> > > > > > > -- > > Thomas Mortagne > -- Thomas Mortagne
