On 07/11/2018 08:44, Thomas Mortagne wrote:
On Mon, Nov 5, 2018 at 5:22 PM Vincent Massol <[email protected]> wrote:
Hi devs,
I’m still not sure but FTM I was thinking of having 2 pipeline jobs:
1) Job 1: Execute one functional test only (e.g. MenuIT for now) but on the
maximum number of configurations, in order to flesh out configs that don’t
start properly. For example XWiki on Tomcat 9.x would fail (since the Tomcat
9.x docker image uses java9+). The job would not send a mail on failure but it
would update a report page (could even update a page on xwiki.org directly or
if too complex update some page on maven.xwiki.org somewhere). This job would
run not very often but say once per week. Note that one config takes 3-4
minutes to run, so 50 configs would take 3 hours which is acceptable.
That's interesting indeed. +1
2) Job 2: Execute all functional tests on a subset of supported configs. For
example we don’t need to run all the tests on PostgreSQL/Jetty/Chrome if we
already run on PostgreSQL/Tomcat/FF and MySQL/Tomcat/Chrome. This job will take
a long time to execute. We’ll start with 3-4 configs and will go to about 10
configs when we add more. The tests will take roughly 2 hours to execute per
config I think. So a total of 20 hours when we have 10 configs. If we run those
once per week it should be fine.
Can't those configs be executed in parallel in different agents ?
Note: Once we have job 1 & 2, we won't need to have the smoke tests I add as
part of the platform JenkinsFile.
WDYT?
+1 for both jobs
Thanks
-Vincent
--
Simon Urli
Software Engineer at XWiki SAS
[email protected]
More about us at http://www.xwiki.com
