Hi, It will be very useful to be able to run functional tests on all supported browsers. However I wonder if it will also be as useful for different servlet containers and databases (at least when the database schema doesn't change). I tried to run the tests on my machine but I got an error: https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you might have an idea why I'm getting this). Also it's a bit disturbing to wait for the docker images to be downloaded without knowing it (we should at least have a message saying "Downloading images..." or even better, the docker output).
Thanks, Adel On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <[email protected]> wrote: > Important note: > * We’ll only be able to test Firefox and Chrome browsers since there’s no > Docker containers for windows/mac having IE11/Edge and Safari (they’re not > supported ATM even on windows machines). > * We could work around this by having some agents running on Windows and > using the installed IE11/Edge browser (i.e. not running in a Docker > container). However it’s too complex FTM and I’d instead move this to later > and instead focus on running XWiki on various DBs. > * So this means that we’ll still need manual tests for IE11/Edge/Safari for > the moment... > > Thanks > -Vincent > >> On 30 Jun 2018, at 14:59, Vincent Massol <[email protected]> wrote: >> >> Status as of today: >> >> * I have a working POC for running XWiki functional tests in various >> browsers (chrome and firefox FTM), using the latest Selenium and latest >> Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires >> docker to be installed so I’ll create a “docker” profile in the build so >> that the build doesn’t break on agents which don’t have docker installed >> yet. It’ll allow you to start testing it on your local machines. >> >> * I have updated the design page at >> http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting >> >> Next steps: >> >> * Commit the work one (in the coming days) >> * I have started trying to deploy XWiki in an embedded undertow. Still not >> working at this stage (but I’ve progressed). I still don’t know if this is >> going to work or not. I hope it will since that would save me from having to >> write the equivalent of the Packager plugin. >> * Integrate ability to choose the DB to use for the tests. >> >> Feedback most welcome. >> >> Thanks >> -Vincent >> >>> On 27 May 2018, at 21:44, Vincent Massol <[email protected]> wrote: >>> >>> Hi devs, >>> >>> I’d like to give you some info about what I’ve started working on and >>> verify you like the direction I’m proposing to take for the future of >>> functional testing on the xwiki project. >>> >>> Needs >>> ===== >>> >>> * Be able to test xwiki on multiple environments >>> >>> Context >>> ====== >>> >>> * Right now we test only in 1 env (Jetty+HSQLDB) >>> * I've started some docker images in xwiki-contrib >>> * I’ve also started some experiment through >>> https://jira.xwiki.org/browse/XWIKI-14929 and >>> https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread >>> "[Brainstorming] Implementing multi-environment tests - Take 2” and >>> https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This >>> email supersedes the "[Brainstorming] Implementing multi-environment tests >>> - Take 2” thread. >>> * Initially I imagined doing the multi env testing in Jenkins thanks to the >>> Jenkins Docker plugin/library. However I realized that it would be better >>> to be able to run that on the dev machines and thus decided instead to >>> implement it at the maven level thanks to the Fabric8 Maven plugin. >>> >>> Proposal >>> ======= >>> >>> * The new proposal is to stop trying to do it at the maven level and >>> instead do it at the Java level, i.e. be able to control (start/stop the >>> various docker images for the DB, Servlet Container/XWiki and the Browser >>> from within the java junit/selenium tests). >>> * There are several java libraries existing to control docker from within >>> java. For example: https://github.com/docker-java/docker-java >>> * I got convinced when finding this awesome library that combines >>> JUnit5/Selenium and Docker for multi-browser testing: >>> https://bonigarcia.github.io/selenium-jupiter/ >>> ** Note that this relies on the browser docker images provided by the >>> Selenoid project: https://aerokube.com/selenoid/latest/ >>> * So the idea is to extend that to be able to control the other 2 docker >>> containers for the DB + ServletContainer/XWiki. >>> >>> Pros >>> ==== >>> >>> * Very simple setup to start/stop functional tests (and to debug them). >>> Only requires Docker to be installed locally. >>> * Very simple to test any combination of DB/Servlet Container/Browser. >>> * Always up to date images with the latest version (we can depend on LATEST >>> of Browser images, MySQL, Tomcat, etc). >>> * Using JUnit5 and thus the latest features >>> * Moving to the latest Selenium version too >>> * Also supports manually executing tests in a given running xwiki instance >>> >>> Implementation >>> ============ >>> >>> Something like: >>> >>> --> XWikiSeleniumExtension extends SeleniumExtension >>> >>> @ExtendWith(XWikiSeleniumExtension.class) >>> public class Test >>> >>> @Test >>> public void xxx(XWikiWebDriver driver) >>> { >>> … >>> } >>> >>> And be able to configure the DB to use, the Servlet container to use, and >>> the packaging to use from system properties (and also from the test itself, >>> see https://bonigarcia.github.io/selenium-jupiter/#generic-driver). >>> >>> The idea is to reimplement the XWiki Packaging Maven plugin as a java lib >>> using Aether and to just start our functional tests using pure junit >>> without anymore more. All the hard work will be performed by the JUnit5 >>> extension (create the packaging if not already exist, update some part of >>> it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, >>> download the docker images). >>> >>> The packaging will be configurable. Some ideas of options: >>> * use an already running xwiki instance >>> * docker created from full XS zip from URL >>> * docker created from XS zip from maven artifact >>> * docker created from computed based on pom in current dir >>> >>> Migration >>> ======= >>> >>> Once a first version is working, it’ll be easy to use it only for a single >>> platform functional tests and then slowly move each module to use the new >>> way for its functional tests. >>> >>> WDYT? >>> >>> I’m planning to continue my investigation/development of this. So please >>> let me know if you have feedback. >>> >>> Thanks >>> -Vincent >>> >>> >> >
