On Mon, Jul 2, 2018 at 10:52 AM, Adel Atallah <adel.atal...@xwiki.com> wrote: > Hi, > > It will be very useful to be able to run functional tests on all > supported browsers. However I wonder if it will also be as useful for > different servlet containers and databases (at least when the database > schema doesn't change).
I don't think it's very useful for different containers but we very often have issues with differences between databases that we discover only after the release (like Oracle and its f... empty string=null or MySQL and its weird handling of escaping) so it will definitely be useful. > I tried to run the tests on my machine but I got an error: > https://pastebin.com/raw/cYjec5Bm (I haven't analyzed it yet but you > might have an idea why I'm getting this). > Also it's a bit disturbing to wait for the docker images to be > downloaded without knowing it (we should at least have a message > saying "Downloading images..." or even better, the docker output). > > Thanks, > Adel > > > On Mon, Jul 2, 2018 at 10:40 AM, Vincent Massol <vinc...@massol.net> wrote: >> Important note: >> * We’ll only be able to test Firefox and Chrome browsers since there’s no >> Docker containers for windows/mac having IE11/Edge and Safari (they’re not >> supported ATM even on windows machines). >> * We could work around this by having some agents running on Windows and >> using the installed IE11/Edge browser (i.e. not running in a Docker >> container). However it’s too complex FTM and I’d instead move this to later >> and instead focus on running XWiki on various DBs. >> * So this means that we’ll still need manual tests for IE11/Edge/Safari for >> the moment... >> >> Thanks >> -Vincent >> >>> On 30 Jun 2018, at 14:59, Vincent Massol <vinc...@massol.net> wrote: >>> >>> Status as of today: >>> >>> * I have a working POC for running XWiki functional tests in various >>> browsers (chrome and firefox FTM), using the latest Selenium and latest >>> Chrome/Firefox. I’ll commit today or in the coming days. Ofc it requires >>> docker to be installed so I’ll create a “docker” profile in the build so >>> that the build doesn’t break on agents which don’t have docker installed >>> yet. It’ll allow you to start testing it on your local machines. >>> >>> * I have updated the design page at >>> http://design.xwiki.org/xwiki/bin/view/Proposal/ConfigurationTesting >>> >>> Next steps: >>> >>> * Commit the work one (in the coming days) >>> * I have started trying to deploy XWiki in an embedded undertow. Still not >>> working at this stage (but I’ve progressed). I still don’t know if this is >>> going to work or not. I hope it will since that would save me from having >>> to write the equivalent of the Packager plugin. >>> * Integrate ability to choose the DB to use for the tests. >>> >>> Feedback most welcome. >>> >>> Thanks >>> -Vincent >>> >>>> On 27 May 2018, at 21:44, Vincent Massol <vinc...@massol.net> wrote: >>>> >>>> Hi devs, >>>> >>>> I’d like to give you some info about what I’ve started working on and >>>> verify you like the direction I’m proposing to take for the future of >>>> functional testing on the xwiki project. >>>> >>>> Needs >>>> ===== >>>> >>>> * Be able to test xwiki on multiple environments >>>> >>>> Context >>>> ====== >>>> >>>> * Right now we test only in 1 env (Jetty+HSQLDB) >>>> * I've started some docker images in xwiki-contrib >>>> * I’ve also started some experiment through >>>> https://jira.xwiki.org/browse/XWIKI-14929 and >>>> https://jira.xwiki.org/browse/XWIKI-14930 (see also email thread >>>> "[Brainstorming] Implementing multi-environment tests - Take 2” and >>>> https://github.com/xwiki/xwiki-platform/compare/XWIKI-14929-14930). This >>>> email supersedes the "[Brainstorming] Implementing multi-environment tests >>>> - Take 2” thread. >>>> * Initially I imagined doing the multi env testing in Jenkins thanks to >>>> the Jenkins Docker plugin/library. However I realized that it would be >>>> better to be able to run that on the dev machines and thus decided instead >>>> to implement it at the maven level thanks to the Fabric8 Maven plugin. >>>> >>>> Proposal >>>> ======= >>>> >>>> * The new proposal is to stop trying to do it at the maven level and >>>> instead do it at the Java level, i.e. be able to control (start/stop the >>>> various docker images for the DB, Servlet Container/XWiki and the Browser >>>> from within the java junit/selenium tests). >>>> * There are several java libraries existing to control docker from within >>>> java. For example: https://github.com/docker-java/docker-java >>>> * I got convinced when finding this awesome library that combines >>>> JUnit5/Selenium and Docker for multi-browser testing: >>>> https://bonigarcia.github.io/selenium-jupiter/ >>>> ** Note that this relies on the browser docker images provided by the >>>> Selenoid project: https://aerokube.com/selenoid/latest/ >>>> * So the idea is to extend that to be able to control the other 2 docker >>>> containers for the DB + ServletContainer/XWiki. >>>> >>>> Pros >>>> ==== >>>> >>>> * Very simple setup to start/stop functional tests (and to debug them). >>>> Only requires Docker to be installed locally. >>>> * Very simple to test any combination of DB/Servlet Container/Browser. >>>> * Always up to date images with the latest version (we can depend on >>>> LATEST of Browser images, MySQL, Tomcat, etc). >>>> * Using JUnit5 and thus the latest features >>>> * Moving to the latest Selenium version too >>>> * Also supports manually executing tests in a given running xwiki instance >>>> >>>> Implementation >>>> ============ >>>> >>>> Something like: >>>> >>>> --> XWikiSeleniumExtension extends SeleniumExtension >>>> >>>> @ExtendWith(XWikiSeleniumExtension.class) >>>> public class Test >>>> >>>> @Test >>>> public void xxx(XWikiWebDriver driver) >>>> { >>>> … >>>> } >>>> >>>> And be able to configure the DB to use, the Servlet container to use, and >>>> the packaging to use from system properties (and also from the test >>>> itself, see https://bonigarcia.github.io/selenium-jupiter/#generic-driver). >>>> >>>> The idea is to reimplement the XWiki Packaging Maven plugin as a java lib >>>> using Aether and to just start our functional tests using pure junit >>>> without anymore more. All the hard work will be performed by the JUnit5 >>>> extension (create the packaging if not already exist, update some part of >>>> it if files have been modifier, start/stop DB+Servlet+Browser+Selenium, >>>> download the docker images). >>>> >>>> The packaging will be configurable. Some ideas of options: >>>> * use an already running xwiki instance >>>> * docker created from full XS zip from URL >>>> * docker created from XS zip from maven artifact >>>> * docker created from computed based on pom in current dir >>>> >>>> Migration >>>> ======= >>>> >>>> Once a first version is working, it’ll be easy to use it only for a single >>>> platform functional tests and then slowly move each module to use the new >>>> way for its functional tests. >>>> >>>> WDYT? >>>> >>>> I’m planning to continue my investigation/development of this. So please >>>> let me know if you have feedback. >>>> >>>> Thanks >>>> -Vincent >>>> >>>> >>> >> -- Thomas Mortagne
