On Nov 22, 2012, at 11:41 AM, Vincent Massol <[email protected]> wrote:
> > On Nov 22, 2012, at 11:28 AM, Vincent Massol <[email protected]> wrote: > >> >> On Feb 15, 2012, at 4:58 PM, Vincent Massol <[email protected]> wrote: >> >>> >>> On Feb 15, 2012, at 4:10 PM, Vincent Massol wrote: >>> >>>> >>>> On Jan 25, 2012, at 5:02 PM, Vincent Massol wrote: >>>> >>>>> >>>>> On Jan 15, 2012, at 11:27 AM, Vincent Massol wrote: >>>>> >>>>>> Hi devs, >>>>>> >>>>>> I've started an experiment to have colocated functional tests (CFT), >>>>>> which means having the functional tests located where the functional >>>>>> domain sources are located instead of in XE. >>>>>> >>>>>> For example for the linkchecker module we have the following directories: >>>>>> >>>>>> xwiki-platform-linkchecker/ >>>>>> |_ xwiki-platform-linkchecker-refresher (JAR) >>>>>> |_ xwiki-platform-linkchecker-ui (XAR) >>>>>> |_ xwiki-platform-linkchecker-tests (functional tests) >>>>>> >>>>>> The rationale for this was: >>>>>> * Have everything about a functional domain self-contained (source and >>>>>> all tests) >>>>>> * Making it easy to run only tests for a given functional domain >>>>>> * Move page objects to the functional domain too >>>>>> >>>>>> Here are some findings about this experiment: >>>>>> >>>>>> A - It takes about 30 seconds to generate the adhoc packaging and start >>>>>> XWiki. This would be done for each module having functional tests >>>>>> compared to only once if all tests were executed in XE >>>>>> B- The package mojo created to generate a full packaging is quite nice >>>>>> and I plan to reuse it in lots of other places in our build >>>>>> (distributions, database, places where we need XWiki configuration files) >>>>>> C- We will not be able to run platform builds in Maven multithreaded >>>>>> mode since it would mean that several XWiki instance could be started at >>>>>> the same time on the same port >>>>>> D- The colocated functional test module >>>>>> >>>>>> Solutions/ideas: >>>>>> >>>>>> * One idea to overcome A and C would to have the following setup: >>>>>> ** Keep functional test modules colocated but have them generate a test >>>>>> JAR >>>>>> ** Still allow running functional tests from the colocated module (this >>>>>> makes it easy to verify no regression was introduced when making changes >>>>>> to a given domain) >>>>>> ** Have functional tests in XE depend on the colocated functional test >>>>>> module JARs and configure Jenkins to run all functional tests from XE >>>>>> only >>>>>> >>>>>> * Another solution to overcome C is to auto-discover the port to use in >>>>>> our XWiki startup script (and save it in a file so that the stop script >>>>>> can use it). >>>>>> >>>>>> I think the first proposal is the best one and brings the best of the 2 >>>>>> worlds. >>>>> >>>>> Actually there's another one: >>>>> >>>>> * Run functional tests in platform >>>>> * Configure jenkins so that we have one jenkins job per platform top >>>>> module >>>>> >>>>> This would allow several things: >>>>> * Faster feedback to user: for example if a commit is done in a module >>>>> that happens late in the build order the whole platform wouldn't need to >>>>> be rebuilt before noticing the problem) - Note: Jenkins supports a "Local >>>>> subdirectory for repo" feature that would make this possible I think) >>>> >>>> After researching this a bit I don't think it's possible… >>>> >>>> This "Local subdirectory for repo" is something else: "Specify a local >>>> directory (relative to the workspace root) where the Git repository will >>>> be checked out. If left empty, the workspace root itself will be used." >>>> >>>> We could check out the whole Git repo but 1) it'll take space (a lot of >>>> space) but more importantly 2) GitHub will trigger a build on all jobs for >>>> that repo which isn't good. >> >> Regarding space, I've tested it and I've also tested the "shallow clone" >> option of Jenkins. >> >> So for platform: >> * with shallow clone: 281MB >> * without shallow clone (ie full clone): 346MB >> >> We have 91 modules in platform so that's 91*281=25571MB == 25GB >> Even with the full clone that's 31GB >> >> We need to add the target/ dir size, so let's say 1GB more in total, to 26GB >> or 32GB. That's doable! :) >> >> So I guess it's ok and that's not our biggest issue. >> >> There are 2 more issues: >> * Automatic creation of jenkins job. That's doable and almost done: >> http://jira.xwiki.org/jira/browse/XCOMMONS-87 >> * Triggers. This is the hardest since we'll need to create a manual ordering >> of jobs as otherwise they'll all start building at the same time when >> there's a github event coming in… It's not hard to do but it's maintenance >> intensive. Of course this ordering would be defined in the automated jenkins >> job creations but it still needs to be maintained... >> >> In conclusion it's doable and we need to work on >> http://jira.xwiki.org/jira/browse/XCOMMONS-87 first. > > Actually maybe a better thing to do (much simpler) would simply be to use > parallel builds for platform and instead ensure that our functional tests > there use different ports so that they can run in //... > > That's provided the agents are powerful enough to support a large # of > threads. Would be interesting to compute the ideal # of threads based on our > dependency graph :) Yet another solution is to build platform without the -Pintegration-tests profile with parallel builds and then create one job per functional test in platform. -Vincent > > Thanks > -Vincent > >>>> So I think we'll have to drop the idea of having a single job per module. >>>> >>>> See also >>>> http://stackoverflow.com/questions/8922857/jenkins-how-to-build-multiple-top-level-projects-from-a-git-repository >>> >>> hmm actually this is what we already do for functional tests but we don't >>> trigger than on Git Hub notifications, we trigger them on xwiki-enterprise >>> (after it's been built). >>> >>> I guess we could do the same, i.e.: >>> >>> * Only trigger the top level module in a given repo using a GitHub push >>> * Only trigger the other modules on "Build whenever a SNAPSHOT dependency >>> is built" >>> >>> The problem is that it would still use a lot of space. For example my >>> xwiki-platform/ dir takes 1.2GB of disk space (with target/ dirs) >>> So with roughly 230+ modules that's 276 GB just for xwiki-platform… >> >> This computation is wrong, see above. >> >> Thanks >> -Vincent >> >>> Combined with commons, rendering and enterprise that's probably around >>> 500GB easily… >>> >>> In conclusion it doesn't seem like a good idea… >>> >>> Thanks >>> -Vincent >>> >>>> Thanks >>>> -Vincent >>>> >>>>> * Use the full power of our agents by distributing the jobs >>>>> * Since functional tests run at the same time as unit tests if the module >>>>> build works we can be sure it's good >>>>> >>>>> Thus even though packaging and starting XE takes a lot of time (I've >>>>> timed it to roughly 1.5 minutes on my machine) the balancing of jobs on >>>>> multiple agents might overcome this - it would still be great to be able >>>>> to improve the XE start time. >>>>> >>>>> Note that creating jenkins job for so many modules might seem daunting >>>>> but this can be easily automated with http://ci.xwiki.org/api/ or even >>>>> better http://evgeny-goldin.com/wiki/Maven-jenkins-plugin >>>>> >>>>> I'm still hesitating between the previous idea and this one but thought I >>>>> would share my current thoughts with you in case you have some >>>>> idea/preference… :) >>>>> >>>>> Thanks >>>>> -Vincent >>>>> >>>>>> WDYT? >>>>>> >>>>>> Thanks >>>>>> -Vincent >>>>>> >>>>> >>>> >>> >> > _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
