Hi all, On Thu, Apr 19, 2012 at 11:17:21AM +0200, Sebastien Bacher wrote: > I agree that libreoffice would be a good fit, especially if it has a > test suite already. Bjoern, do you think it's feasible?
So, we have 3 possible sources of bugs/regressions/test failures for LibreOffice on Ubuntu: - LibreOffice Upstream (http://cgit.freedesktop.org/libreoffice/core/) - Ubuntu platform (that is: all the packages that LibreOffice depends on directly or indirectly) - LibreOffice packaging itself (http://anonscm.debian.org/gitweb/?p=pkg-openoffice/libreoffice.git;a=summary) As for test suites, at LibreOffice we have multiple ones: 1) unittests - run during the build - most cant run against an installation, because they are depending on symbols that are not exported - easy to debug 2) smoketest - basic integration test - runs against an installation - hard to debug 3) complex tests - manually written tests in junit that specific scenarios over the UNO-API - runs against an installation - unusally good test code, but limited coverage - harder to debug (test and tested code in different processes, UNO-bridge) 4) unoapi tests - automatically created tests that test every UNO-interface - rather mechanical test code, but better code coverage - runs against an installation - harder to debug (test and tested code in different processes, UNO-bridge) 5) testtool tests (defunct) - testing LO/OOo by hooking into the toolkit and creating synthetic event - fragile - unreliable - unmaintainable - runs against an installation - hard to debug - defunct now at LibreOffice (good riddance!) 3+4 together are also called "subsequenttests" see http://permalink.gmane.org/gmane.comp.documentfoundation.libreoffice.devel/8746 for details. So what do we do now? During each build we run the tests 1,2,3 and 4, but 2-4 against an installation _before_ we pack the product. That helps us against bugs/regressions caused by changes in: - LibreOffice Upstream - Ubuntu platform at the time of the build it does not find all bugs in LibreOffice packaging as these tests are run _before_ packing. What can we do additionally? 1) just build the latest LibreOffice package available daily or weekly This would show us if there is a bug/regression/test failure caused by an updated package. LibreOffice has been ftbfs multiple times already by innocent looking updates to its dependencies (which are a lot). I consider building the LibreOffice package a testcase in itself, esp. since a lot of tests are run during that. 2) build the latest LibreOffice package available with a brand new checkout of the upstream sources. This would help detecting breakages early from upstream changes, but I think that would not make sense to do before the release branch is branched off (too instable, too much adjustment in packaging needed before branchoff). see http://wiki.documentfoundation.org/ReleasePlan/3.6 for details: branchoff is scheduled for Week 23, 2012 (starting June 4th, 2012) around Alpha1 3) Run tests against a version packed and unpacked/installed into the system. This would be the tests 3+4 from above. This would finally also cover the LibreOffice packaging itself. This would still need a build and an installation. 4) Provide a tinderbox to LibreOffice upstream: http://tinderbox.libreoffice.org/MASTER/status.html This would esp. make sense for example for armhf or somesuch to see nasty platformspecific breakages early (those are mostly independant from packaging, which this would not test). What do I think to be sensible? IMHO: 1) until branchoff of the release branch 2) from branchoff 3) all the time 4) all the time I think we will find a lot of stuff just by building LibreOffice regularly (1+2) -- it is a rather good test case and easy/trivial to implement, even though it is not exactly ressourcefriendly (although when using ccache and not building all l10ns, it shouldnt be too hard on modern hardware). 3 would be a bit more work to implement, and might be questionable in the long term as the Junit/UNO-Tests are not expected survive the LibreOffice-4.0-API change without much work. Still, if we do 1 and 2, it should not be too much additional work. 4 is rather indepedant of 1-3, still worth considering. I would have to choose just one because of hardware limitations, I would go for: 1 up until branchoff, 2 from there -- both with ccache(*) and limited l10n. I believe that would give us already some very good coverage compared to what we have now. I will create a blueprint for UDS-Q this. Best, Bjoern (*) ccache cleared weekly to catch errors by compiler updates -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop