On Monday 05 August 2013 21:54:33 Allen Winter wrote: > On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote: > > @people, please keep both lists on the loop. > > > > One of the things we agreed we wanted for the 4.11 release was no failing > > tests on jenkins. > > > > We've almost achieved that. > > > > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false > > > > We have only 8 failing tests, all of them in kdepim-*.
Yep, although not entirely surprising considering that's where the majority of all our tests seem to be (>1000). That's no excuse of course, just statistics ;) Looking at the fail graphs for kdepim*, you can also see that this plan had some positive effects :) > > My question to the kdepim people: > > * Why are those tests failing? Looking at kdepimlibs-stable: >>> TestSuite.kblog-testmetaweblog IIRC that's an online test, probably the remote site changed (happened with a few others from kblog as well), deactivate/expected fail IMHO. >>> TestSuite.Compat-KOrganizer_3.1.ics >>> TestSuite.Compat-KOrganizer_3.2.ics IIRC those were fixed with libical 1.0, right Allen? Looking at kdepim-runtime-stable: >>> TestSuite.kolabconvertertest Broken, wasn't adjusted to the kolab resource rewrite I think, should be deactivated/expected failed. Christian, can you look into fixing this one please? Looking at kdepim-stable: >>> TestSuite.ktimezonecomboboxtest >>> TestSuite.summaryeventtest those fail with ktimezoned error messages here, probably test setup errors? Who could look into that? >>> TestSuite.messageviewer-rendertest >>> TestSuite.messagecomposer-messagefactorytest >>> TestSuite.messagecomposer-maintextjobtest those are in theory valid tests, but fail for technical reasons (crypto setup), non-trivial to fix correctly I think, so temporarily disable/expected fail until someone has time to fix the test harness there, I'd say. > > My question to release-team and kdepim people: > > * Should we delay the release until tests are fixed? IMHO that's not necessary. > > * Should we simply QFAIL/remove the tests since it seems they are not > > that important? see above, yes. > > * Should we let them fail and still release? > > > > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1 > > one can go to jenkins, check if everything is green and have a very high > > level overview that with some luck maybe nothing regressed. yes, makes sense > I agree with you. We need 0% failing. Akonadi is at 0% already, can we subscribe kde-pim to Jenkins errors for this one already (and add the rest as soon as we hit 0% there for the first time)? > 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer > ones) I'm surprised at this. Can someone tell me which version of libical > is installed on the jenkins machines? IIRC < 1.0, which should explain this. > We should QFAIL the kblog test. I can't remember the last time that was > working. They use online service and test accounts on these, that's not going to keep working long-term without maintenance. regards, Volker
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team