Re: [Kde-pim] 4.11.0 release and tests
On Tue, Aug 6, 2013 at 7:47 PM, Volker Krause wrote: > On Monday 05 August 2013 21:54:33 Allen Winter wrote: >> On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote: >> > @people, please keep both lists on the loop. >> > >> > One of the things we agreed we wanted for the 4.11 release was no failing >> > tests on jenkins. >> > >> > We've almost achieved that. >> > >> > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false >> > >> > We have only 8 failing tests, all of them in kdepim-*. > > Yep, although not entirely surprising considering that's where the majority of > all our tests seem to be (>1000). That's no excuse of course, just statistics > ;) > > Looking at the fail graphs for kdepim*, you can also see that this plan had > some positive effects :) > >> > My question to the kdepim people: >> > * Why are those tests failing? > > Looking at kdepimlibs-stable: > TestSuite.kblog-testmetaweblog > > IIRC that's an online test, probably the remote site changed (happened with a > few others from kblog as well), deactivate/expected fail IMHO. > TestSuite.Compat-KOrganizer_3.1.ics TestSuite.Compat-KOrganizer_3.2.ics > > IIRC those were fixed with libical 1.0, right Allen? > > > Looking at kdepim-runtime-stable: > TestSuite.kolabconvertertest > > Broken, wasn't adjusted to the kolab resource rewrite I think, should be > deactivated/expected failed. Christian, can you look into fixing this one > please? > > > Looking at kdepim-stable: > TestSuite.ktimezonecomboboxtest TestSuite.summaryeventtest > > those fail with ktimezoned error messages here, probably test setup errors? > Who could look into that? That is quite an unstable test according to Jenkins - see the below link. http://build.kde.org/job/kdepim_master/lastCompletedBuild/testReport/(root)/TestSuite/ktimezonecomboboxtest/history/? > TestSuite.messageviewer-rendertest TestSuite.messagecomposer-messagefactorytest TestSuite.messagecomposer-maintextjobtest > > those are in theory valid tests, but fail for technical reasons (crypto > setup), non-trivial to fix correctly I think, so temporarily disable/expected > fail until someone has time to fix the test harness there, I'd say. messageviewer-rendertest seems to be failing due to changes in the HTML code itself. messagecomposer-maintextjobtest appears to be related to charset issues. messagecomposer-messagefactorytest fails due to header changes from what I can tell. You can navigate through the Jenkins interface to get the logs from the test executions if you wish. > >> > My question to release-team and kdepim people: >> > * Should we delay the release until tests are fixed? > > IMHO that's not necessary. > >> > * Should we simply QFAIL/remove the tests since it seems they are not >> > that important? > > see above, yes. > >> > * Should we let them fail and still release? >> > >> > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1 >> > one can go to jenkins, check if everything is green and have a very high >> > level overview that with some luck maybe nothing regressed. > > yes, makes sense > >> I agree with you. We need 0% failing. > > Akonadi is at 0% already, can we subscribe kde-pim to Jenkins errors for this > one already (and add the rest as soon as we hit 0% there for the first time)? Both Akonadi master and Akonadi stable have now been configured to send emails to kde-...@kde.org when tests begin to fail, or if the build fails. > >> 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer >> ones) I'm surprised at this. Can someone tell me which version of libical >> is installed on the jenkins machines? > > IIRC < 1.0, which should explain this. > >> We should QFAIL the kblog test. I can't remember the last time that was >> working. > > They use online service and test accounts on these, that's not going to keep > working long-term without maintenance. > > regards, > Volker Thanks, Ben Cooksley KDE Sysadmin > > ___ > KDE PIM mailing list kde-...@kde.org > https://mail.kde.org/mailman/listinfo/kde-pim > KDE PIM home page at http://pim.kde.org/ ___ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team
Re: [Kde-pim] 4.11.0 release and tests
On Tuesday 06 August 2013 09.47:05 Volker Krause wrote: > On Monday 05 August 2013 21:54:33 Allen Winter wrote: > > On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote: > > > @people, please keep both lists on the loop. > > > > > Looking at kdepim-runtime-stable: > >>> TestSuite.kolabconvertertest > > Broken, wasn't adjusted to the kolab resource rewrite I think, should be > deactivated/expected failed. Christian, can you look into fixing this one > please? > That's fixed now. Thanks for the heads-up. There's still some unused stuff in there (all the .es files), but otherwise I removed what is no longer required, and the test should pass now. Cheers, Christian ___ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team
Re: [Kde-pim] 4.11.0 release and tests
On Tuesday, August 06, 2013 09:47:05 AM Volker Krause wrote: > Looking at kdepim-stable: > >>> TestSuite.ktimezonecomboboxtest > >>> TestSuite.summaryeventtest summaryeventtest is fixed now, will look at the ktimezonecombo later -- Sérgio ___ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team
Re: [Kde-pim] 4.11.0 release and tests
On Monday 05 August 2013 21:54:33 Allen Winter wrote: > On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote: > > @people, please keep both lists on the loop. > > > > One of the things we agreed we wanted for the 4.11 release was no failing > > tests on jenkins. > > > > We've almost achieved that. > > > > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false > > > > We have only 8 failing tests, all of them in kdepim-*. Yep, although not entirely surprising considering that's where the majority of all our tests seem to be (>1000). That's no excuse of course, just statistics ;) Looking at the fail graphs for kdepim*, you can also see that this plan had some positive effects :) > > My question to the kdepim people: > > * Why are those tests failing? Looking at kdepimlibs-stable: >>> TestSuite.kblog-testmetaweblog IIRC that's an online test, probably the remote site changed (happened with a few others from kblog as well), deactivate/expected fail IMHO. >>> TestSuite.Compat-KOrganizer_3.1.ics >>> TestSuite.Compat-KOrganizer_3.2.ics IIRC those were fixed with libical 1.0, right Allen? Looking at kdepim-runtime-stable: >>> TestSuite.kolabconvertertest Broken, wasn't adjusted to the kolab resource rewrite I think, should be deactivated/expected failed. Christian, can you look into fixing this one please? Looking at kdepim-stable: >>> TestSuite.ktimezonecomboboxtest >>> TestSuite.summaryeventtest those fail with ktimezoned error messages here, probably test setup errors? Who could look into that? >>> TestSuite.messageviewer-rendertest >>> TestSuite.messagecomposer-messagefactorytest >>> TestSuite.messagecomposer-maintextjobtest those are in theory valid tests, but fail for technical reasons (crypto setup), non-trivial to fix correctly I think, so temporarily disable/expected fail until someone has time to fix the test harness there, I'd say. > > My question to release-team and kdepim people: > > * Should we delay the release until tests are fixed? IMHO that's not necessary. > > * Should we simply QFAIL/remove the tests since it seems they are not > > that important? see above, yes. > > * Should we let them fail and still release? > > > > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1 > > one can go to jenkins, check if everything is green and have a very high > > level overview that with some luck maybe nothing regressed. yes, makes sense > I agree with you. We need 0% failing. Akonadi is at 0% already, can we subscribe kde-pim to Jenkins errors for this one already (and add the rest as soon as we hit 0% there for the first time)? > 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer > ones) I'm surprised at this. Can someone tell me which version of libical > is installed on the jenkins machines? IIRC < 1.0, which should explain this. > We should QFAIL the kblog test. I can't remember the last time that was > working. They use online service and test accounts on these, that's not going to keep working long-term without maintenance. regards, Volker signature.asc Description: This is a digitally signed message part. ___ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team
Re: [Kde-pim] 4.11.0 release and tests
On Monday 05 August 2013 21.54.33 Allen Winter wrote: > On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote: > > @people, please keep both lists on the loop. > > > > One of the things we agreed we wanted for the 4.11 release was no failing > > tests on jenkins. > > > > We've almost achieved that. > > > > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false > > > > We have only 8 failing tests, all of them in kdepim-*. > > > > My question to the kdepim people: > > * Why are those tests failing? > > > > My question to release-team and kdepim people: > > * Should we delay the release until tests are fixed? > > * Should we simply QFAIL/remove the tests since it seems they are not > > that > > > > important? > > > > * Should we let them fail and still release? > > > > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1 > > one can go to jenkins, check if everything is green and have a very high > > level overview that with some luck maybe nothing regressed. > > I agree with you. We need 0% failing. Same opinion here. > 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer > ones) I'm surprised at this. Can someone tell me which version of libical > is installed on the jenkins machines? 0.46-5.1.2 according to zypper. /Regards Torgny ___ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team
Re: [Kde-pim] 4.11.0 release and tests
On Tuesday, August 06, 2013 01:04:08 AM Albert Astals Cid wrote: > @people, please keep both lists on the loop. > > One of the things we agreed we wanted for the 4.11 release was no failing > tests on jenkins. > > We've almost achieved that. > > http://build.kde.org/view/KDE%20SC%20stable/?auto_refresh=false > > We have only 8 failing tests, all of them in kdepim-*. > > My question to the kdepim people: > * Why are those tests failing? > > My question to release-team and kdepim people: > * Should we delay the release until tests are fixed? > * Should we simply QFAIL/remove the tests since it seems they are not that > important? > * Should we let them fail and still release? > > Personally i'd like the tests either fixed or QFailed, this way for 4.11.1 > one > can go to jenkins, check if everything is green and have a very high level > overview that with some luck maybe nothing regressed. > I agree with you. We need 0% failing. 2 of the kdepimlibs fails are from kcalcore (the TestSuite.Compat.KOrganizer ones) I'm surprised at this. Can someone tell me which version of libical is installed on the jenkins machines? We should QFAIL the kblog test. I can't remember the last time that was working. ___ release-team mailing list release-team@kde.org https://mail.kde.org/mailman/listinfo/release-team
