Re: [xwiki-devs] [VOTE] Add new check to measure quality of tests
If you’re curious I’ve also added some more information at https://github.com/STAMP-project/pitest-descartes/issues/54 ATM, and even with a time of “only” 10mn to run pitest/descartes on xwiki-commons, it’s still too much IMO so I’ll work on turning it off by default but having a job to run it on the CI. Let me know if you have remarks. Thanks -Vincent > On 27 Mar 2018, at 20:09, Vincent Massol wrote: > > > >> On 27 Mar 2018, at 19:32, Vincent Massol wrote: >> >> FYI I’ve implemented it locally for all modules of xwiki-commons and did >> some build time measurements: >> >> * With pitest/descartes: 37:16 minutes >> * Without pitest/descartes 5:10 minutes > > Actually I was able to reduce the time to 15:12 minutes with configuring > pitest with 4 threads. > > Thanks > -Vincent > >> >> So that’s a pretty important hit…. >> >> So I think one strategy could be to not run pitest/descartes by default in >> the quality profile (i.e. have it off by default with >> true) and run it on the CI, from time >> to time, like once per day for example, or once per week. >> >> Small issue: I need to find/test a way to run a crontab type of job in a >> Jenkins pipeline script. I know how to do in theory but I need to test it >> and verify it works. I still have some doubts ATM... >> >> WDYT? >> >> Thanks >> -Vincent >> >>> On 15 Mar 2018, at 09:30, Vincent Massol wrote: >>> >>> Hi devs, >>> >>> As part of the STAMP research project, we’ve developed a new tool >>> (Descartes, based on Pitest) to measure the quality of tests. It generates >>> a mutation score for your tests, defining how good the tests are. Technical >>> Descartes performs some extreme mutations on the code under test (e.g. >>> remove content of void methods, return true for methods returning a >>> boolean, etc - See https://github.com/STAMP-project/pitest-descartes). If >>> the test continues to pass then it means it’s not killing the mutant and >>> thus its mutation score decreases. >>> >>> So in short: >>> * Jacoco/Clover: measure how much of the code is tested >>> * Pitest/Descartes: measure how good the tests are >>> >>> Both provide a percentage value. >>> >>> I’m proposing to compute the current mutation scores for xwiki-commons and >>> xwiki-rendering and fail the build when new code is added that reduce the >>> mutation score threshold (exactly the same as our jacoco threshold and >>> strategy). >>> >>> I consider this is an experiment to push the limit of software engineering >>> a bit further. I don’t know how well it’ll work or not. I propose to do the >>> work and test this for over 2-3 months and see how well it works or not. At >>> that time we can then decide whether it works or not (i.e whether the gains >>> it brings are more important than the problems it causes). >>> >>> Here’s my +1 to try this out. >>> >>> Some links: >>> * pitest: http://pitest.org/ >>> * descartes: https://github.com/STAMP-project/pitest-descartes >>> * http://massol.myxwiki.org/xwiki/bin/view/Blog/ControllingTestQuality >>> * http://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes >>> >>> If you’re curious, you can see a screenshot of a mutation score report at >>> http://massol.myxwiki.org/xwiki/bin/download/Blog/MutationTestingDescartes/report.png >>> >>> Please cast your votes. >>> >>> Thanks >>> -Vincent >> >
Re: [xwiki-devs] [VOTE] Add new check to measure quality of tests
On Wed, Mar 28, 2018 at 1:25 PM, Vincent Massol wrote: > If you’re curious I’ve also added some more information at > https://github.com/STAMP-project/pitest-descartes/issues/54 > > ATM, and even with a time of “only” 10mn to run pitest/descartes on > xwiki-commons, it’s still too much IMO so I’ll work on turning it off by > default but having a job to run it on the CI. Sounds good. I agree that we can't enable it all the time given the time lost. > > Let me know if you have remarks. > > Thanks > -Vincent > >> On 27 Mar 2018, at 20:09, Vincent Massol wrote: >> >> >> >>> On 27 Mar 2018, at 19:32, Vincent Massol wrote: >>> >>> FYI I’ve implemented it locally for all modules of xwiki-commons and did >>> some build time measurements: >>> >>> * With pitest/descartes: 37:16 minutes >>> * Without pitest/descartes 5:10 minutes >> >> Actually I was able to reduce the time to 15:12 minutes with configuring >> pitest with 4 threads. >> >> Thanks >> -Vincent >> >>> >>> So that’s a pretty important hit…. >>> >>> So I think one strategy could be to not run pitest/descartes by default in >>> the quality profile (i.e. have it off by default with >>> true) and run it on the CI, from >>> time to time, like once per day for example, or once per week. >>> >>> Small issue: I need to find/test a way to run a crontab type of job in a >>> Jenkins pipeline script. I know how to do in theory but I need to test it >>> and verify it works. I still have some doubts ATM... >>> >>> WDYT? >>> >>> Thanks >>> -Vincent >>> On 15 Mar 2018, at 09:30, Vincent Massol wrote: Hi devs, As part of the STAMP research project, we’ve developed a new tool (Descartes, based on Pitest) to measure the quality of tests. It generates a mutation score for your tests, defining how good the tests are. Technical Descartes performs some extreme mutations on the code under test (e.g. remove content of void methods, return true for methods returning a boolean, etc - See https://github.com/STAMP-project/pitest-descartes). If the test continues to pass then it means it’s not killing the mutant and thus its mutation score decreases. So in short: * Jacoco/Clover: measure how much of the code is tested * Pitest/Descartes: measure how good the tests are Both provide a percentage value. I’m proposing to compute the current mutation scores for xwiki-commons and xwiki-rendering and fail the build when new code is added that reduce the mutation score threshold (exactly the same as our jacoco threshold and strategy). I consider this is an experiment to push the limit of software engineering a bit further. I don’t know how well it’ll work or not. I propose to do the work and test this for over 2-3 months and see how well it works or not. At that time we can then decide whether it works or not (i.e whether the gains it brings are more important than the problems it causes). Here’s my +1 to try this out. Some links: * pitest: http://pitest.org/ * descartes: https://github.com/STAMP-project/pitest-descartes * http://massol.myxwiki.org/xwiki/bin/view/Blog/ControllingTestQuality * http://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes If you’re curious, you can see a screenshot of a mutation score report at http://massol.myxwiki.org/xwiki/bin/download/Blog/MutationTestingDescartes/report.png Please cast your votes. Thanks -Vincent >>> >> > -- Thomas Mortagne
Re: [xwiki-devs] Add a REST API to XWiki Notifications
> For now, we are only using a pull model for notifications in the wiki. > Implementing a push model on top of that would be indeed a cool idea and > cover more use cases, but I'm having doubts about the scope of this and > the REST API project ; WDYT ? First of all sorry for the late,in the last project i worked on we made a Rest API ,we had a push notification model and something like pull notification model because we created it with firebase notifications and i think that is not the most completed way. I think that push model it must be from WebSocket that is more complexity but is the most completed way.I dont know how much computing resource "eats" Websocket because i think that must be connected with the app the whole time. Something else that i want to ask i have do an acc to try Xwiki platform and to see how it works the existing one notification system but in my bell i dont have notifications option i see instruction to select from my profile but same... The notification system that exist is on stable mode? Nick
[xwiki-devs] [XWiki Day] BFD#171
Hello devs, This Thursday is BFD#171: http://dev.xwiki.org/xwiki/bin/view/Community/XWikiDays#HBugfixingdays Our current status is: * -47 bugs over 120 days (4 months), i.e. we need to close 47 bugs to have created bugs # = closed bugs # * -96 bugs over 365 days (1 year) * -111 bugs over 500 days (between 1 and 2 years) * -313 bugs over 1600 days (4.3 years) * -720 bugs since the beginning of XWiki See https://jira.xwiki.org/secure/Dashboard.jspa?selectPageId=10352 Here's the BFD#171 dashboard to follow the progress during the day: https://jira.xwiki.org/secure/Dashboard.jspa?selectPageId=14093 Happy Bug Fixing Day, Alex
