True. However, knowing the coverage of parserTests and knowing which code isn't even being visited by it is the first step in understanding where the holes are in testing. Code coverage is a primitive metric. But, it's a place to start.
--- On Thu, 7/30/09, Victor Vasiliev <vasi...@gmail.com> wrote: > From: Victor Vasiliev <vasi...@gmail.com> > Subject: Re: [Wikitech-l] parserTests code coverage statistics > To: "Wikimedia developers" <wikitech-l@lists.wikimedia.org> > Date: Thursday, July 30, 2009, 1:28 AM > dan nessett wrote: > > I decided to investigate how well parserTests > exercises the MW code. So, I threw together a couple of > MacGyver tools that use xdebug's code coverage capability > and analyzed the results. The results are very, very > preliminary, but I thought I would get them out so others > can look them over. In the next couple of days I hope to > post more detailed results and the tools themselves on the > Mediawiki wiki. (If someone could tell me the appropriate > page to use that would be useful. Otherwise, I will just > create a page in my own namespace). > > > > The statistics (again very preliminary) are: > > > > Number of files exercised: 141 Number of lines > in those files: 85606 > > Lines covered: 59489 Lines not covered: > 26117 Percentage covered: 0.694916244188 > > > > So, parserTests is getting (at best) about 70% code > coverage. This is better than I expected, but still it means > parserTests does not test 26117 lines of code. What I mean > by "at best" is xdebug just notes whether a line of code is > visited. It doesn't do any logic analysis on which branches > are taken. Furthermore, parserTests may not visit some files > that are critical to the operation of the MW software. > Obviously, xdebug can only gather statistics on visited > files. > > > > I want to emphasize that there may be errors in these > results due to bad assumptions on my part or bad coding. > However, it is a place to start. > > > > Well, they are *parser* tests, they are not intended to > cover > Special:Version or something else. > > --vvv > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
