Test coverage, was: Process and content for next release?
On Aug 16, 2006, at 2:49 PM, Jim Marino wrote: 2) SCA Core (spi, core, hostutil, test, plus apis once that refactor is done) Features I would like to see complete before we consider this stable are: Class loading changes Integration of databinding framework Support for async callbacks Support for complex properties Transitive dependency support I'd also like to see much better test coverage than what we have. This is hard to quantify, but while code coverage does not guarantee good tests, it is an indicator. So, to have a metric, I'd like to see core (and other extensions) at 75% coverage when run through Clover. I picked Clover since it is a decent tool and license-friendly but if someone would like to suggest an alternative we could look at it as well. I think this goal is worth pursuing and would add that as a criteria for the next release. Apache has a license for Clover so we can all use it, Cobertura would be another alternative - any preference here? Whatever we use, I don't think this should be part of the build right now (although that could change later) but that the tool should be run periodically and the results published somewhere (e.g. on our site). Now Jim here only mentioned the core but this would apply to other extensions as well - I would be inclined to extend this requirement to any extension we consider "baseline" - any objections? -- Jeremy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Test coverage, was: Process and content for next release?
On Sep 5, 2006, at 8:14 AM, Jeremy Boynes wrote: On Aug 16, 2006, at 2:49 PM, Jim Marino wrote: 2) SCA Core (spi, core, hostutil, test, plus apis once that refactor is done) Features I would like to see complete before we consider this stable are: Class loading changes Integration of databinding framework Support for async callbacks Support for complex properties Transitive dependency support I'd also like to see much better test coverage than what we have. This is hard to quantify, but while code coverage does not guarantee good tests, it is an indicator. So, to have a metric, I'd like to see core (and other extensions) at 75% coverage when run through Clover. I picked Clover since it is a decent tool and license-friendly but if someone would like to suggest an alternative we could look at it as well. I think this goal is worth pursuing and would add that as a criteria for the next release. Apache has a license for Clover so we can all use it, Cobertura would be another alternative - any preference here? Whatever we use, I don't think this should be part of the build right now (although that could change later) but that the tool should be run periodically and the results published somewhere (e.g. on our site). I prefer Clover as it also has nice IDE integration. I also think test coverage should be run as part of an integration build and published since it is a general indication of areas that need work. Now Jim here only mentioned the core but this would apply to other extensions as well - I would be inclined to extend this requirement to any extension we consider "baseline" - any objections? For extensions considered baseline, I think "respectable" code coverage (~75%) is definitely a worthy goal. For baseline extensions, I would also add that we should also have a minimum bar in terms of what assembly features they support (e.g. state management, non- blocking, etc.). -- Jeremy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Test coverage, was: Process and content for next release?
Hi, Clover sounds good to me and I have tried it before. +1 on the 75% test coverage. Raymond - Original Message - From: "Jim Marino" <[EMAIL PROTECTED]> To: Sent: Tuesday, September 05, 2006 8:48 AM Subject: Re: Test coverage, was: Process and content for next release? On Sep 5, 2006, at 8:14 AM, Jeremy Boynes wrote: On Aug 16, 2006, at 2:49 PM, Jim Marino wrote: 2) SCA Core (spi, core, hostutil, test, plus apis once that refactor is done) Features I would like to see complete before we consider this stable are: Class loading changes Integration of databinding framework Support for async callbacks Support for complex properties Transitive dependency support I'd also like to see much better test coverage than what we have. This is hard to quantify, but while code coverage does not guarantee good tests, it is an indicator. So, to have a metric, I'd like to see core (and other extensions) at 75% coverage when run through Clover. I picked Clover since it is a decent tool and license-friendly but if someone would like to suggest an alternative we could look at it as well. I think this goal is worth pursuing and would add that as a criteria for the next release. Apache has a license for Clover so we can all use it, Cobertura would be another alternative - any preference here? Whatever we use, I don't think this should be part of the build right now (although that could change later) but that the tool should be run periodically and the results published somewhere (e.g. on our site). I prefer Clover as it also has nice IDE integration. I also think test coverage should be run as part of an integration build and published since it is a general indication of areas that need work. Now Jim here only mentioned the core but this would apply to other extensions as well - I would be inclined to extend this requirement to any extension we consider "baseline" - any objections? For extensions considered baseline, I think "respectable" code coverage (~75%) is definitely a worthy goal. For baseline extensions, I would also add that we should also have a minimum bar in terms of what assembly features they support (e.g. state management, non- blocking, etc.). -- Jeremy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Test coverage, was: Process and content for next release?
I've used Clover in the past and Cobertura since Dan mentioned it here a while ago. Cobertura is really really easy and works today, just use mvn cobertura:cobertura and it generates coverage reports in target\site\cobertura. Thats so easy I'd really prefer this over Clover unless Clover can also be done that easily. There's nothing stopping anyone who prefers Clover to also use that themselves. 75% coverage (or even quite a bit higher for a 1.0 release) is a fine goal but I think its a bit early to start excluding things from the next releases for not reaching that percentage. I'd much prefer we focus things like getting functional and integration testing frameworks in place first. ...ant On 9/5/06, Raymond Feng <[EMAIL PROTECTED]> wrote: Hi, Clover sounds good to me and I have tried it before. +1 on the 75% test coverage. Raymond - Original Message - From: "Jim Marino" <[EMAIL PROTECTED]> To: Sent: Tuesday, September 05, 2006 8:48 AM Subject: Re: Test coverage, was: Process and content for next release? > > On Sep 5, 2006, at 8:14 AM, Jeremy Boynes wrote: > >> On Aug 16, 2006, at 2:49 PM, Jim Marino wrote: >>>> 2) SCA Core (spi, core, hostutil, test, plus apis once that >>>> refactor is done) >>>>Features I would like to see complete before we consider this >>>> stable are: >>>> Class loading changes >>>> Integration of databinding framework >>>> Support for async callbacks >>>> Support for complex properties >>>> Transitive dependency support >>>> >>> I'd also like to see much better test coverage than what we have. >>> This is hard to quantify, but while code coverage does not >>> guarantee good tests, it is an indicator. So, to have a metric, >>> I'd like to see core (and other extensions) at 75% coverage when >>> run through Clover. I picked Clover since it is a decent tool and >>> license-friendly but if someone would like to suggest an >>> alternative we could look at it as well. >> >> I think this goal is worth pursuing and would add that as a >> criteria for the next release. Apache has a license for Clover so >> we can all use it, Cobertura would be another alternative - any >> preference here? Whatever we use, I don't think this should be part >> of the build right now (although that could change later) but that >> the tool should be run periodically and the results published >> somewhere (e.g. on our site). >> > I prefer Clover as it also has nice IDE integration. I also think > test coverage should be run as part of an integration build and > published since it is a general indication of areas that need work. > >> Now Jim here only mentioned the core but this would apply to other >> extensions as well - I would be inclined to extend this requirement >> to any extension we consider "baseline" - any objections? >> > For extensions considered baseline, I think "respectable" code > coverage (~75%) is definitely a worthy goal. For baseline extensions, > I would also add that we should also have a minimum bar in terms of > what assembly features they support (e.g. state management, non- > blocking, etc.). > >> -- >> Jeremy >> >> >> - >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Test coverage, was: Process and content for next release?
On Sep 6, 2006, at 9:49 AM, ant elder wrote: I've used Clover in the past and Cobertura since Dan mentioned it here a while ago. Cobertura is really really easy and works today, just use mvn cobertura:cobertura and it generates coverage reports in target\site\cobertura. Thats so easy I'd really prefer this over Clover unless Clover can also be done that easily. There's nothing stopping anyone who prefers Clover to also use that themselves. Clover integrates really nicely and also has great IDE integration, including a visual display of non-covered code inline. Does Cobertura do that? 75% coverage (or even quite a bit higher for a 1.0 release) is a fine goal but I think its a bit early to start excluding things from the next releases for not reaching that percentage. I'd much prefer we focus things like getting functional and integration testing frameworks in place first. Could we make it a requirement that we also have functional and integration testing in place prior to any release? I don't see how we can realistically get quality without unit testing and while code coverage is not a guarantee, it is a good quantitative indication of where we stand. Currently, we have many half-completed and untested extensions and I don't think they should be released in their current state lest we risk gaining a reputation of shipping poor code. If we can't get acceptable coverage, I think we should hold off shipping the extensions and just ship kernel as that offers useful functionality (assuming we reach at least 75% which is definitely achievable given the current amount of coverage). Jim ...ant On 9/5/06, Raymond Feng <[EMAIL PROTECTED]> wrote: Hi, Clover sounds good to me and I have tried it before. +1 on the 75% test coverage. Raymond - Original Message - From: "Jim Marino" <[EMAIL PROTECTED]> To: Sent: Tuesday, September 05, 2006 8:48 AM Subject: Re: Test coverage, was: Process and content for next release? > > On Sep 5, 2006, at 8:14 AM, Jeremy Boynes wrote: > >> On Aug 16, 2006, at 2:49 PM, Jim Marino wrote: >>>> 2) SCA Core (spi, core, hostutil, test, plus apis once that >>>> refactor is done) >>>>Features I would like to see complete before we consider this >>>> stable are: >>>> Class loading changes >>>> Integration of databinding framework >>>> Support for async callbacks >>>> Support for complex properties >>>> Transitive dependency support >>>> >>> I'd also like to see much better test coverage than what we have. >>> This is hard to quantify, but while code coverage does not >>> guarantee good tests, it is an indicator. So, to have a metric, >>> I'd like to see core (and other extensions) at 75% coverage when >>> run through Clover. I picked Clover since it is a decent tool and >>> license-friendly but if someone would like to suggest an >>> alternative we could look at it as well. >> >> I think this goal is worth pursuing and would add that as a >> criteria for the next release. Apache has a license for Clover so >> we can all use it, Cobertura would be another alternative - any >> preference here? Whatever we use, I don't think this should be part >> of the build right now (although that could change later) but that >> the tool should be run periodically and the results published >> somewhere (e.g. on our site). >> > I prefer Clover as it also has nice IDE integration. I also think > test coverage should be run as part of an integration build and > published since it is a general indication of areas that need work. > >> Now Jim here only mentioned the core but this would apply to other >> extensions as well - I would be inclined to extend this requirement >> to any extension we consider "baseline" - any objections? >> > For extensions considered baseline, I think "respectable" code > coverage (~75%) is definitely a worthy goal. For baseline extensions, > I would also add that we should also have a minimum bar in terms of > what assembly features they support (e.g. state management, non- > blocking, etc.). > >> -- >> Jeremy >> >> >> - >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
