subject:"\"Some thoughts on quality\""

On Thu, Aug 15, 2013 at 8:29 AM, Jürgen Schmidt  wrote:
> On 8/15/13 1:33 PM, Rob Weir wrote:
>> On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt  
>> wrote:
>>> On 8/15/13 12:19 PM, janI wrote:
 On Aug 15, 2013 11:14 AM, "Jürgen Schmidt"  wrote:
>
> On 8/14/13 8:30 PM, Rob Weir wrote:
>> On Wed, Aug 14, 2013 at 1:55 PM, janI  wrote:
>>> On 14 August 2013 19:36, Edwin Sharp  wrote:
>>>
 Dear Rob
 The 4.0 release was too ambitious - we should advance in smaller
 steps.
 Nothing compares to general public testing - betas and release
 candidates
 should not be avoided.
 TestLink cases should be less comprehesive (in terms of feature
 coverage)
 and more stress testing oriented.
 Regards,
 Edwin

 On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The
 fact
> that we're doing this, and their are no arguments against it, shows
> that we value quality.   I'd like to take this a step further, and
 see
> what we can learn from the defects in AOO 4.0.0 and what we can do
> going forward to improve.
>
> Quality, in the end, is a process, not a state of grace.  We improve
> by working smarter, not working harder.  The goal should be to learn
> and improve, as individuals and as a community.
>
> Every regression that made it into 4.0.0 was added there by a
> programmer.  And the defect went undetected by testers.  This is not
> to blame.  It just means that we're all human.  We know that.  We all
> make mistakes.  I make mistakes.  A quality process is not about
> becoming perfect, but about acknowledging that we make mistakes and
> that certain formal and informal practices are needed to prevent and
> detect these mistakes.
>
> But enough about generalities.  I'm hoping you'll join with me in
> examining the 32 confirmed 4.0.0 regression defects and answering a
> few questions:
>
> 1) What caused the bug?   What was the "root cause"?  Note:
> "programmer error" is not really a cause.  We should ask what caused
> the error.
>
> 2) What can we do to prevent bugs like this from being checked in?
>
> 3) Why wasn't the bug found during testing?  Was it not covered by
 any
> existing test case?  Was a test case run but the defect was not
> recognized?  Was the defect introduced into the software after the
> tests had already been executed?
>
> 4) What can we do to ensure that bugs like this are caught during
 testing?
>
> So 2 basic questions -- what went wrong and how can we prevent it in
> the future, looked at from perspective of programmers and testers.
  If
> we can keep these questions in mind, and try to answer them, we may
 be
> able to find some patterns that can lead to some process changes for
> AOO 4.1.
>
> You can find the 4.0.0 regressions in Bugzilla here:
>
>

 https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>
>
> Regards,
>
> -Rob

>>>
>>> I strongly believe that one of the things that went wrong is our
 limited
>>> possibility to retest (due to resources), when I look at our current
 manual
>>
>> I wonder about that as well.  That's one reason it would be good to
>> know how many of the confirmed regressions were introduced late in the
>> release process, and thus missed coverage in our full test pass.
>>
>>> testcases, a lot of those could be automated, e.g. with a simple UI
 macro,
>>> that would enable us to run these test cases with every build. It may
 sound
>>> like a dream but where I come from, we did that every night, and it
 caught
>>> a lot of regression bugs and sideeffects.
>>>
>>
>> This begs the question:  Is the functionality of the regressions
>> covered by our test cases?  Or are they covered but we didn't execute
>> them?  Or we executed them but didn't recognize the defect?  I don't
>> know (yet).
>>
>>> A simple start, if to request that every bug fix, is issued with at
 least
>>> one test case (automated or manual).
>>>
>>
>> Often there is, though this information lives in Bugzilla.  One thing
>> we did on another (non open source) project is to mark defects in our
>> bugtracking system that should become test cases.   Not every bug did
>> that.  For example, a defect report to update a mispelling in the UI
>> would not lead to a new test case.  But many would.
>
>

Re: Some thoughts on quality

On Thu, Aug 15, 2013 at 4:08 AM, Oliver-Rainer Wittmann
 wrote:
> Hi,
>
>
> On 14.08.2013 18:59, Rob Weir wrote:
>>
>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The fact
>> that we're doing this, and their are no arguments against it, shows
>> that we value quality.   I'd like to take this a step further, and see
>> what we can learn from the defects in AOO 4.0.0 and what we can do
>> going forward to improve.
>>
>> Quality, in the end, is a process, not a state of grace.  We improve
>> by working smarter, not working harder.  The goal should be to learn
>> and improve, as individuals and as a community.
>>
>> Every regression that made it into 4.0.0 was added there by a
>> programmer.  And the defect went undetected by testers.  This is not
>> to blame.  It just means that we're all human.  We know that.  We all
>> make mistakes.  I make mistakes.  A quality process is not about
>> becoming perfect, but about acknowledging that we make mistakes and
>> that certain formal and informal practices are needed to prevent and
>> detect these mistakes.
>>
>> But enough about generalities.  I'm hoping you'll join with me in
>> examining the 32 confirmed 4.0.0 regression defects and answering a
>> few questions:
>>
>> 1) What caused the bug?   What was the "root cause"?  Note:
>> "programmer error" is not really a cause.  We should ask what caused
>> the error.
>>
>> 2) What can we do to prevent bugs like this from being checked in?
>>
>> 3) Why wasn't the bug found during testing?  Was it not covered by any
>> existing test case?  Was a test case run but the defect was not
>> recognized?  Was the defect introduced into the software after the
>> tests had already been executed?
>>
>> 4) What can we do to ensure that bugs like this are caught during testing?
>>
>> So 2 basic questions -- what went wrong and how can we prevent it in
>> the future, looked at from perspective of programmers and testers.  If
>> we can keep these questions in mind, and try to answer them, we may be
>> able to find some patterns that can lead to some process changes for
>> AOO 4.1.
>>
>> You can find the 4.0.0 regressions in Bugzilla here:
>>
>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>
>
> Please include also issue with status ACCEPTED.
>

Done.  That now gives us 36.

-Rob


> Best regards, Oliver.
>
>>
>> Regards,
>>
>> -Rob
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
>> For additional commands, e-mail: dev-h...@openoffice.apache.org
>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt  wrote:
> On 8/15/13 12:19 PM, janI wrote:
>> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt"  wrote:
>>>
>>> On 8/14/13 8:30 PM, Rob Weir wrote:
 On Wed, Aug 14, 2013 at 1:55 PM, janI  wrote:
> On 14 August 2013 19:36, Edwin Sharp  wrote:
>
>> Dear Rob
>> The 4.0 release was too ambitious - we should advance in smaller
>> steps.
>> Nothing compares to general public testing - betas and release
>> candidates
>> should not be avoided.
>> TestLink cases should be less comprehesive (in terms of feature
>> coverage)
>> and more stress testing oriented.
>> Regards,
>> Edwin
>>
>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The
>> fact
>>> that we're doing this, and their are no arguments against it, shows
>>> that we value quality.   I'd like to take this a step further, and
>> see
>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>> going forward to improve.
>>>
>>> Quality, in the end, is a process, not a state of grace.  We improve
>>> by working smarter, not working harder.  The goal should be to learn
>>> and improve, as individuals and as a community.
>>>
>>> Every regression that made it into 4.0.0 was added there by a
>>> programmer.  And the defect went undetected by testers.  This is not
>>> to blame.  It just means that we're all human.  We know that.  We all
>>> make mistakes.  I make mistakes.  A quality process is not about
>>> becoming perfect, but about acknowledging that we make mistakes and
>>> that certain formal and informal practices are needed to prevent and
>>> detect these mistakes.
>>>
>>> But enough about generalities.  I'm hoping you'll join with me in
>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>> few questions:
>>>
>>> 1) What caused the bug?   What was the "root cause"?  Note:
>>> "programmer error" is not really a cause.  We should ask what caused
>>> the error.
>>>
>>> 2) What can we do to prevent bugs like this from being checked in?
>>>
>>> 3) Why wasn't the bug found during testing?  Was it not covered by
>> any
>>> existing test case?  Was a test case run but the defect was not
>>> recognized?  Was the defect introduced into the software after the
>>> tests had already been executed?
>>>
>>> 4) What can we do to ensure that bugs like this are caught during
>> testing?
>>>
>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>> the future, looked at from perspective of programmers and testers.
>>  If
>>> we can keep these questions in mind, and try to answer them, we may
>> be
>>> able to find some patterns that can lead to some process changes for
>>> AOO 4.1.
>>>
>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>
>>>
>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>
>>>
>>> Regards,
>>>
>>> -Rob
>>
>
> I strongly believe that one of the things that went wrong is our
>> limited
> possibility to retest (due to resources), when I look at our current
>> manual

 I wonder about that as well.  That's one reason it would be good to
 know how many of the confirmed regressions were introduced late in the
 release process, and thus missed coverage in our full test pass.

> testcases, a lot of those could be automated, e.g. with a simple UI
>> macro,
> that would enable us to run these test cases with every build. It may
>> sound
> like a dream but where I come from, we did that every night, and it
>> caught
> a lot of regression bugs and sideeffects.
>

 This begs the question:  Is the functionality of the regressions
 covered by our test cases?  Or are they covered but we didn't execute
 them?  Or we executed them but didn't recognize the defect?  I don't
 know (yet).

> A simple start, if to request that every bug fix, is issued with at
>> least
> one test case (automated or manual).
>

 Often there is, though this information lives in Bugzilla.  One thing
 we did on another (non open source) project is to mark defects in our
 bugtracking system that should become test cases.   Not every bug did
 that.  For example, a defect report to update a mispelling in the UI
 would not lead to a new test case.  But many would.
>>>
>>> we have the automated test framework that needs some more attention and
>>> polishing. And of course the tests have to improved to get satisfying
>>> result.
>>>
>>> We have
>>>
>>> BVT - build verification test
>>> FVT - functional verification test
>>> PVT - performance verification test
>>> SVT - system verification tes

Re: Some thoughts on quality

On Thu, Aug 15, 2013 at 2:59 AM, Edwin Sharp  wrote:
>
>
> On Wed, Aug 14, 2013, at 22:58, Rob Weir wrote:
>>
>> Finally, I think we can all point to a similar open source project
>> that has numerous betas, but still suffers from poor quality.  So a
>> public beta, by itself, is not sufficient.  We need some upstream
>> improvements as well, I think.  But we should do a beta as well.  But
>> aim to have the highest quality beta we can, right?
>>
>> Regards,
>>
>> -Rob
>
> This comparison is not fair because the release frequency of the two projects 
> is totally different.
>

To the original question you had suggested more frequent releases.
Raphael and Hagar suggested beta releases.  So it is entirely relevant
to point out that a sister project that had adopted both of those
practices has not achieved any great quality improvements.

When you think of it, why should release pacing have any impact on
quality?  Quality is increased if you prevent bugs, detect them before
you release, or fix them before you release.  Anything that claims to
improve quality should have a direct impact on one of those three
actions.  Does a release prevent bugs?  Detect them?  Fix them?  No,
not really.

What a release can do is prompt increase QA attention as the release
approaches.  But this is an indirect and not obligatory practice.  We
could have frequent releases that are tested less, because of the less
time available for testing.  This would lead to lower quality.  Or we
could have multiple full test passes with a longer release, taking
better advantage of the time, and leading to higher quality.

So the thing that matters here is the testing time, testing coverage,
testing efficiency, etc.  The release pace has no direct effect on
quality.

> Quality can also be improved by better community culture. No offense, but I 
> found this inappropriate:
>

Culture is important, yes, to the extent it leads to the adoption of
quality practices.  One good cultural point would be a community that
is not offended by facts and fact-based reasoning.

Regards,

-Rob

> On Fri, Jun 7, 2013, at 3:33, Rob Weir wrote:
>>
>> I'll try to clean up a few tonight while I watch TV.
>>
>
>
>
>
> -
> To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: qa-h...@openoffice.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-15 Thread Jürgen Schmidt

On 8/15/13 12:19 PM, janI wrote:
> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt"  wrote:
>>
>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>> On Wed, Aug 14, 2013 at 1:55 PM, janI  wrote:
 On 14 August 2013 19:36, Edwin Sharp  wrote:

> Dear Rob
> The 4.0 release was too ambitious - we should advance in smaller
> steps.
> Nothing compares to general public testing - betas and release
> candidates
> should not be avoided.
> TestLink cases should be less comprehesive (in terms of feature
> coverage)
> and more stress testing oriented.
> Regards,
> Edwin
>
> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The
> fact
>> that we're doing this, and their are no arguments against it, shows
>> that we value quality.   I'd like to take this a step further, and
> see
>> what we can learn from the defects in AOO 4.0.0 and what we can do
>> going forward to improve.
>>
>> Quality, in the end, is a process, not a state of grace.  We improve
>> by working smarter, not working harder.  The goal should be to learn
>> and improve, as individuals and as a community.
>>
>> Every regression that made it into 4.0.0 was added there by a
>> programmer.  And the defect went undetected by testers.  This is not
>> to blame.  It just means that we're all human.  We know that.  We all
>> make mistakes.  I make mistakes.  A quality process is not about
>> becoming perfect, but about acknowledging that we make mistakes and
>> that certain formal and informal practices are needed to prevent and
>> detect these mistakes.
>>
>> But enough about generalities.  I'm hoping you'll join with me in
>> examining the 32 confirmed 4.0.0 regression defects and answering a
>> few questions:
>>
>> 1) What caused the bug?   What was the "root cause"?  Note:
>> "programmer error" is not really a cause.  We should ask what caused
>> the error.
>>
>> 2) What can we do to prevent bugs like this from being checked in?
>>
>> 3) Why wasn't the bug found during testing?  Was it not covered by
> any
>> existing test case?  Was a test case run but the defect was not
>> recognized?  Was the defect introduced into the software after the
>> tests had already been executed?
>>
>> 4) What can we do to ensure that bugs like this are caught during
> testing?
>>
>> So 2 basic questions -- what went wrong and how can we prevent it in
>> the future, looked at from perspective of programmers and testers.
>  If
>> we can keep these questions in mind, and try to answer them, we may
> be
>> able to find some patterns that can lead to some process changes for
>> AOO 4.1.
>>
>> You can find the 4.0.0 regressions in Bugzilla here:
>>
>>
>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>
>>
>> Regards,
>>
>> -Rob
>

 I strongly believe that one of the things that went wrong is our
> limited
 possibility to retest (due to resources), when I look at our current
> manual
>>>
>>> I wonder about that as well.  That's one reason it would be good to
>>> know how many of the confirmed regressions were introduced late in the
>>> release process, and thus missed coverage in our full test pass.
>>>
 testcases, a lot of those could be automated, e.g. with a simple UI
> macro,
 that would enable us to run these test cases with every build. It may
> sound
 like a dream but where I come from, we did that every night, and it
> caught
 a lot of regression bugs and sideeffects.

>>>
>>> This begs the question:  Is the functionality of the regressions
>>> covered by our test cases?  Or are they covered but we didn't execute
>>> them?  Or we executed them but didn't recognize the defect?  I don't
>>> know (yet).
>>>
 A simple start, if to request that every bug fix, is issued with at
> least
 one test case (automated or manual).

>>>
>>> Often there is, though this information lives in Bugzilla.  One thing
>>> we did on another (non open source) project is to mark defects in our
>>> bugtracking system that should become test cases.   Not every bug did
>>> that.  For example, a defect report to update a mispelling in the UI
>>> would not lead to a new test case.  But many would.
>>
>> we have the automated test framework that needs some more attention and
>> polishing. And of course the tests have to improved to get satisfying
>> result.
>>
>> We have
>>
>> BVT - build verification test
>> FVT - functional verification test
>> PVT - performance verification test
>> SVT - system verification test
>>
>> But I have to confess that I have limited knowledge about it yet
> 
> I aware that we ha a limited automated framework, at least thats what I
> found and played with.
> 
>

Re: Some thoughts on quality

2013-08-15 Thread janI

On Aug 15, 2013 11:14 AM, "Jürgen Schmidt"  wrote:
>
> On 8/14/13 8:30 PM, Rob Weir wrote:
> > On Wed, Aug 14, 2013 at 1:55 PM, janI  wrote:
> >> On 14 August 2013 19:36, Edwin Sharp  wrote:
> >>
> >>> Dear Rob
> >>> The 4.0 release was too ambitious - we should advance in smaller
steps.
> >>> Nothing compares to general public testing - betas and release
candidates
> >>> should not be avoided.
> >>> TestLink cases should be less comprehesive (in terms of feature
coverage)
> >>> and more stress testing oriented.
> >>> Regards,
> >>> Edwin
> >>>
> >>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>  We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The
fact
>  that we're doing this, and their are no arguments against it, shows
>  that we value quality.   I'd like to take this a step further, and
see
>  what we can learn from the defects in AOO 4.0.0 and what we can do
>  going forward to improve.
> 
>  Quality, in the end, is a process, not a state of grace.  We improve
>  by working smarter, not working harder.  The goal should be to learn
>  and improve, as individuals and as a community.
> 
>  Every regression that made it into 4.0.0 was added there by a
>  programmer.  And the defect went undetected by testers.  This is not
>  to blame.  It just means that we're all human.  We know that.  We all
>  make mistakes.  I make mistakes.  A quality process is not about
>  becoming perfect, but about acknowledging that we make mistakes and
>  that certain formal and informal practices are needed to prevent and
>  detect these mistakes.
> 
>  But enough about generalities.  I'm hoping you'll join with me in
>  examining the 32 confirmed 4.0.0 regression defects and answering a
>  few questions:
> 
>  1) What caused the bug?   What was the "root cause"?  Note:
>  "programmer error" is not really a cause.  We should ask what caused
>  the error.
> 
>  2) What can we do to prevent bugs like this from being checked in?
> 
>  3) Why wasn't the bug found during testing?  Was it not covered by
any
>  existing test case?  Was a test case run but the defect was not
>  recognized?  Was the defect introduced into the software after the
>  tests had already been executed?
> 
>  4) What can we do to ensure that bugs like this are caught during
> >>> testing?
> 
>  So 2 basic questions -- what went wrong and how can we prevent it in
>  the future, looked at from perspective of programmers and testers.
 If
>  we can keep these questions in mind, and try to answer them, we may
be
>  able to find some patterns that can lead to some process changes for
>  AOO 4.1.
> 
>  You can find the 4.0.0 regressions in Bugzilla here:
> 
> 
> >>>
https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
> 
> 
>  Regards,
> 
>  -Rob
> >>>
> >>
> >> I strongly believe that one of the things that went wrong is our
limited
> >> possibility to retest (due to resources), when I look at our current
manual
> >
> > I wonder about that as well.  That's one reason it would be good to
> > know how many of the confirmed regressions were introduced late in the
> > release process, and thus missed coverage in our full test pass.
> >
> >> testcases, a lot of those could be automated, e.g. with a simple UI
macro,
> >> that would enable us to run these test cases with every build. It may
sound
> >> like a dream but where I come from, we did that every night, and it
caught
> >> a lot of regression bugs and sideeffects.
> >>
> >
> > This begs the question:  Is the functionality of the regressions
> > covered by our test cases?  Or are they covered but we didn't execute
> > them?  Or we executed them but didn't recognize the defect?  I don't
> > know (yet).
> >
> >> A simple start, if to request that every bug fix, is issued with at
least
> >> one test case (automated or manual).
> >>
> >
> > Often there is, though this information lives in Bugzilla.  One thing
> > we did on another (non open source) project is to mark defects in our
> > bugtracking system that should become test cases.   Not every bug did
> > that.  For example, a defect report to update a mispelling in the UI
> > would not lead to a new test case.  But many would.
>
> we have the automated test framework that needs some more attention and
> polishing. And of course the tests have to improved to get satisfying
> result.
>
> We have
>
> BVT - build verification test
> FVT - functional verification test
> PVT - performance verification test
> SVT - system verification test
>
> But I have to confess that I have limited knowledge about it yet

I aware that we ha a limited automated framework, at least thats what I
found and played with.

but, it is not integrated into our build, or our buildbot. Especially
testing in buildbot

Re: Some thoughts on quality

2013-08-15 Thread Jürgen Schmidt

On 8/14/13 8:30 PM, Rob Weir wrote:
> On Wed, Aug 14, 2013 at 1:55 PM, janI  wrote:
>> On 14 August 2013 19:36, Edwin Sharp  wrote:
>>
>>> Dear Rob
>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>> Nothing compares to general public testing - betas and release candidates
>>> should not be avoided.
>>> TestLink cases should be less comprehesive (in terms of feature coverage)
>>> and more stress testing oriented.
>>> Regards,
>>> Edwin
>>>
>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
 We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The fact
 that we're doing this, and their are no arguments against it, shows
 that we value quality.   I'd like to take this a step further, and see
 what we can learn from the defects in AOO 4.0.0 and what we can do
 going forward to improve.

 Quality, in the end, is a process, not a state of grace.  We improve
 by working smarter, not working harder.  The goal should be to learn
 and improve, as individuals and as a community.

 Every regression that made it into 4.0.0 was added there by a
 programmer.  And the defect went undetected by testers.  This is not
 to blame.  It just means that we're all human.  We know that.  We all
 make mistakes.  I make mistakes.  A quality process is not about
 becoming perfect, but about acknowledging that we make mistakes and
 that certain formal and informal practices are needed to prevent and
 detect these mistakes.

 But enough about generalities.  I'm hoping you'll join with me in
 examining the 32 confirmed 4.0.0 regression defects and answering a
 few questions:

 1) What caused the bug?   What was the "root cause"?  Note:
 "programmer error" is not really a cause.  We should ask what caused
 the error.

 2) What can we do to prevent bugs like this from being checked in?

 3) Why wasn't the bug found during testing?  Was it not covered by any
 existing test case?  Was a test case run but the defect was not
 recognized?  Was the defect introduced into the software after the
 tests had already been executed?

 4) What can we do to ensure that bugs like this are caught during
>>> testing?

 So 2 basic questions -- what went wrong and how can we prevent it in
 the future, looked at from perspective of programmers and testers.  If
 we can keep these questions in mind, and try to answer them, we may be
 able to find some patterns that can lead to some process changes for
 AOO 4.1.

 You can find the 4.0.0 regressions in Bugzilla here:

>>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834

 Regards,

 -Rob
>>>
>>
>> I strongly believe that one of the things that went wrong is our limited
>> possibility to retest (due to resources), when I look at our current manual
> 
> I wonder about that as well.  That's one reason it would be good to
> know how many of the confirmed regressions were introduced late in the
> release process, and thus missed coverage in our full test pass.
> 
>> testcases, a lot of those could be automated, e.g. with a simple UI macro,
>> that would enable us to run these test cases with every build. It may sound
>> like a dream but where I come from, we did that every night, and it caught
>> a lot of regression bugs and sideeffects.
>>
> 
> This begs the question:  Is the functionality of the regressions
> covered by our test cases?  Or are they covered but we didn't execute
> them?  Or we executed them but didn't recognize the defect?  I don't
> know (yet).
> 
>> A simple start, if to request that every bug fix, is issued with at least
>> one test case (automated or manual).
>>
> 
> Often there is, though this information lives in Bugzilla.  One thing
> we did on another (non open source) project is to mark defects in our
> bugtracking system that should become test cases.   Not every bug did
> that.  For example, a defect report to update a mispelling in the UI
> would not lead to a new test case.  But many would.

we have the automated test framework that needs some more attention and
polishing. And of course the tests have to improved to get satisfying
result.

We have

BVT - build verification test
FVT - functional verification test
PVT - performance verification test
SVT - system verification test

But I have to confess that I have limited knowledge about it yet

Juergen

> 
> Regards,
> 
> -Rob
> 
>> rgds
>> jan I.
>>
>>

 -
 To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
 For additional commands, e-mail: qa-h...@openoffice.apache.org

>>>
>>> -
>>> To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
>>> For additional co

Re: Some thoughts on quality

2013-08-15 Thread Oliver-Rainer Wittmann

Hi,

On 14.08.2013 18:59, Rob Weir wrote:

We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
that we're doing this, and their are no arguments against it, shows
that we value quality. I'd like to take this a step further, and see
what we can learn from the defects in AOO 4.0.0 and what we can do
going forward to improve.

Quality, in the end, is a process, not a state of grace. We improve
by working smarter, not working harder. The goal should be to learn
and improve, as individuals and as a community.

Every regression that made it into 4.0.0 was added there by a
programmer. And the defect went undetected by testers. This is not
to blame. It just means that we're all human. We know that. We all
make mistakes. I make mistakes. A quality process is not about
becoming perfect, but about acknowledging that we make mistakes and
that certain formal and informal practices are needed to prevent and
detect these mistakes.

But enough about generalities. I'm hoping you'll join with me in
examining the 32 confirmed 4.0.0 regression defects and answering a
few questions:

1) What caused the bug? What was the "root cause"? Note:
"programmer error" is not really a cause. We should ask what caused
the error.

2) What can we do to prevent bugs like this from being checked in?

3) Why wasn't the bug found during testing? Was it not covered by any
existing test case? Was a test case run but the defect was not
recognized? Was the defect introduced into the software after the
tests had already been executed?

4) What can we do to ensure that bugs like this are caught during testing?

So 2 basic questions -- what went wrong and how can we prevent it in
the future, looked at from perspective of programmers and testers. If
we can keep these questions in mind, and try to answer them, we may be
able to find some patterns that can lead to some process changes for
AOO 4.1.

You can find the 4.0.0 regressions in Bugzilla here:

https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834

Please include also issue with status ACCEPTED.

Best regards, Oliver.

Regards,

-Rob

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-14 Thread Kay Schenk

On Wed, Aug 14, 2013 at 12:51 PM, Marcus (OOo)  wrote:

> Am 08/14/2013 09:39 PM, schrieb Rob Weir:
>
>  On Wed, Aug 14, 2013 at 3:33 PM, Hagar 
> Delest>
>>  wrote:
>>
>>> Le 14/08/2013 21:29, Hagar Delest a écrit :
>>>
>>>  Le 14/08/2013 21:01, Raphael Bircher a écrit :

  I like the idea of a public beta.  But consider the numbers.  The 40
>> or so regressions that were reported came from an install base (based
>> on download figures since 4.0.0 was released) of around 3 million
>> users.  Realistically, can we expect anywhere near that number in a
>> public beta?  Or is it more likely that a beta program has 10,000
>> users or fewer?  I don't know the answer here.   But certainly a
>> well-publicized and used beta will find more than a beta used by just
>> a few hundred users.
>>
>
> The public beta is from my point of view realy important. Even you have
> only 10'000 Downloads of a beta, you have normaly verry experianced
> Users
> there, like power users from Companies. They provide realy valua
> feedback.
> So from my point of view, this is one of the moast important changes
> we have
> to do. For all Feature release a beta version. And don't forget,
> people are
> realy happy to do beta tests. but many of them are maybe not willing to
> follow a mailing list.
>


 +1.
 OOo used to have RC versions strongly advertised, it could go up to 6 RC
 before going final and it was very useful to spot the main bugs.

  And I forgot: the 2 main bugs (slow saving in MS Office formats and
>>> the Calc
>>> display issue under Windows) were reported in the forum only 2 days and 5
>>> days after the release! and we have had many topics for both afterward.
>>> A RC
>>> would have clearly spared the hassle.
>>>
>>> See:
>>> http://forum.openoffice.org/**en/forum/viewtopic.php?f=9&t=**63082
>>> http://forum.openoffice.org/**en/forum/viewtopic.php?f=9&t=**63161
>>>
>>>
>>
>> Maybe we need to call an earlier build the "RC" so it will get more
>> attention?   We had a complete test build that we were testing for
>> over a month.  But maybe it is ignored unless we call it an "RC"?  In
>> other words, there were many opportunities for users to help try 4.0
>> before it was released, but maybe there opportunities were not well
>> known.
>>
>
> Déjà vu ;-)
>
> I remember that we had the same problem in the old project.
>
> Call it "Developer Snapshot" and you will have a few hundreads downloads
> with respective number of feedback. But call it (and announce it!) with a
> name that sounds more familar (like Early Access, Preview, Beta, RC) then
> we had much more. So, going public more early should bring us a higher
> number of feedback.
>
> Marcus


If this would help, then we should do it. Hopefully this will get users'
attention more.



>
>
>
> --**--**-
> To unsubscribe, e-mail: 
> dev-unsubscribe@openoffice.**apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
>
>


-- 
-
MzK

"When in doubt, cop an attitude."
-- Cat laws

Re: Some thoughts on quality

2013-08-14 Thread sebb

On 14 August 2013 23:10, Rob Weir  wrote:
> On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo)  wrote:
>> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>>
>>> Le 14/08/2013 21:39, Rob Weir a écrit :

 Maybe we need to call an earlier build the "RC" so it will get more
 attention? We had a complete test build that we were testing for
 over a month. But maybe it is ignored unless we call it an "RC"? In
 other words, there were many opportunities for users to help try 4.0
 before it was released, but maybe there opportunities were not well
 known.
>>>
>>>
>>> I think that it was too difficult to get to the dev versions and it was
>>> too much complicated. There was no clear link to download a dev version
>>> (I had to rely on the url in the messages from the dev list).
>>
>>
>> This was intended.
>>
>> We hadn't published the dev builds via Apache or SF mirrors but only on 2
>> people accounts. Apache policy says it's not allowed to publish them for a
>> wider audience to save the servers from a high traffic load. It's the job of
>> the mirrors to handle this.
>>
>>
>>> What I see (from a standard user point of view) for a RC:
>>> - When a dev version is almost done, rename it RC and make it known
>>> (blog and we would relay the announcement in the forums of course)
>>> - Have a link visible under the main download button of the download
>>
>>
>> Both can be done, depending where the install files are located.
>>
>>
>>> page (perhaps a similar button as a dedicated entry)
>>> - Make that RC installable in parallel with a stable version
>>> - No file association allowed for that RC by design
>>
>>
>> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC
>> anymore. One of the RC attributes is to change it into the final release
>> with, e.g., just a file name change. But this has to be done without any
>> code changes. Otherwise you have to change code parts, build again, test
>> again, ... ;-)
>>
>> But a Beta release could go this separated way.
>>
>
> Right.  A release is a release is a release.  The basic requirements
> for every release still apply:
>
> 1) 3 PMC +1 votes
> 2) Must include source files
> 3) Digital signatures, hash files, etc.
>
> But we can have a "beta release", that follows these rules, and it
> would be acceptable.  We can then host on the mirrors, publicize, etc.

There would likely be some restrictions on how many extra downloads
are permitted.
For example, the ASF mirrors probably could not cope with a set of
betas of all the languages for all the OSes in addition to the current
GA release.

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo)  wrote:
> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>
>> Le 14/08/2013 21:39, Rob Weir a écrit :
>>>
>>> Maybe we need to call an earlier build the "RC" so it will get more
>>> attention? We had a complete test build that we were testing for
>>> over a month. But maybe it is ignored unless we call it an "RC"? In
>>> other words, there were many opportunities for users to help try 4.0
>>> before it was released, but maybe there opportunities were not well
>>> known.
>>
>>
>> I think that it was too difficult to get to the dev versions and it was
>> too much complicated. There was no clear link to download a dev version
>> (I had to rely on the url in the messages from the dev list).
>
>
> This was intended.
>
> We hadn't published the dev builds via Apache or SF mirrors but only on 2
> people accounts. Apache policy says it's not allowed to publish them for a
> wider audience to save the servers from a high traffic load. It's the job of
> the mirrors to handle this.
>
>
>> What I see (from a standard user point of view) for a RC:
>> - When a dev version is almost done, rename it RC and make it known
>> (blog and we would relay the announcement in the forums of course)
>> - Have a link visible under the main download button of the download
>
>
> Both can be done, depending where the install files are located.
>
>
>> page (perhaps a similar button as a dedicated entry)
>> - Make that RC installable in parallel with a stable version
>> - No file association allowed for that RC by design
>
>
> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC
> anymore. One of the RC attributes is to change it into the final release
> with, e.g., just a file name change. But this has to be done without any
> code changes. Otherwise you have to change code parts, build again, test
> again, ... ;-)
>
> But a Beta release could go this separated way.
>

Right.  A release is a release is a release.  The basic requirements
for every release still apply:

1) 3 PMC +1 votes
2) Must include source files
3) Digital signatures, hash files, etc.

But we can have a "beta release", that follows these rules, and it
would be acceptable.  We can then host on the mirrors, publicize, etc.

However, back to the original topic of this thread:   We should look
to see when the bugs we're fixing in 4.0.1 were added to the code.
Not to blame or make anyone feel bad.  But to understand.  If these
were late bugs then an earlier beta would not have found them.

-Rob

>
>> The wiki page for the dev builds were too complicated and sounded to
>> much beta to make the users confident in using them (when they managed
>> to get on that page!).
>
>
> Marcus
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-14 Thread David Gerard

On 14 August 2013 20:39, Rob Weir  wrote:

> Maybe we need to call an earlier build the "RC" so it will get more
> attention?   We had a complete test build that we were testing for
> over a month.  But maybe it is ignored unless we call it an "RC"?  In
> other words, there were many opportunities for users to help try 4.0
> before it was released, but maybe there opportunities were not well
> known.
>

It basically wasn't publicised. I knew about it because I actually went
looking for it (goal: to put an image on the Wikipedia article of the 4.0
sidebar). It took a bit of ferreting about to find the prereleases. I've
followed the list all this year, so I knew (a) it existed (b) what to ask
about; the percentage of AOO users on the dev list is all but invisible.

Suggestion 1: note prereleases on the blog and in the social media channels.

Suggestion 2: do an RC for 4.0.1 as well. Even if you don't think there's a
need to, and fully expect the RC to be byte-for-byte identical to the final
release, people will appreciate being asked.

The Cathedral and the Bazaar still applies. "Release early, release often."
You could have had six months' intense testing from users who were
seriously interested.

- d.

Re: Some thoughts on quality


Am 08/14/2013 10:24 PM, schrieb Marcus (OOo):

Am 08/14/2013 10:05 PM, schrieb Hagar Delest:

Le 14/08/2013 21:39, Rob Weir a écrit :

Maybe we need to call an earlier build the "RC" so it will get more
attention? We had a complete test build that we were testing for
over a month. But maybe it is ignored unless we call it an "RC"? In
other words, there were many opportunities for users to help try 4.0
before it was released, but maybe there opportunities were not well
known.


I think that it was too difficult to get to the dev versions and it was
too much complicated. There was no clear link to download a dev version
(I had to rely on the url in the messages from the dev list).


This was intended.

We hadn't published the dev builds via Apache or SF mirrors but only on
2 people accounts. Apache policy says it's not allowed to publish them
for a wider audience to save the servers from a high traffic load. It's
the job of the mirrors to handle this.


What I see (from a standard user point of view) for a RC:
- When a dev version is almost done, rename it RC and make it known
(blog and we would relay the announcement in the forums of course)
- Have a link visible under the main download button of the download


Both can be done, depending where the install files are located.


page (perhaps a similar button as a dedicated entry)
- Make that RC installable in parallel with a stable version
- No file association allowed for that RC by design


IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
RC anymore. One of the RC attributes is to change it into the final
release with, e.g., just a file name change. But this has to be done
without any code changes. Otherwise you have to change code parts, build
again, test again, ... ;-)

But a Beta release could go this separated way.


The wiki page for the dev builds were too complicated and sounded to
much beta to make the users confident in using them (when they managed
to get on that page!).


Marcus


Sorry, forgot the link:

[1] 
http://en.wikipedia.org/wiki/Software_release_life_cycle#Release_candidate



-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Am 08/14/2013 10:05 PM, schrieb Hagar Delest:

Le 14/08/2013 21:39, Rob Weir a écrit :

Maybe we need to call an earlier build the "RC" so it will get more
attention? We had a complete test build that we were testing for
over a month. But maybe it is ignored unless we call it an "RC"? In
other words, there were many opportunities for users to help try 4.0
before it was released, but maybe there opportunities were not well
known.


I think that it was too difficult to get to the dev versions and it was
too much complicated. There was no clear link to download a dev version
(I had to rely on the url in the messages from the dev list).


This was intended.

We hadn't published the dev builds via Apache or SF mirrors but only on 
2 people accounts. Apache policy says it's not allowed to publish them 
for a wider audience to save the servers from a high traffic load. It's 
the job of the mirrors to handle this.



What I see (from a standard user point of view) for a RC:
- When a dev version is almost done, rename it RC and make it known
(blog and we would relay the announcement in the forums of course)
- Have a link visible under the main download button of the download


Both can be done, depending where the install files are located.


page (perhaps a similar button as a dedicated entry)
- Make that RC installable in parallel with a stable version
- No file association allowed for that RC by design


IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a 
RC anymore. One of the RC attributes is to change it into the final 
release with, e.g., just a file name change. But this has to be done 
without any code changes. Otherwise you have to change code parts, build 
again, test again, ... ;-)


But a Beta release could go this separated way.


The wiki page for the dev builds were too complicated and sounded to
much beta to make the users confident in using them (when they managed
to get on that page!).


Marcus


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Am 08/14/2013 09:58 PM, schrieb Rob Weir:

On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo)  wrote:

Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:


Am 14.08.13 20:21, schrieb Rob Weir:


On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp  wrote:


Dear Rob
The 4.0 release was too ambitious - we should advance in smaller steps.
Nothing compares to general public testing - betas and release
candidates should not be avoided.
TestLink cases should be less comprehesive (in terms of feature
coverage) and more stress testing oriented.


The number to consider here is how many defects were found and fixed
during the 4.0.0 testing, before the general public users had access?
I assume it was quite substantial. If so, the TestLink usage was
effective. In other words, we might have found fewer bugs without
using it.



In general, my feeling is that it's too early to do a retrospective and ask
"what was good, what needs to be improved?" (to say it with SCRUM words ;-)
) Just after the first major release.



Memories are still fresh and programmers are looking at the relevant
code.  This is the best time to answer these questions.


We should look on the BZ query you have mentioned and see if there is one or
more hotspots that should be improved fast. That's it.



That would be fine.  I'm not suggesting anything radical.  Gradual,
but constant improvements are the way to high quality.


Then this should be sufficiant IMHO.


This is important to keep in mind: we want to prevent or find more
bugs, but we're not starting from zero. We're starting from a process
that does a lot of things right.

I like the idea of a public beta. But consider the numbers. The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users. Realistically, can we expect anywhere near that number in a
public beta? Or is it more likely that a beta program has 10,000
users or fewer? I don't know the answer here. But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.


The public beta is from my point of view realy important. Even you have
only 10'000 Downloads of a beta, you have normaly verry experianced
Users there, like power users from Companies. They provide realy valua
feedback. So from my point of view, this is one of the moast important
changes we have to do. For all Feature release a beta version. And don't
forget, people are realy happy to do beta tests. but many of them are
maybe not willing to follow a mailing list.



A public beta release is of course not the golden solution but could
activate some power users that give us the feedback we want and need.

So, +1 for going this way.

After the 4.1 release is done we can see if this was much better - and ask
ourself why? :-)



But several of the bugs in 4.0.0 should never have made it to even a
beta.  For example, the very slow saving in Excel format.  What
happened there?

Sure, this could be fixed later in the process, in a beta, or in a
4.0.1 as we're doing now.  But this really should have been detected
and fixed much earlier in the process.

What are betas good for?  Betas are good for expanding the set of
configurations. Platforms, languages, extensions, etc.But the
informal "tests" done by real users tend to be shallow.  Also, we have
no way of determining what the test coverage is.  In particular we
have no basis for telling the difference between the coverage of a 2
week beta versus a 4 week beta.  But with out formal test cases we can
easily look at how many test cases were executed. We could even look
at code coverage if we wanted to.

Maybe if there was a way to record what features beta testers were
using...   Or even have a little survey where they report what
platform they ran on, etc.

But very slow saving to XLS files?  A defect like that should have
been caught by us before a beta and before a RC.  We shouldn't expect
to find bugs like that in a beta.


Of course. I've said nothing different. ;-)


Finally, I think we can all point to a similar open source project
that has numerous betas, but still suffers from poor quality.  So a
public beta, by itself, is not sufficient.  We need some upstream
improvements as well, I think.  But we should do a beta as well.  But
aim to have the highest quality beta we can, right?


Sure, the XLS bug has nothing to with "this wouldn't happen with a 
Beta". As I answered to Hagar this is one hotspot that should be looked 
at closer.


Marcus


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Le 14/08/2013 21:39, Rob Weir a écrit :

Maybe we need to call an earlier build the "RC" so it will get more
attention?   We had a complete test build that we were testing for
over a month.  But maybe it is ignored unless we call it an "RC"?  In
other words, there were many opportunities for users to help try 4.0
before it was released, but maybe there opportunities were not well
known.


I think that it was too difficult to get to the dev versions and it was too 
much complicated. There was no clear link to download a dev version (I had to 
rely on the url in the messages from the dev list).

What I see (from a standard user point of view) for a RC:
- When a dev version is almost done, rename it RC and make it known (blog and 
we would relay the announcement in the forums of course)
- Have a link visible under the main download button of the download page 
(perhaps a similar button as a dedicated entry)
- Make that RC installable in parallel with a stable version
- No file association allowed for that RC by design

The wiki page for the dev builds were too complicated and sounded to much beta 
to make the users confident in using them (when they managed to get on that 
page!).

Hagar

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Am 08/14/2013 09:50 PM, schrieb Hagar Delest:

Le 14/08/2013 21:37, Marcus (OOo) a écrit :

In general, my feeling is that it's too early to do a retrospective
and ask "what was good, what needs to be improved?" (to say it with
SCRUM words ;-) ) Just after the first major release.

Well, I visit forums and contribute since OOo 2.0 and it's the first
time we (volunteers in forums) advise to downgrade so often, even for a
major upgrade (because of the slow saving time and the Calc display
issue mainly).
So there has been clearly a quality issue.


Sure, I don't want to deny this.

However, IMHO it's too early to look at the big picture and try to find 
the black spots.



About test case, it's strange that the 2 bugs above were not seen.


That seems indeen one hotspot I mentioned.

Marcus


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo)  wrote:
> Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
>
>> Am 14.08.13 20:21, schrieb Rob Weir:
>>>
>>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp  wrote:

 Dear Rob
 The 4.0 release was too ambitious - we should advance in smaller steps.
 Nothing compares to general public testing - betas and release
 candidates should not be avoided.
 TestLink cases should be less comprehesive (in terms of feature
 coverage) and more stress testing oriented.
>>>
>>> The number to consider here is how many defects were found and fixed
>>> during the 4.0.0 testing, before the general public users had access?
>>> I assume it was quite substantial. If so, the TestLink usage was
>>> effective. In other words, we might have found fewer bugs without
>>> using it.
>
>
> In general, my feeling is that it's too early to do a retrospective and ask
> "what was good, what needs to be improved?" (to say it with SCRUM words ;-)
> ) Just after the first major release.
>

Memories are still fresh and programmers are looking at the relevant
code.  This is the best time to answer these questions.

> We should look on the BZ query you have mentioned and see if there is one or
> more hotspots that should be improved fast. That's it.
>

That would be fine.  I'm not suggesting anything radical.  Gradual,
but constant improvements are the way to high quality.

>
>>> This is important to keep in mind: we want to prevent or find more
>>> bugs, but we're not starting from zero. We're starting from a process
>>> that does a lot of things right.
>>>
>>> I like the idea of a public beta. But consider the numbers. The 40
>>> or so regressions that were reported came from an install base (based
>>> on download figures since 4.0.0 was released) of around 3 million
>>> users. Realistically, can we expect anywhere near that number in a
>>> public beta? Or is it more likely that a beta program has 10,000
>>> users or fewer? I don't know the answer here. But certainly a
>>> well-publicized and used beta will find more than a beta used by just
>>> a few hundred users.
>>
>> The public beta is from my point of view realy important. Even you have
>> only 10'000 Downloads of a beta, you have normaly verry experianced
>> Users there, like power users from Companies. They provide realy valua
>> feedback. So from my point of view, this is one of the moast important
>> changes we have to do. For all Feature release a beta version. And don't
>> forget, people are realy happy to do beta tests. but many of them are
>> maybe not willing to follow a mailing list.
>
>
> A public beta release is of course not the golden solution but could
> activate some power users that give us the feedback we want and need.
>
> So, +1 for going this way.
>
> After the 4.1 release is done we can see if this was much better - and ask
> ourself why? :-)
>

But several of the bugs in 4.0.0 should never have made it to even a
beta.  For example, the very slow saving in Excel format.  What
happened there?

Sure, this could be fixed later in the process, in a beta, or in a
4.0.1 as we're doing now.  But this really should have been detected
and fixed much earlier in the process.

What are betas good for?  Betas are good for expanding the set of
configurations. Platforms, languages, extensions, etc.But the
informal "tests" done by real users tend to be shallow.  Also, we have
no way of determining what the test coverage is.  In particular we
have no basis for telling the difference between the coverage of a 2
week beta versus a 4 week beta.  But with out formal test cases we can
easily look at how many test cases were executed. We could even look
at code coverage if we wanted to.

Maybe if there was a way to record what features beta testers were
using...   Or even have a little survey where they report what
platform they ran on, etc.

But very slow saving to XLS files?  A defect like that should have
been caught by us before a beta and before a RC.  We shouldn't expect
to find bugs like that in a beta.

Finally, I think we can all point to a similar open source project
that has numerous betas, but still suffers from poor quality.  So a
public beta, by itself, is not sufficient.  We need some upstream
improvements as well, I think.  But we should do a beta as well.  But
aim to have the highest quality beta we can, right?

Regards,

-Rob

> Marcus
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Am 08/14/2013 09:39 PM, schrieb Rob Weir:

On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delest  wrote:

Le 14/08/2013 21:29, Hagar Delest a écrit :


Le 14/08/2013 21:01, Raphael Bircher a écrit :


I like the idea of a public beta.  But consider the numbers.  The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users.  Realistically, can we expect anywhere near that number in a
public beta?  Or is it more likely that a beta program has 10,000
users or fewer?  I don't know the answer here.   But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.


The public beta is from my point of view realy important. Even you have
only 10'000 Downloads of a beta, you have normaly verry experianced Users
there, like power users from Companies. They provide realy valua feedback.
So from my point of view, this is one of the moast important changes we have
to do. For all Feature release a beta version. And don't forget, people are
realy happy to do beta tests. but many of them are maybe not willing to
follow a mailing list.



+1.
OOo used to have RC versions strongly advertised, it could go up to 6 RC
before going final and it was very useful to spot the main bugs.


And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc
display issue under Windows) were reported in the forum only 2 days and 5
days after the release! and we have had many topics for both afterward. A RC
would have clearly spared the hassle.

See:
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161




Maybe we need to call an earlier build the "RC" so it will get more
attention?   We had a complete test build that we were testing for
over a month.  But maybe it is ignored unless we call it an "RC"?  In
other words, there were many opportunities for users to help try 4.0
before it was released, but maybe there opportunities were not well
known.


Déjà vu ;-)

I remember that we had the same problem in the old project.

Call it "Developer Snapshot" and you will have a few hundreads downloads 
with respective number of feedback. But call it (and announce it!) with 
a name that sounds more familar (like Early Access, Preview, Beta, RC) 
then we had much more. So, going public more early should bring us a 
higher number of feedback.


Marcus


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Le 14/08/2013 21:37, Marcus (OOo) a écrit :

In general, my feeling is that it's too early to do a retrospective and ask "what 
was good, what needs to be improved?" (to say it with SCRUM words ;-) ) Just after 
the first major release.

Well, I visit forums and contribute since OOo 2.0 and it's the first time we 
(volunteers in forums) advise to downgrade so often, even for a major upgrade 
(because of the slow saving time and the Calc display issue mainly).
So there has been clearly a quality issue.

About test case, it's strange that the 2 bugs above were not seen.

Hagar

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delest  wrote:
> Le 14/08/2013 21:29, Hagar Delest a écrit :
>
>> Le 14/08/2013 21:01, Raphael Bircher a écrit :
>>
 I like the idea of a public beta.  But consider the numbers.  The 40
 or so regressions that were reported came from an install base (based
 on download figures since 4.0.0 was released) of around 3 million
 users.  Realistically, can we expect anywhere near that number in a
 public beta?  Or is it more likely that a beta program has 10,000
 users or fewer?  I don't know the answer here.   But certainly a
 well-publicized and used beta will find more than a beta used by just
 a few hundred users.
>>>
>>> The public beta is from my point of view realy important. Even you have
>>> only 10'000 Downloads of a beta, you have normaly verry experianced Users
>>> there, like power users from Companies. They provide realy valua feedback.
>>> So from my point of view, this is one of the moast important changes we have
>>> to do. For all Feature release a beta version. And don't forget, people are
>>> realy happy to do beta tests. but many of them are maybe not willing to
>>> follow a mailing list.
>>
>>
>> +1.
>> OOo used to have RC versions strongly advertised, it could go up to 6 RC
>> before going final and it was very useful to spot the main bugs.
>>
> And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc
> display issue under Windows) were reported in the forum only 2 days and 5
> days after the release! and we have had many topics for both afterward. A RC
> would have clearly spared the hassle.
>
> See:
> http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082
> http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161
>


Maybe we need to call an earlier build the "RC" so it will get more
attention?   We had a complete test build that we were testing for
over a month.  But maybe it is ignored unless we call it an "RC"?  In
other words, there were many opportunities for users to help try 4.0
before it was released, but maybe there opportunities were not well
known.

-Rob

>
> Hagar
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: dev-h...@openoffice.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:

Am 14.08.13 20:21, schrieb Rob Weir:

On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp  wrote:

Dear Rob
The 4.0 release was too ambitious - we should advance in smaller steps.
Nothing compares to general public testing - betas and release
candidates should not be avoided.
TestLink cases should be less comprehesive (in terms of feature
coverage) and more stress testing oriented.

The number to consider here is how many defects were found and fixed
during the 4.0.0 testing, before the general public users had access?
I assume it was quite substantial. If so, the TestLink usage was
effective. In other words, we might have found fewer bugs without
using it.


In general, my feeling is that it's too early to do a retrospective and 
ask "what was good, what needs to be improved?" (to say it with SCRUM 
words ;-) ) Just after the first major release.


We should look on the BZ query you have mentioned and see if there is 
one or more hotspots that should be improved fast. That's it.



This is important to keep in mind: we want to prevent or find more
bugs, but we're not starting from zero. We're starting from a process
that does a lot of things right.

I like the idea of a public beta. But consider the numbers. The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users. Realistically, can we expect anywhere near that number in a
public beta? Or is it more likely that a beta program has 10,000
users or fewer? I don't know the answer here. But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.

The public beta is from my point of view realy important. Even you have
only 10'000 Downloads of a beta, you have normaly verry experianced
Users there, like power users from Companies. They provide realy valua
feedback. So from my point of view, this is one of the moast important
changes we have to do. For all Feature release a beta version. And don't
forget, people are realy happy to do beta tests. but many of them are
maybe not willing to follow a mailing list.


A public beta release is of course not the golden solution but could 
activate some power users that give us the feedback we want and need.


So, +1 for going this way.

After the 4.1 release is done we can see if this was much better - and 
ask ourself why? :-)


Marcus


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Le 14/08/2013 21:29, Hagar Delest a écrit :

Le 14/08/2013 21:01, Raphael Bircher a écrit :


I like the idea of a public beta.  But consider the numbers.  The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users.  Realistically, can we expect anywhere near that number in a
public beta?  Or is it more likely that a beta program has 10,000
users or fewer?  I don't know the answer here.   But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.

The public beta is from my point of view realy important. Even you have only 
10'000 Downloads of a beta, you have normaly verry experianced Users there, 
like power users from Companies. They provide realy valua feedback. So from my 
point of view, this is one of the moast important changes we have to do. For 
all Feature release a beta version. And don't forget, people are realy happy to 
do beta tests. but many of them are maybe not willing to follow a mailing list.


+1.
OOo used to have RC versions strongly advertised, it could go up to 6 RC before 
going final and it was very useful to spot the main bugs.


And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc 
display issue under Windows) were reported in the forum only 2 days and 5 days 
after the release! and we have had many topics for both afterward. A RC would 
have clearly spared the hassle.

See:
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161

Hagar

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality


Le 14/08/2013 21:01, Raphael Bircher a écrit :


I like the idea of a public beta.  But consider the numbers.  The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users.  Realistically, can we expect anywhere near that number in a
public beta?  Or is it more likely that a beta program has 10,000
users or fewer?  I don't know the answer here.   But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.

The public beta is from my point of view realy important. Even you have only 
10'000 Downloads of a beta, you have normaly verry experianced Users there, 
like power users from Companies. They provide realy valua feedback. So from my 
point of view, this is one of the moast important changes we have to do. For 
all Feature release a beta version. And don't forget, people are realy happy to 
do beta tests. but many of them are maybe not willing to follow a mailing list.


+1.
OOo used to have RC versions strongly advertised, it could go up to 6 RC before 
going final and it was very useful to spot the main bugs.

Hagar

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

2013-08-14 Thread Raphael Bircher


Am 14.08.13 20:21, schrieb Rob Weir:

On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp  wrote:

Dear Rob
The 4.0 release was too ambitious - we should advance in smaller steps.
Nothing compares to general public testing - betas and release candidates 
should not be avoided.
TestLink cases should be less comprehesive (in terms of feature coverage) and 
more stress testing oriented.

The number to consider here is how many defects were found and fixed
during the 4.0.0 testing, before the general public users had access?
I assume it was quite substantial.  If so, the TestLink usage was
effective.  In other words, we might have found fewer bugs without
using it.

This is important to keep in mind:  we want to prevent or find more
bugs, but we're not starting from zero. We're starting from a process
that does a lot of things right.

I like the idea of a public beta.  But consider the numbers.  The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users.  Realistically, can we expect anywhere near that number in a
public beta?  Or is it more likely that a beta program has 10,000
users or fewer?  I don't know the answer here.   But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.
The public beta is from my point of view realy important. Even you have 
only 10'000 Downloads of a beta, you have normaly verry experianced 
Users there, like power users from Companies. They provide realy valua 
feedback. So from my point of view, this is one of the moast important 
changes we have to do. For all Feature release a beta version. And don't 
forget, people are realy happy to do beta tests. but many of them are 
maybe not willing to follow a mailing list.


Greetings Raphael


-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

On Wed, Aug 14, 2013 at 1:55 PM, janI  wrote:
> On 14 August 2013 19:36, Edwin Sharp  wrote:
>
>> Dear Rob
>> The 4.0 release was too ambitious - we should advance in smaller steps.
>> Nothing compares to general public testing - betas and release candidates
>> should not be avoided.
>> TestLink cases should be less comprehesive (in terms of feature coverage)
>> and more stress testing oriented.
>> Regards,
>> Edwin
>>
>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> > We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The fact
>> > that we're doing this, and their are no arguments against it, shows
>> > that we value quality.   I'd like to take this a step further, and see
>> > what we can learn from the defects in AOO 4.0.0 and what we can do
>> > going forward to improve.
>> >
>> > Quality, in the end, is a process, not a state of grace.  We improve
>> > by working smarter, not working harder.  The goal should be to learn
>> > and improve, as individuals and as a community.
>> >
>> > Every regression that made it into 4.0.0 was added there by a
>> > programmer.  And the defect went undetected by testers.  This is not
>> > to blame.  It just means that we're all human.  We know that.  We all
>> > make mistakes.  I make mistakes.  A quality process is not about
>> > becoming perfect, but about acknowledging that we make mistakes and
>> > that certain formal and informal practices are needed to prevent and
>> > detect these mistakes.
>> >
>> > But enough about generalities.  I'm hoping you'll join with me in
>> > examining the 32 confirmed 4.0.0 regression defects and answering a
>> > few questions:
>> >
>> > 1) What caused the bug?   What was the "root cause"?  Note:
>> > "programmer error" is not really a cause.  We should ask what caused
>> > the error.
>> >
>> > 2) What can we do to prevent bugs like this from being checked in?
>> >
>> > 3) Why wasn't the bug found during testing?  Was it not covered by any
>> > existing test case?  Was a test case run but the defect was not
>> > recognized?  Was the defect introduced into the software after the
>> > tests had already been executed?
>> >
>> > 4) What can we do to ensure that bugs like this are caught during
>> testing?
>> >
>> > So 2 basic questions -- what went wrong and how can we prevent it in
>> > the future, looked at from perspective of programmers and testers.  If
>> > we can keep these questions in mind, and try to answer them, we may be
>> > able to find some patterns that can lead to some process changes for
>> > AOO 4.1.
>> >
>> > You can find the 4.0.0 regressions in Bugzilla here:
>> >
>> >
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>> >
>> >
>> > Regards,
>> >
>> > -Rob
>>
>
> I strongly believe that one of the things that went wrong is our limited
> possibility to retest (due to resources), when I look at our current manual

I wonder about that as well.  That's one reason it would be good to
know how many of the confirmed regressions were introduced late in the
release process, and thus missed coverage in our full test pass.

> testcases, a lot of those could be automated, e.g. with a simple UI macro,
> that would enable us to run these test cases with every build. It may sound
> like a dream but where I come from, we did that every night, and it caught
> a lot of regression bugs and sideeffects.
>

This begs the question:  Is the functionality of the regressions
covered by our test cases?  Or are they covered but we didn't execute
them?  Or we executed them but didn't recognize the defect?  I don't
know (yet).

> A simple start, if to request that every bug fix, is issued with at least
> one test case (automated or manual).
>

Often there is, though this information lives in Bugzilla.  One thing
we did on another (non open source) project is to mark defects in our
bugtracking system that should become test cases.   Not every bug did
that.  For example, a defect report to update a mispelling in the UI
would not lead to a new test case.  But many would.

Regards,

-Rob

> rgds
> jan I.
>
>
>> >
>> > -
>> > To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
>> > For additional commands, e-mail: qa-h...@openoffice.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
>> For additional commands, e-mail: qa-h...@openoffice.apache.org
>>
>>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp  wrote:
> Dear Rob
> The 4.0 release was too ambitious - we should advance in smaller steps.
> Nothing compares to general public testing - betas and release candidates 
> should not be avoided.
> TestLink cases should be less comprehesive (in terms of feature coverage) and 
> more stress testing oriented.

The number to consider here is how many defects were found and fixed
during the 4.0.0 testing, before the general public users had access?
I assume it was quite substantial.  If so, the TestLink usage was
effective.  In other words, we might have found fewer bugs without
using it.

This is important to keep in mind:  we want to prevent or find more
bugs, but we're not starting from zero. We're starting from a process
that does a lot of things right.

I like the idea of a public beta.  But consider the numbers.  The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users.  Realistically, can we expect anywhere near that number in a
public beta?  Or is it more likely that a beta program has 10,000
users or fewer?  I don't know the answer here.   But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.

Regards,

-Rob


> Regards,
> Edwin
>
> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The fact
>> that we're doing this, and their are no arguments against it, shows
>> that we value quality.   I'd like to take this a step further, and see
>> what we can learn from the defects in AOO 4.0.0 and what we can do
>> going forward to improve.
>>
>> Quality, in the end, is a process, not a state of grace.  We improve
>> by working smarter, not working harder.  The goal should be to learn
>> and improve, as individuals and as a community.
>>
>> Every regression that made it into 4.0.0 was added there by a
>> programmer.  And the defect went undetected by testers.  This is not
>> to blame.  It just means that we're all human.  We know that.  We all
>> make mistakes.  I make mistakes.  A quality process is not about
>> becoming perfect, but about acknowledging that we make mistakes and
>> that certain formal and informal practices are needed to prevent and
>> detect these mistakes.
>>
>> But enough about generalities.  I'm hoping you'll join with me in
>> examining the 32 confirmed 4.0.0 regression defects and answering a
>> few questions:
>>
>> 1) What caused the bug?   What was the "root cause"?  Note:
>> "programmer error" is not really a cause.  We should ask what caused
>> the error.
>>
>> 2) What can we do to prevent bugs like this from being checked in?
>>
>> 3) Why wasn't the bug found during testing?  Was it not covered by any
>> existing test case?  Was a test case run but the defect was not
>> recognized?  Was the defect introduced into the software after the
>> tests had already been executed?
>>
>> 4) What can we do to ensure that bugs like this are caught during testing?
>>
>> So 2 basic questions -- what went wrong and how can we prevent it in
>> the future, looked at from perspective of programmers and testers.  If
>> we can keep these questions in mind, and try to answer them, we may be
>> able to find some patterns that can lead to some process changes for
>> AOO 4.1.
>>
>> You can find the 4.0.0 regressions in Bugzilla here:
>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>
>>
>> Regards,
>>
>> -Rob
>>
>> -
>> To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
>> For additional commands, e-mail: qa-h...@openoffice.apache.org
>>
>
> -
> To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org
> For additional commands, e-mail: qa-h...@openoffice.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Re: Some thoughts on quality

I apologize in advance if my note was note clear.  I'm not at all
interested in off-the-cuff opinions.  We all have our opinions.  But
I'm only interested in fact-based analysis of the actual regressions
reported in BZ.   Specifically:  what caused the actually defects that
ended up in 4.0.0 and what could have been done to prevent it.
General recommendations, like "more time", not backed by specific
analysis, are not very useful.  And remember, there will never be
enough time to improve quality with a suboptimal process.  The goal
should be (IMHO) to improve the process, i.e., work smarter, not
harder.

Regards,

-Rob

On Wed, Aug 14, 2013 at 12:59 PM, Rob Weir  wrote:
> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0.  The fact
> that we're doing this, and their are no arguments against it, shows
> that we value quality.   I'd like to take this a step further, and see
> what we can learn from the defects in AOO 4.0.0 and what we can do
> going forward to improve.
>
> Quality, in the end, is a process, not a state of grace.  We improve
> by working smarter, not working harder.  The goal should be to learn
> and improve, as individuals and as a community.
>
> Every regression that made it into 4.0.0 was added there by a
> programmer.  And the defect went undetected by testers.  This is not
> to blame.  It just means that we're all human.  We know that.  We all
> make mistakes.  I make mistakes.  A quality process is not about
> becoming perfect, but about acknowledging that we make mistakes and
> that certain formal and informal practices are needed to prevent and
> detect these mistakes.
>
> But enough about generalities.  I'm hoping you'll join with me in
> examining the 32 confirmed 4.0.0 regression defects and answering a
> few questions:
>
> 1) What caused the bug?   What was the "root cause"?  Note:
> "programmer error" is not really a cause.  We should ask what caused
> the error.
>
> 2) What can we do to prevent bugs like this from being checked in?
>
> 3) Why wasn't the bug found during testing?  Was it not covered by any
> existing test case?  Was a test case run but the defect was not
> recognized?  Was the defect introduced into the software after the
> tests had already been executed?
>
> 4) What can we do to ensure that bugs like this are caught during testing?
>
> So 2 basic questions -- what went wrong and how can we prevent it in
> the future, looked at from perspective of programmers and testers.  If
> we can keep these questions in mind, and try to answer them, we may be
> able to find some patterns that can lead to some process changes for
> AOO 4.1.
>
> You can find the 4.0.0 regressions in Bugzilla here:
>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>
>
> Regards,
>
> -Rob

-
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org

Some thoughts on quality