Re: Some thoughts on quality
On 15/08/2013 Rob Weir wrote: A thought experiment: If we ran the existing test automation on 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think would be detected? I would expect this 4.0.1 blocker issue (wrong results in certain Calc functions) to be detectable by automated testing: https://issues.apache.org/ooo/show_bug.cgi?id=122997 Regards, Andrea. - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Wed, Aug 14, 2013 at 3:35 PM, sebb seb...@gmail.com wrote: On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote: On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. Right. A release is a release is a release. The basic requirements for every release still apply: 1) 3 PMC +1 votes 2) Must include source files 3) Digital signatures, hash files, etc. But we can have a beta release, that follows these rules, and it would be acceptable. We can then host on the mirrors, publicize, etc. There would likely be some restrictions on how many extra downloads are permitted. For example, the ASF mirrors probably could not cope with a set of betas of all the languages for all the OSes in addition to the current GA release. Looking again at this thread, and knowing the amount of time and resources it takes to actually mount an approved release, I wonder if some of the issues we had with the 4.0 release could have been addressed by a longer RC vote timeframe -- like 2 weeks or so. The vote this time was relatively short as I recall on both the original RC and RC2. We have over 400 people on this list. I would hope given a longer test period, we might receive better testing and feedback if we took a bit longer for approval. I realize this is not the same as a beta -- we would still be using development snapshots, but the RC could be provided in all languages we intend to use, etc. - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org -- - MzK When in doubt, cop an attitude. -- Cat laws
Re: Some thoughts on quality
On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk kay.sch...@gmail.com wrote: On Wed, Aug 14, 2013 at 3:35 PM, sebb seb...@gmail.com wrote: On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote: On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. Right. A release is a release is a release. The basic requirements for every release still apply: 1) 3 PMC +1 votes 2) Must include source files 3) Digital signatures, hash files, etc. But we can have a beta release, that follows these rules, and it would be acceptable. We can then host on the mirrors, publicize, etc. There would likely be some restrictions on how many extra downloads are permitted. For example, the ASF mirrors probably could not cope with a set of betas of all the languages for all the OSes in addition to the current GA release. Looking again at this thread, and knowing the amount of time and resources it takes to actually mount an approved release, I wonder if some of the issues we had with the 4.0 release could have been addressed by a longer RC vote timeframe -- like 2 weeks or so. The vote this time was relatively short as I recall on both the original RC and RC2. We have over 400 people on this list. I would hope given a longer test period, we might receive better testing and feedback if we took a bit longer for approval. You can answer that question yourself. Look at the bugs that we're fixing in 4.0.1. Which of those would have been found if the same project members spent two more weeks reviewing the same code? I doubt it would have made much of a difference. The slow Excel saving bug, for example, was in the code since November 2012. So this is not question of time. I'm hoping we can all look closely at the facts and come to a similar conclusion. The basic concept here is defect yield, which is a measure of what % of the defects in the code are found by a given testing technique. We can consider the formal QA test cases as one technique. We can consider the PMC voting period as another testing technique, although an informal one. We could consider beta testing, automated testing, etc., all as different techniques. Hopefully it is clear that the defect yield of a technique does not increase greatly by spending more time doing it. For example, suppose our test cases were capable of finding 25% of the bugs in the code. So we run the test cases once and find 23% of the bugs. We don't get 25% because of human error in testing. We could double the amount of time spent testing and run all the tests again. That might get us to 24%. Not a very efficient use of incremental time. Ditto for PMC voting. The informal testing is going to just poke at the surface. It will have some bug yield, less than the formal testing did. But we all have our usage patterns, the subset of features that we use. It is very likely that in two weeks we exercise only slightly more of the features than we did in a one week voting period. Diminishing returns. In any case, we need to escape the incorrect notion that a RC vote is an invitation to start testing. It
Re: Some thoughts on quality
On Fri, Aug 16, 2013 at 6:55 AM, Andrea Pescetti pesce...@apache.org wrote: On 15/08/2013 Rob Weir wrote: A thought experiment: If we ran the existing test automation on 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think would be detected? I would expect this 4.0.1 blocker issue (wrong results in certain Calc functions) to be detectable by automated testing: https://issues.apache.org/ooo/show_bug.cgi?id=122997 I don't believe this is actually covered in our test automation. So maybe the actual root causes is that there were no test cases for this function. If there were test cases then the defect could have been detected by manual or automated testing. I agree with you that spreadsheets functions lend themselves to automated testing. But the problem here was not lack of automation. It was lack of a test case, period. -Rob Regards, Andrea. - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Fri, Aug 16, 2013 at 1:06 PM, Kay Schenk kay.sch...@gmail.com wrote: On Fri, Aug 16, 2013 at 9:47 AM, Rob Weir robw...@apache.org wrote: On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk kay.sch...@gmail.com wrote: On Wed, Aug 14, 2013 at 3:35 PM, sebb seb...@gmail.com wrote: On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote: On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. Right. A release is a release is a release. The basic requirements for every release still apply: 1) 3 PMC +1 votes 2) Must include source files 3) Digital signatures, hash files, etc. But we can have a beta release, that follows these rules, and it would be acceptable. We can then host on the mirrors, publicize, etc. There would likely be some restrictions on how many extra downloads are permitted. For example, the ASF mirrors probably could not cope with a set of betas of all the languages for all the OSes in addition to the current GA release. Looking again at this thread, and knowing the amount of time and resources it takes to actually mount an approved release, I wonder if some of the issues we had with the 4.0 release could have been addressed by a longer RC vote timeframe -- like 2 weeks or so. The vote this time was relatively short as I recall on both the original RC and RC2. We have over 400 people on this list. I would hope given a longer test period, we might receive better testing and feedback if we took a bit longer for approval. You can answer that question yourself. Look at the bugs that we're fixing in 4.0.1. Which of those would have been found if the same project members spent two more weeks reviewing the same code? Well this was my point. Not the same members, but different members on this list who, due to the timeframe we had for voting on the RC, were not available for more extensive testing. It is the psychological difference between a developer snapshot and an RC candidate. But then you have the fact that one of the more serious bugs was in the code since November. So time alone is not the solution. It is how we use the time that counts. -Rob I doubt it would have made much of a difference. The slow Excel saving bug, for example, was in the code since November 2012. So this is not question of time. I'm hoping we can all look closely at the facts and come to a similar conclusion. The basic concept here is defect yield, which is a measure of what % of the defects in the code are found by a given testing technique. We can consider the formal QA test cases as one technique. We can consider the PMC voting period as another testing technique, although an informal one. We could consider beta testing, automated testing, etc., all as different techniques. Hopefully it is clear that the defect yield of a technique does not increase greatly by spending more time doing it. For example, suppose our test cases were capable of finding 25% of the bugs in the code. So we run the test cases once
Re: Some thoughts on quality
Hi, On 14.08.2013 18:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Please include also issue with status ACCEPTED. Best regards, Oliver. Regards, -Rob - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet Juergen Regards, -Rob rgds jan I. - To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org For additional commands, e-mail: qa-h...@openoffice.apache.org - To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org For additional commands, e-mail: qa-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet I aware that we ha a limited automated framework, at least thats what I found and played with. but, it is not integrated into our build, or our buildbot. Especially testing in buildbot gives better qa. An manually controlled automated test is not really an ideal solution. Take a look at the manual test cases, a lot could be automated, and free qa resources for more complex testing. rgds jan I Juergen Regards, -Rob rgds jan I.
Re: Some thoughts on quality
On 8/15/13 12:19 PM, janI wrote: On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet I aware that we ha a limited automated framework, at least thats what I found and played with. but, it is not integrated into our build, or our buildbot. Especially testing in buildbot gives better qa. An manually controlled automated test is not really an ideal solution. +1 and I think it was the intended idea behind this, have it run on a regular basis ideally on the build bots. The work is not finished and have to be done as so many open work items. If thet run more stable and the results are in good shape we can rpobably quite easy includ ethem in our
Re: Some thoughts on quality
On Thu, Aug 15, 2013 at 2:59 AM, Edwin Sharp el...@mail-page.com wrote: On Wed, Aug 14, 2013, at 22:58, Rob Weir wrote: Finally, I think we can all point to a similar open source project that has numerous betas, but still suffers from poor quality. So a public beta, by itself, is not sufficient. We need some upstream improvements as well, I think. But we should do a beta as well. But aim to have the highest quality beta we can, right? Regards, -Rob This comparison is not fair because the release frequency of the two projects is totally different. To the original question you had suggested more frequent releases. Raphael and Hagar suggested beta releases. So it is entirely relevant to point out that a sister project that had adopted both of those practices has not achieved any great quality improvements. When you think of it, why should release pacing have any impact on quality? Quality is increased if you prevent bugs, detect them before you release, or fix them before you release. Anything that claims to improve quality should have a direct impact on one of those three actions. Does a release prevent bugs? Detect them? Fix them? No, not really. What a release can do is prompt increase QA attention as the release approaches. But this is an indirect and not obligatory practice. We could have frequent releases that are tested less, because of the less time available for testing. This would lead to lower quality. Or we could have multiple full test passes with a longer release, taking better advantage of the time, and leading to higher quality. So the thing that matters here is the testing time, testing coverage, testing efficiency, etc. The release pace has no direct effect on quality. Quality can also be improved by better community culture. No offense, but I found this inappropriate: Culture is important, yes, to the extent it leads to the adoption of quality practices. One good cultural point would be a community that is not offended by facts and fact-based reasoning. Regards, -Rob On Fri, Jun 7, 2013, at 3:33, Rob Weir wrote: I'll try to clean up a few tonight while I watch TV. - To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org For additional commands, e-mail: qa-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/15/13 12:19 PM, janI wrote: On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet I aware that we ha a limited automated framework, at least thats what I found and played with. but, it is not integrated into our build, or our buildbot. Especially testing in buildbot gives better qa. An manually controlled automated test is not really an ideal solution. +1 and I think it was the intended idea behind this, have it run on a regular basis ideally on the build bots. The work is not finished and have to be done as so many open work items. If thet run more stable
Re: Some thoughts on quality
On Thu, Aug 15, 2013 at 4:08 AM, Oliver-Rainer Wittmann orwittm...@googlemail.com wrote: Hi, On 14.08.2013 18:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Please include also issue with status ACCEPTED. Done. That now gives us 36. -Rob Best regards, Oliver. Regards, -Rob - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Thu, Aug 15, 2013 at 8:29 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/15/13 1:33 PM, Rob Weir wrote: On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/15/13 12:19 PM, janI wrote: On Aug 15, 2013 11:14 AM, Jürgen Schmidt jogischm...@gmail.com wrote: On 8/14/13 8:30 PM, Rob Weir wrote: On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. we have the automated test framework that needs some more attention and polishing. And of course the tests have to improved to get satisfying result. We have BVT - build verification test FVT - functional verification test PVT - performance verification test SVT - system verification test But I have to confess that I have limited knowledge about it yet I aware that we ha a limited automated framework, at least thats what I found and played with. but, it is not integrated into our build, or our buildbot. Especially testing in buildbot gives better qa. An manually controlled automated test is not really an ideal solution. +1 and I think it was the intended idea behind this, have it run on a regular basis ideally on
Some thoughts on quality
We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
I apologize in advance if my note was note clear. I'm not at all interested in off-the-cuff opinions. We all have our opinions. But I'm only interested in fact-based analysis of the actual regressions reported in BZ. Specifically: what caused the actually defects that ended up in 4.0.0 and what could have been done to prevent it. General recommendations, like more time, not backed by specific analysis, are not very useful. And remember, there will never be enough time to improve quality with a suboptimal process. The goal should be (IMHO) to improve the process, i.e., work smarter, not harder. Regards, -Rob On Wed, Aug 14, 2013 at 12:59 PM, Rob Weir robw...@apache.org wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Wed, Aug 14, 2013 at 1:55 PM, janI j...@apache.org wrote: On 14 August 2013 19:36, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. Regards, Edwin On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote: We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact that we're doing this, and their are no arguments against it, shows that we value quality. I'd like to take this a step further, and see what we can learn from the defects in AOO 4.0.0 and what we can do going forward to improve. Quality, in the end, is a process, not a state of grace. We improve by working smarter, not working harder. The goal should be to learn and improve, as individuals and as a community. Every regression that made it into 4.0.0 was added there by a programmer. And the defect went undetected by testers. This is not to blame. It just means that we're all human. We know that. We all make mistakes. I make mistakes. A quality process is not about becoming perfect, but about acknowledging that we make mistakes and that certain formal and informal practices are needed to prevent and detect these mistakes. But enough about generalities. I'm hoping you'll join with me in examining the 32 confirmed 4.0.0 regression defects and answering a few questions: 1) What caused the bug? What was the root cause? Note: programmer error is not really a cause. We should ask what caused the error. 2) What can we do to prevent bugs like this from being checked in? 3) Why wasn't the bug found during testing? Was it not covered by any existing test case? Was a test case run but the defect was not recognized? Was the defect introduced into the software after the tests had already been executed? 4) What can we do to ensure that bugs like this are caught during testing? So 2 basic questions -- what went wrong and how can we prevent it in the future, looked at from perspective of programmers and testers. If we can keep these questions in mind, and try to answer them, we may be able to find some patterns that can lead to some process changes for AOO 4.1. You can find the 4.0.0 regressions in Bugzilla here: https://issues.apache.org/ooo/buglist.cgi?cmdtype=doremremaction=runnamedcmd=400_regressionssharer_id=248521list_id=80834 Regards, -Rob I strongly believe that one of the things that went wrong is our limited possibility to retest (due to resources), when I look at our current manual I wonder about that as well. That's one reason it would be good to know how many of the confirmed regressions were introduced late in the release process, and thus missed coverage in our full test pass. testcases, a lot of those could be automated, e.g. with a simple UI macro, that would enable us to run these test cases with every build. It may sound like a dream but where I come from, we did that every night, and it caught a lot of regression bugs and sideeffects. This begs the question: Is the functionality of the regressions covered by our test cases? Or are they covered but we didn't execute them? Or we executed them but didn't recognize the defect? I don't know (yet). A simple start, if to request that every bug fix, is issued with at least one test case (automated or manual). Often there is, though this information lives in Bugzilla. One thing we did on another (non open source) project is to mark defects in our bugtracking system that should become test cases. Not every bug did that. For example, a defect report to update a mispelling in the UI would not lead to a new test case. But many would. Regards, -Rob rgds jan I. - To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org For additional commands, e-mail: qa-h...@openoffice.apache.org - To unsubscribe, e-mail: qa-unsubscr...@openoffice.apache.org For additional commands, e-mail: qa-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 14.08.13 20:21, schrieb Rob Weir: On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. The number to consider here is how many defects were found and fixed during the 4.0.0 testing, before the general public users had access? I assume it was quite substantial. If so, the TestLink usage was effective. In other words, we might have found fewer bugs without using it. This is important to keep in mind: we want to prevent or find more bugs, but we're not starting from zero. We're starting from a process that does a lot of things right. I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. Greetings Raphael - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Le 14/08/2013 21:01, Raphael Bircher a écrit : I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. +1. OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs. Hagar - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Le 14/08/2013 21:29, Hagar Delest a écrit : Le 14/08/2013 21:01, Raphael Bircher a écrit : I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. +1. OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs. And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc display issue under Windows) were reported in the forum only 2 days and 5 days after the release! and we have had many topics for both afterward. A RC would have clearly spared the hassle. See: http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63082 http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63161 Hagar - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 08/14/2013 09:01 PM, schrieb Raphael Bircher: Am 14.08.13 20:21, schrieb Rob Weir: On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. The number to consider here is how many defects were found and fixed during the 4.0.0 testing, before the general public users had access? I assume it was quite substantial. If so, the TestLink usage was effective. In other words, we might have found fewer bugs without using it. In general, my feeling is that it's too early to do a retrospective and ask what was good, what needs to be improved? (to say it with SCRUM words ;-) ) Just after the first major release. We should look on the BZ query you have mentioned and see if there is one or more hotspots that should be improved fast. That's it. This is important to keep in mind: we want to prevent or find more bugs, but we're not starting from zero. We're starting from a process that does a lot of things right. I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. A public beta release is of course not the golden solution but could activate some power users that give us the feedback we want and need. So, +1 for going this way. After the 4.1 release is done we can see if this was much better - and ask ourself why? :-) Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delest hagar.del...@laposte.net wrote: Le 14/08/2013 21:29, Hagar Delest a écrit : Le 14/08/2013 21:01, Raphael Bircher a écrit : I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. +1. OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs. And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc display issue under Windows) were reported in the forum only 2 days and 5 days after the release! and we have had many topics for both afterward. A RC would have clearly spared the hassle. See: http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63082 http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63161 Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. -Rob Hagar - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Le 14/08/2013 21:37, Marcus (OOo) a écrit : In general, my feeling is that it's too early to do a retrospective and ask what was good, what needs to be improved? (to say it with SCRUM words ;-) ) Just after the first major release. Well, I visit forums and contribute since OOo 2.0 and it's the first time we (volunteers in forums) advise to downgrade so often, even for a major upgrade (because of the slow saving time and the Calc display issue mainly). So there has been clearly a quality issue. About test case, it's strange that the 2 bugs above were not seen. Hagar - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 08/14/2013 09:39 PM, schrieb Rob Weir: On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delesthagar.del...@laposte.net wrote: Le 14/08/2013 21:29, Hagar Delest a écrit : Le 14/08/2013 21:01, Raphael Bircher a écrit : I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. +1. OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs. And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc display issue under Windows) were reported in the forum only 2 days and 5 days after the release! and we have had many topics for both afterward. A RC would have clearly spared the hassle. See: http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63082 http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63161 Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. Déjà vu ;-) I remember that we had the same problem in the old project. Call it Developer Snapshot and you will have a few hundreads downloads with respective number of feedback. But call it (and announce it!) with a name that sounds more familar (like Early Access, Preview, Beta, RC) then we had much more. So, going public more early should bring us a higher number of feedback. Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 09:01 PM, schrieb Raphael Bircher: Am 14.08.13 20:21, schrieb Rob Weir: On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp el...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. The number to consider here is how many defects were found and fixed during the 4.0.0 testing, before the general public users had access? I assume it was quite substantial. If so, the TestLink usage was effective. In other words, we might have found fewer bugs without using it. In general, my feeling is that it's too early to do a retrospective and ask what was good, what needs to be improved? (to say it with SCRUM words ;-) ) Just after the first major release. Memories are still fresh and programmers are looking at the relevant code. This is the best time to answer these questions. We should look on the BZ query you have mentioned and see if there is one or more hotspots that should be improved fast. That's it. That would be fine. I'm not suggesting anything radical. Gradual, but constant improvements are the way to high quality. This is important to keep in mind: we want to prevent or find more bugs, but we're not starting from zero. We're starting from a process that does a lot of things right. I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. A public beta release is of course not the golden solution but could activate some power users that give us the feedback we want and need. So, +1 for going this way. After the 4.1 release is done we can see if this was much better - and ask ourself why? :-) But several of the bugs in 4.0.0 should never have made it to even a beta. For example, the very slow saving in Excel format. What happened there? Sure, this could be fixed later in the process, in a beta, or in a 4.0.1 as we're doing now. But this really should have been detected and fixed much earlier in the process. What are betas good for? Betas are good for expanding the set of configurations. Platforms, languages, extensions, etc.But the informal tests done by real users tend to be shallow. Also, we have no way of determining what the test coverage is. In particular we have no basis for telling the difference between the coverage of a 2 week beta versus a 4 week beta. But with out formal test cases we can easily look at how many test cases were executed. We could even look at code coverage if we wanted to. Maybe if there was a way to record what features beta testers were using... Or even have a little survey where they report what platform they ran on, etc. But very slow saving to XLS files? A defect like that should have been caught by us before a beta and before a RC. We shouldn't expect to find bugs like that in a beta. Finally, I think we can all point to a similar open source project that has numerous betas, but still suffers from poor quality. So a public beta, by itself, is not sufficient. We need some upstream improvements as well, I think. But we should do a beta as well. But aim to have the highest quality beta we can, right? Regards, -Rob Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 08/14/2013 09:50 PM, schrieb Hagar Delest: Le 14/08/2013 21:37, Marcus (OOo) a écrit : In general, my feeling is that it's too early to do a retrospective and ask what was good, what needs to be improved? (to say it with SCRUM words ;-) ) Just after the first major release. Well, I visit forums and contribute since OOo 2.0 and it's the first time we (volunteers in forums) advise to downgrade so often, even for a major upgrade (because of the slow saving time and the Calc display issue mainly). So there has been clearly a quality issue. Sure, I don't want to deny this. However, IMHO it's too early to look at the big picture and try to find the black spots. About test case, it's strange that the 2 bugs above were not seen. That seems indeen one hotspot I mentioned. Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design The wiki page for the dev builds were too complicated and sounded to much beta to make the users confident in using them (when they managed to get on that page!). Hagar - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 08/14/2013 09:58 PM, schrieb Rob Weir: On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo)marcus.m...@wtnet.de wrote: Am 08/14/2013 09:01 PM, schrieb Raphael Bircher: Am 14.08.13 20:21, schrieb Rob Weir: On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharpel...@mail-page.com wrote: Dear Rob The 4.0 release was too ambitious - we should advance in smaller steps. Nothing compares to general public testing - betas and release candidates should not be avoided. TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented. The number to consider here is how many defects were found and fixed during the 4.0.0 testing, before the general public users had access? I assume it was quite substantial. If so, the TestLink usage was effective. In other words, we might have found fewer bugs without using it. In general, my feeling is that it's too early to do a retrospective and ask what was good, what needs to be improved? (to say it with SCRUM words ;-) ) Just after the first major release. Memories are still fresh and programmers are looking at the relevant code. This is the best time to answer these questions. We should look on the BZ query you have mentioned and see if there is one or more hotspots that should be improved fast. That's it. That would be fine. I'm not suggesting anything radical. Gradual, but constant improvements are the way to high quality. Then this should be sufficiant IMHO. This is important to keep in mind: we want to prevent or find more bugs, but we're not starting from zero. We're starting from a process that does a lot of things right. I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. A public beta release is of course not the golden solution but could activate some power users that give us the feedback we want and need. So, +1 for going this way. After the 4.1 release is done we can see if this was much better - and ask ourself why? :-) But several of the bugs in 4.0.0 should never have made it to even a beta. For example, the very slow saving in Excel format. What happened there? Sure, this could be fixed later in the process, in a beta, or in a 4.0.1 as we're doing now. But this really should have been detected and fixed much earlier in the process. What are betas good for? Betas are good for expanding the set of configurations. Platforms, languages, extensions, etc.But the informal tests done by real users tend to be shallow. Also, we have no way of determining what the test coverage is. In particular we have no basis for telling the difference between the coverage of a 2 week beta versus a 4 week beta. But with out formal test cases we can easily look at how many test cases were executed. We could even look at code coverage if we wanted to. Maybe if there was a way to record what features beta testers were using... Or even have a little survey where they report what platform they ran on, etc. But very slow saving to XLS files? A defect like that should have been caught by us before a beta and before a RC. We shouldn't expect to find bugs like that in a beta. Of course. I've said nothing different. ;-) Finally, I think we can all point to a similar open source project that has numerous betas, but still suffers from poor quality. So a public beta, by itself, is not sufficient. We need some upstream improvements as well, I think. But we should do a beta as well. But aim to have the highest quality beta we can, right? Sure, the XLS bug has nothing to with this wouldn't happen with a Beta. As I answered to Hagar this is one hotspot that should be looked at closer. Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. The wiki page for the dev builds were too complicated and sounded to much beta to make the users confident in using them (when they managed to get on that page!). Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
Am 08/14/2013 10:24 PM, schrieb Marcus (OOo): Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. The wiki page for the dev builds were too complicated and sounded to much beta to make the users confident in using them (when they managed to get on that page!). Marcus Sorry, forgot the link: [1] http://en.wikipedia.org/wiki/Software_release_life_cycle#Release_candidate - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On 14 August 2013 20:39, Rob Weir robw...@apache.org wrote: Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. It basically wasn't publicised. I knew about it because I actually went looking for it (goal: to put an image on the Wikipedia article of the 4.0 sidebar). It took a bit of ferreting about to find the prereleases. I've followed the list all this year, so I knew (a) it existed (b) what to ask about; the percentage of AOO users on the dev list is all but invisible. Suggestion 1: note prereleases on the blog and in the social media channels. Suggestion 2: do an RC for 4.0.1 as well. Even if you don't think there's a need to, and fully expect the RC to be byte-for-byte identical to the final release, people will appreciate being asked. The Cathedral and the Bazaar still applies. Release early, release often. You could have had six months' intense testing from users who were seriously interested. - d.
Re: Some thoughts on quality
On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. Right. A release is a release is a release. The basic requirements for every release still apply: 1) 3 PMC +1 votes 2) Must include source files 3) Digital signatures, hash files, etc. But we can have a beta release, that follows these rules, and it would be acceptable. We can then host on the mirrors, publicize, etc. However, back to the original topic of this thread: We should look to see when the bugs we're fixing in 4.0.1 were added to the code. Not to blame or make anyone feel bad. But to understand. If these were late bugs then an earlier beta would not have found them. -Rob The wiki page for the dev builds were too complicated and sounded to much beta to make the users confident in using them (when they managed to get on that page!). Marcus - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On 14 August 2013 23:10, Rob Weir robw...@apache.org wrote: On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 10:05 PM, schrieb Hagar Delest: Le 14/08/2013 21:39, Rob Weir a écrit : Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list). This was intended. We hadn't published the dev builds via Apache or SF mirrors but only on 2 people accounts. Apache policy says it's not allowed to publish them for a wider audience to save the servers from a high traffic load. It's the job of the mirrors to handle this. What I see (from a standard user point of view) for a RC: - When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course) - Have a link visible under the main download button of the download Both can be done, depending where the install files are located. page (perhaps a similar button as a dedicated entry) - Make that RC installable in parallel with a stable version - No file association allowed for that RC by design IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC anymore. One of the RC attributes is to change it into the final release with, e.g., just a file name change. But this has to be done without any code changes. Otherwise you have to change code parts, build again, test again, ... ;-) But a Beta release could go this separated way. Right. A release is a release is a release. The basic requirements for every release still apply: 1) 3 PMC +1 votes 2) Must include source files 3) Digital signatures, hash files, etc. But we can have a beta release, that follows these rules, and it would be acceptable. We can then host on the mirrors, publicize, etc. There would likely be some restrictions on how many extra downloads are permitted. For example, the ASF mirrors probably could not cope with a set of betas of all the languages for all the OSes in addition to the current GA release. - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Some thoughts on quality
On Wed, Aug 14, 2013 at 12:51 PM, Marcus (OOo) marcus.m...@wtnet.de wrote: Am 08/14/2013 09:39 PM, schrieb Rob Weir: On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delesthagar.delest@laposte.**nethagar.del...@laposte.net wrote: Le 14/08/2013 21:29, Hagar Delest a écrit : Le 14/08/2013 21:01, Raphael Bircher a écrit : I like the idea of a public beta. But consider the numbers. The 40 or so regressions that were reported came from an install base (based on download figures since 4.0.0 was released) of around 3 million users. Realistically, can we expect anywhere near that number in a public beta? Or is it more likely that a beta program has 10,000 users or fewer? I don't know the answer here. But certainly a well-publicized and used beta will find more than a beta used by just a few hundred users. The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list. +1. OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs. And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc display issue under Windows) were reported in the forum only 2 days and 5 days after the release! and we have had many topics for both afterward. A RC would have clearly spared the hassle. See: http://forum.openoffice.org/**en/forum/viewtopic.php?f=9t=**63082http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63082 http://forum.openoffice.org/**en/forum/viewtopic.php?f=9t=**63161http://forum.openoffice.org/en/forum/viewtopic.php?f=9t=63161 Maybe we need to call an earlier build the RC so it will get more attention? We had a complete test build that we were testing for over a month. But maybe it is ignored unless we call it an RC? In other words, there were many opportunities for users to help try 4.0 before it was released, but maybe there opportunities were not well known. Déjà vu ;-) I remember that we had the same problem in the old project. Call it Developer Snapshot and you will have a few hundreads downloads with respective number of feedback. But call it (and announce it!) with a name that sounds more familar (like Early Access, Preview, Beta, RC) then we had much more. So, going public more early should bring us a higher number of feedback. Marcus If this would help, then we should do it. Hopefully this will get users' attention more. --**--**- To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.orgdev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org -- - MzK When in doubt, cop an attitude. -- Cat laws