Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, May 21, 2021 at 2:14 PM Thomas Huth wrote: > > On 21/05/2021 16.29, Peter Maydell wrote: > > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé > > wrote: > >> If you think these tests belong to tests/tcg/, I am OK to put > >> them they, but I don't think adding the Avocado buildsys > >> machinery to the already-complex tests/tcg/ Makefiles is going > >> to help us... > > > > This does raise the question of what we're actually trying > > to distinguish. It seems to me somewhat that what tests/acceptance/ > > actually contains that makes it interestingly different from other > > tests/ stuff is that it's specifically "tests using the Avocado > > framework". On that theory we might name it tests/avocado/. > > I think there are two aspects: > > 1) These tests are using the avocado framework > > 2) These tests are downloading other stuff from the internet (unlike the > other tests that we have) > After Peter's reply, I noticed QEMU does not organize tests under the tests folder by software engineering test category but by the mechanism/machinery the tests run on. This makes me think that we may need to handle the folders name and the CI jobs name differently: 1 - Change the current "test/acceptance" folder name to "test/(avocado or avoqado)." Change the "make check-acceptance" to "make check-validation," and the GitLab CI job names to "validation," meaning that, in a promising future, other tests running on a different framework and acting like validation tests would run in the same make command and same GitLab CI job. 2 - Change the current "test/acceptance" folder name to "test/(avocado or avoqado)." Change the "make check-acceptance" to "make check-(avocado or avoqaco)" and the GitLab CI job names to "(avocado or avoqado)," meaning that, in a promising future, we can categorize validation jobs inside the CI and run each of the different validation tests supported by a framework on its own GitLab CI job. Personally, I prefer option 2 as it gives more flexibility to decide how to set a GitLab CI job or run it when testing locally. > > Or we could just leave it as it is -- is the current naming > > actually confusing anybody? :-) > > Yes, I think "acceptance" is rather confusing. So far they haven't been part > of your PR acceptance tests (well, now they are part of the gitlab-CI, > though), and it's also not about tests that have been set up by customers, > which is what you normally think of when hearing "acceptance tests". So a > different name would be adequate. > > I think I'd vote for either "avocado", "avoqado" or "validation". > Even laughing every time I read "avoqado" (and thanks for that), I liked the idea as there is supplementary code added inside "tests/acceptance/avocado_qemu" to support the tests, meaning they are not "pure" avocado. > Thomas >
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 5/21/21 7:14 PM, Thomas Huth wrote: > On 21/05/2021 16.29, Peter Maydell wrote: >> On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé >> wrote: >>> If you think these tests belong to tests/tcg/, I am OK to put >>> them they, but I don't think adding the Avocado buildsys >>> machinery to the already-complex tests/tcg/ Makefiles is going >>> to help us... >> >> This does raise the question of what we're actually trying >> to distinguish. It seems to me somewhat that what tests/acceptance/ >> actually contains that makes it interestingly different from other >> tests/ stuff is that it's specifically "tests using the Avocado >> framework". On that theory we might name it tests/avocado/. > > I think there are two aspects: > > 1) These tests are using the avocado framework > > 2) These tests are downloading other stuff from the internet (unlike the > other tests that we have) > >> Or we could just leave it as it is -- is the current naming >> actually confusing anybody? :-) > > Yes, I think "acceptance" is rather confusing. So far they haven't been > part of your PR acceptance tests (well, now they are part of the > gitlab-CI, though), and it's also not about tests that have been set up > by customers, which is what you normally think of when hearing > "acceptance tests". So a different name would be adequate. IIUC the current "acceptance tests" are the ones Peter runs, which are *gating* the merge process. They can not fail. The current tests in tests/acceptance/ use Avocado (as said Thomas, to easily download artifacts) and shouldn't be considered gating; they could fail. This is my confusion so far. It should be OK to add tests using the Avocado framework which might fail and aren't gating.
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 21/05/2021 16.29, Peter Maydell wrote: On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé wrote: If you think these tests belong to tests/tcg/, I am OK to put them they, but I don't think adding the Avocado buildsys machinery to the already-complex tests/tcg/ Makefiles is going to help us... This does raise the question of what we're actually trying to distinguish. It seems to me somewhat that what tests/acceptance/ actually contains that makes it interestingly different from other tests/ stuff is that it's specifically "tests using the Avocado framework". On that theory we might name it tests/avocado/. I think there are two aspects: 1) These tests are using the avocado framework 2) These tests are downloading other stuff from the internet (unlike the other tests that we have) Or we could just leave it as it is -- is the current naming actually confusing anybody? :-) Yes, I think "acceptance" is rather confusing. So far they haven't been part of your PR acceptance tests (well, now they are part of the gitlab-CI, though), and it's also not about tests that have been set up by customers, which is what you normally think of when hearing "acceptance tests". So a different name would be adequate. I think I'd vote for either "avocado", "avoqado" or "validation". Thomas
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, May 21, 2021 at 12:23 PM Peter Maydell wrote: > > On Fri, 21 May 2021 at 16:13, Willian Rampazzo wrote: > > On Fri, May 21, 2021 at 11:29 AM Peter Maydell > > wrote: > > > This does raise the question of what we're actually trying > > > to distinguish. It seems to me somewhat that what tests/acceptance/ > > > actually contains that makes it interestingly different from other > > > tests/ stuff is that it's specifically "tests using the Avocado > > > framework". On that theory we might name it tests/avocado/. > > > > I think the updated README.rst from this RFC, inside the system > > (originally acceptance) folder, is a good description of what these > > tests should be: "This directory contains system tests. They're > > usually higher level, and may interact with external resources and > > with various guest operating systems." I can improve it, if needed. > > > > We are using Avocado Framework as a tool to accomplish the above > > description, but I don't think we should strictly use it if there is > > another way to accomplish what those tests are supposed to be. Calling > > them "avocado" tests may restrict the intent of them, in my opinion. > > But the main reason IMHO that we have them in a separate directory is > because that's where we have all the avocado machinery. I think the > sharing of the machinery is what dictates whether a test winds up in > tests/acceptance, tests/qtest, tests/tcg or tests/qemu-iotests > much more than whether it is "usually higher level" or more of a > unit test or whatever. If we ever added some other test framework for > doing system tests, we'd probably want to put it in its own > directory rather than lumping all its support machinery and > build files in together with the avocado based tests. Okay, I understand your point. With the organization of tests under the test folder that QEMU has today, I agree with you. > > thanks > -- PMM >
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, 21 May 2021 at 16:13, Willian Rampazzo wrote: > On Fri, May 21, 2021 at 11:29 AM Peter Maydell > wrote: > > This does raise the question of what we're actually trying > > to distinguish. It seems to me somewhat that what tests/acceptance/ > > actually contains that makes it interestingly different from other > > tests/ stuff is that it's specifically "tests using the Avocado > > framework". On that theory we might name it tests/avocado/. > > I think the updated README.rst from this RFC, inside the system > (originally acceptance) folder, is a good description of what these > tests should be: "This directory contains system tests. They're > usually higher level, and may interact with external resources and > with various guest operating systems." I can improve it, if needed. > > We are using Avocado Framework as a tool to accomplish the above > description, but I don't think we should strictly use it if there is > another way to accomplish what those tests are supposed to be. Calling > them "avocado" tests may restrict the intent of them, in my opinion. But the main reason IMHO that we have them in a separate directory is because that's where we have all the avocado machinery. I think the sharing of the machinery is what dictates whether a test winds up in tests/acceptance, tests/qtest, tests/tcg or tests/qemu-iotests much more than whether it is "usually higher level" or more of a unit test or whatever. If we ever added some other test framework for doing system tests, we'd probably want to put it in its own directory rather than lumping all its support machinery and build files in together with the avocado based tests. thanks -- PMM
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, May 21, 2021 at 11:29 AM Peter Maydell wrote: > > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé wrote: > > If you think these tests belong to tests/tcg/, I am OK to put > > them they, but I don't think adding the Avocado buildsys > > machinery to the already-complex tests/tcg/ Makefiles is going > > to help us... > > This does raise the question of what we're actually trying > to distinguish. It seems to me somewhat that what tests/acceptance/ > actually contains that makes it interestingly different from other > tests/ stuff is that it's specifically "tests using the Avocado > framework". On that theory we might name it tests/avocado/. I think the updated README.rst from this RFC, inside the system (originally acceptance) folder, is a good description of what these tests should be: "This directory contains system tests. They're usually higher level, and may interact with external resources and with various guest operating systems." I can improve it, if needed. We are using Avocado Framework as a tool to accomplish the above description, but I don't think we should strictly use it if there is another way to accomplish what those tests are supposed to be. Calling them "avocado" tests may restrict the intent of them, in my opinion. > > Or we could just leave it as it is -- is the current naming > actually confusing anybody? :-) > > thanks > -- PMM >
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, May 21, 2021 at 11:29 AM Peter Maydell wrote: > > On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé wrote: > > If you think these tests belong to tests/tcg/, I am OK to put > > them they, but I don't think adding the Avocado buildsys > > machinery to the already-complex tests/tcg/ Makefiles is going > > to help us... > > This does raise the question of what we're actually trying > to distinguish. It seems to me somewhat that what tests/acceptance/ > actually contains that makes it interestingly different from other > tests/ stuff is that it's specifically "tests using the Avocado > framework". On that theory we might name it tests/avocado/. > > Or we could just leave it as it is -- is the current naming > actually confusing anybody? :-) Conceptually speaking, it isn't very clear because acceptance tests are meant to test features requested by the customer during the requisite phase for a specific software release. The QEMU project does not have a formal requisite phase where features added to one software version are discussed with the customers. In this case, the QEMU acceptance tests are not accepting a use case or requested feature of a release. Eventually, those tests may not be a blocker to a release if some of them fail. Thus, calling them acceptance, may confuse people working with tests. > > thanks > -- PMM >
Re: [RFC 1/1] acceptance tests: rename acceptance to system
Philippe Mathieu-Daudé writes: > On 5/21/21 3:03 PM, Alex Bennée wrote: >> Philippe Mathieu-Daudé writes: >>> On 5/21/21 2:28 PM, Willian Rampazzo wrote: On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: > > On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote: >> On 5/20/21 9:53 PM, Willian Rampazzo wrote: >>> Conceptually speaking, acceptance tests "are a series of specific tests >>> conducted by the customer in an attempt to uncover product errors before >>> accepting the software from the developer. Conducted by the end-user >>> rather >>> than software engineers, acceptance testing can range from an informal >>> “test drive” to a planned and systematically executed series of scripted >>> tests" [1]. Every time Pressman refers to the term "acceptance >>> testing," he >>> also refers to user's agreement in the final state of an implemented >>> feature. >>> Today, QEMU is not implementing user acceptance tests as described by >>> Pressman. >>> >>> There are other three possible terms we could use to describe what is >>> currently >>> QEMU "acceptance" tests: >>> >>>1 - Integration tests: >>>- "Integration testing is a systematic technique for >>> constructing the >>> software architecture while at the same time conducting tests >>> to >>> uncover errors associated with interfacing. The objective is >>> to take >>> unit-tested components and build a program structure that has >>> been >>> dictated by design." [2] >>>* Note: Sommerville does not have a clear definition of >>> integration >>> testing. He refers to incremental integration of components >>> inside >>> the system testing (see [3]). > > After thinking about this for a while, I agree with you that renaming the > "acceptance" tests to "integration" tests is also not a good idea. When I > hear "integration" test in the context of the virt stack, I'd rather > expect > a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe > virt-manager on top and tests them all together. So we should look for a > different name indeed. > >>>2 - Validation tests: >>>- "Validation testing begins at the culmination of integration >>> testing, >>> when individual components have been exercised, the software >>> is >>> completely assembled as a package, and interfacing errors >>> have been >>> uncovered and corrected. At the validation or system level, >>> the >>> distinction between different software categories disappears. >>> Testing >>> focuses on user-visible actions and user-recognizable output >>> from the >>> system." [4] >>>- "where you expect the system to perform correctly using a set >>> of test >>> cases that reflect the system’s expected use." [5] >>>* Note: the definition of "validation testing" from Sommerville >>> reflects >>> the same definition found around the Internet, as one of the >>> processes >>> inside the "Verification & Validation (V)." In this concept, >>> validation testing is a high-level definition that covers unit >>> testing, >>> functional testing, integration testing, system testing, and >>> acceptance >>> testing. >>> >>>3 - System tests: >>>- "verifies that all elements mesh properly and that overall >>> system >>> function and performance is achieved." [6] >>>- "involves integrating components to create a version of the >>> system and >>> then testing the integrated system. System testing checks that >>> components are compatible, interact correctly, and transfer >>> the right >>> data at the right time across their interfaces." [7] >>> >>> The tests implemented inside the QEMU "acceptance" directory depend on >>> the >>> software completely assembled and, sometimes, on other elements, like >>> operating >>> system images. In this case, the proposal here is to rename the current >>> "acceptance" directory to "system." >> >> Are user-mode tests using Avocado also system tests? >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html > > We've indeed got the problem that the word "system" is a little bit > overloaded in the context of QEMU. We often talk about "system" when > referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user > emulator binaries). For example, the "--disable-system" switch of the > configure script, or the "build-system" and "check-system" jobs in the >
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, 21 May 2021 at 15:19, Philippe Mathieu-Daudé wrote: > If you think these tests belong to tests/tcg/, I am OK to put > them they, but I don't think adding the Avocado buildsys > machinery to the already-complex tests/tcg/ Makefiles is going > to help us... This does raise the question of what we're actually trying to distinguish. It seems to me somewhat that what tests/acceptance/ actually contains that makes it interestingly different from other tests/ stuff is that it's specifically "tests using the Avocado framework". On that theory we might name it tests/avocado/. Or we could just leave it as it is -- is the current naming actually confusing anybody? :-) thanks -- PMM
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 5/21/21 3:03 PM, Alex Bennée wrote: > Philippe Mathieu-Daudé writes: >> On 5/21/21 2:28 PM, Willian Rampazzo wrote: >>> On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote: > On 5/20/21 9:53 PM, Willian Rampazzo wrote: >> Conceptually speaking, acceptance tests "are a series of specific tests >> conducted by the customer in an attempt to uncover product errors before >> accepting the software from the developer. Conducted by the end-user >> rather >> than software engineers, acceptance testing can range from an informal >> “test drive” to a planned and systematically executed series of scripted >> tests" [1]. Every time Pressman refers to the term "acceptance testing," >> he >> also refers to user's agreement in the final state of an implemented >> feature. >> Today, QEMU is not implementing user acceptance tests as described by >> Pressman. >> >> There are other three possible terms we could use to describe what is >> currently >> QEMU "acceptance" tests: >> >>1 - Integration tests: >>- "Integration testing is a systematic technique for constructing >> the >> software architecture while at the same time conducting tests >> to >> uncover errors associated with interfacing. The objective is >> to take >> unit-tested components and build a program structure that has >> been >> dictated by design." [2] >>* Note: Sommerville does not have a clear definition of >> integration >> testing. He refers to incremental integration of components >> inside >> the system testing (see [3]). After thinking about this for a while, I agree with you that renaming the "acceptance" tests to "integration" tests is also not a good idea. When I hear "integration" test in the context of the virt stack, I'd rather expect a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe virt-manager on top and tests them all together. So we should look for a different name indeed. >>2 - Validation tests: >>- "Validation testing begins at the culmination of integration >> testing, >> when individual components have been exercised, the software is >> completely assembled as a package, and interfacing errors have >> been >> uncovered and corrected. At the validation or system level, the >> distinction between different software categories disappears. >> Testing >> focuses on user-visible actions and user-recognizable output >> from the >> system." [4] >>- "where you expect the system to perform correctly using a set >> of test >> cases that reflect the system’s expected use." [5] >>* Note: the definition of "validation testing" from Sommerville >> reflects >> the same definition found around the Internet, as one of the >> processes >> inside the "Verification & Validation (V)." In this concept, >> validation testing is a high-level definition that covers unit >> testing, >> functional testing, integration testing, system testing, and >> acceptance >> testing. >> >>3 - System tests: >>- "verifies that all elements mesh properly and that overall >> system >> function and performance is achieved." [6] >>- "involves integrating components to create a version of the >> system and >> then testing the integrated system. System testing checks that >> components are compatible, interact correctly, and transfer >> the right >> data at the right time across their interfaces." [7] >> >> The tests implemented inside the QEMU "acceptance" directory depend on >> the >> software completely assembled and, sometimes, on other elements, like >> operating >> system images. In this case, the proposal here is to rename the current >> "acceptance" directory to "system." > > Are user-mode tests using Avocado also system tests? > https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html We've indeed got the problem that the word "system" is a little bit overloaded in the context of QEMU. We often talk about "system" when referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user emulator binaries). For example, the "--disable-system" switch of the configure script, or the "build-system" and "check-system" jobs in the .gitlab-ci.yml file ... thus this could get quite confusing in the .gitlab-ci.yml file afterwards. >>> >>> I agree with you here. After I made the changes to the code,
Re: [RFC 1/1] acceptance tests: rename acceptance to system
Thomas Huth writes: > On 21/05/2021 14.28, Willian Rampazzo wrote: >> On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: > [...] >>> Alternatively, what about renaming the "acceptance" tests to "validation" >>> instead? That word does not have a duplicated definition in the context of >>> QEMU yet, so I think it would be less confusing. >> While at the beginning of your reply, I started thinking if >> "validation" would cause less confusion for the QEMU project. Although >> validation testing is a broader concept inside the Verification & >> Validation process, encompassing unit testing, functional testing, >> integration testing, system testing, and acceptance testing, it may be >> an option for the QEMU project. >> While system testing would be the correct terminology to use, if it >> causes more confusion, using a less strict terminology, like >> validation testing, is valid, in my opinion. > > > Or we could come up with a new, artificial name instead (like > "qtests"). It certainly need a "q" in the name. For example what > about: > > - avoqado > - valiqation > - aqueptance > - inteqration make check-quixotic? > > ? > > > Thomas -- Alex Bennée
Re: [RFC 1/1] acceptance tests: rename acceptance to system
Philippe Mathieu-Daudé writes: > On 5/21/21 2:28 PM, Willian Rampazzo wrote: >> On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: >>> >>> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote: On 5/20/21 9:53 PM, Willian Rampazzo wrote: > Conceptually speaking, acceptance tests "are a series of specific tests > conducted by the customer in an attempt to uncover product errors before > accepting the software from the developer. Conducted by the end-user > rather > than software engineers, acceptance testing can range from an informal > “test drive” to a planned and systematically executed series of scripted > tests" [1]. Every time Pressman refers to the term "acceptance testing," > he > also refers to user's agreement in the final state of an implemented > feature. > Today, QEMU is not implementing user acceptance tests as described by > Pressman. > > There are other three possible terms we could use to describe what is > currently > QEMU "acceptance" tests: > >1 - Integration tests: >- "Integration testing is a systematic technique for constructing > the > software architecture while at the same time conducting tests to > uncover errors associated with interfacing. The objective is to > take > unit-tested components and build a program structure that has > been > dictated by design." [2] >* Note: Sommerville does not have a clear definition of integration > testing. He refers to incremental integration of components > inside > the system testing (see [3]). >>> >>> After thinking about this for a while, I agree with you that renaming the >>> "acceptance" tests to "integration" tests is also not a good idea. When I >>> hear "integration" test in the context of the virt stack, I'd rather expect >>> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe >>> virt-manager on top and tests them all together. So we should look for a >>> different name indeed. >>> >2 - Validation tests: >- "Validation testing begins at the culmination of integration > testing, > when individual components have been exercised, the software is > completely assembled as a package, and interfacing errors have > been > uncovered and corrected. At the validation or system level, the > distinction between different software categories disappears. > Testing > focuses on user-visible actions and user-recognizable output > from the > system." [4] >- "where you expect the system to perform correctly using a set of > test > cases that reflect the system’s expected use." [5] >* Note: the definition of "validation testing" from Sommerville > reflects > the same definition found around the Internet, as one of the > processes > inside the "Verification & Validation (V)." In this concept, > validation testing is a high-level definition that covers unit > testing, > functional testing, integration testing, system testing, and > acceptance > testing. > >3 - System tests: >- "verifies that all elements mesh properly and that overall system > function and performance is achieved." [6] >- "involves integrating components to create a version of the > system and > then testing the integrated system. System testing checks that > components are compatible, interact correctly, and transfer the > right > data at the right time across their interfaces." [7] > > The tests implemented inside the QEMU "acceptance" directory depend on the > software completely assembled and, sometimes, on other elements, like > operating > system images. In this case, the proposal here is to rename the current > "acceptance" directory to "system." Are user-mode tests using Avocado also system tests? https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html >>> >>> We've indeed got the problem that the word "system" is a little bit >>> overloaded in the context of QEMU. We often talk about "system" when >>> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user >>> emulator binaries). For example, the "--disable-system" switch of the >>> configure script, or the "build-system" and "check-system" jobs in the >>> .gitlab-ci.yml file ... thus this could get quite confusing in the >>> .gitlab-ci.yml file afterwards. >> >> I agree with you here. After I made the changes to the code, I noticed >> QEMU has the "system" word spread all over the place. That may confuse >> people looking at the "system tests" without much interaction with >> software
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 5/21/21 2:42 PM, Thomas Huth wrote: > On 21/05/2021 14.28, Willian Rampazzo wrote: >> On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: > [...] >>> Alternatively, what about renaming the "acceptance" tests to >>> "validation" >>> instead? That word does not have a duplicated definition in the >>> context of >>> QEMU yet, so I think it would be less confusing. >> >> While at the beginning of your reply, I started thinking if >> "validation" would cause less confusion for the QEMU project. Although >> validation testing is a broader concept inside the Verification & >> Validation process, encompassing unit testing, functional testing, >> integration testing, system testing, and acceptance testing, it may be >> an option for the QEMU project. >> >> While system testing would be the correct terminology to use, if it >> causes more confusion, using a less strict terminology, like >> validation testing, is valid, in my opinion. > > > Or we could come up with a new, artificial name instead (like "qtests"). > It certainly need a "q" in the name. For example what about: > > - avoqado +1 for 'make check-avoqado' :)
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 21/05/2021 14.28, Willian Rampazzo wrote: On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: [...] Alternatively, what about renaming the "acceptance" tests to "validation" instead? That word does not have a duplicated definition in the context of QEMU yet, so I think it would be less confusing. While at the beginning of your reply, I started thinking if "validation" would cause less confusion for the QEMU project. Although validation testing is a broader concept inside the Verification & Validation process, encompassing unit testing, functional testing, integration testing, system testing, and acceptance testing, it may be an option for the QEMU project. While system testing would be the correct terminology to use, if it causes more confusion, using a less strict terminology, like validation testing, is valid, in my opinion. Or we could come up with a new, artificial name instead (like "qtests"). It certainly need a "q" in the name. For example what about: - avoqado - valiqation - aqueptance - inteqration ? Thomas
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 5/21/21 2:28 PM, Willian Rampazzo wrote: > On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: >> >> On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote: >>> On 5/20/21 9:53 PM, Willian Rampazzo wrote: Conceptually speaking, acceptance tests "are a series of specific tests conducted by the customer in an attempt to uncover product errors before accepting the software from the developer. Conducted by the end-user rather than software engineers, acceptance testing can range from an informal “test drive” to a planned and systematically executed series of scripted tests" [1]. Every time Pressman refers to the term "acceptance testing," he also refers to user's agreement in the final state of an implemented feature. Today, QEMU is not implementing user acceptance tests as described by Pressman. There are other three possible terms we could use to describe what is currently QEMU "acceptance" tests: 1 - Integration tests: - "Integration testing is a systematic technique for constructing the software architecture while at the same time conducting tests to uncover errors associated with interfacing. The objective is to take unit-tested components and build a program structure that has been dictated by design." [2] * Note: Sommerville does not have a clear definition of integration testing. He refers to incremental integration of components inside the system testing (see [3]). >> >> After thinking about this for a while, I agree with you that renaming the >> "acceptance" tests to "integration" tests is also not a good idea. When I >> hear "integration" test in the context of the virt stack, I'd rather expect >> a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe >> virt-manager on top and tests them all together. So we should look for a >> different name indeed. >> 2 - Validation tests: - "Validation testing begins at the culmination of integration testing, when individual components have been exercised, the software is completely assembled as a package, and interfacing errors have been uncovered and corrected. At the validation or system level, the distinction between different software categories disappears. Testing focuses on user-visible actions and user-recognizable output from the system." [4] - "where you expect the system to perform correctly using a set of test cases that reflect the system’s expected use." [5] * Note: the definition of "validation testing" from Sommerville reflects the same definition found around the Internet, as one of the processes inside the "Verification & Validation (V)." In this concept, validation testing is a high-level definition that covers unit testing, functional testing, integration testing, system testing, and acceptance testing. 3 - System tests: - "verifies that all elements mesh properly and that overall system function and performance is achieved." [6] - "involves integrating components to create a version of the system and then testing the integrated system. System testing checks that components are compatible, interact correctly, and transfer the right data at the right time across their interfaces." [7] The tests implemented inside the QEMU "acceptance" directory depend on the software completely assembled and, sometimes, on other elements, like operating system images. In this case, the proposal here is to rename the current "acceptance" directory to "system." >>> >>> Are user-mode tests using Avocado also system tests? >>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html >> >> We've indeed got the problem that the word "system" is a little bit >> overloaded in the context of QEMU. We often talk about "system" when >> referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user >> emulator binaries). For example, the "--disable-system" switch of the >> configure script, or the "build-system" and "check-system" jobs in the >> .gitlab-ci.yml file ... thus this could get quite confusing in the >> .gitlab-ci.yml file afterwards. > > I agree with you here. After I made the changes to the code, I noticed > QEMU has the "system" word spread all over the place. That may confuse > people looking at the "system tests" without much interaction with > software testing terminology. > >> >> So I think renaming "acceptance" to "system" is especially ok if we only >> keep the "softmmu"-related tests in that
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Fri, May 21, 2021 at 4:16 AM Thomas Huth wrote: > > On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote: > > On 5/20/21 9:53 PM, Willian Rampazzo wrote: > >> Conceptually speaking, acceptance tests "are a series of specific tests > >> conducted by the customer in an attempt to uncover product errors before > >> accepting the software from the developer. Conducted by the end-user rather > >> than software engineers, acceptance testing can range from an informal > >> “test drive” to a planned and systematically executed series of scripted > >> tests" [1]. Every time Pressman refers to the term "acceptance testing," he > >> also refers to user's agreement in the final state of an implemented > >> feature. > >> Today, QEMU is not implementing user acceptance tests as described by > >> Pressman. > >> > >> There are other three possible terms we could use to describe what is > >> currently > >> QEMU "acceptance" tests: > >> > >>1 - Integration tests: > >>- "Integration testing is a systematic technique for constructing > >> the > >> software architecture while at the same time conducting tests to > >> uncover errors associated with interfacing. The objective is to > >> take > >> unit-tested components and build a program structure that has > >> been > >> dictated by design." [2] > >>* Note: Sommerville does not have a clear definition of integration > >> testing. He refers to incremental integration of components inside > >> the system testing (see [3]). > > After thinking about this for a while, I agree with you that renaming the > "acceptance" tests to "integration" tests is also not a good idea. When I > hear "integration" test in the context of the virt stack, I'd rather expect > a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe > virt-manager on top and tests them all together. So we should look for a > different name indeed. > > >>2 - Validation tests: > >>- "Validation testing begins at the culmination of integration > >> testing, > >> when individual components have been exercised, the software is > >> completely assembled as a package, and interfacing errors have > >> been > >> uncovered and corrected. At the validation or system level, the > >> distinction between different software categories disappears. > >> Testing > >> focuses on user-visible actions and user-recognizable output > >> from the > >> system." [4] > >>- "where you expect the system to perform correctly using a set of > >> test > >> cases that reflect the system’s expected use." [5] > >>* Note: the definition of "validation testing" from Sommerville > >> reflects > >> the same definition found around the Internet, as one of the > >> processes > >> inside the "Verification & Validation (V)." In this concept, > >> validation testing is a high-level definition that covers unit > >> testing, > >> functional testing, integration testing, system testing, and > >> acceptance > >> testing. > >> > >>3 - System tests: > >>- "verifies that all elements mesh properly and that overall system > >> function and performance is achieved." [6] > >>- "involves integrating components to create a version of the > >> system and > >> then testing the integrated system. System testing checks that > >> components are compatible, interact correctly, and transfer the > >> right > >> data at the right time across their interfaces." [7] > >> > >> The tests implemented inside the QEMU "acceptance" directory depend on the > >> software completely assembled and, sometimes, on other elements, like > >> operating > >> system images. In this case, the proposal here is to rename the current > >> "acceptance" directory to "system." > > > > Are user-mode tests using Avocado also system tests? > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html > > We've indeed got the problem that the word "system" is a little bit > overloaded in the context of QEMU. We often talk about "system" when > referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user > emulator binaries). For example, the "--disable-system" switch of the > configure script, or the "build-system" and "check-system" jobs in the > .gitlab-ci.yml file ... thus this could get quite confusing in the > .gitlab-ci.yml file afterwards. I agree with you here. After I made the changes to the code, I noticed QEMU has the "system" word spread all over the place. That may confuse people looking at the "system tests" without much interaction with software testing terminology. > > So I think renaming "acceptance" to "system" is especially ok if we only > keep the "softmmu"-related tests in that folder... would it maybe make sense > to add the linux-user related tests in a
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On Thu, May 20, 2021 at 5:28 PM Philippe Mathieu-Daudé wrote: > > On 5/20/21 9:53 PM, Willian Rampazzo wrote: > > Conceptually speaking, acceptance tests "are a series of specific tests > > conducted by the customer in an attempt to uncover product errors before > > accepting the software from the developer. Conducted by the end-user rather > > than software engineers, acceptance testing can range from an informal > > “test drive” to a planned and systematically executed series of scripted > > tests" [1]. Every time Pressman refers to the term "acceptance testing," he > > also refers to user's agreement in the final state of an implemented > > feature. > > Today, QEMU is not implementing user acceptance tests as described by > > Pressman. > > > > There are other three possible terms we could use to describe what is > > currently > > QEMU "acceptance" tests: > > > > 1 - Integration tests: > > - "Integration testing is a systematic technique for constructing the > > software architecture while at the same time conducting tests to > > uncover errors associated with interfacing. The objective is to > > take > > unit-tested components and build a program structure that has been > > dictated by design." [2] > > * Note: Sommerville does not have a clear definition of integration > > testing. He refers to incremental integration of components inside > > the system testing (see [3]). > > > > 2 - Validation tests: > > - "Validation testing begins at the culmination of integration > > testing, > > when individual components have been exercised, the software is > > completely assembled as a package, and interfacing errors have been > > uncovered and corrected. At the validation or system level, the > > distinction between different software categories disappears. > > Testing > > focuses on user-visible actions and user-recognizable output from > > the > > system." [4] > > - "where you expect the system to perform correctly using a set of > > test > > cases that reflect the system’s expected use." [5] > > * Note: the definition of "validation testing" from Sommerville > > reflects > > the same definition found around the Internet, as one of the > > processes > > inside the "Verification & Validation (V)." In this concept, > > validation testing is a high-level definition that covers unit > > testing, > > functional testing, integration testing, system testing, and > > acceptance > > testing. > > > > 3 - System tests: > > - "verifies that all elements mesh properly and that overall system > > function and performance is achieved." [6] > > - "involves integrating components to create a version of the system > > and > > then testing the integrated system. System testing checks that > > components are compatible, interact correctly, and transfer the > > right > > data at the right time across their interfaces." [7] > > > > The tests implemented inside the QEMU "acceptance" directory depend on the > > software completely assembled and, sometimes, on other elements, like > > operating > > system images. In this case, the proposal here is to rename the current > > "acceptance" directory to "system." > > Are user-mode tests using Avocado also system tests? > https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html Yes, they are considered system tests because in software engineering, system testing means your software built and tested using external test artifacts, just like your example. > > > [1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. p. 430. > > [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. Software Engineering, p. 398. > > [3] Sommerville, Ian (2016). Software Engineering. p. 240-242. > > [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. Software Engineering, p. 407. > > [5] Sommerville, Ian (2016). Software Engineering. p. 227. > > [6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > > Practitioner’s Approach. Software Engineering, p. 377. > > [7] Sommerville, Ian (2016). Software Engineering. p. 240. > > > > Signed-off-by: Willian Rampazzo > > --- > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > > index f718b61fa7..c5de3c9fd5 100644 > > --- a/.gitlab-ci.yml > > +++ b/.gitlab-ci.yml > > @@ -52,7 +52,7 @@ include: > > # Avoid recompiling by hiding ninja with NINJA=":" > > - make NINJA=":" $MAKE_CHECK_ARGS > > > > -.acceptance_template: _definition > > +.system_template: _definition > > .system_test_template: _test_definition ? Agreed, keeps the consistency. > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > index
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 20/05/2021 22.28, Philippe Mathieu-Daudé wrote: On 5/20/21 9:53 PM, Willian Rampazzo wrote: Conceptually speaking, acceptance tests "are a series of specific tests conducted by the customer in an attempt to uncover product errors before accepting the software from the developer. Conducted by the end-user rather than software engineers, acceptance testing can range from an informal “test drive” to a planned and systematically executed series of scripted tests" [1]. Every time Pressman refers to the term "acceptance testing," he also refers to user's agreement in the final state of an implemented feature. Today, QEMU is not implementing user acceptance tests as described by Pressman. There are other three possible terms we could use to describe what is currently QEMU "acceptance" tests: 1 - Integration tests: - "Integration testing is a systematic technique for constructing the software architecture while at the same time conducting tests to uncover errors associated with interfacing. The objective is to take unit-tested components and build a program structure that has been dictated by design." [2] * Note: Sommerville does not have a clear definition of integration testing. He refers to incremental integration of components inside the system testing (see [3]). After thinking about this for a while, I agree with you that renaming the "acceptance" tests to "integration" tests is also not a good idea. When I hear "integration" test in the context of the virt stack, I'd rather expect a test suite that picks KVM (i.e. a kernel), QEMU, libvirt and maybe virt-manager on top and tests them all together. So we should look for a different name indeed. 2 - Validation tests: - "Validation testing begins at the culmination of integration testing, when individual components have been exercised, the software is completely assembled as a package, and interfacing errors have been uncovered and corrected. At the validation or system level, the distinction between different software categories disappears. Testing focuses on user-visible actions and user-recognizable output from the system." [4] - "where you expect the system to perform correctly using a set of test cases that reflect the system’s expected use." [5] * Note: the definition of "validation testing" from Sommerville reflects the same definition found around the Internet, as one of the processes inside the "Verification & Validation (V)." In this concept, validation testing is a high-level definition that covers unit testing, functional testing, integration testing, system testing, and acceptance testing. 3 - System tests: - "verifies that all elements mesh properly and that overall system function and performance is achieved." [6] - "involves integrating components to create a version of the system and then testing the integrated system. System testing checks that components are compatible, interact correctly, and transfer the right data at the right time across their interfaces." [7] The tests implemented inside the QEMU "acceptance" directory depend on the software completely assembled and, sometimes, on other elements, like operating system images. In this case, the proposal here is to rename the current "acceptance" directory to "system." Are user-mode tests using Avocado also system tests? https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html We've indeed got the problem that the word "system" is a little bit overloaded in the context of QEMU. We often talk about "system" when referring to the qemu-softmmu-xxx emulators (in contrast to the linux-user emulator binaries). For example, the "--disable-system" switch of the configure script, or the "build-system" and "check-system" jobs in the .gitlab-ci.yml file ... thus this could get quite confusing in the .gitlab-ci.yml file afterwards. So I think renaming "acceptance" to "system" is especially ok if we only keep the "softmmu"-related tests in that folder... would it maybe make sense to add the linux-user related tests in a separate folder called tests/user/ instead, Philippe? And we should likely rename the current build-system and check-system jobs in our gitlab-CI to build-softmmu and check-softmmu or so? Alternatively, what about renaming the "acceptance" tests to "validation" instead? That word does not have a duplicated definition in the context of QEMU yet, so I think it would be less confusing. Thomas
Re: [RFC 1/1] acceptance tests: rename acceptance to system
On 5/20/21 9:53 PM, Willian Rampazzo wrote: > Conceptually speaking, acceptance tests "are a series of specific tests > conducted by the customer in an attempt to uncover product errors before > accepting the software from the developer. Conducted by the end-user rather > than software engineers, acceptance testing can range from an informal > “test drive” to a planned and systematically executed series of scripted > tests" [1]. Every time Pressman refers to the term "acceptance testing," he > also refers to user's agreement in the final state of an implemented feature. > Today, QEMU is not implementing user acceptance tests as described by > Pressman. > > There are other three possible terms we could use to describe what is > currently > QEMU "acceptance" tests: > > 1 - Integration tests: > - "Integration testing is a systematic technique for constructing the > software architecture while at the same time conducting tests to > uncover errors associated with interfacing. The objective is to take > unit-tested components and build a program structure that has been > dictated by design." [2] > * Note: Sommerville does not have a clear definition of integration > testing. He refers to incremental integration of components inside > the system testing (see [3]). > > 2 - Validation tests: > - "Validation testing begins at the culmination of integration testing, > when individual components have been exercised, the software is > completely assembled as a package, and interfacing errors have been > uncovered and corrected. At the validation or system level, the > distinction between different software categories disappears. Testing > focuses on user-visible actions and user-recognizable output from the > system." [4] > - "where you expect the system to perform correctly using a set of test > cases that reflect the system’s expected use." [5] > * Note: the definition of "validation testing" from Sommerville reflects > the same definition found around the Internet, as one of the processes > inside the "Verification & Validation (V)." In this concept, > validation testing is a high-level definition that covers unit > testing, > functional testing, integration testing, system testing, and > acceptance > testing. > > 3 - System tests: > - "verifies that all elements mesh properly and that overall system > function and performance is achieved." [6] > - "involves integrating components to create a version of the system and > then testing the integrated system. System testing checks that > components are compatible, interact correctly, and transfer the right > data at the right time across their interfaces." [7] > > The tests implemented inside the QEMU "acceptance" directory depend on the > software completely assembled and, sometimes, on other elements, like > operating > system images. In this case, the proposal here is to rename the current > "acceptance" directory to "system." Are user-mode tests using Avocado also system tests? https://www.mail-archive.com/qemu-devel@nongnu.org/msg782505.html > [1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. p. 430. > [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. Software Engineering, p. 398. > [3] Sommerville, Ian (2016). Software Engineering. p. 240-242. > [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. Software Engineering, p. 407. > [5] Sommerville, Ian (2016). Software Engineering. p. 227. > [6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A > Practitioner’s Approach. Software Engineering, p. 377. > [7] Sommerville, Ian (2016). Software Engineering. p. 240. > > Signed-off-by: Willian Rampazzo > --- > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index f718b61fa7..c5de3c9fd5 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -52,7 +52,7 @@ include: > # Avoid recompiling by hiding ninja with NINJA=":" > - make NINJA=":" $MAKE_CHECK_ARGS > > -.acceptance_template: _definition > +.system_template: _definition .system_test_template: _test_definition ? > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 8f220e15d1..c580292bb5 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -16,7 +16,7 @@ ifneq ($(filter $(all-check-targets), check-softfloat),) > @echo " $(MAKE) check-tcgRun TCG tests" > @echo " $(MAKE) check-softfloat Run FPU emulation tests" > endif > - @echo " $(MAKE) check-acceptance Run all acceptance (functional) > tests" > + @echo " $(MAKE) check-system Run all system tests"
[RFC 1/1] acceptance tests: rename acceptance to system
Conceptually speaking, acceptance tests "are a series of specific tests
conducted by the customer in an attempt to uncover product errors before
accepting the software from the developer. Conducted by the end-user rather
than software engineers, acceptance testing can range from an informal
“test drive” to a planned and systematically executed series of scripted
tests" [1]. Every time Pressman refers to the term "acceptance testing," he
also refers to user's agreement in the final state of an implemented feature.
Today, QEMU is not implementing user acceptance tests as described by Pressman.
There are other three possible terms we could use to describe what is currently
QEMU "acceptance" tests:
1 - Integration tests:
- "Integration testing is a systematic technique for constructing the
software architecture while at the same time conducting tests to
uncover errors associated with interfacing. The objective is to take
unit-tested components and build a program structure that has been
dictated by design." [2]
* Note: Sommerville does not have a clear definition of integration
testing. He refers to incremental integration of components inside
the system testing (see [3]).
2 - Validation tests:
- "Validation testing begins at the culmination of integration testing,
when individual components have been exercised, the software is
completely assembled as a package, and interfacing errors have been
uncovered and corrected. At the validation or system level, the
distinction between different software categories disappears. Testing
focuses on user-visible actions and user-recognizable output from the
system." [4]
- "where you expect the system to perform correctly using a set of test
cases that reflect the system’s expected use." [5]
* Note: the definition of "validation testing" from Sommerville reflects
the same definition found around the Internet, as one of the processes
inside the "Verification & Validation (V)." In this concept,
validation testing is a high-level definition that covers unit testing,
functional testing, integration testing, system testing, and acceptance
testing.
3 - System tests:
- "verifies that all elements mesh properly and that overall system
function and performance is achieved." [6]
- "involves integrating components to create a version of the system and
then testing the integrated system. System testing checks that
components are compatible, interact correctly, and transfer the right
data at the right time across their interfaces." [7]
The tests implemented inside the QEMU "acceptance" directory depend on the
software completely assembled and, sometimes, on other elements, like operating
system images. In this case, the proposal here is to rename the current
"acceptance" directory to "system."
[1] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
Practitioner’s Approach. p. 430.
[2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
Practitioner’s Approach. Software Engineering, p. 398.
[3] Sommerville, Ian (2016). Software Engineering. p. 240-242.
[4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
Practitioner’s Approach. Software Engineering, p. 407.
[5] Sommerville, Ian (2016). Software Engineering. p. 227.
[6] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering, A
Practitioner’s Approach. Software Engineering, p. 377.
[7] Sommerville, Ian (2016). Software Engineering. p. 240.
Signed-off-by: Willian Rampazzo
---
.gitlab-ci.yml| 58 +--
MAINTAINERS | 40 ++---
configure | 2 +-
docs/devel/build-system.rst | 2 +-
docs/devel/testing.rst| 23
docs/system/arm/orangepi.rst | 8 +--
tests/Makefile.include| 14 ++---
tests/acceptance/README.rst | 10
tests/system/README.rst | 10
.../avocado_qemu/__init__.py | 2 +-
tests/{acceptance => system}/boot_linux.py| 0
.../boot_linux_console.py | 0
tests/{acceptance => system}/boot_xen.py | 0
tests/{acceptance => system}/cpu_queries.py | 0
.../{acceptance => system}/empty_cpu_model.py | 0
tests/{acceptance => system}/linux_initrd.py | 2 +-
.../linux_ssh_mips_malta.py | 0
.../machine_arm_canona1100.py | 0
.../machine_arm_integratorcp.py | 0
.../machine_arm_n8x0.py | 0
tests/{acceptance => system}/machine_avr6.py | 2 +-
.../machine_m68k_nextcube.py | 0
.../machine_microblaze.py
