Perhaps automation needs to be implemented. CC-ing Pavlin that I think can advise about automation strategies.
Johannes Lips wrote on 1/8/19 12:17 PM: > > > On Tue, Jan 8, 2019 at 11:08 AM Lukas Ruzicka <[email protected] > <mailto:[email protected]>> wrote: > > Hello KDE and XFCE teams, > > I am writing to you on behalf of the Fedora QA team, because you are the > stakeholders of release-blocking desktop environments in Fedora. > > We believe that there is need to revisit and review some of the > approaches we > have in desktop testing. I hope that I can clarify in the text. > > *How do we test desktop applications?* > > Currently, there are three desktop environments, which block the Fedora > releases: GNOME, KDE, and XFCE (on ARM). Among others, there is a > test case > called “Desktop Menus” [ > https://fedoraproject.org/wiki/QA:Testcase_desktop_menus ] which is > used for all three desktop environments. > > Why is Xfce tested only on ARM devices? I mean if that's one of the main > reasons it's slow and tedious, why not change the architecture? > Additionally, it would be probably worthwhile to create two different > sets of programs, one group has to start and thus needs to be tested and > the other is a nice > to have, but doesn't need to start. At least the criteria could be a > little bit less strict. > > Johannes > > *What is the purpose of the Desktop Menus test case?* > > The purpose of the “Desktop Menus” test case is to test that > applications listed > in the menu do generally work. That means that each application in > the default > system menu must launch without crashing and withstand basic usage. [2] > > *What problems can arise when trying to follow the test case?* > > This approach leaves a lot of space for variability, because the > terms are not > clearly given, especially when talking about “basic functionality” > of the > desktop applications. In the past, there have been some disputes > during Blocker > Review meetings, or during Go/No go meetings whether basic > functionality is > broken or not. It's usually up to group judgement to decide what is > a basic > functionality and what it beyond that, which means as testers we > need to err on > the side of doing more testing rather than less. > > Testing the “Desktop Menus” test case is one of the most time > consuming test > case we currently have. In GNOME, there are currently about 40 > applications in > the default menu and all need to be started and their basic > functionality > tested. In KDE, the number of applications is even higher, and some > application > types (like web browsers) are even present as several different > applications. > The scope of this test case also covers testing the subcomponents in > the Control > Center (or a similar equivalent) and, usually, such subcomponents > are numerous. > > Testing XFCE on ARM is also rather time consuming, because the ARM > devices we > can use for testing are very slow and there are some application > that make > little sense on an ARM motherboard (e.g. a CD writing software). > Thus, doing a > thorough Desktop Menus test case requires a rather longish time > frame and we > lose time we could devote to different problems. > > Besides that, the test case is also very boring, so it usually gets > tested by > Fedora QA members only, with few inputs from the community. So, > before the Beta > and Final release we are able to do such tests a couple of times, > but the > overall coverage is not big. > > *What could we do about it?* > > At Fedora QA, we have been trying to come up with some ideas to make the > situation a little better than it is now. The goal is that the > important areas > get enough attention and testing they deserve, and we avoid shipping > broken and > under-tested software. See the ideas below and please tell us if you > could adopt > all/some of them, or if you have different suggestions to improve > the current > state, we'd love to hear them too. > > > * The list of pre-installed applications could be pruned of those > which don't make much sense for that particular purpose (e.g. CD > writing software on ARM, or perhaps on any system nowadays). > * Pre-installed applications could be deduplicated where possible > (e.g. not having several web browsers pre-installed by default). > * We could define a list of "important" applications that would be > tested thoroughly and the basic functionality criterion would > apply just to them. That would tell QA where to focus. (This > would obviously contain applications like system settings, file > browser, text editor, browser, terminal emulator, etc. But e.g. > a screenshot utility would probably not fall into the list. > Also, in case of duplicated apps, only one of those could be > marked as "important", and the others would* not). The rest of > the applications would be just best effort in terms of QA, and > wouldn't block the release. > * Last but not least, you could help us more with release > validation before relevant milestones (Beta, Final) :-) If > there's anything that should be improved on our part > (communication, instructions, etc), we'd love to hear that. > > Please, let me know what you think about the ideas proposed above. > Do you have > any other ideas that could help us improve the quality (with the > same amount of > testing force)? > > > Best regards, > > -- > > Lukáš Růžička > > FEDORA QE, RHCE > > Red Hat > > <https://www.redhat.com> > > Purkyňova 115 > > 612 45 Brno - Královo Pole > > [email protected] <mailto:[email protected]> > > > TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted> > > _______________________________________________ > xfce mailing list -- [email protected] > <mailto:[email protected]> > To unsubscribe send an email to [email protected] > <mailto:[email protected]> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > > > _______________________________________________ > xfce mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/[email protected] > _______________________________________________ xfce mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected]
