Thanks for the reply, I decided to reply to this e-mail and not to
previous one to keep answers in one place.
On 10/4/18 9:15 AM, Josef Reidinger wrote:
On Thu, 4 Oct 2018 09:02:46 +0200
Rodion Iafarov <[email protected]> wrote:
On 10/3/18 11:00 AM, Ladislav Slezak wrote:
Dne 02. 10. 18 v 11:18 Lukas Ocilka napsal(a):
-------- Forwarded Message --------
Subject: YaST GUI testing framework
Date: Mon, 24 Sep 2018 18:37:18 +0200
From: Rodion Iafarov <[email protected]>
To: [email protected]
Dear YaSTees,
I would like to kindly ask for your help. In order to make next game
changing step for YaST GUI testing, we need to improve macro player
functionality.
My first question would be: is the macro player good enough?
If we want to invest some time in this area, maybe we could use
something else.
Yes, as for the recorder it could speed up test development a bit, but
in general we just need player.
In case we cannot get enough support and development is too heavy, we
may use squish or ldtp, which have certain limitations, but will server
the purpose.
I have spent two hackweeks developing an automatic YaST UI testing
framework. It's based on a REST API (HTTP with JSON format), it can not only
send the user input actions but can also test the content of the
current dialog.
The advantage is that the REST API is not bound to any programming language,
you can write a client driving an YaST installation in any language. With help
of
"curl" and "jq" it would be possible even in shell... (but I'd avoid it)
The YaST UI actions which can be done by user are quite limited (e.g. you can
press a button, select a checkbox, write something to an input field... and
that's basically it). So made a simple Cucumber wrapper which defines such
steps.
Unfortunately, this was initial reason why we still on current state of
things. Even with simplest yast module (e.g. hostnames), we need
interaction with tables, selecting rows, checking values in the cells,
etc. So this is a must to proceed. That's also the reason why we can
consider other tools which can already operate on qt.
Problem of tools operating only on qt is that we really have to test also
ncurses as many corporate customers uses ncurses ( info we get from support and
sale guys ), so it is also very vital to test it ( like that everything fits
into screen, all is accessible, no trash in output and so on ).
That's true, but we need to start somewhere. If we cover qt for
functional, we can at least limit coverage with ncurses, as performing
same action will trigger same code underneath.
Of course, if you do not like Cucumber you can use any other framework, you
could extend the current openQA or start something new from scratch. That's
a big advantage of the generic REST API, if the testing frontend does not
fit any more you can easily change it without affecting YaST itself. We could
even
use a different frontend in our CI (Travis/Jenkins) and different one in
openQA built on top of the same REST backend.
See more details about the REST API in my hackweek blog posts [1], [2].
If you want to give it a try I have prepared RPM packages for SLE15/Leap15
in [3], how to install the packages is described in the Cucumber wrapper
at GitHub [4].
(Note: I built the packages several months ago, if something does not work
now just ping me.)
This project is far from complete, there are still some missing parts
like support for some specific widgets, security improvements (authentication
and encryption, isolating to a separate plugin so the REST API can be installed
only when needed, not included in the default installation, etc...).
But the prototype is already able to test the default openSUSE/SLE installation
well,
(see the screencasts in the blog) so I think it would make sense to at least
think
about this possibility as an alternative to the UI macro player.
If you have any questions about this project just ask me.
I've actually seen those two projects and they look really promising, so
I'm with Lukas here, seems like a good solution.
However, I started this discussion to confirm preferable way, so we
could collaborate.
I think first step can be to write down your use cases and also your
requirements, so hopefully we can somehow prioritize work on it.
These is combined list for the needs for your previous email:
1. Operating on all existing widgets, including tables, trees,
comboboxes, tabs. Being able to perform same set of actions like user
can do normally.
2. Possibility to assess widgets properties e.g. values in the table
cells, labels text, UI properties (if control is visible/disabled)
3. Some way of integration with openQA (that's important for QA team, as
we have to use it, but it should not be an issue as we can simply create
file with results and parse it)
4. There is not to complex way to launch tests when running installation
(e.g. booting from installer DVD and triggering test)
5. There is a way to run tests with local changes without configuring
any kind of complex setup
6. Solution can be used to run CI per branch
7. Can be used for ncurses and qt GUIs
8. Running tests minimizes influence on SUT (we should keep difference
of real installation vs integration one as little as possible not
installing hundreds of the dependencies).
As we cannot get it all, my proposal would be to start with something
more simple, e.g. yast module instead of installer. But even for
simplest ones e.g. timezones and hostnames, we lack functionality at the
moment.
Also, I would like to mention, we as QA department are really interested
in the development and considered driving it with the support of the
YaST team, as I fully understand lack of resources for it. But we need
your support to decide on a solution and helping to create tasks so we
could pick them up.
Josef
Note: Because this REST API is available for any libyui application and is not
YaST
specific we could possibly cooperate with the Mageia (formerly Mandriva,
formerly
Mandrake) developers, their management tools also use the libyui [5] and having
a
testing framework would probably help them as well...
Ladislav
[1] https://blog.ladslezak.cz/2018/07/13/hackweek-17/
[2] https://blog.ladslezak.cz/2017/12/06/hackweek-16-yast-ui-rest-api/
[3] https://build.opensuse.org/project/show/home:lslezak:libyui-rest-api
[4] https://github.com/lslezak/cucumber-yast
[5] https://madb.mageia.org/package/show/name/manatools
--
Best Regards
Ladislav Slezák
Yast Developer
------------------------------------------------------------------------
SUSE LINUX, s.r.o. e-mail: [email protected]
Lihovarská 1060/12 tel: +420 284 028 960
190 00 Prague 9 fax: +420 284 028 951
Czech Republic http://www.suse.cz/
--
Kind regards,
Rodion Iafarov <[email protected]>
QA Engineer
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [email protected]
To contact the owner, e-mail: [email protected]
