Dne 2.8.2017 v 13:10 Josef Reidinger napsal(a): > So I think at first we should agreed what is goal of such tests. If it is > smoke > test, then it should run almost automatic without writting too much ( ideally > nothing, just run ncurses check if there is not error and exit ).
Um, I'm not sure if we can make such automatic check generic enough. I mean some modules display a popup at start, some want to install additional packages, etc. How can I "exit" safely and reliably? Maybe there is "Abort" button, maybe "Cancel", maybe it's "OK", maybe you have to "Cancel" the popup first then "Abort" the main dialog... Killing running YaST also does not look nice to me. But yes, having such an automatic check without need to touch the sources would be nice. > If goal is to run integration tests, then we already have tools around and > question is why we should use it and the most important decide what to use, as > having three various integration tests tools is very confusing and time > consuming. > So e.g. for rear example, smoke test can find problem. Of course if we write > integration test it will also detect it, but question is why write new tool > for > it when we can have such test even in openQA which we currently using? Um, there is *no* openQA test for Rear (otherwise it would have found the issue). Do you really want to write and maintain openQA tests? IIRC we discussed it several times and the response was always "no". Also running openQA is not trivial, how I can run an openQA test on my machine before I open a pull request? What about the community developers who open a pull request from a fork? Travis works out of box with very small effort on our side... > So to sum it up, if we use it as smoke test, then it have to run without > modification of target module git repo and run it automatic for given repo ( > e.g. > take command from desktop file, start it in ncurses, check for error and then > abort ). If we want to use it as integration test, then it have to be > properly > discussed why to use this and not openQA as having multiple tools for same > purpose does not look like good idea for me. Um, personally I do not like openQA as writing and maintaining the tests is difficult and running them locally is almost impossible (at least for me). Look at the storage-ng case, Christopher is fighting with the openQA and OBS all the time and it requires really high effort just to keep it working. With this lightweight approach the effort should be much lower. (But don't get we wrong, openQA still has a great value as it can test the whole installation, test several HW configurations, etc...) And the most importantly: And as I mentioned, if we use a simple solution (e.g. a simple shell script) openQA could run the very same script. So there would be actually no duplication and we could even save some work for the openQA team. My proposal is: - Use unit tests whenever you can - Use Travis for smoke tests and integration testing with external tools (simple scenarios where it's possible) - For the rest use openQA - Run the Travis integration test also in openQA So far we have unit test (small scope, quick, simple to maintain) and openQA (full distro scope, slow, difficult to maintain), IMHO it makes sense to have something in the middle with the advantages of the both systems. -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: [email protected] To contact the owner, e-mail: [email protected]
