Re: Automatic testing: openqa.debian.net

[Holger Levsen]

resend after fixing the debian-edu mailing list address…

Hi Phil,

(dropping debian-boot@ and adding debian-edu@l.d.o to the recipients,
and leaving context for the latter...)

On Fri, Nov 24, 2017 at 01:35:41PM +0100, Philip Hands wrote:
> If you look here:
>   https://openqa.debian.net/
> 
> You'll see that I've been testing d-i daily images for a while.
> 
> The scripts that drive those tests are available here:
>   https://anonscm.debian.org/cgit/collab-maint/openqa-tests-debian.git/
 
very cool. (how) can I subscribe to commit notifications for this repo?

> As you can see from the README.md this is based on Fedora's tests.  The
> README helpfully points at the original documentation for os-autoinst,
> which is the thing that does the work.
> 
> It's possible that the README says things work that I've since broken in
> order to make it work for the Debian tests.  Please point that out to me
> if you notice, and I'll either fix things, or fix the README, as
> appropriate.
> 
> So far I've been focused on testing d-i up to the point where we can see
> that it's possible to login, and see whatever should be expected for
> each of our desktops.

I see you also seem to have tests for Debian Edu!
(at http://openqa.debian.net/group_overview/6 )

> There is no reason to limit ourselves to that, and since we're
> generating newly installed VM images regularly, it's completely fine to
> write tests that use those as a starting point.  It's also possible to
> write tests that use ssh or the serial console, so that yo don't need to
> hunt for things in screenshots.
> 
> Currently it's all running in one VM (with nested VMs), but the
> os-autoinst is able to run additional workers, so we should be able to
> scale up as required.
> 
> At some point I'll want to reinstall everything, when all the bits are
> available as packages (which might be already true -- I'll check shortly).
> 
> BTW In order to log in, you'll currently need an OpenSUSE SSO account
> (because that works out of the box, and I've no idea what needs to be
> done to make things work with sso.debian.org, say -- all hints
> gratefully accepted :-) )
> 
> There's lots of things left to do here, with the most important thing
> probably being making it possible to add tests without needing root
> access to the machine (which is currently needed for some bits) so
> please pester me about what you would like to test, and that will force
> me to make it possible for you to do it without my intervention
> (eventually ;-) ).

I'd like it to get into a state ASAP so that we can turn of the
"g-i-installation" tests on jenkins.debian.net - how can I help with
that?

If i look at the job group "Debian" I cannot (yet?) clearly see which
Debian images are tested how?

I suppose it would be good to setup tests for stable and
testing/unstable, and use the former to tests the tests and the latter
to test Debian... :) (and both combined to test+develop the UI)


-- 
cheers,
Holger


signature.asc
Description: PGP signature

Re: Automatic testing: openqa.debian.net

[Philip Hands]

On Wed, 06 Dec 2017, intrigeri  wrote:
> Hi,
>
> Philip Hands:
>> Having jenkins in the mix makes things at least twice as painful.
>
> I would certainly never recommend *developing* such tests directly on
> Jenkins. At Tails we develop tests locally and use Jenkins only
> for CI.

Yeah, that was the thing that I was doing most wrong. I somehow never
succeeded in getting SikuliX to run locally -- I'm sure that would help
a lot.

In the mean time I've developed a pretty profound dislike of jenkins, so
being able to largely avoid that is a bonus from my point of view (I
guess we'll be triggering jobs from it, but at least I'll not be having
to fight with jenkins in order to see the results of the tests).

>> OpenQA really makes it a lot easier, orders of magnitude quicker, and
>> much more fun.
>
> :)
>
>> I can imagine getting OpenQA to a point where people can do drive-by
>> test creation when they're testing bugs in random packages, and that not
>> only would it be less effort than testing by hand, but should build into
>> a nice regression suite -- I doubt that was ever going to happen with
>> cucumber etc.
>
> Glad you've found something that works better for Debian.

The thing that I've not yet managed to do, that I had working nicely in
cucumber is talking to the VM's serial port, but it seems that they have
some nice scripting for dealing with that (which I'm not quite
understanding as yet). I certainly liked that aspect of the tails tests.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Automatic testing: openqa.debian.net

[intrigeri]

Hi,

Philip Hands:
> Having jenkins in the mix makes things at least twice as painful.

I would certainly never recommend *developing* such tests directly on
Jenkins. At Tails we develop tests locally and use Jenkins only
for CI.

> OpenQA really makes it a lot easier, orders of magnitude quicker, and
> much more fun.

:)

> I can imagine getting OpenQA to a point where people can do drive-by
> test creation when they're testing bugs in random packages, and that not
> only would it be less effort than testing by hand, but should build into
> a nice regression suite -- I doubt that was ever going to happen with
> cucumber etc.

Glad you've found something that works better for Debian.

Cheers,
-- 
intrigeri

Re: Automatic testing: openqa.debian.net

[Philip Hands]

On Tue, 05 Dec 2017, Raphael Hertzog  wrote:
> Hi fil,
>
> On Fri, 24 Nov 2017, Philip Hands wrote:
>> If you look here:
>> 
>>   https://openqa.debian.net/
>> 
>> You'll see that I've been testing d-i daily images for a while.
>>
>> The scripts that drive those tests are available here:
>> 
>>   https://anonscm.debian.org/cgit/collab-maint/openqa-tests-debian.git/
>
> In what way is openqa/os-autoinst better than the jenkins jobs
> that you and Holger have created to do similar tests in the past?
>
> I know Holger was at some point considering to replicate what Tails
> has done:
> https://tails.boum.org/contribute/release_process/test/automated_tests/

That's what I started from with what I was doing before.

> Have you looked into this and how does openqa/os-autoinst compare
> to this solution?

It's entirely possible that I wasn't doing that in the most
straight-forward manner, and that it's possible to make it easier to
work with cucumber/sikuli, but I doubt that it's ever possible to make
it a pleasant experience. Sikuli on its own seems like it's probably a
nice thing, and cucumber seems very useful if one is doing BDD, but the
cucumber web site has a big fat warning on it saying that if you are
using it for what we were using it for, then you're doing it wrong.

I was already agreeing with them about that before I saw OpenQA, so it
was a great relief to discover that I could stop.

Having jenkins in the mix makes things at least twice as painful.

OpenQA really makes it a lot easier, orders of magnitude quicker, and
much more fun.

I can imagine getting OpenQA to a point where people can do drive-by
test creation when they're testing bugs in random packages, and that not
only would it be less effort than testing by hand, but should build into
a nice regression suite -- I doubt that was ever going to happen with
cucumber etc.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Automatic testing: openqa.debian.net

[Raphael Hertzog]

Hi fil,

On Fri, 24 Nov 2017, Philip Hands wrote:
> If you look here:
> 
>   https://openqa.debian.net/
> 
> You'll see that I've been testing d-i daily images for a while.
>
> The scripts that drive those tests are available here:
> 
>   https://anonscm.debian.org/cgit/collab-maint/openqa-tests-debian.git/

In what way is openqa/os-autoinst better than the jenkins jobs
that you and Holger have created to do similar tests in the past?

I know Holger was at some point considering to replicate what Tails
has done:
https://tails.boum.org/contribute/release_process/test/automated_tests/

Have you looked into this and how does openqa/os-autoinst compare
to this solution?

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Re: Automatic testing: openqa.debian.net

[Philip Hands]

On Sun, 03 Dec 2017, Holger Levsen  wrote:
> Hi Phil,
>
> (dropping debian-boot@ and adding debian-edu@l.d.o to the recipients,
> and leaving context for the latter...)
>
> On Fri, Nov 24, 2017 at 01:35:41PM +0100, Philip Hands wrote:
>> If you look here:
>>   https://openqa.debian.net/
>> 
>> You'll see that I've been testing d-i daily images for a while.
>> 
>> The scripts that drive those tests are available here:
>>   https://anonscm.debian.org/cgit/collab-maint/openqa-tests-debian.git/
>  
> very cool. (how) can I subscribe to commit notifications for this
> repo?

I guess we should plumb it up with a link to IRC?

At present I'm mostly being naughty and doing my flailing attempts to
make things work directly in the live directory, so there are only
commits when things start working properly (so pretty rare ;-) )

>> As you can see from the README.md this is based on Fedora's tests.  The
>> README helpfully points at the original documentation for os-autoinst,
>> which is the thing that does the work.
>> 
>> It's possible that the README says things work that I've since broken in
>> order to make it work for the Debian tests.  Please point that out to me
>> if you notice, and I'll either fix things, or fix the README, as
>> appropriate.
>> 
>> So far I've been focused on testing d-i up to the point where we can see
>> that it's possible to login, and see whatever should be expected for
>> each of our desktops.
>
> I see you also seem to have tests for Debian Edu!
> (at http://openqa.debian.net/group_overview/6 )

Yeah -- that was mostly to see how things work once you add another
thing to test.  It only does a simple install so far, and it gets a bit
befuddled by Firefox automatically starting, but it does work.

If there are regularly produced betas to test somewhere, that would make
it a much more interesting test.

>> There is no reason to limit ourselves to that, and since we're
>> generating newly installed VM images regularly, it's completely fine to
>> write tests that use those as a starting point.  It's also possible to
>> write tests that use ssh or the serial console, so that yo don't need to
>> hunt for things in screenshots.
>> 
>> Currently it's all running in one VM (with nested VMs), but the
>> os-autoinst is able to run additional workers, so we should be able to
>> scale up as required.
>> 
>> At some point I'll want to reinstall everything, when all the bits are
>> available as packages (which might be already true -- I'll check shortly).
>> 
>> BTW In order to log in, you'll currently need an OpenSUSE SSO account
>> (because that works out of the box, and I've no idea what needs to be
>> done to make things work with sso.debian.org, say -- all hints
>> gratefully accepted :-) )
>> 
>> There's lots of things left to do here, with the most important thing
>> probably being making it possible to add tests without needing root
>> access to the machine (which is currently needed for some bits) so
>> please pester me about what you would like to test, and that will force
>> me to make it possible for you to do it without my intervention
>> (eventually ;-) ).
>
> I'd like it to get into a state ASAP so that we can turn of the
> "g-i-installation" tests on jenkins.debian.net - how can I help with
> that?

Me too -- my time is currently being consumed by the fact that my local
kindergarten is infested with some sort of vomiting virus, hence I
decided Mathilda would be better off using me as a climbing frame (not
great for productivity, so don't expect quick replies ;-) )

> If i look at the job group "Debian" I cannot (yet?) clearly see which
> Debian images are tested how?

The images are available in the assets tab, but you're right -- they are
the daily sid images.

> I suppose it would be good to setup tests for stable and
> testing/unstable, and use the former to tests the tests and the latter
> to test Debian... :) (and both combined to test+develop the UI)

The current tests should work with stable/testing, so that's just a
question of launching the tests.

I'd really like to do this using some mechanism for throwing images at
the test system, so that when the images are built, the tests can be
triggered.

Scripting it otherwise has proven rather fragile, and always seems to
need a helping hand when releases happen, which is a bit of a bore.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Automatic testing: openqa.debian.net

[Holger Levsen]

Hi Phil,

(dropping debian-boot@ and adding debian-edu@l.d.o to the recipients,
and leaving context for the latter...)

On Fri, Nov 24, 2017 at 01:35:41PM +0100, Philip Hands wrote:
> If you look here:
>   https://openqa.debian.net/
> 
> You'll see that I've been testing d-i daily images for a while.
> 
> The scripts that drive those tests are available here:
>   https://anonscm.debian.org/cgit/collab-maint/openqa-tests-debian.git/
 
very cool. (how) can I subscribe to commit notifications for this repo?

> As you can see from the README.md this is based on Fedora's tests.  The
> README helpfully points at the original documentation for os-autoinst,
> which is the thing that does the work.
> 
> It's possible that the README says things work that I've since broken in
> order to make it work for the Debian tests.  Please point that out to me
> if you notice, and I'll either fix things, or fix the README, as
> appropriate.
> 
> So far I've been focused on testing d-i up to the point where we can see
> that it's possible to login, and see whatever should be expected for
> each of our desktops.

I see you also seem to have tests for Debian Edu!
(at http://openqa.debian.net/group_overview/6 )

> There is no reason to limit ourselves to that, and since we're
> generating newly installed VM images regularly, it's completely fine to
> write tests that use those as a starting point.  It's also possible to
> write tests that use ssh or the serial console, so that yo don't need to
> hunt for things in screenshots.
> 
> Currently it's all running in one VM (with nested VMs), but the
> os-autoinst is able to run additional workers, so we should be able to
> scale up as required.
> 
> At some point I'll want to reinstall everything, when all the bits are
> available as packages (which might be already true -- I'll check shortly).
> 
> BTW In order to log in, you'll currently need an OpenSUSE SSO account
> (because that works out of the box, and I've no idea what needs to be
> done to make things work with sso.debian.org, say -- all hints
> gratefully accepted :-) )
> 
> There's lots of things left to do here, with the most important thing
> probably being making it possible to add tests without needing root
> access to the machine (which is currently needed for some bits) so
> please pester me about what you would like to test, and that will force
> me to make it possible for you to do it without my intervention
> (eventually ;-) ).

I'd like it to get into a state ASAP so that we can turn of the
"g-i-installation" tests on jenkins.debian.net - how can I help with
that?

If i look at the job group "Debian" I cannot (yet?) clearly see which
Debian images are tested how?

I suppose it would be good to setup tests for stable and
testing/unstable, and use the former to tests the tests and the latter
to test Debian... :) (and both combined to test+develop the UI)


-- 
cheers,
Holger


signature.asc
Description: PGP signature