Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Andreas Färber]

Am 28.07.2007 um 12:36 schrieb Thiemo Seufer:


qemu seems to expect everyone to patch it herself instead of fixing
problems in the repository at their origin. Major example being qemu
clinging on to GCC 3.x instead of supporting both 3 and 4 by  
updating and

integrating the existing patches that worked for 0.9.0.


The patches I remember did either break other compilers or were a
maintenance nightmare of postprocessed assembler output.


I didn't say the particular patch was perfect, I can't tell. All I  
know is that there are platforms that don't have a GCC 3.x (the Intel  
Macs for instance), and syncing the existing 0.9.0 patches with CVS  
HEAD is like shooting for a running target - I didn't manage to  
successfully update the patch, it crashed badly. Having some initial,  
conditional support for GCC 4 (e.g. --enable-gcc4 or depending on $CC  
version) would be good. Do you know where exactly the problem is  
between the versions and what needs to be done to have it working  
(again)?


Andreas

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Sunil Amitkumar Janki]

Thiemo Seufer wrote:

The problem is the linker (presumably /usr/bin/ld) does not
find the math library (likely name is /lib/libm.so.6). You can
add "-v" to the gcc command line to find out what it looks for.

JFTR, building qemu on Debian mips worked for me some days ago,
but the resulting binary was broken (as expected with current CVS).
  


I have rebuilt in a pure Slackware 12.0 chroot in which I have replaced the
GCC 4.1.2 system compiler that runs from /usr/bin  with a previously 
built GCC
3.4.6 that also runs from /usr/bin and I agree that it builds this way. 
I will keep
building in this chroot then for the foreseeable future until QEMU is 
ready for

GCC 4.

Do you have any idea of what else is needed to get QEMU to a usable state on
MIPS hosts? Are there any other ideas for virtualisation such as Xen or 
VMware/
SimOS? I have asked on the Xen-devel list but nobody gave any feedback 
there.

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Thiemo Seufer]

Andreas Färber wrote:
>
> Am 27.07.2007 um 16:58 schrieb Sunil Amitkumar Janki:
>
>> Dan Shearer wrote:
>>> I see this as a different class of testing
>>> [...], checking out and compiling
>>> under different circumstances. That's very valuable, and will help
>>> address a problem that drives away a lot of potential QEMU users - they
>>> just can't compile it.
>>
>> That would be great. I am regularly compiling and using QEMU CVS
>> snapshots because features get added all the time. A lot of the time
>> it happens that these build fine on x86 whereas they don't work on
>> other architectures because the developers don't have access to those.
>>
>> I have been running self-built Slackware 12.0 on MIPS for the last
>> few months and would like to run QEMU on that architecture. previously
>> it built but didn't run, but now it doesn't even compile anymore. The
>> error message I get is the following: [...]
>
> I recently reported build failure of the ppc target on OS X due to an 
> unsupported GCC feature but no-one has shown any interest in fixing it, 
> neither applying my workaround patch nor proposing a different solution.
>
> It's a strange phenomenon that many other open source projects don't suffer 
> - qemu seems to expect everyone to patch it herself instead of fixing 
> problems in the repository at their origin. Major example being qemu 
> clinging on to GCC 3.x instead of supporting both 3 and 4 by updating and 
> integrating the existing patches that worked for 0.9.0.

The patches I remember did either break other compilers or were a
maintenance nightmare of postprocessed assembler output.


Thiemo

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Thiemo Seufer]

Sunil Amitkumar Janki wrote:
> Thiemo Seufer wrote:
>> A missing libm sounds like a build environment problem.
>>
>> Thiemo
>
> Do you have a test for this because I reinstalled the entire
> system from scratch and have had no problems building
> lots of packages over the last week except for those
> applications that assume that all the world is an x86?
>
> The only thing I have had to do is getting a GCC 3.4.6
> compiler setup which I did by configuring and packaging
> it to  "/opt/gcc/3.4.6" and including it in the path and
> making it the default using a script in "/usr/bin" and
> reversing that again.

The problem is the linker (presumably /usr/bin/ld) does not
find the math library (likely name is /lib/libm.so.6). You can
add "-v" to the gcc command line to find out what it looks for.

JFTR, building qemu on Debian mips worked for me some days ago,
but the resulting binary was broken (as expected with current CVS).

> I *do* have it configured with "--with-arch=r4600", which
> may have led to problems. It's really painful for me to
> go back to some hacked up previous os release, that was
> not done by me, on which I can rebuild GCC 3.4.6 without
> this option, but I will if I really have to.

A plain upstream tarball and the classic
"./configure && make && make install"
should work for that.


Thiemo

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[n schembr]

- Original Message 
From: Paul Brook <[EMAIL PROTECTED]>
To: qemu-devel@nongnu.org
Cc: n schembr <[EMAIL PROTECTED]>
Sent: Friday, July 27, 2007 1:38:19 PM
Subject: Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h 
hw/acpi.c hw/adlib.c ...)

> >As Qemu cannot use multicore CPUs (partially due to missing thread
> >safety), yet, you won't benefit from this unless you want to run
> >multiple instances of Qemu in parallel.
>
> how close is thread safey?

In a useful form: a fair way off.

It's relatively simple to hack something together than runs. Making it work 
correctly and go fast is much harder though. My current prototype (running on 
2 cores) runs about a quarter the speed of normal qemu, and dies shortly 
after booting because the guest atomic synchronisation primitives don't work 
right.

Either way, for a tinderbox or automated testing yo *do* have lots of 
different tests you can run in parallel, so it doesn't matter that each qemu 
instance only uses one core. We're only talking about small SMP here - maybe 
16 cores, but not thousands.

Paul
-

Is the the issue I had with qemu moving from cpu to cpu?  taskset -c 1 qemu or 
notsc

http://lists.gnu.org/archive/html/qemu-devel/2007-03/msg00652.html  
http://qemu-forum.ipi.fi/viewtopic.php?t=3714&highlight=taskset



What about preforking the processes one per virtual cpu? 

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Paul Brook]

> >> how close is thread safey?
> >
> > In a useful form: a fair way off.
> >
> > It's relatively simple to hack something together than runs. Making it
> > work correctly and go fast is much harder though. My current prototype
> > (running on 2 cores) runs about a quarter the speed of normal qemu, and
> > dies shortly after booting because the guest atomic synchronisation
> > primitives don't work right.
>
> This later problem seems like the hardest to solve to me.  Did you have
> any ideas here that don't involve hand coding the translation for atomic
> instructions?

Yes. I do exclusive access locking at the TLB level. i.e. creating a TLB entry 
for a writable page forces that page to be flushed from all the other CPU 
TLBs. 

There's some wiggle room in the definition of a "writable" page. If other 
critera are met it should be sufficient to just do this exclusion for atomic 
accesses.

If necessary the same technique can be used to avoid write ordering and 
coherency problems without having to accurately map guest barriers onto the 
equivalent host operations. This is handy when most of the guest barriers are 
implicit, e.g. when emulating a strictly ordered guest on a weakly ordered 
host.

In theory this could be taken to extremes and used to split emulation of a 
single machine over multiple address spaces/nodes.  In practice the 
contention on a normal SMP operating system is high enough that this is not 
practical.

Paul

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Sunil Amitkumar Janki]

Thiemo Seufer wrote:

A missing libm sounds like a build environment problem.

Thiemo


Do you have a test for this because I reinstalled the entire
system from scratch and have had no problems building
lots of packages over the last week except for those
applications that assume that all the world is an x86?

The only thing I have had to do is getting a GCC 3.4.6
compiler setup which I did by configuring and packaging
it to  "/opt/gcc/3.4.6" and including it in the path and
making it the default using a script in "/usr/bin" and
reversing that again.

I *do* have it configured with "--with-arch=r4600", which
may have led to problems. It's really painful for me to
go back to some hacked up previous os release, that was
not done by me, on which I can rebuild GCC 3.4.6 without
this option, but I will if I really have to.

It works well for several packages that don't compile with
GCC 4.1.2, which is the system compiler and seems to
be the accepted default nowadays in all distributions.

I guess the best solution would be to change QEMU to
make it build with GCC4 but I understand that may not
be an easy endeavour to undertake because of fundamental
design issues.

Sunil

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Thiemo Seufer]

Sunil Amitkumar Janki wrote:
[snip]
>> gcc -g  -Wl,-T,/root/tmp/qemu-20070727/mipsel.ld   -o qemu-i386 main.o 
>> syscall.o mmap.o signal.o path.o osdep.o thunk.o elfload.o linuxload.o 
>> vm86.o libqemu.a gdbstub.o   -lm -lrt -lasound
>> /usr/bin/ld: cannot find -lm
>> collect2: ld returned 1 exit status
>> make[1]: *** [qemu-i386] Error 1

A missing libm sounds like a build environment problem.


Thiemo

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Andreas Färber]

Am 27.07.2007 um 16:58 schrieb Sunil Amitkumar Janki:


Dan Shearer wrote:

I see this as a different class of testing
[...], checking out and compiling
under different circumstances. That's very valuable, and will help
address a problem that drives away a lot of potential QEMU users -  
they

just can't compile it.


That would be great. I am regularly compiling and using QEMU CVS
snapshots because features get added all the time. A lot of the time
it happens that these build fine on x86 whereas they don't work on
other architectures because the developers don't have access to those.

I have been running self-built Slackware 12.0 on MIPS for the last
few months and would like to run QEMU on that architecture. previously
it built but didn't run, but now it doesn't even compile anymore. The
error message I get is the following: [...]


I recently reported build failure of the ppc target on OS X due to an  
unsupported GCC feature but no-one has shown any interest in fixing  
it, neither applying my workaround patch nor proposing a different  
solution.


It's a strange phenomenon that many other open source projects don't  
suffer - qemu seems to expect everyone to patch it herself instead of  
fixing problems in the repository at their origin. Major example  
being qemu clinging on to GCC 3.x instead of supporting both 3 and 4  
by updating and integrating the existing patches that worked for 0.9.0.


Andreas

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Anthony Liguori]

Paul Brook wrote:

As Qemu cannot use multicore CPUs (partially due to missing thread
safety), yet, you won't benefit from this unless you want to run
multiple instances of Qemu in parallel.
  

how close is thread safey?



In a useful form: a fair way off.

It's relatively simple to hack something together than runs. Making it work 
correctly and go fast is much harder though. My current prototype (running on 
2 cores) runs about a quarter the speed of normal qemu, and dies shortly 
after booting because the guest atomic synchronisation primitives don't work 
right.
  


This later problem seems like the hardest to solve to me.  Did you have 
any ideas here that don't involve hand coding the translation for atomic 
instructions?


Regards,

Anthony Liguori

Either way, for a tinderbox or automated testing yo *do* have lots of 
different tests you can run in parallel, so it doesn't matter that each qemu 
instance only uses one core. We're only talking about small SMP here - maybe 
16 cores, but not thousands.


Paul



  

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Paul Brook]

> >As Qemu cannot use multicore CPUs (partially due to missing thread
> >safety), yet, you won't benefit from this unless you want to run
> >multiple instances of Qemu in parallel.
>
> how close is thread safey?

In a useful form: a fair way off.

It's relatively simple to hack something together than runs. Making it work 
correctly and go fast is much harder though. My current prototype (running on 
2 cores) runs about a quarter the speed of normal qemu, and dies shortly 
after booting because the guest atomic synchronisation primitives don't work 
right.

Either way, for a tinderbox or automated testing yo *do* have lots of 
different tests you can run in parallel, so it doesn't matter that each qemu 
instance only uses one core. We're only talking about small SMP here - maybe 
16 cores, but not thousands.

Paul

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[n schembr]

- Original Message 
From: Jan Marten Simons <[EMAIL PROTECTED]>
To: qemu-devel@nongnu.org
Sent: Friday, July 27, 2007 12:50:51 PM
Subject: Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h 
hw/acpi.c hw/adlib.c ...)

Sunil Amitkumar Janki schrieb:
> Dan Shearer wrote:
>> You do also have the architectures provided to you by QEMU :-) :-)
>>   
> 
> I have been able to build modular Xorg for Armedslack in QEMU without
> having the hardware but when I tried to port Slackware 12.0 to SPARC in
> QEMU I found it was way too slow and started looking for some real 
> hardware.
> 
> Maybe in the future where we have all kinds of cheap multicore processors
> this will not matter much but for the moment limited resources are still 
> some
> kind of hindrance to realizing the benefits of multi-architecture 
> emulation.
> 
> Maybe I should trade in my single-core Athlon system for some serious
> quadcore x86_64 chip but I'd rather wait for 16-core Loongson 3 :-).
> 

>As Qemu cannot use multicore CPUs (partially due to missing thread 
>safety), yet, you won't benefit from this unless you want to run 
>multiple instances of Qemu in parallel.

>regs,
>Jan

how close is thread safey?

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Jan Marten Simons]

Sunil Amitkumar Janki schrieb:

Dan Shearer wrote:

You do also have the architectures provided to you by QEMU :-) :-)
  


I have been able to build modular Xorg for Armedslack in QEMU without
having the hardware but when I tried to port Slackware 12.0 to SPARC in
QEMU I found it was way too slow and started looking for some real 
hardware.


Maybe in the future where we have all kinds of cheap multicore processors
this will not matter much but for the moment limited resources are still 
some
kind of hindrance to realizing the benefits of multi-architecture 
emulation.


Maybe I should trade in my single-core Athlon system for some serious
quadcore x86_64 chip but I'd rather wait for 16-core Loongson 3 :-).



As Qemu cannot use multicore CPUs (partially due to missing thread 
safety), yet, you won't benefit from this unless you want to run 
multiple instances of Qemu in parallel.


regs,
Jan

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Dan Shearer]

On Fri, Jul 27, 2007 at 05:50:05PM +0200, Sunil Amitkumar Janki wrote:
> Dan Shearer wrote:
> >You do also have the architectures provided to you by QEMU :-) :-)
> 
> Maybe in the future where we have all kinds of cheap multicore processors
> this will not matter much but for the moment limited resources are still 
> some
> kind of hindrance to realizing the benefits of multi-architecture emulation.

In the special case of repeat automated test compiles where the only
output is a short status message from time to time, the lack of speed is
less of a problem. You do end up with a machine that creates the maximum
amount of noise the hardware is capable of, and runs very hot :-)

-- 
Dan Shearer
[EMAIL PROTECTED]

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Sunil Amitkumar Janki]

Dan Shearer wrote:

You do also have the architectures provided to you by QEMU :-) :-)
  


I have been able to build modular Xorg for Armedslack in QEMU without
having the hardware but when I tried to port Slackware 12.0 to SPARC in
QEMU I found it was way too slow and started looking for some real hardware.

Maybe in the future where we have all kinds of cheap multicore processors
this will not matter much but for the moment limited resources are still 
some

kind of hindrance to realizing the benefits of multi-architecture emulation.

Maybe I should trade in my single-core Athlon system for some serious
quadcore x86_64 chip but I'd rather wait for 16-core Loongson 3 :-).

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Dan Shearer]

On Fri, Jul 27, 2007 at 04:58:19PM +0200, Sunil Amitkumar Janki wrote:
> Dan Shearer wrote:
> > 
> >I see this as a different class of testing to the tinderbox-style
> >testing Natalia Portillo was talking about, checking out and compiling
> >under different circumstances. That's very valuable, and will help
> >address a problem that drives away a lot of potential QEMU users - they
> >just can't compile it.
> 
> That would be great. I am regularly compiling and using QEMU CVS
> snapshots because features get added all the time. A lot of the time
> it happens that these build fine on x86 whereas they don't work on
> other architectures because the developers don't have access to those.
> 
> I have been running self-built Slackware 12.0 on MIPS for the last
> few months and would like to run QEMU on that architecture. previously
> it built but didn't run, but now it doesn't even compile anymore. The
> error message I get is the following:

This is one of the three reasons why I say it is better to do
tinerbox-type testing inside a VM. If you could make the precise QEMU
MIPS target image that fails to build QEMU available online then it is
easier for everyone to see the context. (The other two reasons are that
it is a good workout for the simulator, and general dogfood-ness for the
developers who are forced to become aware of problems that would
otherwise only be noticed by minority users such as you :-)

> I will try to get SPARC Linux and Solaris running on my newly acquired Sun
> Enterprise 250 server and compile there as well so I can test snapshots on
> all the architectures I have at my disposal.

You do also have the architectures provided to you by QEMU :-) :-)

-- 
Dan Shearer
[EMAIL PROTECTED]

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Sunil Amitkumar Janki]

Dan Shearer wrote:

On Fri, Jul 27, 2007 at 09:29:11AM -0500, Anthony Liguori wrote:
  
FYI, I've started building a VNC based automated tester.  You provide it 
a series of screenshots with masks of data that's likely to be 



That's excellent.

I see this as a different class of testing to the tinderbox-style
testing Natalia Portillo was talking about, checking out and compiling
under different circumstances. That's very valuable, and will help
address a problem that drives away a lot of potential QEMU users - they
just can't compile it.


That would be great. I am regularly compiling and using QEMU CVS
snapshots because features get added all the time. A lot of the time
it happens that these build fine on x86 whereas they don't work on
other architectures because the developers don't have access to those.

I have been running self-built Slackware 12.0 on MIPS for the last
few months and would like to run QEMU on that architecture. previously
it built but didn't run, but now it doesn't even compile anymore. The
error message I get is the following:

gcc -O2 -march=r4600 -Wall -O2 -g -fno-strict-aliasing -I. -I.. 
-I/root/tmp/qemu-20070727/target-i386 -I/root/tmp/qemu-20070727 
-I/root/tmp/qemu-20070727/linux-user 
-I/root/tmp/qemu-20070727/linux-user/i386 -D_GNU_SOURCE 
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 
-I/root/tmp/qemu-20070727/fpu -DHAS_AUDIO -DHAS_AUDIO_CHOICE 
-I/root/tmp/qemu-20070727/slirp-c -o gdbstub.o 
/root/tmp/qemu-20070727/gdbstub.c
gcc -g  -Wl,-T,/root/tmp/qemu-20070727/mipsel.ld   -o qemu-i386 main.o 
syscall.o mmap.o signal.o path.o osdep.o thunk.o elfload.o linuxload.o 
vm86.o libqemu.a gdbstub.o   -lm -lrt -lasound

/usr/bin/ld: cannot find -lm
collect2: ld returned 1 exit status
make[1]: *** [qemu-i386] Error 1
make[1]: Leaving directory `/root/tmp/qemu-20070727/i386-linux-user'
make: *** [subdir-i386-linux-user] Error 2
make -C i386-linux-user all
make[1]: Entering directory `/root/tmp/qemu-20070727/i386-linux-user'
gcc -g  -Wl,-T,/root/tmp/qemu-20070727/mipsel.ld   -o qemu-i386 main.o 
syscall.o mmap.o signal.o path.o osdep.o thunk.o elfload.o linuxload.o 
vm86.o libqemu.a gdbstub.o   -lm -lrt -lasound

/usr/bin/ld: cannot find -lm
collect2: ld returned 1 exit status
make[1]: *** [qemu-i386] Error 1
make[1]: Leaving directory `/root/tmp/qemu-20070727/i386-linux-user'
make: *** [subdir-i386-linux-user] Error 2


I will try to get SPARC Linux and Solaris running on my newly acquired Sun
Enterprise 250 server and compile there as well so I can test snapshots on
all the architectures I have at my disposal.

Sunil

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Dan Shearer]

On Fri, Jul 27, 2007 at 09:29:11AM -0500, Anthony Liguori wrote:
> FYI, I've started building a VNC based automated tester.  You provide it 
> a series of screenshots with masks of data that's likely to be 

That's excellent.

I see this as a different class of testing to the tinderbox-style
testing Natalia Portillo was talking about, checking out and compiling
under different circumstances. That's very valuable, and will help
address a problem that drives away a lot of potential QEMU users - they
just can't compile it.

-- 
Dan Shearer
[EMAIL PROTECTED]

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Anthony Liguori]

FYI, I've started building a VNC based automated tester.  You provide it 
a series of screenshots with masks of data that's likely to be 
semi-random and it'll wait for each screen shot.  It also has the smarts 
to find where the mouse is so that you can move it to an exact location.


I'll have something functional (along with a paper) within a month.

Regards,

Anthony Liguori

Dan Shearer wrote:

On Sun, Apr 08, 2007 at 09:43:20PM +0100, Natalia Portillo wrote:

Whoops, slightly late reply :-)

  

I have a huge list of operating systems (both closed and open source, that
works and that doesn't work under QEMU) that can be used to check that QEMU
doesn't broke (or even, that it corrects a non-working state).



Where the problem with QEMU is not something basic (eg doesn't boot) it
can be useful to use nested virtualisation. Although QEMU in QEMU is of
course slow because that case hasn't been optimised, if you have a
single image within which are many other images savevm'd just before
the error condition occurs, you can run a repeatable testsuite quite
quickly. And since you can snapshot the outer image, you know that you
are running these tests in exactly the same way every time. (There are a
lot of other uses for recursive virtualisation, I'm a fan of it!)

  

And I think it is better to check an installed image that to test it only
boots or installs (faster at least).



Agreed. As to installs I had to do something similar for different VM
systems a while ago and I found it useful to have a frozen image just
before the typical "detecting hardware" phase of an installation. The
point of this was not to compare the hardware that was detected, just to
detect major malfunctions. Hardware probing like this is uniquely
stressful. It isn't hard to notice if an installer crashes, or the VM
crashes, or there's lot's of errors. 

  

Fabrice and me discussed about taking screenshot per some seconds to compare
with known took screenshots and if something fails shutdown the VM and send
an email.

But that required some macro interface "click at x,y, wait some seconds,
press 'k' key", that is not currently under QEMU.



Why can't you redirect the QEMU monitor to something easily accessible,
eg a virtual serial port, and then use mouse_move and sendkey monitor
commands to drive testing, and screendump to save images for comparison?
Those commands have been in the monitor for years, so is there something
I'm not understanding here?

  

The best solution I think is to get a way to send QEMU the screenshot to
file command some times and stop the VM when both are equal, then send the
last took screenshot to a mail address so breakability can be checked. (If
it is the login window, great, it BOOTS!, if not, it doesn't)

What do you think?



The problem is synchronisation given that QEMU has no guaranteed time
domain, so you don't know if your target is failing or just being slow
so you keep on taking screenshots for a ridiculously long time.  You can
take fewer screenshots if you have some out-of-band signal. For example
knowing to expect a particular screenshot around the time that a
particular file is touched or a particular hardware device is
initialised, or a magic QEMU-specific CPU instruction is executed.

  

I have enough spare space to hold the boot images (600Gibibytes free), and a
machine that should be able to automatically checkout and compile QEMU
(Athlon XP 2000+, 768Mb RAM, 600GiB free, Gentoo Linux).



Compiling is really important. Even without testing targets, this would
be a very good community service if you want to do even just this much.
Compiling a simulator under itself in all supported configurations is an
important set of tests (compile/run MIPS under IA32, IA32 under PPC32,
etc especially 32/64 mixes, then the different supported compiler
versions, and then a couple of different Linux distributions with their
different choices of library versions and configurations.) This is
probably 24 hours of hard work. I have seen simulators choke on these
sorts of tests.

  

Just, I have not the knowledge to make the script that boots qemu, says qemu
to take the screenshot, compares the took one with the last one, stops qemu,
sends the last screenshot by email, compresses all took screenshots, goes to
next VM, so on. (Preferibly without X11 running, as the machine is a mostly
headless P2P and File server)



I think this can be simplified. Care to send a copy offlist of what
you've done and i'll have a look at it?

--
Dan Shearer
[EMAIL PROTECTED]



  

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Dan Shearer]

On Sun, Apr 08, 2007 at 09:43:20PM +0100, Natalia Portillo wrote:

Whoops, slightly late reply :-)

> I have a huge list of operating systems (both closed and open source, that
> works and that doesn't work under QEMU) that can be used to check that QEMU
> doesn't broke (or even, that it corrects a non-working state).

Where the problem with QEMU is not something basic (eg doesn't boot) it
can be useful to use nested virtualisation. Although QEMU in QEMU is of
course slow because that case hasn't been optimised, if you have a
single image within which are many other images savevm'd just before
the error condition occurs, you can run a repeatable testsuite quite
quickly. And since you can snapshot the outer image, you know that you
are running these tests in exactly the same way every time. (There are a
lot of other uses for recursive virtualisation, I'm a fan of it!)

> And I think it is better to check an installed image that to test it only
> boots or installs (faster at least).

Agreed. As to installs I had to do something similar for different VM
systems a while ago and I found it useful to have a frozen image just
before the typical "detecting hardware" phase of an installation. The
point of this was not to compare the hardware that was detected, just to
detect major malfunctions. Hardware probing like this is uniquely
stressful. It isn't hard to notice if an installer crashes, or the VM
crashes, or there's lot's of errors. 

> Fabrice and me discussed about taking screenshot per some seconds to compare
> with known took screenshots and if something fails shutdown the VM and send
> an email.
> 
> But that required some macro interface "click at x,y, wait some seconds,
> press 'k' key", that is not currently under QEMU.

Why can't you redirect the QEMU monitor to something easily accessible,
eg a virtual serial port, and then use mouse_move and sendkey monitor
commands to drive testing, and screendump to save images for comparison?
Those commands have been in the monitor for years, so is there something
I'm not understanding here?

> The best solution I think is to get a way to send QEMU the screenshot to
> file command some times and stop the VM when both are equal, then send the
> last took screenshot to a mail address so breakability can be checked. (If
> it is the login window, great, it BOOTS!, if not, it doesn't)
> 
> What do you think?

The problem is synchronisation given that QEMU has no guaranteed time
domain, so you don't know if your target is failing or just being slow
so you keep on taking screenshots for a ridiculously long time.  You can
take fewer screenshots if you have some out-of-band signal. For example
knowing to expect a particular screenshot around the time that a
particular file is touched or a particular hardware device is
initialised, or a magic QEMU-specific CPU instruction is executed.

> I have enough spare space to hold the boot images (600Gibibytes free), and a
> machine that should be able to automatically checkout and compile QEMU
> (Athlon XP 2000+, 768Mb RAM, 600GiB free, Gentoo Linux).

Compiling is really important. Even without testing targets, this would
be a very good community service if you want to do even just this much.
Compiling a simulator under itself in all supported configurations is an
important set of tests (compile/run MIPS under IA32, IA32 under PPC32,
etc especially 32/64 mixes, then the different supported compiler
versions, and then a couple of different Linux distributions with their
different choices of library versions and configurations.) This is
probably 24 hours of hard work. I have seen simulators choke on these
sorts of tests.

> Just, I have not the knowledge to make the script that boots qemu, says qemu
> to take the screenshot, compares the took one with the last one, stops qemu,
> sends the last screenshot by email, compresses all took screenshots, goes to
> next VM, so on. (Preferibly without X11 running, as the machine is a mostly
> headless P2P and File server)

I think this can be simplified. Care to send a copy offlist of what
you've done and i'll have a look at it?

--
Dan Shearer
[EMAIL PROTECTED]

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Pierre d'Herbemont]

On 8 avr. 07, at 22:43, Natalia Portillo wrote:

But that required some macro interface "click at x,y, wait some  
seconds,

press 'k' key", that is not currently under QEMU.


There is cxtest (GPL) which has been created for that matter, it  
should be interesting to give it a try:

http://www.cxtest.org/

"Cxtest is an open source project that provides visual regression  
testing facilities for X11 based systems. It can automate basic X  
Window functionality, including finding X windows by title or by  
graphic picture"



Pierre.

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Jamie Lokier]

Natalia Portillo wrote:
> But that required some macro interface "click at x,y, wait some seconds,
> press 'k' key", that is not currently under QEMU.
> 
> The best solution I think is to get a way to send QEMU the screenshot to
> file command some times and stop the VM when both are equal, then send the
> last took screenshot to a mail address so breakability can be checked. (If
> it is the login window, great, it BOOTS!, if not, it doesn't)

If there was a way to get QEMU to stop when the screen matches a
particular stored screenshot, that would make it a lot more scriptable.

For example, you could record the steps to install a particular OS
from installation media as a series of screenshots showing dialog
boxes, and script commands like pressing keys or clicking at x, y when
those screens match.

Another useful event to stop on would be when the processor hits an
x86 HLT instruction or whatever is the equivalent on other
architectures, or particular recognised waiting loops - in conjunction
with there being no outstanding I/O.  Maybe that's an appropriate time
to check for matching screenshots.

That kind of script would be nice when you want to create a new fresh
installation of some OS, but with a few options changed.  The script
would not be too hard to stop at a certain point, and set different
option in the installation, then continue.  Currently I do this
manually and it's quite tedious.

Similarly for actually running applications in a VM.

-- Jamie

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Rob Landley]

On Sunday 08 April 2007 4:43 pm, Natalia Portillo wrote:
> Hi all,
> 
> I have a huge list of operating systems (both closed and open source, that
> works and that doesn't work under QEMU) that can be used to check that QEMU
> doesn't broke (or even, that it corrects a non-working state).

I vaguely intend to get Firmware Linux to rebuild itself under qemu.  Right 
now it's building target images for nine different platforms.  (Although 
that's not as impressive as it sounds.  Two of them, sparc and powerpc, don't 
actually run under qemu yet.  i586 and armv5l are trivial variants of i686 
and armv4l.  And and mips big endian just got added today and wasn't in the 
last release.)

Anyway, I've found compiling large software packages to be a reasonable stress 
test for most systems, and it's entirely scriptable.  Not much of a test of 
any of the peripheral hardware, though.

Rob
-- 
Penguicon 5.0 Apr 20-22, Linux Expo/SF Convention.  Bruce Schneier, Christine 
Peterson, Steve Jackson, Randy Milholland, Elizabeth Bear, Charlie Stross...

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Eduardo Felipe]

Sorry, I'm afraid I badly mixed up things in my mind.

vncrec and pyvnc2swf record VNC server responses, not client actions, so
they will hardly be of any help.

To record and replay client actions these tools are more appropriate:

http://cyberelk.net/tim/rfbproxy/
http://cyberelk.net/tim/rfbplaymacro/

Although they seem not to be available at this moment...

Regards,
Eduardo

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Natalia Portillo]

It doesn't fullfil the requirements for an automated testing,  
however, it is a good idea, but not inside the plannings I proposed.


El 08/04/2007, a las 23:07, Eduardo Felipe escribió:



But that required some macro interface "click at x,y, wait some  
seconds,

press 'k' key", that is not currently under QEMU.

It does exist... it's called VNC server. Maybe you can have a look at:

http://www.sodan.org/~penny/vncrec/

or

http://suif.stanford.edu/vncplay/freenix05-html/

I don't know any of them, but they can be useful.

Also this one:

http://www.unixuser.org/~euske/vnc2swf/pyvnc2swf.html

It's vncrec compatible and can record and replay sessions.

The best solution I think is to get a way to send QEMU the  
screenshot to
file command some times and stop the VM when both are equal, then  
send the
last took screenshot to a mail address so breakability can be  
checked. (If

it is the login window, great, it BOOTS!, if not, it doesn't)


pyvnc2swf has the ability to create image files from VNC sessions.  
It is easily customizable and a special instruction could be  
recorded into the .vnc files meaning:


1) Request a full screen update
2) Turn screen buffer into an image
3) Compare image to reference

Control points could be chosen in recording time and inserted into  
the .vnc by some key combination.


Regards,
Eduardo

Re: QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Eduardo Felipe]

But that required some macro interface "click at x,y, wait some seconds,
press 'k' key", that is not currently under QEMU.



It does exist... it's called VNC server. Maybe you can have a look at:

http://www.sodan.org/~penny/vncrec/ 

or

http://suif.stanford.edu/vncplay/freenix05-html/

I don't know any of them, but they can be useful.

Also this one:

http://www.unixuser.org/~euske/vnc2swf/pyvnc2swf.html


It's vncrec compatible and can record and replay sessions.

The best solution I think is to get a way to send QEMU the screenshot to

file command some times and stop the VM when both are equal, then send the
last took screenshot to a mail address so breakability can be checked. (If
it is the login window, great, it BOOTS!, if not, it doesn't)



pyvnc2swf has the ability to create image files from VNC sessions. It is
easily customizable and a special instruction could be recorded into the
.vnc files meaning:

1) Request a full screen update
2) Turn screen buffer into an image
3) Compare image to reference

Control points could be chosen in recording time and inserted into the .vnc
by some key combination.

Regards,
Eduardo

QEMU Automated Testing (was [Qemu-devel] qemu Makefile.target vl.h hw/acpi.c hw/adlib.c ...)

[Natalia Portillo]

Hi all,

I have a huge list of operating systems (both closed and open source, that
works and that doesn't work under QEMU) that can be used to check that QEMU
doesn't broke (or even, that it corrects a non-working state).

I've already discussed that with Fabrice.

And I think it is better to check an installed image that to test it only
boots or installs (faster at least).

Fabrice and me discussed about taking screenshot per some seconds to compare
with known took screenshots and if something fails shutdown the VM and send
an email.

But that required some macro interface "click at x,y, wait some seconds,
press 'k' key", that is not currently under QEMU.

The best solution I think is to get a way to send QEMU the screenshot to
file command some times and stop the VM when both are equal, then send the
last took screenshot to a mail address so breakability can be checked. (If
it is the login window, great, it BOOTS!, if not, it doesn't)

What do you think?

I have enough spare space to hold the boot images (600Gibibytes free), and a
machine that should be able to automatically checkout and compile QEMU
(Athlon XP 2000+, 768Mb RAM, 600GiB free, Gentoo Linux).

Just, I have not the knowledge to make the script that boots qemu, says qemu
to take the screenshot, compares the took one with the last one, stops qemu,
sends the last screenshot by email, compresses all took screenshots, goes to
next VM, so on. (Preferibly without X11 running, as the machine is a mostly
headless P2P and File server)

If anyone can do this, I will make the boot images for all my OSes and start
it.

I think, it is a great idea (and a better way to update my -currently dead
but no more spammed- official OS support list).

Regards,
Natalia Portillo