On 12/27/2011 11:37 PM, Anthony Liguori wrote:
On 12/27/2011 04:35 PM, Cleber Rosa wrote:
On 12/26/2011 08:00 PM, Dor Laor wrote:
On 12/26/2011 05:12 PM, Anthony Liguori wrote:
Hi Dor,
Merry Christmas Anthony,
On 12/25/2011 09:19 AM, Dor Laor wrote:
On 12/19/2011 07:13 PM, Anthony Liguori wrote:
Well, I'm still not convinced that a new standalone package should
handle these
cases instead of kvm autotest. I'll be happy to integrate the
tests to
kvm
autotest anyway and the more the merrier but imho it's a duplicate.
I'm sure kvm autotest could be taught to do exactly what qemu-test is
doing. But why does kvm autotest have to do everything? I doubt there
would be much code reuse.
I think it's not a bad thing to have multiple test suites when there
isn't considerable overlap.
I think the main goal of qemu-tests (may be implicit) is to be quick
and simple.
qemu-test doesn't have a main goal. My goal is to improve QEMU's
quality. qemu-test is just a tool to help achieve that goal.
Maybe I've used the wrong wording. I got the feeling that, besides
testing qemu the way you need it, keeping qemu-test simple was really
important. If not, I'd imagine you'd try to re-use something that
already exists (yes, kvm autotest).
That is indeed great, but if one thinks that's all we'll ever going
to need,
that thought is pretty naive.
I don't know who "we" is, but I can tell you that qemu-test is exactly
what *I* need. Consider that I spent a good portion of every single
day testing QEMU with either my own or other people's patches, making
that job easier and more automated is fairly important to me.
"We" is everyone that somehow contributes to QEMU, that is, the QEMU
community. If you're only concerned about what *you* need, then you're
on the right track. If, besides that, you feel it'd be nice to *try to*
concentrate our efforts, then we're all on the same track.
I'm sharing it because I suspect that a lot of other developers have a
similar need.
And it may be true that there's room for both test
suites... or that, as busy developers, we're refusing to deal with
the added
complexity (qemu alone accounts for a lot) and delaying to fix the
fixable. I
believe on the later.
One example: kvm-autotest has a complex configuration file format
with a steep
learning curve, while a test such as qemu-tests/tests/simple-ping.sh
would have
to be tweaked if somehow the kernel detects the first ethernet
interface as em1
(such as recent Fedora systems do). Simple, but not scalable.
I can tell by this comment that you don't actually understand how
qemu-test works. Please take a look at it before jumping to
conclusions about whether it should or shouldn't be part of kvm-autotest.
Hint: qemu-test always uses the same kernel because it builds it as
part of the test suite. The behavior of how a nic test will never
change unless someone explicitly changes the kernel.
I can tell you did not get my point: I'm giving some reasons of why
current kvm autotest is somehow complex, and how qemu-tests gets away
and keeps it simple.
BTW, I did not jumped at any conclusion. I'm just trying to enrich the
discussion, which may end up proving that there's no other way to have
what qemu-tests does.
1) It builds a custom kernel and initramfs based on busybox. This is
fairly important to ensure that we can run tests with no device
pre-requisites.
This can be done easily w/ autotest too.
The Python requirement inside the guest is true *if* we want to run
regular
autotest tests inside the guest (see
autotest/client/virt/tests/autotest.py) and
this accounts for very very little of kvm autotest usage. All other
tests
interact with the monitor directly and with the guest via
ssh/telnet/serial.
qemu-test does not require any specific hardware to be used in the
guest which lets it test a wider variety of scenarios in QEMU. So you
cannot assume there is ssh/telnet/serial available.
I really thought we could rely on, at least, a serial connection. If
not, then indeed the current kvm autotest approach is not compatible
with that test environment. That is not to say that kvm autotest
couldn't incorporate the qemu-tests approach/functionality.
BTW, I just don't like the idea of having lots of functionalities/tests
implemented on two test suites for a single piece of software, unless
proven that there's no way around it. To me, this is the whole point of
this discussion.
So, I see no reason for not using a more expressive language,
I seriously doubt you can build a useful initramfs that contains
python without doing something crazy like livecd-tools does....
You're right. Again, I was thinking we could rely at least on a serial
connection. Can we not?
Actually, kvm-autotest has various layers of abstraction in how QEMU
ends up being launched. As you mention below, those layers are
there to
allow for things like using libvirt.
Indeed the qemu command line parameters gets generated depending on many
configuration parameters. It'd be *really* simple to add a configuration
parameters that overwrites the qemu command with an static one.
But if you're a QEMU developer, you want to have as much control of
the command line as possible. For instance, one of the tests in
qemu-test makes sure to test invocations without -device as this
triggers a different code path (there was a recent regression in this
too). You can't just add arguments to reproduce this behavior.
It goes beyond that, since it also related to the monitor interface
as well.
That's desirable when you're doing "virt testing", but not so
desirably
when you're trying to write specific unit tests against QEMU.
True, one may not need it at all but it's nice that a test for
migration/stress/hotplug will be tested directly w/ qemu and libvirt
w/ the
same effort.
5) The tests execute very quickly, can be run stand alone, and do
not
require root privileges
ditto for kvm auotest. It's possible to configure it w/o root too
which is not a
huge issue.
When I say, "run quickly", I mean, they execute very quickly.
/me too
$ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64
tests/virtio-serial.sh
real 0m4.385s
user 0m1.460s
sys 0m1.860s
That's impressive but it's more of a function of the guest being
used - if
instead of running a full Fedora install, you'll chose your busybox
image w/
-kernel/initrd you'll get a similar result.
I also think so. Maybe kvm-autotest would take a little more time
because of the
different approach we take when communicating with the guest, but I
bet it'd be
irrelevant.
I don't see any reason why everything needs to live in kvm-autotest...
but if you really feel that way, please provide patches that
demonstrate how this would work.
If it's technically viable, I think that having it as part of kvm
autotest, shows that the project is more cohesive, attracts more
contributions, and makes better use of our efforts.
We could argue indefinitely about how things could work, it's much
better to compare how things actually do work :-)
Sure, that's also what I suggested when I mentioned it'd be a "nice
exercise for all of us".
I agree autotest is not perfect but it likes to be such.
If you wish, you can challenge Lucas and Cleber w/ these type of
requirements
and we'll all improve as a result.
Yes, I believe it'd be a nice exercise for all of us.
The only thing I ask is that we bear at least with some of the
complexity that
kvm-autotest inherently holds...
I think there's something of a knee jerk reaction here. The existence
of qemu-test does not take anything away from kvm-autotest. It's just
another tool in our arsenal to achieve our joint goal of making QEMU
(and KVM) higher quality.
You're right, It does not take anything away from kvm autotest today.
But suppose we can prove that kvm autotest can indeed absorve all of
qemu-tests functionalities, it'd be itself a reason for doing so. It'd
avoid finding ourselves with two evolved test tools that do some of the
same things, but are separate implementations.
autotest is made to invoke third party tests so the two tools can
co-exist in a complimentary way.
That's a no brainier and everyone so far agrees on that. The delicate
issue is whether qemu-tests functionality could live in kvm autotest in
a nice way.
Regards,
Anthony Liguori
the last requests we had was to get rid of all
the complexity, while retaining all the other nice characteristics.
Pretty hard,
so I think we failed, or maybe only half-succeeded at it.
Cheers,
Dor
Regards,
Anthony Liguori
6) They are random by nature with the ability to fix the seed in
order
to be used in git-bisect.
I think Gerd had been looking at doing something similar with a
custom
initrd.
I've tried to consider other architectures and had hoped that we
could
commit the vmlinuz and initramfs into git so that it was easy to
test
other architectures without having a full build environment.
Unfortunately, busybox doesn't link statically with glibc and I
can't
see an obvious way to commit binaries while respecting the GPL
since we
need to pull glibc into the initramfs.
I know buildroot exists specifically to deal with this but in my
experience, buildroot is very unreliable and extremely heavy weight
since it rebuilds gcc multiple times in order to bootstrap a ulibc
environment.
Anyway, the code is available at:
http://git.qemu.org/qemu-test.git
See the README for instructions on how to use it.
Regards,
Anthony Liguori