Re: [PATCH v2] tests/avocado: re-factor igb test to avoid timeouts

2023-03-23 Thread Alex Bennée 


Akihiko Odaki  writes:

> On 2023/03/22 23:55, Alex Bennée wrote:
>> The core of the test was utilising "ethtool -t eth1 offline" to run
>> through a test sequence. For reasons unknown the test hangs under some
>> configurations of the build on centos8-stream. Fundamentally running
>> the old fedora-31 cloud-init is just too much for something that is
>> directed at testing one device. So we:
>>- replace fedora with a custom kernel + buildroot rootfs
>>- rename the test from IGB to NetDevEthtool
>>- re-factor the common code, add (currently skipped) tests for other
>>   devices which support ethtool
>>- remove the KVM limitation as its fast enough to run in KVM or TCG
>> Signed-off-by: Alex Bennée 
>> Reviewed-by: Philippe Mathieu-Daudé 
>> Cc: Akihiko Odaki 
>> ---
>> v2
>>- use squashfs instead of largely empty ext4 device
>>- use read-only cdrom
>>- don't bother with login favour of direct call from init
>>- kill VM once test is passed
>>- add explicit kvm option
>
> Why did you add explicit kvm option? Is there something not likely
> covered with TCG?

I realised it was the case that the previous igb tested so I added for
completeness. What I really wanted to do was to make the test agnostic
so it would use KVM when available and fall back to TCG when it
couldn't. However my attempt to specify --accel kvm,tcg didn't work.

But yes I doubt there is much coverage difference between the two -
certainly in the emulation side.

>
> Regards,
> Akihiko Odaki
>
>>- add tags for device type
>> ---
>>   tests/avocado/igb.py|  38 ---
>>   tests/avocado/netdev-ethtool.py | 116 
>>   2 files changed, 116 insertions(+), 38 deletions(-)
>>   delete mode 100644 tests/avocado/igb.py
>>   create mode 100644 tests/avocado/netdev-ethtool.py
>> diff --git a/tests/avocado/igb.py b/tests/avocado/igb.py
>> deleted file mode 100644
>> index abf5dfa07f..00
>> --- a/tests/avocado/igb.py
>> +++ /dev/null
>> @@ -1,38 +0,0 @@
>> -# SPDX-License-Identifier: GPL-2.0-or-later
>> -# ethtool tests for igb registers, interrupts, etc
>> -
>> -from avocado_qemu import LinuxTest
>> -
>> -class IGB(LinuxTest):
>> -"""
>> -:avocado: tags=accel:kvm
>> -:avocado: tags=arch:x86_64
>> -:avocado: tags=distro:fedora
>> -:avocado: tags=distro_version:31
>> -:avocado: tags=machine:q35
>> -"""
>> -
>> -timeout = 180
>> -
>> -def test(self):
>> -self.require_accelerator('kvm')
>> -kernel_url = self.distro.pxeboot_url + 'vmlinuz'
>> -kernel_hash = '5b6f6876e1b5bda314f93893271da0d5777b1f3c'
>> -kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
>> -initrd_url = self.distro.pxeboot_url + 'initrd.img'
>> -initrd_hash = 'dd0340a1b39bd28f88532babd4581c67649ec5b1'
>> -initrd_path = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
>> -
>> -# Ideally we want to test MSI as well, but it is blocked by a bug
>> -# fixed with:
>> -# 
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=28e96556baca7056d11d9fb3cdd0aba4483e00d8
>> -kernel_params = self.distro.default_kernel_params + ' pci=nomsi'
>> -
>> -self.vm.add_args('-kernel', kernel_path,
>> - '-initrd', initrd_path,
>> - '-append', kernel_params,
>> - '-accel', 'kvm',
>> - '-device', 'igb')
>> -self.launch_and_wait()
>> -self.ssh_command('dnf -y install ethtool')
>> -self.ssh_command('ethtool -t eth1 offline')
>> diff --git a/tests/avocado/netdev-ethtool.py 
>> b/tests/avocado/netdev-ethtool.py
>> new file mode 100644
>> index 00..f7e9464184
>> --- /dev/null
>> +++ b/tests/avocado/netdev-ethtool.py
>> @@ -0,0 +1,116 @@
>> +# ethtool tests for emulated network devices
>> +#
>> +# This test leverages ethtool's --test sequence to validate network
>> +# device behaviour.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-or-late
>> +
>> +from avocado import skip
>> +from avocado_qemu import QemuSystemTest
>> +from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
>> +from avocado_qemu import wait_for_console_pattern
>> +
>> +class NetDevEthtool(QemuSystemTest):
>> +"""
>> +:avocado: tags=arch:x86_64
>> +:avocado: tags=machine:q35
>> +"""
>> +
>> +# Runs in about 17s under KVM, 19s under TCG, 25s under GCOV
>> +timeout = 45
>> +
>> +# Fetch assets from the netdev-ethtool subdir of my shared test
>> +# images directory on fileserver.linaro.org.
>> +def get_asset(self, name, sha1):
>> +base_url = ('https://fileserver.linaro.org/s/'
>> +'kE4nCFLdQcoBF9t/download?'
>> +'path=%2Fnetdev-ethtool=' )
>> +url = base_url + name
>> +# use explicit name rather than failing to neatly parse the
>> +# URL

Re: [PATCH v2] tests/avocado: re-factor igb test to avoid timeouts

2023-03-22 Thread Akihiko Odaki 

On 2023/03/22 23:55, Alex Bennée wrote:

The core of the test was utilising "ethtool -t eth1 offline" to run
through a test sequence. For reasons unknown the test hangs under some
configurations of the build on centos8-stream. Fundamentally running
the old fedora-31 cloud-init is just too much for something that is
directed at testing one device. So we:

   - replace fedora with a custom kernel + buildroot rootfs
   - rename the test from IGB to NetDevEthtool
   - re-factor the common code, add (currently skipped) tests for other
  devices which support ethtool
   - remove the KVM limitation as its fast enough to run in KVM or TCG

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Cc: Akihiko Odaki 

---
v2
   - use squashfs instead of largely empty ext4 device
   - use read-only cdrom
   - don't bother with login favour of direct call from init
   - kill VM once test is passed
   - add explicit kvm option


Why did you add explicit kvm option? Is there something not likely 
covered with TCG?


Regards,
Akihiko Odaki


   - add tags for device type
---
  tests/avocado/igb.py|  38 ---
  tests/avocado/netdev-ethtool.py | 116 
  2 files changed, 116 insertions(+), 38 deletions(-)
  delete mode 100644 tests/avocado/igb.py
  create mode 100644 tests/avocado/netdev-ethtool.py

diff --git a/tests/avocado/igb.py b/tests/avocado/igb.py
deleted file mode 100644
index abf5dfa07f..00
--- a/tests/avocado/igb.py
+++ /dev/null
@@ -1,38 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-# ethtool tests for igb registers, interrupts, etc
-
-from avocado_qemu import LinuxTest
-
-class IGB(LinuxTest):
-"""
-:avocado: tags=accel:kvm
-:avocado: tags=arch:x86_64
-:avocado: tags=distro:fedora
-:avocado: tags=distro_version:31
-:avocado: tags=machine:q35
-"""
-
-timeout = 180
-
-def test(self):
-self.require_accelerator('kvm')
-kernel_url = self.distro.pxeboot_url + 'vmlinuz'
-kernel_hash = '5b6f6876e1b5bda314f93893271da0d5777b1f3c'
-kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
-initrd_url = self.distro.pxeboot_url + 'initrd.img'
-initrd_hash = 'dd0340a1b39bd28f88532babd4581c67649ec5b1'
-initrd_path = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
-
-# Ideally we want to test MSI as well, but it is blocked by a bug
-# fixed with:
-# 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=28e96556baca7056d11d9fb3cdd0aba4483e00d8
-kernel_params = self.distro.default_kernel_params + ' pci=nomsi'
-
-self.vm.add_args('-kernel', kernel_path,
- '-initrd', initrd_path,
- '-append', kernel_params,
- '-accel', 'kvm',
- '-device', 'igb')
-self.launch_and_wait()
-self.ssh_command('dnf -y install ethtool')
-self.ssh_command('ethtool -t eth1 offline')
diff --git a/tests/avocado/netdev-ethtool.py b/tests/avocado/netdev-ethtool.py
new file mode 100644
index 00..f7e9464184
--- /dev/null
+++ b/tests/avocado/netdev-ethtool.py
@@ -0,0 +1,116 @@
+# ethtool tests for emulated network devices
+#
+# This test leverages ethtool's --test sequence to validate network
+# device behaviour.
+#
+# SPDX-License-Identifier: GPL-2.0-or-late
+
+from avocado import skip
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
+from avocado_qemu import wait_for_console_pattern
+
+class NetDevEthtool(QemuSystemTest):
+"""
+:avocado: tags=arch:x86_64
+:avocado: tags=machine:q35
+"""
+
+# Runs in about 17s under KVM, 19s under TCG, 25s under GCOV
+timeout = 45
+
+# Fetch assets from the netdev-ethtool subdir of my shared test
+# images directory on fileserver.linaro.org.
+def get_asset(self, name, sha1):
+base_url = ('https://fileserver.linaro.org/s/'
+'kE4nCFLdQcoBF9t/download?'
+'path=%2Fnetdev-ethtool=' )
+url = base_url + name
+# use explicit name rather than failing to neatly parse the
+# URL into a unique one
+return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+
+def common_test_code(self, netdev, extra_args=None, kvm=False):
+
+# This custom kernel has drivers for all the supported network
+# devices we can emulate in QEMU
+kernel = self.get_asset("bzImage",
+"33469d7802732d5815226166581442395cb289e2")
+
+rootfs = self.get_asset("rootfs.squashfs",
+"9793cea7021414ae844bda51f558bd6565b50cdc")
+
+append = 'printk.time=0 console=ttyS0 '
+append += 'root=/dev/sr0 rootfstype=squashfs '
+
+# any additional kernel tweaks for the test
+if extra_args:
+

[PATCH v2] tests/avocado: re-factor igb test to avoid timeouts

2023-03-22 Thread Alex Bennée 
The core of the test was utilising "ethtool -t eth1 offline" to run
through a test sequence. For reasons unknown the test hangs under some
configurations of the build on centos8-stream. Fundamentally running
the old fedora-31 cloud-init is just too much for something that is
directed at testing one device. So we:

  - replace fedora with a custom kernel + buildroot rootfs
  - rename the test from IGB to NetDevEthtool
  - re-factor the common code, add (currently skipped) tests for other
 devices which support ethtool
  - remove the KVM limitation as its fast enough to run in KVM or TCG

Signed-off-by: Alex Bennée 
Reviewed-by: Philippe Mathieu-Daudé 
Cc: Akihiko Odaki 

---
v2
  - use squashfs instead of largely empty ext4 device
  - use read-only cdrom
  - don't bother with login favour of direct call from init
  - kill VM once test is passed
  - add explicit kvm option
  - add tags for device type
---
 tests/avocado/igb.py|  38 ---
 tests/avocado/netdev-ethtool.py | 116 
 2 files changed, 116 insertions(+), 38 deletions(-)
 delete mode 100644 tests/avocado/igb.py
 create mode 100644 tests/avocado/netdev-ethtool.py

diff --git a/tests/avocado/igb.py b/tests/avocado/igb.py
deleted file mode 100644
index abf5dfa07f..00
--- a/tests/avocado/igb.py
+++ /dev/null
@@ -1,38 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-# ethtool tests for igb registers, interrupts, etc
-
-from avocado_qemu import LinuxTest
-
-class IGB(LinuxTest):
-"""
-:avocado: tags=accel:kvm
-:avocado: tags=arch:x86_64
-:avocado: tags=distro:fedora
-:avocado: tags=distro_version:31
-:avocado: tags=machine:q35
-"""
-
-timeout = 180
-
-def test(self):
-self.require_accelerator('kvm')
-kernel_url = self.distro.pxeboot_url + 'vmlinuz'
-kernel_hash = '5b6f6876e1b5bda314f93893271da0d5777b1f3c'
-kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
-initrd_url = self.distro.pxeboot_url + 'initrd.img'
-initrd_hash = 'dd0340a1b39bd28f88532babd4581c67649ec5b1'
-initrd_path = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
-
-# Ideally we want to test MSI as well, but it is blocked by a bug
-# fixed with:
-# 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=28e96556baca7056d11d9fb3cdd0aba4483e00d8
-kernel_params = self.distro.default_kernel_params + ' pci=nomsi'
-
-self.vm.add_args('-kernel', kernel_path,
- '-initrd', initrd_path,
- '-append', kernel_params,
- '-accel', 'kvm',
- '-device', 'igb')
-self.launch_and_wait()
-self.ssh_command('dnf -y install ethtool')
-self.ssh_command('ethtool -t eth1 offline')
diff --git a/tests/avocado/netdev-ethtool.py b/tests/avocado/netdev-ethtool.py
new file mode 100644
index 00..f7e9464184
--- /dev/null
+++ b/tests/avocado/netdev-ethtool.py
@@ -0,0 +1,116 @@
+# ethtool tests for emulated network devices
+#
+# This test leverages ethtool's --test sequence to validate network
+# device behaviour.
+#
+# SPDX-License-Identifier: GPL-2.0-or-late
+
+from avocado import skip
+from avocado_qemu import QemuSystemTest
+from avocado_qemu import exec_command, exec_command_and_wait_for_pattern
+from avocado_qemu import wait_for_console_pattern
+
+class NetDevEthtool(QemuSystemTest):
+"""
+:avocado: tags=arch:x86_64
+:avocado: tags=machine:q35
+"""
+
+# Runs in about 17s under KVM, 19s under TCG, 25s under GCOV
+timeout = 45
+
+# Fetch assets from the netdev-ethtool subdir of my shared test
+# images directory on fileserver.linaro.org.
+def get_asset(self, name, sha1):
+base_url = ('https://fileserver.linaro.org/s/'
+'kE4nCFLdQcoBF9t/download?'
+'path=%2Fnetdev-ethtool=' )
+url = base_url + name
+# use explicit name rather than failing to neatly parse the
+# URL into a unique one
+return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+
+def common_test_code(self, netdev, extra_args=None, kvm=False):
+
+# This custom kernel has drivers for all the supported network
+# devices we can emulate in QEMU
+kernel = self.get_asset("bzImage",
+"33469d7802732d5815226166581442395cb289e2")
+
+rootfs = self.get_asset("rootfs.squashfs",
+"9793cea7021414ae844bda51f558bd6565b50cdc")
+
+append = 'printk.time=0 console=ttyS0 '
+append += 'root=/dev/sr0 rootfstype=squashfs '
+
+# any additional kernel tweaks for the test
+if extra_args:
+append += extra_args
+
+# finally invoke ethtool directly
+append += ' init=/usr/sbin/ethtool -- -t eth1 offline'
+
+# add the rootfs via a readonly