On Fri, Jul 02, 2021 at 11:14:00PM +0300, Alper Nebi Yasak wrote: > On 02/07/2021 23:03, Alper Nebi Yasak wrote: > > On 02/07/2021 22:01, Tom Rini wrote: > >> Hey all, > >> > >> I started taking a look at moving to guestfish to see if this resolves > >> the latest problem I've run in to: > >> https://source.denx.de/u-boot/u-boot/-/jobs/284763#L307 > >> which I think is due to guestmount not being done in time for the test. > > > > That failing test's setup uses virt-make-fs, different from what you're > > changing below. I locally only see that failure for the clang build, and > > it still fails after adding time.sleep(300) after its virt-make-fs > > calls. I don't think it's an issue in the test setup. > > > >> So I started converting things to use guestfish directly: > >> diff --git a/test/py/tests/test_fs/conftest.py > >> b/test/py/tests/test_fs/conftest.py > >> index 7325486cdb1a..e8899cfdd118 100644 > >> --- a/test/py/tests/test_fs/conftest.py > >> +++ b/test/py/tests/test_fs/conftest.py > >> @@ -265,10 +265,10 @@ def fs_obj_basic(request, u_boot_config): > >> fs_ubtype = fstype_to_ubname(fs_type) > >> check_ubconfig(u_boot_config, fs_ubtype) > >> > >> - mount_dir = u_boot_config.persistent_data_dir + '/mnt' > >> + data_dir = u_boot_config.persistent_data_dir + '/data' > >> > >> - small_file = mount_dir + '/' + SMALL_FILE > >> - big_file = mount_dir + '/' + BIG_FILE > >> + small_file = data_dir + '/' + SMALL_FILE > >> + big_file = data_dir + '/' + BIG_FILE > >> > >> try: > >> > >> @@ -279,26 +279,14 @@ def fs_obj_basic(request, u_boot_config): > >> return > >> > >> try: > >> - check_call('mkdir -p %s' % mount_dir, shell=True) > >> + check_call('mkdir -p %s' % data_dir, shell=True) > >> except CalledProcessError as err: > >> pytest.skip('Preparing mount folder failed for filesystem: ' + > >> fs_type + '. {}'.format(err)) > >> call('rm -f %s' % fs_img, shell=True) > >> return > >> > >> try: > >> - # Mount the image so we can populate it. > >> - mount_fs(fs_type, fs_img, mount_dir) > >> - except CalledProcessError as err: > >> - pytest.skip('Mounting to folder failed for filesystem: ' + > >> fs_type + '. {}'.format(err)) > >> - call('rmdir %s' % mount_dir, shell=True) > >> - call('rm -f %s' % fs_img, shell=True) > >> - return > >> - > >> - try: > >> - # Create a subdirectory. > >> - check_call('mkdir %s/SUBDIR' % mount_dir, shell=True) > >> - > >> - # Create big file in this image. > >> + # Create big file to copy in to the image. > >> # Note that we work only on the start 1MB, couple MBs in the 2GB > >> range > >> # and the last 1 MB of the huge 2.5GB file. > >> # So, just put random values only in those areas. > >> @@ -309,10 +297,14 @@ def fs_obj_basic(request, u_boot_config): > >> check_call('dd if=/dev/urandom of=%s bs=1M count=1 seek=2499' > >> % big_file, shell=True) > >> > >> - # Create a small file in this image. > >> + # Create a small file to copy in to the image. > >> check_call('dd if=/dev/urandom of=%s bs=1M count=1' > >> % small_file, shell=True) > >> > >> + # Copy the files in to the image and add a subdirectory. > >> + # Create a subdirectory. > >> + check_call('guestfish add %s : run : mount /dev/sda / : mkdir > >> /SUBDIR : copy-in %s %s /' > >> + % (fs_img, big_file, small_file), shell=True) > > > > It could be faster to do things within guestfish as much as possible, > > instead of preparing the files outside and copying them in. > > > > Also it looks like python bindings are available as python3-guestfs on > > Debian and Ubuntu, just not on pypi.org. > > > >> # Delete the small file copies which possibly are written as part > >> of a > >> # previous test. > >> # check_call('rm -f "%s.w"' % MB1, shell=True) > >> @@ -357,13 +349,11 @@ def fs_obj_basic(request, u_boot_config): > >> > >> except CalledProcessError as err: > >> pytest.skip('Setup failed for filesystem: ' + fs_type + '. > >> {}'.format(err)) > >> - umount_fs(mount_dir) > >> return > >> else: > >> - umount_fs(mount_dir) > >> yield [fs_ubtype, fs_img, md5val] > >> finally: > >> - call('rmdir %s' % mount_dir, shell=True) > >> + call('rmdir %s' % data_dir, shell=True) > >> call('rm -f %s' % fs_img, shell=True) > >> > >> # > >> > >> The problem here is that a test run went from taking about 5 minutes to > >> taking about 17 minutes. I can reduce this to closer to 15 minutes with > >> LIBGUESTFS_BACKEND=direct and using libguestfs-make-fixed-appliance to > >> make an appliance we reuse. But that's still too long to be usable. > >> I'm hoping someone has some ideas here on how to improve things. > > > > If libguestfs is falling back to slow emulation because /dev/kvm isn't > > available, maybe it's appropriate to check for that and skip the fs tests... > > Also the User-Mode Linux backend might be worth exploring.
Trying to use user-mode-linux under Docker and I'm running in to another round of failures and annoyances. This might be easier with a Debian rather than Ubuntu based container, so I'm not giving up yet, but it's not a trivial switch anyhow. -- Tom
signature.asc
Description: PGP signature