On Tue, Jan 19, 2021 at 4:05 PM Marcin Sobczyk <msobc...@redhat.com> wrote: > > Hi, > > This looks like something wrong with CI agents, in the part which scrubs > chroot we can find: > > [2021-01-19T12:10:44.669Z] Finish: scrub ['chroot'] > [2021-01-19T12:10:44.669Z] Traceback (most recent call last): > [2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 1059, in <module> > [2021-01-19T12:10:44.669Z] exitStatus = main() > [2021-01-19T12:10:44.669Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.669Z] result = func(*args, **kw) > [2021-01-19T12:10:44.669Z] File "/dev/fd/6", line 825, in main > [2021-01-19T12:10:44.669Z] result = run_command(options, args, > config_opts, commands, buildroot, state) > [2021-01-19T12:10:44.669Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.669Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File "/dev/fd/6", line 859, in run_command > [2021-01-19T12:10:44.670Z] commands.scrub(options.scrub) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/backend.py", line 125, in scrub > [2021-01-19T12:10:44.670Z] self.buildroot.delete() > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/buildroot.py", line 869, in > delete > [2021-01-19T12:10:44.670Z] file_util.rmtree(self.basedir, > selinux=self.selinux) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in > rmtree > [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, > exclude=exclude) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in > rmtree > [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, > exclude=exclude) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in > rmtree > [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, > exclude=exclude) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in > rmtree > [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, > exclude=exclude) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 59, in > rmtree > [2021-01-19T12:10:44.670Z] rmtree(fullname, selinux=selinux, > exclude=exclude) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/trace_decorator.py", line > 93, in trace > [2021-01-19T12:10:44.670Z] result = func(*args, **kw) > [2021-01-19T12:10:44.670Z] File > "/usr/lib/python3.6/site-packages/mockbuild/file_util.py", line 68, in > rmtree > [2021-01-19T12:10:44.670Z] os.rmdir(path) > [2021-01-19T12:10:44.670Z] OSError: [Errno 16] Device or resource busy: > '/var/lib/mock/epel-8-x86_64-8e6e57af00668dc4b1252bf1bf4d754c-2645436/root/var/tmp/vdsm-storage/mount.file-512' > [2021-01-19T12:10:44.670Z] Scrub chroot took 10 seconds > > Evgheni, could you please take a look at it? > > Regards, Marcin > > > On 1/19/21 1:49 PM, Nir Soffer wrote: > > We seem to have an issue in the CI, starting this week. > > > > All tests pass, but creating coverage report fail: > > > > + generate_combined_coverage_report > > + pushd tests > > ~/tests ~ > > + pwd > > /home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests > > + ls .coverage-gluster .coverage-lib .coverage-network .coverage-nose > > .coverage-storage .coverage-virt > > .coverage-gluster > > .coverage-lib > > .coverage-network > > .coverage-nose > > .coverage-storage > > .coverage-virt > > + python3 -m coverage combine .coverage-gluster .coverage-lib > > .coverage-network .coverage-nose .coverage-storage .coverage-virt > > No usable data files > > Coverage.py warning: Couldn't read data from > > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-gluster': > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > > 106: invalid start byte > > Coverage.py warning: Couldn't read data from > > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-lib': > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > > 106: invalid start byte > > Coverage.py warning: Couldn't read data from > > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-network': > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > > 106: invalid start byte > > Coverage.py warning: Couldn't read data from > > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-nose': > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > > 106: invalid start byte > > Coverage.py warning: Couldn't read data from > > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-storage': > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > > 106: invalid start byte > > Coverage.py warning: Couldn't read data from > > '/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/tests/.coverage-virt': > > UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position > > 106: invalid start byte > > > > This looks like a bug in coverage, or maybe we use the wrong version > > of this tool? > > > > + teardown > > + res=1 > > + '[' 1 -ne 0 ']' > > + echo '*** err: 1' > > *** err: 1 > > > > Additionally, we have this very strange error: > > > > + teardown_storage > > + python3 tests/storage/userstorage.py teardown > > python3: can't open file 'tests/storage/userstorage.py': [Errno 2] No > > such file or directory > > > > This error is impossible, since tests/storage/userstorage.py is part > > of the source. > > > > Is it possible that some other code has deleted the source while a > > build was running? > > > > + echo 'WARNING: Ingoring error while tearing down user storage' > > WARNING: Ingoring error while tearing down user storage > > > > We were careful not to fail the build because this issue, but it failed > > because > > coverage report could not be created. > > > > There are multiple instance of this issue: > > - > > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26107//artifact/check-patch.tests-py3.el8.x86_64/mock_logs/script/stdout_stderr.log > > - > > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26103//artifact/check-patch.tests-py3.el8.x86_64/mock_logs/script/stdout_stderr.log > > - > > https://jenkins.ovirt.org/job/vdsm_standard-check-patch/26102//artifact/check-patch.tests-py3.el8.x86_64/mock_logs/script/stdout_stderr.log > > > > I think Eyal reported another instance, but I cannot find it now. > > > > 2 weeks ago we updated pytest and pytest-cov to latest version, but we did > > not see this issue in the CI when we tested the change: > > https://gerrit.ovirt.org/c/vdsm/+/112604 > > > > And CI was fine since then, so this may not be related, unless this is > > a regression > > in latest coverage or pytest-cov.
Issue was solved, vdsm CI is green again. Many thanks to Vojta for investigating this! If your vdsm patch fails in the CI, please rebase on master. Cheers, NIr _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WTMSSTPU5BG53DC2BRC554W4DQX45E4A/
