Re: Test failures on macOS 12
On Mon, Aug 08, 2022 at 08:54:14PM +0200, Christophe de Dinechin wrote:
> On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
> > The other issue is in qemuxml2argvtest:
> >
> > error : virCommandWait:2752 : internal error: Child process
> >(/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
> >unexpected fatal signal 6: dyld[8896]: symbol not found in flat
> >namespace '_virQEMUCapsGet'
> > error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
> >find a satisfying vhost-user-gpu
>
> I don’t see this one.
Dang :(
> What I see instead is failures on qemucapabilitiestest.
> The tests seems to depend on a particular install path for the
> qemu-system- binaries:
>
> binary = g_strdup_printf("/usr/bin/qemu-system-%s",
> data->archName);
>
> On macOS, there is something called “system integrity protection”
> which makes it a bit more difficult to add stuff to /usr/bin.
>
> However, I don’t really think that’s the problem.
We definitely shouldn't be looking at the contents of the build
machine's actual /usr/bin directory in our test suite! That'd be a
bug in our mocking machinery.
> I ran some tests with:
>
> VIR_TEST_DEBUG=1
> VIR_TEST_RANGE=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69
> /Users/ddd/Work/libvirt/build/tests/qemucapabilitiestest
>
> The error I get seem to point to ‘hvf’ been unexected in the output:
>
> 15) 6.2.0 (x86_64)...
> In '/Users/ddd/Work/libvirt/tests/qemucapabilitiesdata/caps_6.2.0.x86_64.xml':
> Offset 7557
> Expect [c]
> Actual [hvf'/>
>
> I do not really understand what the test is supposed to do. It seems
> to be comparing with reference files, but where does it get the
> qemu capabilities to compare with from?
The test reads a .replies file from tests/qemucapabilitiesdata/,
parses it as if a QEMU binary had replied that way to our poking it,
figure out what the corresponding emulator capabilities are, format
them to XML and compare the result to the matching .xml file.
> How should I proceed?
I've prepared patches that should address the issue you're seeing:
https://listman.redhat.com/archives/libvir-list/2022-August/233645.html
It'd be great if you could confirm that they work as intended.
--
Andrea Bolognani / Red Hat / Virtualization
Re: Test failures on macOS 12
> On 9 Aug 2022, at 11:50, Andrea Bolognani wrote: > > On Tue, Aug 09, 2022 at 11:34:20AM +0200, Christophe de Dinechin wrote: >> On 9 Aug 2022, at 11:28, Andrea Bolognani wrote: >>> Yeah, this seems to help and the change makes sense to me. >>> >>> I wonder why we didn't run into this much earlier though? As I >>> mentioned, the test runs successfully as-is on macOS 11. Plus, many >>> other tests rely on library injection and yet work okay even without >>> this change. I must admit that this puzzled me a bit too. I spent a bit of time checking for dyld warnings or anything else. One explanation could be if in other cases, the symbols are marked as weak? I did not check that. And I don’t have a macOS 11 machine to compare anymore. >>> >>> Anyway, I'm happy to add my >>> >>> Reviewed-by: Andrea Bolognani >>> >>> to this patch and push it. The authorship information looks a bit >>> funky though, with the two S-o-bs... >> >> I did not know which one you’d prefer (in case there is a policy). >> If I get to choose, assign that to Red Hat (and change the author >> accordingly). >> >> (and I’ll change my libvirt gitconfig accordingly in the future) > > Done. I'll push once CI has passed. > > It would be great if you could use git-publish for future code > submissions: that way patches can be applied locally more > conveniently by the reviewer. I was able to make it work regardless, > it just took a bit more effort :) Ack. https://gitlab.com/c3d/libvirt/-/pipelines/608168172 > > -- > Andrea Bolognani / Red Hat / Virtualization >
Re: Test failures on macOS 12
On Tue, Aug 09, 2022 at 11:34:20AM +0200, Christophe de Dinechin wrote: > On 9 Aug 2022, at 11:28, Andrea Bolognani wrote: > > Yeah, this seems to help and the change makes sense to me. > > > > I wonder why we didn't run into this much earlier though? As I > > mentioned, the test runs successfully as-is on macOS 11. Plus, many > > other tests rely on library injection and yet work okay even without > > this change. > > > > Anyway, I'm happy to add my > > > > Reviewed-by: Andrea Bolognani > > > > to this patch and push it. The authorship information looks a bit > > funky though, with the two S-o-bs... > > I did not know which one you’d prefer (in case there is a policy). > If I get to choose, assign that to Red Hat (and change the author > accordingly). > > (and I’ll change my libvirt gitconfig accordingly in the future) Done. I'll push once CI has passed. It would be great if you could use git-publish for future code submissions: that way patches can be applied locally more conveniently by the reviewer. I was able to make it work regardless, it just took a bit more effort :) -- Andrea Bolognani / Red Hat / Virtualization
Re: Test failures on macOS 12
> On 9 Aug 2022, at 11:28, Andrea Bolognani wrote: > > On Mon, Aug 08, 2022 at 08:19:28PM +0200, Christophe de Dinechin wrote: >> Hi Andrea, >> >> I saw your call to Sergio and Marc-André on IRC, and I thought I would >> spend a few minutes inviestigating. > > Thanks, I appreciate that! > I'm trying to enable CI coverage for macOS 12, but I'm running into a couple of issues that I'm not sure how to handle. Note that the test suite currently passes on macOS 11[1], so these failures have to be a consequence to changes made to macOS that we haven't yet learned how to cope with. The first one is in vircryptotest: Encrypt aes265cbc ... Expected ciphertext doesn't match I've added some debug statements and it looks like the generated data is different every time, which seems like a pretty good indication that virrandommock is not being picked up correctly. This is not the only test program that uses that specific mock though, so I'm not sure what makes it fail when all the others are succeeding. >> >> I believe that the following patch fixes this one: >> >> From: Christophe de Dinechin >> Date: Mon, 8 Aug 2022 20:14:08 +0200 >> Subject: [PATCH] tests: Pass the flat_namespace option to the linker >> >> This fixes vircryptotest on macOS 12 (Monterey). >> >> The test relies on library injection (using DYLD_INSERT_LIBRARIES) >> to replace the normal random functions with functions giving predictable >> results, defined in virrandommock.c. However, using DYLD_INSERT_LIBRARIES >> only works when building with flat namespaces. >> >> Adding the -Wl,-flat_namespace option to the linker fixes the problem. >> The option was already defined in the top-level meson.build, but had been >> forgotten in the test linker arguments. >> >> Signed-off-by: Christophe de Dinechin >> Signed-off-by: Christophe de Dinechin >> --- >> tests/meson.build | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/tests/meson.build b/tests/meson.build >> index bc9d8ccc4c..d6b1bb2bf0 100644 >> --- a/tests/meson.build >> +++ b/tests/meson.build >> @@ -28,6 +28,7 @@ tests_dep = declare_dependency( >> ], >> link_args: ( >> libvirt_export_dynamic >> ++ libvirt_flat_namespace >> + coverage_flags >> ), >> ) >> -- >> 2.37.1 >> >> >> Could you please check? > > Yeah, this seems to help and the change makes sense to me. > > I wonder why we didn't run into this much earlier though? As I > mentioned, the test runs successfully as-is on macOS 11. Plus, many > other tests rely on library injection and yet work okay even without > this change. > > Anyway, I'm happy to add my > > Reviewed-by: Andrea Bolognani > > to this patch and push it. The authorship information looks a bit > funky though, with the two S-o-bs... I did not know which one you’d prefer (in case there is a policy). If I get to choose, assign that to Red Hat (and change the author accordingly). (and I’ll change my libvirt gitconfig accordingly in the future) Thanks, Christophe > Should I drop the @redhat.com > one and only keep the one with your personal email address, matching > the commit authorship information? > > -- > Andrea Bolognani / Red Hat / Virtualization >
Re: Test failures on macOS 12
On Mon, Aug 08, 2022 at 08:19:28PM +0200, Christophe de Dinechin wrote: > Hi Andrea, > > I saw your call to Sergio and Marc-André on IRC, and I thought I would > spend a few minutes inviestigating. Thanks, I appreciate that! > >> I'm trying to enable CI coverage for macOS 12, but I'm running into a > >> couple of issues that I'm not sure how to handle. > >> > >> Note that the test suite currently passes on macOS 11[1], so these > >> failures have to be a consequence to changes made to macOS that we > >> haven't yet learned how to cope with. > >> > >> The first one is in vircryptotest: > >> > >> Encrypt aes265cbc ... Expected ciphertext doesn't match > >> > >> I've added some debug statements and it looks like the generated data > >> is different every time, which seems like a pretty good indication > >> that virrandommock is not being picked up correctly. This is not the > >> only test program that uses that specific mock though, so I'm not > >> sure what makes it fail when all the others are succeeding. > > I believe that the following patch fixes this one: > > From: Christophe de Dinechin > Date: Mon, 8 Aug 2022 20:14:08 +0200 > Subject: [PATCH] tests: Pass the flat_namespace option to the linker > > This fixes vircryptotest on macOS 12 (Monterey). > > The test relies on library injection (using DYLD_INSERT_LIBRARIES) > to replace the normal random functions with functions giving predictable > results, defined in virrandommock.c. However, using DYLD_INSERT_LIBRARIES > only works when building with flat namespaces. > > Adding the -Wl,-flat_namespace option to the linker fixes the problem. > The option was already defined in the top-level meson.build, but had been > forgotten in the test linker arguments. > > Signed-off-by: Christophe de Dinechin > Signed-off-by: Christophe de Dinechin > --- > tests/meson.build | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/tests/meson.build b/tests/meson.build > index bc9d8ccc4c..d6b1bb2bf0 100644 > --- a/tests/meson.build > +++ b/tests/meson.build > @@ -28,6 +28,7 @@ tests_dep = declare_dependency( >], >link_args: ( > libvirt_export_dynamic > ++ libvirt_flat_namespace > + coverage_flags >), > ) > -- > 2.37.1 > > > Could you please check? Yeah, this seems to help and the change makes sense to me. I wonder why we didn't run into this much earlier though? As I mentioned, the test runs successfully as-is on macOS 11. Plus, many other tests rely on library injection and yet work okay even without this change. Anyway, I'm happy to add my Reviewed-by: Andrea Bolognani to this patch and push it. The authorship information looks a bit funky though, with the two S-o-bs... Should I drop the @redhat.com one and only keep the one with your personal email address, matching the commit authorship information? -- Andrea Bolognani / Red Hat / Virtualization
Re: Test failures on macOS 12
> On 10 Jun 2022, at 11:34, Andrea Bolognani wrote:
>
> On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote:
>> I'm trying to enable CI coverage for macOS 12, but I'm running into a
>> couple of issues that I'm not sure how to handle.
>>
>> Note that the test suite currently passes on macOS 11[1], so these
>> failures have to be a consequence to changes made to macOS that we
>> haven't yet learned how to cope with.
>>
>> The first one is in vircryptotest:
>>
>> Encrypt aes265cbc ... Expected ciphertext doesn't match
>>
>> I've added some debug statements and it looks like the generated data
>> is different every time, which seems like a pretty good indication
>> that virrandommock is not being picked up correctly. This is not the
>> only test program that uses that specific mock though, so I'm not
>> sure what makes it fail when all the others are succeeding.
>>
>> The other issue is in qemuxml2argvtest:
>>
>> error : virCommandWait:2752 : internal error: Child process
>>(/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities)
>>unexpected fatal signal 6: dyld[8896]: symbol not found in flat
>>namespace '_virQEMUCapsGet'
>> error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to
>>find a satisfying vhost-user-gpu
I don’t see this one.
What I see instead is failures on qemucapabilitiestest.
The tests seems to depend on a particular install path for the
qemu-system- binaries:
binary = g_strdup_printf("/usr/bin/qemu-system-%s",
data->archName);
On macOS, there is something called “system integrity protection”
which makes it a bit more difficult to add stuff to /usr/bin.
However, I don’t really think that’s the problem. I ran some tests with:
VIR_TEST_DEBUG=1
VIR_TEST_RANGE=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69
/Users/ddd/Work/libvirt/build/tests/qemucapabilitiestest
The error I get seem to point to ‘hvf’ been unexected in the output:
15) 6.2.0 (x86_64)...
In '/Users/ddd/Work/libvirt/tests/qemucapabilitiesdata/caps_6.2.0.x86_64.xml':
Offset 7557
Expect [c]
Actual [hvf'/>
>
>> So the various virFileWrapperAddPrefix() calls that cause the
>> contents of tests/qemuvhostuserdata/ to override the host's own
>> vhostuser configuration are still effective, but for some reason the
>> trivial test-vhost-user-gpu shell script can't be run successfully
>> because an internal libvirt symbol can't be found somehow? Confusing.
>>
>> Roman, does any of this ring a bell? Any chance you could
>> investigate? macOS 12 has been out for a while now so I'd be very
>> keen to have it added to CI.
>
> Roman, any chance you can find some time to investigate the test
> failures documented above? I'm afraid I've reached the limit of my
> ability to debug macOS-specific behavior :(
>
> --
> Andrea Bolognani / Red Hat / Virtualization
>
Re: Test failures on macOS 12
Hi Andrea, I saw your call to Sergio and Marc-André on IRC, and I thought I would spend a few minutes inviestigating. > On 10 Jun 2022, at 11:34, Andrea Bolognani wrote: > > On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote: >> I'm trying to enable CI coverage for macOS 12, but I'm running into a >> couple of issues that I'm not sure how to handle. >> >> Note that the test suite currently passes on macOS 11[1], so these >> failures have to be a consequence to changes made to macOS that we >> haven't yet learned how to cope with. >> >> The first one is in vircryptotest: >> >> Encrypt aes265cbc ... Expected ciphertext doesn't match >> >> I've added some debug statements and it looks like the generated data >> is different every time, which seems like a pretty good indication >> that virrandommock is not being picked up correctly. This is not the >> only test program that uses that specific mock though, so I'm not >> sure what makes it fail when all the others are succeeding. I believe that the following patch fixes this one: From: Christophe de Dinechin Date: Mon, 8 Aug 2022 20:14:08 +0200 Subject: [PATCH] tests: Pass the flat_namespace option to the linker This fixes vircryptotest on macOS 12 (Monterey). The test relies on library injection (using DYLD_INSERT_LIBRARIES) to replace the normal random functions with functions giving predictable results, defined in virrandommock.c. However, using DYLD_INSERT_LIBRARIES only works when building with flat namespaces. Adding the -Wl,-flat_namespace option to the linker fixes the problem. The option was already defined in the top-level meson.build, but had been forgotten in the test linker arguments. Signed-off-by: Christophe de Dinechin Signed-off-by: Christophe de Dinechin --- tests/meson.build | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/meson.build b/tests/meson.build index bc9d8ccc4c..d6b1bb2bf0 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -28,6 +28,7 @@ tests_dep = declare_dependency( ], link_args: ( libvirt_export_dynamic ++ libvirt_flat_namespace + coverage_flags ), ) -- 2.37.1 Could you please check? >> >> The other issue is in qemuxml2argvtest: >> >> error : virCommandWait:2752 : internal error: Child process >>(/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities) >>unexpected fatal signal 6: dyld[8896]: symbol not found in flat >>namespace '_virQEMUCapsGet' >> error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to >>find a satisfying vhost-user-gpu >> >> So the various virFileWrapperAddPrefix() calls that cause the >> contents of tests/qemuvhostuserdata/ to override the host's own >> vhostuser configuration are still effective, but for some reason the >> trivial test-vhost-user-gpu shell script can't be run successfully >> because an internal libvirt symbol can't be found somehow? Confusing. >> >> Roman, does any of this ring a bell? Any chance you could >> investigate? macOS 12 has been out for a while now so I'd be very >> keen to have it added to CI. > > Roman, any chance you can find some time to investigate the test > failures documented above? I'm afraid I've reached the limit of my > ability to debug macOS-specific behavior :( > > -- > Andrea Bolognani / Red Hat / Virtualization >
Re: Test failures on macOS 12
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote: > I'm trying to enable CI coverage for macOS 12, but I'm running into a > couple of issues that I'm not sure how to handle. > > Note that the test suite currently passes on macOS 11[1], so these > failures have to be a consequence to changes made to macOS that we > haven't yet learned how to cope with. > > The first one is in vircryptotest: > > Encrypt aes265cbc ... Expected ciphertext doesn't match > > I've added some debug statements and it looks like the generated data > is different every time, which seems like a pretty good indication > that virrandommock is not being picked up correctly. This is not the > only test program that uses that specific mock though, so I'm not > sure what makes it fail when all the others are succeeding. > > The other issue is in qemuxml2argvtest: > > error : virCommandWait:2752 : internal error: Child process > (/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities) > unexpected fatal signal 6: dyld[8896]: symbol not found in flat > namespace '_virQEMUCapsGet' > error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to > find a satisfying vhost-user-gpu > > So the various virFileWrapperAddPrefix() calls that cause the > contents of tests/qemuvhostuserdata/ to override the host's own > vhostuser configuration are still effective, but for some reason the > trivial test-vhost-user-gpu shell script can't be run successfully > because an internal libvirt symbol can't be found somehow? Confusing. > > Roman, does any of this ring a bell? Any chance you could > investigate? macOS 12 has been out for a while now so I'd be very > keen to have it added to CI. Roman, any chance you can find some time to investigate the test failures documented above? I'm afraid I've reached the limit of my ability to debug macOS-specific behavior :( -- Andrea Bolognani / Red Hat / Virtualization
Re: Test failures on macOS 12
On Fri, May 06, 2022 at 11:10:44AM +0100, Daniel P. Berrangé wrote: > On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote: > > I'm trying to enable CI coverage for macOS 12, but I'm running into a > > couple of issues that I'm not sure how to handle. > > > > Note that the test suite currently passes on macOS 11[1], so these > > failures have to be a consequence to changes made to macOS that we > > haven't yet learned how to cope with. > > Our macOS 11 config askes for 'big-sur-base' and based on logs our > that is getting clang 12.5.1. > > Cirrus CI jobs indicate you can request a 'big-sur-xcode-13' which > gives newer clang. So perhaps try testing that image in our config, > to narrow down whether the problem is macOS 12 vs clang >= 12 Interesting idea! Unfortunately the test suite still passes on macOS 11 even when using Clang 13, so it looks like something really changed at the OS level. https://gitlab.com/abologna/libvirt/-/pipelines/532966833 -- Andrea Bolognani / Red Hat / Virtualization
Re: Test failures on macOS 12
On Fri, May 06, 2022 at 03:00:14AM -0700, Andrea Bolognani wrote: > I'm trying to enable CI coverage for macOS 12, but I'm running into a > couple of issues that I'm not sure how to handle. > > Note that the test suite currently passes on macOS 11[1], so these > failures have to be a consequence to changes made to macOS that we > haven't yet learned how to cope with. Our macOS 11 config askes for 'big-sur-base' and based on logs our that is getting clang 12.5.1. Cirrus CI jobs indicate you can request a 'big-sur-xcode-13' which gives newer clang. So perhaps try testing that image in our config, to narrow down whether the problem is macOS 12 vs clang >= 12 With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Test failures on macOS 12
I'm trying to enable CI coverage for macOS 12, but I'm running into a couple of issues that I'm not sure how to handle. Note that the test suite currently passes on macOS 11[1], so these failures have to be a consequence to changes made to macOS that we haven't yet learned how to cope with. The first one is in vircryptotest: Encrypt aes265cbc ... Expected ciphertext doesn't match I've added some debug statements and it looks like the generated data is different every time, which seems like a pretty good indication that virrandommock is not being picked up correctly. This is not the only test program that uses that specific mock though, so I'm not sure what makes it fail when all the others are succeeding. The other issue is in qemuxml2argvtest: error : virCommandWait:2752 : internal error: Child process (/usr/libexec/qemu/vhost-user/test-vhost-user-gpu --print-capabilities) unexpected fatal signal 6: dyld[8896]: symbol not found in flat namespace '_virQEMUCapsGet' error : qemuVhostUserFillDomainGPU:394 : operation failed: Unable to find a satisfying vhost-user-gpu So the various virFileWrapperAddPrefix() calls that cause the contents of tests/qemuvhostuserdata/ to override the host's own vhostuser configuration are still effective, but for some reason the trivial test-vhost-user-gpu shell script can't be run successfully because an internal libvirt symbol can't be found somehow? Confusing. Roman, does any of this ring a bell? Any chance you could investigate? macOS 12 has been out for a while now so I'd be very keen to have it added to CI. Thanks in advance! [1] https://gitlab.com/libvirt/libvirt/-/jobs/2421455154 -- Andrea Bolognani / Red Hat / Virtualization
