Re: Issue accidentally creating a '--type' defined pool

[Michal Prívozník]

On 9/29/23 11:40, Joshua Blake wrote:
> Hello,
> 
> I seem to have accidentally created a pool called "--type", but it
> doesn't appear in 'virsh pool-list'. See below.
> 
> [root@vh01 /]# virsh pool-define-as --name --type dir --target
> /data/storage/
> Pool --type defined
> 
> [root@vh01 /]# virsh pool-list
>  Name State    Autostart
> --
>  isos active   yes
>  jblake   active   yes
> 
> [root@vh01 /]# virsh pool-destroy '--type'
> error: command 'pool-destroy' doesn't support option --type

You may want to use:

virsh pool-destroy --pool --type


Michal

Re: Help ! libvirt

[Michal Prívozník]

On 9/22/23 10:34, Andrea Bolognani wrote:
> On Fri, Sep 22, 2023 at 06:33:06AM +0100, Bhasker C V wrote:
>> I finally fixed it.
>> The issue seems to be with the tpm-tis/cpu backend  (wonder why it shows up
>> with a different error)
>> For the sake of community, I am attaching the new xml file so that you can
>> do forensics on what changed
>> (I prettymuch did a virt-install --import --print-xml to redo the config
>> keeping everything intact)
> 
> Glad to hear you managed to make it work!
> 
>> I found another issue
>> If I switch the suspend-to-disk enabled="yes"
>> I strangely get an error
>> error: operation failed: Unable to find any firmware to satisfy 'efi'
> 
> I can explain that one.
> 
> suspend-to-disk.enabled=yes requires a firmware image that advertises
> the acpi-s4 feature, and you probably don't have one on your system.
> 
> For example, on my Fedora 38 machine:
> 
>   $ grep acpi-s4 /usr/share/qemu/firmware/*.json
>   $
> 
> Yeah, the error message is not very helpful. Unfortunately, due to
> the way firmware autoselection works, emitting a better one would be
> pretty much impossible :(
> 

You can enable debug logs and then observe why each one FW was rejected
(see those VIR_DEBUG() printings in qemuFirmwareMatchDomain() [1]).

1: 
https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_firmware.c?ref_type=heads#L1174

Michal

Re: Question about encryption and tls

[Michal Prívozník]

On 9/10/23 13:00, Kamil Jońca wrote:
> (Posted few days ago on qemu group but no reactions)
> 
> Do I understand correctly that ssl shoudl be configured independently
> for libvirt and each hypervisor?
> I asked because I configured libvirt connection as
> 
> qemu+tls://bambus.kjonca/system?pkipath=...

This is a libvirt connection. It specifies how libvirt APIs are passed
between client and server.

> 
> (and on bambus in /etc/libvirt/libvirtd.conf) I set
> key_file = ...
> cert_file = ...
> ca_file = ...
> 
> But after connect and lauching (on bambus) vm I tried to snif traffic to
>  bambus:5900 on client) and wireshark was able to detect "VNC"\

This is graphical console connection. It's independent of libvirt
connection (libvirt does not wrap this inside a libvirt connection).
There were some discussions about this IIRC, but the problem is: while
libvirt's RPC has support for streaming data between host and client, it
does not allow multiplexing (i.e. packing multiple streams into a single
channel). That's why it's not used for graphical console, which
typically require more connections. BUT it can be used for "simpler" -
serial consoles (virsh console).

>  protocol (BTW not spice?), so I am confused.

Maybe you configured VNC for your guest?

>  should I configure in  /etc/libvirt/qemu.conf
> 
> spice_tls option and certificates ?

Correct. This is the route you want to go. But since you configured your
guest to use VNC then you want to set vnc_tls* in qemu.conf.

Michal

Re: Does libvirt support intra-host KVM migration?

[Michal Prívozník]

On 8/16/23 22:55, Eric Wheeler wrote:
> Hello all, I'm reposting this to the libvirt-users list:
> 
> I looked around for documentation on intra-host KVM migration but haven't 
> found much.  For example, this could be useful to "migrate" VM to run on 
> an upgraded version of `qemu-kvm` without migrating to a different host 
> and migrating back.
> 
> We tested migrating a VM to the same host on an old version of libvirt
> (el7), and it complained about UUID conflicts if the destination already
> hosts the same VM.
> 
> Changing the name or the UUID in the destination XML during migration
> (`virsh migrate --xml`) gives an error that the UUID or name does not
> match during migration:
> 
> UUID change:
> error: unsupported configuration: Target domain uuid 
> 66a113f4-a101-4db1-8478-49cf088fedb9 does not matchsource 
> b5256b25-e137-45b8-bddb-78545ab55fc4
> 
> Name change:
> error: unsupported configuration: Target domain name 'diskless2' does 
> not match source 'diskless'
> 
> Questions before we do more testing:
> 
> - Do modern versions of libvirt support intra-host migration? 
> - If so, which version?
> - Documentation?
> 
> 
> Any help you can provide would be greatly appreciated!

In general, no. There are plenty of resources that are unique to the
host and migrating within the host would try to reacquire those. For
instance, various sockets, locks, etc.

Having said that, your best choice is to run libvirt inside a container
and then migrate to another container running on the same host. This has
the highest chance of succeeding. But other limitations still apply
(e.g. no PCI passthrough, ideally shared storage, etc.) because from
libvirt's POV it would be migration between two hosts.

Michal

Re: virsh not connecting to libvertd ?

[Michal Prívozník]

On 6/12/23 20:17, Jerry Buburuz wrote:
> 
> Just found my issue.
> 
> After I removed the cephfs mounts it worked!
> 
> I will debug ceph.
> 
> I assumed because I could touch files on mounted cephfs it was working.
> 
> Now virsh list works!

Out of curiosity. Do you perhaps have a storage pool defined over
cephfs? I can see two possible sources for the problem:

1) autostarted storage pool that makes libvirt mount cephfs, or
2) a storage pool defined over a path where cephfs is mounted.

The problem with 1) is obvious (in fact it's not specific to ceph, if it
was NFS/iSCSI and the server wasn't responding then libvirtd would just
hang).

The problem with 2) is that for some types of storage pools ('dir'
typically) libvirt assumes they are always 'running'. And proceeds to
enumerate volumes in that pool (i.e. files under the dir). And if
there's a stale mount point, this might stuck libvirtd. But again, this
is not limited to ceph, any network FS might do this.

Michal

Re: Changing EFI boot order via API

[Michal Prívozník]

On 6/12/23 15:38, Lukas Zapletal wrote:
> Hello,
> 
> I would like to be able to change the boot order of an EFI VM via
> libvirt API. Specifically, I am looking into configuring a VM to boot
> over HTTP UEFI Boot or HTTPS UEFI Boot (also available as HTTP IPv4 in
> the firmware).
> 
> Is this possible? I cannot find anything relevant in the documentation. 
> Thanks.
> 

It is possible. Just use  for individual devices, or:


  


https://libvirt.org/formatdomain.html#specifying-boot-order
https://libvirt.org/formatdomain.html#bios-bootloader

Michal

Re: Invalid value '-1' for 'cpu.max': Invalid argument - a result of?

[Michal Prívozník]

On 5/16/23 14:57, lejeczek wrote:
> hmm, I've decommissioned that test-lab, whichever version is there in
> oVirt stable... I might try again later.

Okay.

> ps. it does not take up much off the underlying hardware - as a VM, a
> few gigs and few cores.
> 

I know. It's not matter of hardware resources, it's matter of my time
resources and willingness to reproduce something only to find it's
already fixed.

Usually, when I'm filing a bug against QEMU, I try to come up with as
minimal reproducer as possible. And the very first step is to reproduce
the bug without libvirt, i.e. via plain command line. Sure, I start with
libvirt generated cmd line but then peel off unnecessary arguments.

This preprocessing then makes it way easier for QEMU developers to
reproduce and fix the issue. And thus increases my chances of getting
the issue fixed.

Michal

Re: Invalid value '-1' for 'cpu.max': Invalid argument - a result of?

[Michal Prívozník]

On 5/15/23 15:35, lejeczek wrote:
> 
> 
> On 15/05/2023 12:06, Michal Prívozník wrote:
>> On 5/14/23 07:35, lejeczek wrote:
>>> Hi guys.
>>>
>>> In hope that an expert read this - what is, can be, the below a
>>> result of?
>>>
>>> 12284 still running (86040)
>>> Invalid value '-1' for 'cpu.max': Invalid argument
>> This looks like a string coming from libvirt. When setting CGroups, from
>> virCgroupSetValueRaw() which seems to be called (transitively) from
>> virCgroupV2SetCpuCfsPeriod().
>>
>>> 12284 still running (86035)
>>> 12284 still running (86030)
>> This string doesn't appear in our code base.
>>
>>> this is a snippet from libvirtd logs which is a consequence of what
>>> ovirt's engine setup is doing.
>>> To troubleshoot ansible playbooks which is what engine setup does, as I
>>> understand it, would be an impossible task for me so I reckoned I should
>>> try this end.
>>>
>> If you don't provide more context from the log I don't think we can help
>> you, sorry. If there isn't more context (which I doubt, because at least
>> common log line prefix was stripped) then set up debug logs, paste them
>> somewhere and provide us with the link.
>>
>> Michal
>>
> I think it might be here or related:
> https://bugzilla.redhat.com/show_bug.cgi?id=2037998

Yep, looks like it's the same bug. It's supposed to be fixed in
libvirt-9.0.0-2 rpm OR libvirt-9.1.0 upstream. What's your libvirt version?

> 
> This should be reproducible easily I'd think as comes from widely - as I
> understand - used VM management platform.
> This is oVirt self hosted engine setup in a kvm-vm for which VM
> bare-metal host is Centos 9 Stream with everything up-to-dayte off the
> distro repos.
> I'd imagine if you/anybody were to try to deploy current-stable oVirt
> node and attempted to deploy hosted engine - will hit this very
> issue/errors.

Unfortunately, I don't have a setup for running oVirt, nor I intent to
create one, sorry.

Michal

Re: ignored ?

[Michal Prívozník]

On 5/15/23 19:07, lejeczek wrote:
> Hi guys.
> 
> I have a domain definition with this:
> ...
>   
>     
>      value='name=opt/com.coreos/config,file=/00-VMs/oshift1node.ign'/>
>   
> 
> 
> at the bottom of xml.
> 'virsh' creates domain a okey but in VM I do not see ignition happened -
> it's fedora coreos - and when I dumpxml running domain, then indeed
> those bits, show above, are not there.
> 
> I'd grateful for any/all thoughts shared on - what is not/happening there.
> Is there something else which is a prerequisite to 'qemu:commandline'
> but if yes and I'm missing those, why would not then 'virsh' and/or
> 'virtqemud' say something?
> 
> many thanks, L.

No need for using  as this is well configurable via
domain XML (for ~3 years):

https://libvirt.org/formatdomain.html#smbios-system-information


  example value
  


Michal

Re: PHP API

[Michal Prívozník]

On 5/11/23 20:19, Simon Fairweather wrote:
> Hi Michal,
> 
> Would you be able to add, if there is a process you follow I can look at
> creating merge requests.
> 
> Regards
> Simon
> 
> int   virDomainBlockPull  (virDomainPtr
> <https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainPtr> dom,
>const char * disk,
>unsigned long bandwidth,
>unsigned int flags)
> 
> 
> int   virDomainBlockCopy  (virDomainPtr
> <https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainPtr> dom,
>const char * disk,
>const char * destxml,
>virTypedParameterPtr
> <https://libvirt.org/html/libvirt-libvirt-common.html#virTypedParameterPtr> 
> params,
>int nparams,
>unsigned int 
> flags)intvirDomainBlockPull (virDomainPtr
> <https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainPtr>dom,
> 
> 
> On Wed, May 3, 2023 at 12:42 PM Michal Prívozník  <mailto:mpriv...@redhat.com>> wrote:
> 
> On 4/30/23 20:29, Simon Fairweather wrote:
> > Hi Michal
> >
> > Are there plans to implement  snapshot create_as version of the
> > following function where XML could be passed?
> 
> Honestly, there are no plans for libvirt-php. I maintain it in my spare
> time and implement features as somebody requests them. Let me implement
> it. And also fix the issue you've filled earlier this week.
> 
> Michal
> 

Posted as:

https://gitlab.com/libvirt/libvirt-php/-/merge_requests/39

Michal

Re: Invalid value '-1' for 'cpu.max': Invalid argument - a result of?

[Michal Prívozník]

On 5/14/23 07:35, lejeczek wrote:
> Hi guys.
> 
> In hope that an expert read this - what is, can be, the below a result of?
> 
> 12284 still running (86040)
> Invalid value '-1' for 'cpu.max': Invalid argument

This looks like a string coming from libvirt. When setting CGroups, from
virCgroupSetValueRaw() which seems to be called (transitively) from
virCgroupV2SetCpuCfsPeriod().

> 12284 still running (86035)
> 12284 still running (86030)

This string doesn't appear in our code base.

> 
> this is a snippet from libvirtd logs which is a consequence of what
> ovirt's engine setup is doing.
> To troubleshoot ansible playbooks which is what engine setup does, as I
> understand it, would be an impossible task for me so I reckoned I should
> try this end.
> 

If you don't provide more context from the log I don't think we can help
you, sorry. If there isn't more context (which I doubt, because at least
common log line prefix was stripped) then set up debug logs, paste them
somewhere and provide us with the link.

Michal

Re: how to get domain name into the logs - ?

[Michal Prívozník]

On 5/8/23 11:02, lejeczek wrote:
> Hi guys.
> 
> I have libvirt/qemu setup with pretty vanilla settings and my logs, a
> snippet, look like this:
> 
> ...
> migration successfully aborted
> internal error: qemu unexpectedly closed the monitor:

This ^^^ is error message produced by libvirt and as such it could
include domain name (I'll post a patch soon), but ...

> 2023-05-08T07:56:37.785886Z qemu-kvm: Failed to load
> virtio_pci/modern_queue_state:desc
> 2023-05-08T07:56:37.786024Z qemu-kvm: Failed to load
> virtio_pci/modern_state:vqs
> 2023-05-08T07:56:37.786029Z qemu-kvm: Failed to load
> virtio/extra_state:extra_state
> 2023-05-08T07:56:37.786033Z qemu-kvm: Failed to load virtio-rng:virtio
> 2023-05-08T07:56:37.786036Z qemu-kvm: error while loading state for
> instance 0x0 of device ':00:02.5:00.0/virtio-rng'
> 2023-05-08T07:56:37.786279Z qemu-kvm: load of migration failed:
> Input/output error

... these come from QEMU itself. And for that we would need to patch
QEMU. I mean, libvirt passes:

  -name guest=${VM_NAME}

onto the cmd line, so it should be possible. But I guess specifying the
name in the first line is good enough, isn't it?

Michal

Re: PHP API

[Michal Prívozník]

On 4/30/23 20:29, Simon Fairweather wrote:
> Hi Michal
> 
> Are there plans to implement  snapshot create_as version of the
> following function where XML could be passed?

Honestly, there are no plans for libvirt-php. I maintain it in my spare
time and implement features as somebody requests them. Let me implement
it. And also fix the issue you've filled earlier this week.

Michal

Re: Option Flags

[Michal Prívozník]

On 3/23/23 10:35, Simon Fairweather wrote:
> Hi 
> 
> Are the flags documented? can this function be used to  specify  same
> as virsh undefine --nvram "name of VM"
> 
> libvirt_domain_undefine_flags($res, $flags)
> 
> [Since version (null)]
> 
> Function is used to undefine(with flags) the domain identified by it's
> resource.
> 
> /@res [resource]/:libvirt domain resource, e.g. from
> libvirt_domain_lookup_by_*()
> /@flags [int]/:   optional flags
> /Returns/:: TRUE for success, FALSE on error
> 
> 
> 

Hey, thanks for the report. I've created a MR that fixes these problems
here:

https://gitlab.com/libvirt/libvirt-php/-/merge_requests/38

Michal

Re: virsh domifaddr --domain domname --source {lease, arp} not showing results with ipv6

[Michal Prívozník]

On 3/21/23 16:50, Natxo Asenjo wrote:
> hi,
> 
> On Tue, Mar 21, 2023 at 3:40 PM Michal Prívozník  <mailto:mpriv...@redhat.com>> wrote:
> 
> On 3/19/23 20:21, Natxo Asenjo wrote:
> > hi,
> >
> > I have configured a routed network on my laptop with a ipv6 subnet and
> > dnsmasq is handing out ipv6 addresses to my vms and it works
> really wel,
> > but finding out which ips have been used is not as easy as with ipv4.
> >
> > [root@lenovo ~]# virsh domifaddr --domain wec --source lease
> >  Name       MAC address          Protocol     Address
> >
> 
> ---
> >
> > [root@lenovo ~]# virsh domifaddr --domain wec --source arp
> >  Name       MAC address          Protocol     Address
> >
> 
> ---
> >
> > When using a ipv4 network, this works:
> >
> > root@lenovo ~]# virsh domifaddr --domain evenng --source arp
> >  Name       MAC address          Protocol     Address
> >
> 
> ---
> >  vnet2      52:54:00:4c:83:98    ipv4         192.168.122.229/0
> <http://192.168.122.229/0>
> > <http://192.168.122.229/0 <http://192.168.122.229/0>>
> >
> > [root@lenovo ~]# virsh domifaddr --domain evenng --source lease
> >  Name       MAC address          Protocol     Address
> >
> 
> ---
> >  vnet2      52:54:00:4c:83:98    ipv4         192.168.122.229/24
> <http://192.168.122.229/24>
> > <http://192.168.122.229/24 <http://192.168.122.229/24>>
> >
> > I can obviously look into the leases file and find out the
> address, but
> > it would be nice to be able to use the virt tooling.
> 
> Looks like you won't find any info for IPv6 there, because that's
> exactly what '--source lease' is doing. Are you sure that your IPv6
> address is not autoconfigured and it indeed is libvirt spawned dnsmasq
> that's assigning the IPv6 address?
> 
> 
> this is the xml:
> 
> # cat networks/ipv6.xml
> 
> 
> 
>   ipv6
>   49957183-e4e4-4541-8746-11be67f0a511
>   
>   
>   
>   
>   
>     
>       
>     
>   
> 
> 
> so it looks like a dhcp server to me ;-)

Yep. That's fine. The only thing missing now is how's the 
configured in domain XML. Because '--source leases' is designed to work
only with . Any other type (e.g.
type='bridge') is ignored.

> 
> I just found out about virsh net-dhcp-leases and that one lists the leases:
> 
> # virsh net-dhcp-leases --network ipv6
>  Expiry Time           MAC address         Protocol   IP address        
>        Hostname          Client ID or DUID
> -
>  2023-03-22 16:35:44   52:54:00:0e:ee:da   ipv6      
> 2a02:xxx:xx:7::1ba/64   WIN-C0HD8RH61MI  
> 00:01:00:01:2b:a7:05:c5:52:54:00:0e:ee:da

And the fact that you can see some leases but virsh doesn't report any
suggests that you may be using type='bridge'.

> 
> 
> For the '--source arp' case - libvirt reads the arp table (effectively
> the same info is exposed under /proc/net/arp file) and finds a matching
> entry there, based on the MAC address. Can you see whether there's a
> match?
> 
> 
> No, that only entry right now there is my router:
> 
> ]# cat /proc/net/arp
> IP address       HW type     Flags       HW address            Mask    
> Device
> xx.x.x.1         0x1         0x2         cafebabecafe     *        wifi0

Yeah, this is why '--source arp' doesn't display anything. And I just
realized that IPv6 doesn't use ARP, so that won't work :-)

> And lastly - there's '--source agent' which is most likely to return
> useful information, but that requires guest agent running inside.
> 
> 
> Indeed, no agent running on the vm, could add it.
> 
> 

If you want to get most consistent answer then that's probably the way
to go.

Michal

Re: virsh domifaddr --domain domname --source {lease, arp} not showing results with ipv6

[Michal Prívozník]

On 3/19/23 20:21, Natxo Asenjo wrote:
> hi,
> 
> I have configured a routed network on my laptop with a ipv6 subnet and
> dnsmasq is handing out ipv6 addresses to my vms and it works really wel,
> but finding out which ips have been used is not as easy as with ipv4.
> 
> [root@lenovo ~]# virsh domifaddr --domain wec --source lease
>  Name       MAC address          Protocol     Address
> ---
> 
> [root@lenovo ~]# virsh domifaddr --domain wec --source arp
>  Name       MAC address          Protocol     Address
> ---
> 
> When using a ipv4 network, this works:
> 
> root@lenovo ~]# virsh domifaddr --domain evenng --source arp
>  Name       MAC address          Protocol     Address
> ---
>  vnet2      52:54:00:4c:83:98    ipv4         192.168.122.229/0
> 
> 
> [root@lenovo ~]# virsh domifaddr --domain evenng --source lease
>  Name       MAC address          Protocol     Address
> ---
>  vnet2      52:54:00:4c:83:98    ipv4         192.168.122.229/24
> 
> 
> I can obviously look into the leases file and find out the address, but
> it would be nice to be able to use the virt tooling.

Looks like you won't find any info for IPv6 there, because that's
exactly what '--source lease' is doing. Are you sure that your IPv6
address is not autoconfigured and it indeed is libvirt spawned dnsmasq
that's assigning the IPv6 address?

For the '--source arp' case - libvirt reads the arp table (effectively
the same info is exposed under /proc/net/arp file) and finds a matching
entry there, based on the MAC address. Can you see whether there's a match?

And lastly - there's '--source agent' which is most likely to return
useful information, but that requires guest agent running inside.

Michal

> 
> This is on a fedora 37 running qemu-kvm-7.0.0-14.fc37.x86_64 and
> dnsmasq-2.89-1.fc37.x86_64, everything intalled from the fedora
> repositories.


So is it safe to assume that libvirt (which is where you're reporting
the issue) is: libvirt-8.6.0-5.fc37 ?

Michal

Re: ACLs problem on /dev/kvm

[Michal Prívozník]

[Please keep the list CCed for benefit of the community]
On 2/27/23 16:58, Sánta, Márton (ext) wrote:
> Dear Michal,
> 
> Thank you for your quick help, this worked for me but I would like to use the 
> latest version if it is possible. Do you know how to upgrade my libvirt 
> version in yocto? I tried to use devtool to upgrade v9.0.0 but there were 
> some issues during the process and I had to do a lot of things manually 
> (downloading the approprieta sources as during the update, the v8.1.0 sources 
> were downloaded etc.) and I do not see what problems can it cause later on. 
> What do you suggest?

Hey, unfortunately I have zero experience with yocto. I believe you need
to contact whoever maintains libvirt there and point them to the fix I
mention in my previous reply. Surely they have a way to backport patches.

Michal

Re: ACLs problem on /dev/kvm

[Michal Prívozník]

On 2/21/23 09:47, Sánta, Márton (ext) wrote:
> Dear All,
> 
>  
> 
> I have started to work with libvirt a few weeks ago, but I have some
> problem with starting a virtual machine. Currently, I am using an
> *embedded arm64 device* with a *Linux* built with *Yocto*. I managed to
> install *lbvirt 8.1.0* in the image and I have a *qemu user* and *qemu
> and kvm groups *in the system. I am using *KVM hypervisor* and I did the
> configuration in the *qemu.conf* and *libvirtd.conf* files, enabled all
> the sockets and services in the system. The xml based definition of the
> virtual machine is simple, but when I try to start it I get the error
> message: *Failed to start domain ’XYZ’* and *Unable to set ACLs on
> /dev/kvm: Invalid argument*. I cannot set ACLs on the /dev/kvm (owner is
> /root/, group is /kvm/ but I have also tried to set it /root:root/) with
> the /setfacl /command, but I gave /rwx/ access to user, group and others
> as well so everybody can use the device. I also uncommented the relevant
> lines in the /qemu.conf/ file (/cgroup_controllers = …/ and
> /cgroup_device_acl = …/) and I also found that /devices/ controller is
> already mounted at //sys/fs/cgroup/devices/. Can anybody help me with
> this issue? Did Anybody have similar problem? I can start a virtual
> machine with *qemu-system-aarch64*, but I would like to use the libvirt
> library to manage the machines.
> 
>  
> 
> Thank you in advance for an early reply.
> 

This is a namespace issue. Basically, when starting a guest (or domain
as we call it), libvirt creates a private /dev for it. It's using mount
namespace to create a private mount table to replace the original /dev,
hence the name of the feature. And this private /dev is populated with
only a handful of nodes (some basic ones, like /dev/zero, /dev/null, ...
and those which are configured in domain XML). Each individual node is
created as an exact copy of the original /dev, including ACL entries. If
you understand C a bit you can see the function that's responsible for
creating the nodes here [1].

Now, there used to be a bug, where libvirt tried to set ACLs even though
the corresponding file had none. It was fixed by the following commit
[2]. unfortunately, the commit is part of newer libvirt than what you
have: v8.8.0.

There is a workaround though: you can disable this namespace feature by
setting the following in /etc/libvirt/qemu.conf:

  namespaces = []


Michal

1: 
https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_namespace.c#L972

2: 
https://gitlab.com/libvirt/libvirt/-/commit/687374959e160dc566bd4b6d43c7bf1beb470c59

Re: Unable to set XATTR trusted.libvirt.security.dac Operation not supported

[Michal Prívozník]

On 2/6/23 10:49, Simon Fairweather wrote:
> Thanks Michal, following your suggestion on FS It looks like a fuse
> 3.13.0 issue, if I use drive mount point it works. Should have checked
> this first.
> 

No problem. Do you have a link to a bug report perhaps? I'd like to
check to see if it's worth changing libvirt so that it does not tackle
this bug.

What worries me is that firstly, libvirt tries to read XATTRs and upon
seeing ENOTSUP it doesn't proceed to setting XATTRs on given file.
Therefore, reading XATTRs must have succeeded but setting them failed.
Was this a read-only FS perhaps?

Michal

Re: Unable to set XATTR trusted.libvirt.security.dac Operation not supported

[Michal Prívozník]

On 2/2/23 20:05, Simon Fairweather wrote:
> Running with Kernel 6.1.8 
> 
> QEMU 7.1 libvirt 8.7
> 
> setting remeber_owner = 0 resolves the issue but was working ok on
> kernel 6.0.x.
> 
> root@computenode:~# cat  /etc/libvirt/qemu.conf | grep reme
> # Whether libvirt should remember and restore the original
> remember_owner = 0
> root@computenode:~# 
> 
> Any known issues for this, is it likely to be QEMU/KVM?

So is there a failure you're seeing (e.g. a domain unable to start) or
is this just a warning printed into logs?

Can you shed more light into what path is libvirt unable to set XATTRs
on and what's the underlying filesystem?

Michal

Re: libvirt error - "cannot set CPU affinity"

[Michal Prívozník]

On 1/31/23 20:17, Dipanjan Das wrote:
> 
> 
> On Tue, 31 Jan 2023 at 22:49, Michal Prívozník  <mailto:mpriv...@redhat.com>> wrote:
> 
> On 1/31/23 13:43, Dipanjan Das wrote:
> > Hello,
> >
> > I am getting the following error message when trying to run a VM with
> > the emulatorpin cpuset set to a value -
> >
> > "cannot set CPU affinity on process 78: Invalid argument".
> >
> > Note that removing the emulatorpin leads to the VM running
> successfully.
> >
> > The core that I am setting for the emulatorpin cpuset is present
> on the
> > system and is not in the isolcpus set of the host.
> 
> The question is, whether the container that libvirtd runs in is allowed
> to run on the selected cpuset. I mean, you can have a cgroup that allows
> a subset of physical CPUs, and when a process running within the cgroup
> tries to set affinity to a CPU that's not allowed it gets EINVAL.
> 
> 
> Makes sense. 
> 
> I am now wondering whether we can check the cgroups on the host or
> inside the containers to confirm that this is the case. Since I am
> setting the emulator pin thread to a known cpu, being able to check
> whether the cpu is in the cpu subset of the cgroup might be a quick way
> to find the problem.

Yeah. I don't know enough about KubeVirt let alone OpenShift to tell you 
ow to do that, sorry. But essentially, you want to check the contents of
cpuset.cpus of the container that libvirtd runs in.

cat $(grep -lr $(pgrep libvirtd) $(mount | grep cpuset | awk '{print $3}') | 
head -n1 | sed 's/cgroup.procs/cpuset.cpus/')

>  
> 
> 
> The function this error message comes from is: virProcessSetAffinity()
> (the linux version)
> 
> 
> https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c#L445 
> <https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c#L445>
> 
> Another reason for the error might be insufficient permissions (though
> the man page of sched_setaffinity() says you'd get an EPERM so maybe
> not).
> 
> >
> > Any suggestions on what could be the problem? Any pointers to how to
> > debug the issue would help greatly too.
> 
> Turning on debug logs is a good start. From there you'd need a trained
> eye to see the code path, but if you post a link to the logs here I'll
> try to help.
> 
> 
> Thank you very much. Should I enable debug logs for any specific
> module(s) or for all of them? 

This should help:

https://libvirt.org/kbase/debuglogs.html

And we're interested in the daemon logs as the daemon is where all the
interesting work takes place.

Michal

Re: libvirt error - "cannot set CPU affinity"

[Michal Prívozník]

On 1/31/23 13:43, Dipanjan Das wrote:
> Hello,
> 
> I am getting the following error message when trying to run a VM with
> the emulatorpin cpuset set to a value -
> 
> "cannot set CPU affinity on process 78: Invalid argument".
> 
> Note that removing the emulatorpin leads to the VM running successfully.
> 
> The core that I am setting for the emulatorpin cpuset is present on the
> system and is not in the isolcpus set of the host.

The question is, whether the container that libvirtd runs in is allowed
to run on the selected cpuset. I mean, you can have a cgroup that allows
a subset of physical CPUs, and when a process running within the cgroup
tries to set affinity to a CPU that's not allowed it gets EINVAL.

The function this error message comes from is: virProcessSetAffinity()
(the linux version)

https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c#L445

Another reason for the error might be insufficient permissions (though
the man page of sched_setaffinity() says you'd get an EPERM so maybe not).

> 
> Any suggestions on what could be the problem? Any pointers to how to
> debug the issue would help greatly too.

Turning on debug logs is a good start. From there you'd need a trained
eye to see the code path, but if you post a link to the logs here I'll
try to help.

Michal

Re: virConnectGetDomainCapabilities

[Michal Prívozník]

On 1/7/23 16:15, Simon Fairweather wrote:
> Hi
> 
> Is there a php api version for this?

There is now:

https://gitlab.com/libvirt/libvirt-php/-/commit/af33dcf01243b81fe1d9b23ab105c1ba157cd9af

Michal

Re: libvirt_connect_get_nic_models

[Michal Prívozník]

On 12/28/22 10:09, Simon Fairweather wrote:
> Hi 
> 
> Can you confirm that I source get an array of these when running this
> api call, currently I just get a return of false
> 

Yeah, this is because as of QEMU's commit [1] (contained in QEMU-4.1.0
release) the format in which list is reported changed. Instead of
everything being on one line separated by commas, now each model is
printed on individual line.

And looking at a function that's related
(libvirt_connect_get_soundhw_models()) I'll need to rework that too,
because in QEMU-7.1 they removed -soundhw argument.

Michal

1: 
https://gitlab.com/qemu-project/qemu/-/commit/7b71e03af98fd7b5e125d7c10f02c804fcc5e5af

Re: libvirt_connect_get_machine_types

[Michal Prívozník]

On 12/7/22 11:27, Simon Fairweather wrote:
> Awesome, thanks! 

Aaand it's done.

https://libvirt.org/sources/php/

Michal

Re: libvirt_connect_get_machine_types

[Michal Prívozník]

On 12/6/22 20:13, Simon Fairweather wrote:
> Thanks for the update. What does this mean?
> 
> The only problem is that QEMU feeds us PPM which is not wired to PHP
> really.
> 
> We are using this patch to remove imageMagik as it is not required for
> our needs.
> Eric - I strip out the requirement for ImageMagick because of all the
> dependancies
> needed for that peice.  ImageMagick is only used for libvirt's
> screenshot API calls
> which we dont even take advantage of in the unRAID web UI.
> --- a/configure.ac   2016-10-05 22:20:39.172968837
> -0500
> +++ b/configure.ac   2016-10-05 22:21:05.645301252
> -0500
> @@ -172,8 +172,6 @@
>  LIBVIRT_CHECK_PHP_EXTENSIONDIR
>  LIBVIRT_CHECK_PHP_CONFDIR
>  
> -LIBVIRT_CHECK_PHP_EXTENSION([imagick])
> -

Yeah, pretty much every distro uses this patch. The problem is, without
imagick the examples won't work:

libvirt-php.git/tests $ php -n -c php.ini -S localhost:8080
../examples/index.php

BUT, here's what I can do. I just found out that QEMU added support for
taking screenshots in PNG format, so I can fix libvirt to prefer that
(PPM is not very well supported anyway - I haven't found a browser that
would render it) and drop imagick completely. Then I can do the release
and distros can then just drop the patch.

Michal

Re: libvirt_connect_get_machine_types

[Michal Prívozník]

On 12/6/22 13:57, Simon Fairweather wrote:
> When will 0.5.7 be released?

Yeah, I should probably do the release soon as there is a lot of
changes. But firstly, I'd like to get rid of imagick dependency as it's
not strictly needed (only used in examples/ for downscaling the
screenshots of running domains). Let me see how quickly I can do that.
The only problem is, that QEMU feeds us PPM which is not wired to PHP
really.

Michal

Re: libvirt_connect_get_machine_types

[Michal Prívozník]

On 12/6/22 09:41, Simon Fairweather wrote:
> Hi Michal,
> 
> There is an issue with keys generated in the function adding null to the
> end of each key, I have created an issue with a patch from Eric at LimeTech.
> 
> https://gitlab.com/libvirt/libvirt-php/-/issues/7
> 
> 
> my testing.

Yep, this is a genuine bug. I've merged the fix now.

Michal

Re: libvirt_connect_get_machine_types

[Michal Prívozník]

On 12/4/22 18:18, Simon Fairweather wrote:
> Using the following
> 
> php: version 8.1.13
> php-libvirt: version 0.5.6 (build 2)
> 
> libvirt 8.7.0
> QEMU 7.1.0
> 
> virsh capabilities works fine.
> 


> 
> Any known issues with php 8 for this function, others seem to be working
> fine.

Hey, I haven't tried php-8 yet. What is the problem you're seeing?
Looking into the source code, the only thing that
libvirt_connect_get_machine_types() does is it fetches capabilities
(libvirt_connect_get_capabilities()) and then runs couple of XPATHs to
construct an associative array of supported machine types. The same can
be achieved in plain PHP.

Michal

Re: domain_qemu_agent_command($domain,$cmd,$timeout,$flags)

[Michal Prívozník]

On 11/16/22 09:07, Simon Fairweather wrote:
> Thanks, But is is not in the list here.
> 
> https://libvirt.org/php/api-reference.html
> 

Ah, fixed now. Thanks for the report.

Michal

Re: domain_qemu_agent_command($domain,$cmd,$timeout,$flags)

[Michal Prívozník]

On 11/15/22 18:21, Simon Fairweather wrote:
> libvirt_domain_interface_addresses()  this doesn't seem to be in the php
> api reference.

I'm not sure what you mean. It was added ~3 years ago:

https://gitlab.com/libvirt/libvirt-php/-/commit/97a86d8428b5d189a0dd09c86be8c7cfee05a463

It's in libvirt-php-0.5.5.

Michal

Re: domain_qemu_agent_command($domain,$cmd,$timeout,$flags)

[Michal Prívozník]

On 11/15/22 14:44, Simon Fairweather wrote:
> Hi Michal,
> 
> Can you advise why the following warning is logged for this command:
> domain_qemu_agent_command($res,
> '{"execute":"guest-network-get-interfaces"}', 10, 0) ;
> 
> warning : qemuDomainObjTaintMsg:6464 : Domain id=1 name='HA'
> uuid=ffbc955e-9a51-f1d1-2501-b4cbdac5dd4c is tainted: custom-ga-command

Because using qemu-ga can change the state behind libvirt's back. So far
we've ignored tainted flag but the idea is to be similar to kernel and
its tainting.

In this specific case - libvirt offers a this particular command as an API:

libvirt_domain_interface_addresses() in PHP, or
virDomainInterfaceAddresses() in C.

Michal

Re: domain_qemu_agent_command($domain,$cmd,$timeout,$flags)

[Michal Prívozník]

On 10/25/22 20:04, Simon Fairweather wrote:
> Should there be a way to suppress errors using the flags?
> 
>  error : qemuDomainAgentAvailable:8411 : Guest agent is not responding:
> QEMU guest agent is not connected
> 
> 
> Or is there a libvirt function to check connection status or another way?

This error message is "harmless" in a sense that there is no split brain
of states between libvirt and QEMU. But yeah, your API is going to fail.

You can check for the agent status in the domain XML, for instance with
the following (rewrite in your favorite language):

virsh dumpxml fedora | xpath -e 
"string(//channel/target[@name='org.qemu.guest_agent.0']/@state)"

Michal

Re: Virtual Networking

[Michal Prívozník]

On 10/25/22 17:03, Armin Lepir wrote:
> Hi 
> 
> I am using libvrt for the first time. Im building a KVM for multiple
> Virtual OS instances.
> 
> The problem i have is with your official documentation for Virtual
> Networking.
> 
> https://wiki.libvirt.org/page/VirtualNetworking
> 
> 

Please note that our wiki page is obsolete and we tried to move
everything into our knowledge base articles:

  https://libvirt.org/kbase/

We're keeping the wiki around though, because maybe not everything was
moved.

> 
> The following is wrong:
> 
> The default mode is BRIDGE + NAT.
> 
> Optional mode is ROUTING.
> 
> 
> It should be:
> 
> The default mode is ROUTING + NAT.
> 
> Optional mode is BRIDGE.
> 
> 
> As far as i know a bridge operates on the Layer2.
> 
> IP and NAT operate on the Layer3.
> 
> NAT can not operate on the L2.
> 
> Please tell me that im wrong and explain how am i wrong.

If we'd be talking about bare metal network components then you are 100%
correct. Except, the Linux bridge is more than just plain L2 bridge. It
can have an IP address, route traffic, serve as network interface (when
a host is sending a packet to a guest, the packet is injected into said
bridge).

What the wiki page is trying to say, that by default you'll get this
'default' network which uses this Linux 'bridge' + NAT. Optionally, you
can define new network, or modify the existing one to switch to so
called routed mode.

Hope this clears things up.

Michal

Re: failed to load seccomp syscall filter in kernel: Operation canceled

[Michal Prívozník]

On 10/25/22 02:38, Jiatong Shen wrote:
> Hello community,
> 
>    I am operating an openstack cluster where applications (libvirt/nova
> etc) are running using containers. The compute node's arch is aarch64
> (phytium 2500), when there are virtual machines around 60 or 70, I
> failed to boot new virtual machines and faced with following error message,
> 
> error: internal error: qemu unexpectedly closed the monitor:
> 2022-10-24T06:23:54.545685Z qemu-system-aarch64: -sandbox
> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny:
> failed to load seccomp syscall filter in kernel: Operation canceled
> 

This error message comes from qemu, which sees seccomp_load() fail.
Looking further [1], seccomp_load() returns -ECANCELED. This is because
qemu does not set SCMP_FLTATR_API_SYSRAWRC attribute and thus public
APIs fail with this meaningless error. I've posted patch here [1].

But the symptoms suggest that you are hitting a limit (for eBPF perhaps?).

1: https://lists.gnu.org/archive/html/qemu-devel/2022-10/msg04509.html

> 
> Interestingly, if I virsh stop one virtual machine, I am able to boot
> another. Besides, I managed to manually boot a virtual machine without
> any issue. So my question is what could be the potential cause of this
> behavior and how can I deal with it? Thank you very much in advance for
> the help.
> 

I wonder whether the fact that openstack runs VMs in container has
something to do with this. Perhaps it touches the limit/sets different
accounting for it?

Michal

Re: hook script for changing disk definition before guest start

[Michal Prívozník]

On 10/21/22 15:30, Gionatan Danti wrote:
> Hi all,
> due to this [1] I am trying to change a guest's disk definition from
> "pool volume" to "file" via hook scripts.

This is downright libvirt bug and needs to be fixed.

For the libvirt developers on CC: when starting a domain, the QEMU
driver sets disk src->path to the actual file path in
virDomainDiskTranslateSourcePool() which is called from
qemuProcessPrepareDomainStorage(). I'm not sure whether the virt-aa
helper sees the translated path in the XML though. So maybe something as
trivial as the following might help?

diff --git i/src/security/virt-aa-helper.c w/src/security/virt-aa-helper.c
index a15971b9bb..f42c3f9a42 100644
--- i/src/security/virt-aa-helper.c
+++ w/src/security/virt-aa-helper.c
@@ -904,6 +904,7 @@ get_files(vahControl * ctl)
 #ifdef WITH_STORAGE
 virDriverLoadModule("storage", "storageRegister", false);
 #endif
+virDomainDiskTranslateSourcePool(ctl->def);
 
 for (i = 0; i < ctl->def->ndisks; i++) {
 virDomainDiskDef *disk = ctl->def->disks[i];

> 
> This seems an easy task via virsh detach-disk and virsh attach-disk, but
> virsh can not be called inside libvirt hook scripts. The best approach
> would be to change the disk definition right when a disk is attached,
> but I can not find any event calling a hook script when a disk is
> attache/detached.

You don't want to translate the path yourself. Libvirt has extensive
code for that.

> 
> Do you have any suggestion short of monitoring the directory
> /etc/libvirt/qemu itself for changed files?
> Thanks.

These are not the only files where libvirt stores its configuration. For
instance, if I'd 'virsh attach-device --live' then this new device won't
appear in the XML stored under /etc/libvirt/qemu.

I hope we can resolve this libvirt bug soon and you can use fixed
version.

Michal

Re: MemoryBacking options

[Michal Prívozník]

[Once again, I ask you to keep the list on the CC. In gmail's web UI
it's the 'Reply to all' button].

On 10/22/22 11:19, Simon Fairweather wrote:
> Is there a way to limit the shared memory allocated via Libvirt to
> memory backing?
> 

Yes, there is.
On the top level there's /domain/memoryBacking [1] that serves as the
default value for the guest. Then, under /domain/cpu/numa/cell [2]
individual guest NUMA nodes are configured and each can have @memAccess
attribute which overrides the top level setting (=memoryBacking). And
finally, under /domain/devices/memory [3] individual RAM modules can be
configured, and each one can have @access attribute which overrides both
top level AND guest NUMA node setting.

For instance:


  8
  5
  

  
  ...
  

  
  

  
  ...
  

  
1
0
  

  


Here, the guest NUMA node 0 is going to be shared, except for DIMM
module attached to it (which is going to be private) and NUMA node 1 is
going to be private again.

Michal

1: https://libvirt.org/formatdomain.html#memory-backing
2: https://libvirt.org/formatdomain.html#cpu-model-and-topology
3: https://libvirt.org/formatdomain.html#memory-devices

> >
> On Mon, Oct 17, 2022 at 9:06 AM Michal Prívozník  <mailto:mpriv...@redhat.com>> wrote:
> 
> [please keep the list on CC for benefit of the community]
> On 10/16/22 09:58, Simon Fairweather wrote:
> > Thanks Michal, Are there any known issues with Memory Backing. We have
> > QEMU 7.1 and libvirt 8.7.
> >
> > If just Memory Backing is set the VM Freezes after a couple of
> hours and
> > the allocated CPUs show 100% on the host.
> >
> > Seems to happen more on AMD Platforms than Intel.
> >
> > Please advise on any info you need, likely this is a QEMU issue.
> >
> >   
> >     
> >     
> >   
> 
> Agreed, this smells like QEMU issue. There's nothing obviously wrong
> with this snipped. I have a VM with just that and run it just fine.
> Although, I run it for couple of hours max.
> 
> Michal
> 

Re: MemoryBacking options

[Michal Prívozník]

[please keep the list on CC for benefit of the community]
On 10/16/22 09:58, Simon Fairweather wrote:
> Thanks Michal, Are there any known issues with Memory Backing. We have
> QEMU 7.1 and libvirt 8.7.
> 
> If just Memory Backing is set the VM Freezes after a couple of hours and
> the allocated CPUs show 100% on the host.
> 
> Seems to happen more on AMD Platforms than Intel.
> 
> Please advise on any info you need, likely this is a QEMU issue.
> 
>   
> 
> 
>   

Agreed, this smells like QEMU issue. There's nothing obviously wrong
with this snipped. I have a VM with just that and run it just fine.
Although, I run it for couple of hours max.

Michal

Re: MemoryBacking options

[Michal Prívozník]

On 10/8/22 11:10, Simon Fairweather wrote:
> Hi,
> 
> Is there any documentation that states which options can be used
> together with each other.
> 
> 
> 
> 
>   ...
>   
> 
>   
>   
> 
> 
> 
> 
> 
> 
> 
>   
>   ...
> 
> 

All of them can be used together because they are orthogonal to each other.

Michal

Re: Why host device disappear in libvirt doman xml?

[Michal Prívozník]

[Once again, please keep the list on CC]
On 9/29/22 04:59, 陈新隆 wrote:
> Thanks for the detail explanation, it's very helpful. 
> 
> I guess :
> 1. virsh dumpxml  outputs the live xml(or active xml) 

Yes, if domain is running then it outputs the live XML, otherwise it
outputs the inactive XML. For running domains you can use --inactive
flag to get the inactive XML.

> 2. virsh edit  edits the inactive xml, so I should cold boot
> domain to take effect

Correct.

Michal

Re: Why host device disappear in libvirt doman xml?

[Michal Prívozník]

[Please, keep the list on CC for benefit of others]
On 9/28/22 10:13, 陈新隆 wrote:
> I'm using the `virsh dumpxml ` to check the xml. When I
> first execute this command, there's two hostdev(GPU) in the xml, but
> next time I execute the same command these two hostdev elements
> disappeared. During this time, the vm didn't restart or stop.  Also,
> within the vm `lspci | grep -i nvidia` command did not print GPU infos.
> 
> So I was wondering if there's a mechanism in libvirtd detach the hostdev
> element without the vm stop or restart. This problem doesn't happen
> often, so I am looking a way to reproduce it. 
> 
> Can you help me with these questions :
> 
> 1. If I edit the xml manually then detached the hostdev elements from
> xml by invoke libvirtd apis, will the vm apply it immediately without
> stop or restart? Or I must restart  the vm to apply the latest xml ?

I'm not sure what you mean. Editing XML manually is different to using
libvirt APIs to detach hostdevs. Here's how it works:

1) a domain is defined (say using virsh define file.xml), libvirt parses
this XML, keeps it in a memory and stores it "somewhere" (it's under
/etc/libvirt/qemu/ but we do not want users to hand edit those files
manually as libvirt reads them only on libvirtd/virtqemud restart). This
is called inactive XML, because it reflects the inactive state of
domain. Sometimes it's also called config XML.

2) when the domain is started (e.g. virsh start), libvirt creates a copy
of inactive XML, populates it with runtime information and saves it
elsewhere (/run/libvirt/qemu/, but again, we do not want users to hand
edit those files). This copy is referred to as live XML.

3) users can alter the live XML using APIs (e.g. to hotplug a device or
hotunplug it). The inactive XML can be altered by providing altered XML
and defining it again (here, domain name and UUID must match already
existing domain).

4) upon domain shutoff, the live XML is thrown away, and finally

5) the inactive XML is never thrown away, until virsh undefine is called.

Now, you can see that there's no real connection between live and
inactive XMLs and changing one has no affect on the other, except when
the domain is cold booted again. Live and inactive XMLs can vary wildly.

Therefore, you can have inactive XML with two -s, and active
XML with no  at all. NB, hotplug APIs can also be used to alter
inactive XML (virsh attach-device --config / virsh detach-device
--config / ...) or both at the same time (virsh attach-device --config
--live / virsh attach-device --config --live / ...).

And what you describe sounds as if those two -s you saw at
domain startup were hotunplugged. The fact that even 'lspci' ran from
inside the domain can't find them only supports this theory.

And no, libvirt never tries to bring inactive and live XMLs together. It
has no intelligence built in to do that and we, developers, do not want
such thing either. We might change something that user specifically
wanted and I believe nobody likes those "smart" tools that get in your way.

> 2. After a host device is hot-unplugged, will libvirtd aware of it then
> remove the related `hostdev` element from xml ? 

Yes. I believe the reasoning is seen in the previous block of my reply.

> 
> On Mon, Sep 26, 2022 at 10:10 PM Michal Prívozník  <mailto:mpriv...@redhat.com>> wrote:
> 
> On 9/26/22 15:06, 陈新隆 wrote:
> >
> > <https://stackoverflow.com/posts/73854544/timeline
> <https://stackoverflow.com/posts/73854544/timeline>>
> >
> > I'm using Kubevirt to manage my virtual machine instances. When I
> using
> > Kubevirt to create a vm(with two GPUs), kubevirt will generate a
> libvirt
> > guest domain xml for this vm which includes two GPUs, the domain
> xml as
> > follow :
> >
> > |  > name='vfio'/>   > function='0x0'/>> type='pci' domain='0x' bus='0x06' slot='0x00' function='0x0'/>
> >> name='vfio'/>   > function='0x0'/>> type='pci' domain='0x' bus='0x07' slot='0x00' function='0x0'/>
> >  |
> >
> > No one ever edit this domain xml, but these two |hostdev| element
> > disappeared in the domain xml. During this time, I've run
> > the |gpu_burn| command to do a stress test for these two GPUs.
> >
> > My question is :
> >
> >   * when will libvirtd change the guest domain xml ?
> >   * why libvirtd delete these two |hostdev| from domain xml ?
> >
> 
> Libvirt d

Re: [Question] Should libvirt update live xml accordingly when guest mac is changed?

[Michal Prívozník]

On 8/30/22 05:22, Fangge Jin wrote:
> Hi
> 
> I met an issue when testing trustGuestRxFilters:
> Attach a macvtap interface with trustGuestRxFilters=’yes’ to vm, then
> change interface mac address in vm.
> Should libvirt update interface mac in live vm xml accordingly? If not,
> vm network will be broken after
> managedsaving and restoring vm.
> 
> BR,
> Fangge Jin

The MAC is updated accordingly. It's just that it does not survive
save+restore (and thus possibly migration). This is a bug in libvirt,
please file one. What we could do is just before resuming vCPUs, we
could query-rx-filters for those s where trustGuestRxFilters
is enabled and change MAC, just like we do when NIC_RX_FILTER_CHANGED
event is delivered.

Michal

Re: Why host device disappear in libvirt doman xml?

[Michal Prívozník]

On 9/26/22 15:06, 陈新隆 wrote:
> 
> 
> 
> I'm using Kubevirt to manage my virtual machine instances. When I using
> Kubevirt to create a vm(with two GPUs), kubevirt will generate a libvirt
> guest domain xml for this vm which includes two GPUs, the domain xml as
> follow :
> 
> |  name='vfio'/>   function='0x0'/>type='pci' domain='0x' bus='0x06' slot='0x00' function='0x0'/>
>name='vfio'/>   function='0x0'/>type='pci' domain='0x' bus='0x07' slot='0x00' function='0x0'/>
>  |
> 
> No one ever edit this domain xml, but these two |hostdev| element
> disappeared in the domain xml. During this time, I've run
> the |gpu_burn| command to do a stress test for these two GPUs.
> 
> My question is :
> 
>   * when will libvirtd change the guest domain xml ?
>   * why libvirtd delete these two |hostdev| from domain xml ?
> 

Libvirt does not remove anything from domain XML (except for the
elements it does not understand, but this is not the case). My suspicion
is that you're looking at live XML instead of inactive XML or vice
versa. Libvirt allows guests to be defined (i.e. libvirt manages their
inactive definition). However, a guest can be started with wildly
different configuration (e.g. without those two -s). OR, they
might have been hot-unplugged.


This article can explain more details:
https://wiki.libvirt.org/page/VM_lifecycle

Michal

Re: NUMA node - Memory Only

[Michal Prívozník]

On 8/9/22 12:55, Jin Huang wrote:
> Hi, everyone
> I built the libvirt 8.6.0 on my Ubuntu 20 system with the options like this:
> meson build -Dsystem=true -Ddriver_interface=enabled
> -Ddriver_libvirtd=enabled -Ddriver_network=enabled -Ddriver_qemu=enabled
> -Ddriver_remote=enabled -Dnumactl=enabled -Dnumad=enabled
> -Dstorage_disk=enabled
> 
> (1)After installation, when I tried to start the libvirtd, I get this
> error message:
> error : virNetworkObjAssignDefLocked:576 : operation failed: network
> 'default' already exists with uuid 7477a9f5-02d3-4fbc-b0e8-d7229d39a6a2
> 
> (2)When try the virsh command, I get this error message:
> virsh: /lib/x86_64-linux-gnu/libvirt-qemu.so.0: version
> `LIBVIRT_QEMU_8.2.0' not found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_8.0.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_8.5.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_6.10.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.7.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.8.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.2.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.1.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_7.3.0' not
> found (required by virsh)
> virsh: /lib/x86_64-linux-gnu/libvirt.so.0: version
> `LIBVIRT_PRIVATE_8.6.0' not found (required by virsh)
> 
> Could anyone give me some suggestions to fix these issues?

This is pretty much expected if you had libvirt installed from your
package manager (which I believe is the case because of the network
error). I don't know what the correct way to build a .deb package is,
but on rpm based distros I usually build a .tar.xz (meson dist) from
which I build a .rpm (rpmbuild -ta) and then install it.

Michal

Re: NUMA node - Memory Only

[Michal Prívozník]

On 8/5/22 11:25, Jin Huang wrote:
> Hi, libvirt-users
> How could I set up a memory-only and no-cpu NUMA node for the qemu-VM
> with the XML file?
> 
> Seems each NUMA cell has to be bundled with some specific cpuids.
> If I write the element like this, it is wrong.
> 
> 
>   
>     
>     
>   
> 
> Also if I ignore the cpus item, it is not acceptable for the virsh command.
> 
> The reason I want the memory-only node is that I want to set up multiple
> memory tiers for the VM, just
> like https://stevescargall.com/2022/06/10/using-linux-kernel-memory-tiering/
> 
> 

What you want to configure is so called HMAT (Heterogeneous Memory
Attribute Table). It can be configured by leaving out the @cpus
attribute completely. For instance like this:



  

  
  


  
  


  
  

  
  

  
  

  
  


  

  


This was introduced in libvirt-6.6.0 so you may need to upgrade.

Michal

Re: how to get vm uptime through libvirt api

[Michal Prívozník]

On 7/26/22 10:07, Jiatong Shen wrote:
> thank you very much for reply. I have got another question which is off
> topic.
> Am I able to get vm's process id through libvirt api?

Unfortunately, no. We specifically try to avoid telling users this
information so that they are not tempted to go behind libvirt's back and
interact with QEMU directly (which can lead to a split brain scenario -
libvirt thinks guest is in a certain state, but in fact it's in a
different state).

But as I said, it should be fairly easy to extend stats to report when a
domain was started. If you don't feel comfortable providing a patch for
that I can try.

Michal

Re: how to get vm uptime through libvirt api

[Michal Prívozník]

On 7/26/22 09:48, Jiatong Shen wrote:
> Thank you very much for reply. Initially my thought of uptime is equal
> to executing to `uptime` inside a virtual machine. 
> as for the second case, where virtual machine is paused for a period,
> does libvirt expose an api to get how long qemu process
> has existed?

No, it doesn't. But that should be pretty easy to implement into
virDomainListGetStats()/virConnectGetAllDomainStats().
Alternatively, libvirt emits an event when a domain is started, so you
may use that to record the time when the domain was started.

Michal

Re: how to get vm uptime through libvirt api

[Michal Prívozník]

On 7/26/22 08:49, Jiatong Shen wrote:
> Hello community,
> 
>    I would like to know if there is an api to get a virtual machine's
> uptime. Thank you in advance for the help.

There's no such API because not even qemu guest agent has an explicit
API for that. However, it has an API to execute binaries:

# virsh qemu-agent-command --pretty $dom '{"execute":"guest-exec",
"arguments":{"path":"uptime", "capture-output":true}}'
{
  "return": {
"pid": 1174
  }
}


# virsh qemu-agent-command --pretty $dom
'{"execute":"guest-exec-status", "arguments":{"pid":1174}}'
{
  "return": {
"exitcode": 0,
"out-data":
"IDA5OjEwOjU4IHVwIDggbWluLCAgMSB1c2VyLCAgbG9hZCBhdmVyYWdlOiAwLjAwLCAwLjA5LCAwLjA4Cg==",
"exited": true
  }
}


$ echo
"IDA5OjEwOjU4IHVwIDggbWluLCAgMSB1c2VyLCAgbG9hZCBhdmVyYWdlOiAwLjAwLCAwLjA5LCAwLjA4Cg=="
| base64 -d
 09:10:58 up 8 min,  1 user,  load average: 0.00, 0.09, 0.08


But it also depends what exactly do you mean by 'uptime'. Because if I
start a guest, then pause it for 5 minutes and then let it run again,
what should 'uptime' refer to?

Michal

Re: Testing DKIM handling

[Michal Prívozník]

On 7/15/22 14:37, dberra...@yahoo.com wrote:
> Hopefully the final test message, as DKIM is believed to be  correctly
> handled on the mailing lists now
> 

Yay! 🎉

Michal

Re: Domain XML of a restored VM

[Michal Prívozník]

On 6/10/22 15:31, Tomáš Golembiovský wrote:
> Hi,
> 
> I am looking for clarification on what exactly is happening to the
> domain XML when resuming a suspended VM. We would like to make sure we
> don't lose any configuration and we also see some unexpected behavior.
> 
> The VM with the name/UUID same as resumed VM may or may not already
> exist. If it does, what role does the original domain XML of a VM play
> when resuming? Is there any merging taking place between the existing
> domain XML and the domain XML stored in save image? Is the original XML
> simply thrown away and replaced?
> 

A guest can have up to two XMLs:

1) defined guests have so called 'inactive' or 'config' XML. This XML
does not contain any runtime information, and is used as the initial
configuration when starting the guest. Once the guest is running, it'll
also have ...

2) live XML. This XML contains runtime information and because of
hotplug and plenty of APIs that tweak various XML knobs
(virDomainSetMemoryParameters(), virDomainSetInterfaceParameters(),
etc.) can be very different to the inactive XML. Changes to the live XML
are thrown away as soon as the guest is shut off and we're back to
square 1 where only the inactive XML is preserved (and used as the
initial config for the next run).

However, this live XML is very important because it reflects the guest
ABI. For instance, some of our APIs offer a way for users to supply
modified domain XML (as you say below, virDomainRestoreFlags() is onw of
them, migration APIs are another). After this user supplied XML is
parsed, it's thrown at an excessive ABI stability checker, who's only
goal is to make sure that the live XML maintained by Libvirt and the
user provided XML differ only in those ways that won't break guest ABI
(for instance, changing the port where SPICE is allowed, removing a
 isn't).

Having said all of this - the XML that's saved among the domain by
virDomainSave() API is the live XML. Because that's the one that
reflects the current state of the saved image. Inactive XML is
disconnected from all of this.

> Is there any difference how the stored XML is treated compared to the
> XML passed in `virDomainRestoreFlags()`? I see some unexpected changes
> here. When I use `virDomainSaveImageGetXMLDesc()` to retrieve the stored
> XML and pass it unchanged to the `virDomainRestoreFlags()` the final
> domain XML is different then when `virDomainRestore()` is used or when
> `virDomainRestoreFlags()` with no `dxml` argument is used. We have:
> 
>
> 
> in the domain XML and the `passwdValidTo` argument disappears when
> domain XML is passed in `dxml` argument but it is preserved in other
> cases. Is this behavior expected? If yes, can we do anything to preserve
> the configuration? What other changes in domain XML can be expected?

No, this behaviour is definitely not expected.
As explained above, runtime values, that don't affect guest ABI are
allowed to change. The rest is not. Password timeout should be allowed
to change, because guest has no idea about the password in the first
place. This is all done in the backed (that part of SPICE that faces the
host/clients).

Michal

Re: something (qemu?) is leaking

[Michal Prívozník]

On 5/23/22 11:19, lejeczek wrote:
> Hi guys.
> 
> I do a simple thing which should be easy to reproduce.
> 
> -> $ virt-install -n rum1 --virt-type kvm --os-variant centos8 --memory
> $((4*1024)) --disk=/VMs3/rum1.qcow2,device=disk,bus=virtio --network
> network=10_3_1,model=virtio --graphics=listen=0.0.0.0 --cpu EPYC-Rome
> --vcpus 3 --cdrom /VMs3/CentOS-Stream-9-latest-x86_64-dvd1.iso
> 
> During manual setup in the VM I set 'hostname' to something and when
> installation begins and disk config stage takes place I can see - and
> later when VM(c9s) is ready can confirm - that VG name is taken from
> another VM defined/running on the host.

I'm no LVM expert, but I always thought that installer has some defaults
built in and thus it's kind of expected if you went with defaults.

But I'm kind of failing why is this a problem since all you're giving to
the guest is a single qcow2 disk which is not shared between two
domains. Maybe I'm misunderstanding something?

Michal

Re: Virtiofs xattr options on domain xml

[Michal Prívozník]

On 5/19/22 00:53, ksobrenat32 wrote:
> Hi!
> 
> I have a debian 11 (bullseye) machine running libvirtd version 7.0.0 and
> a RHEL 9 virtual machine that I need to share a disk and though about
> virtiofs.
> 
> The disk is a btrfs disk and I have successfully mount it with:
> 
>     
>   
>   
>     
>     
>   
>   
>   
>   
>    function='0x0'/>
>     
> 
> The problem I have is with selinux, when I try to change the context of
> a file inside the virtual machine I get a 'Operation not permitted'
> error, I can change the context in the Debian host and see the changes
> in the virtual machine but I would want to be able to change the context
> from the vm to able to use podman containers with selinux enabled.
> 
> I see on the docs
> https://qemu.readthedocs.io/en/latest/tools/virtiofsd.html#selinux-support
> you can run virtiofsd with a xattr option so it is compatible with
> selinux but I do not find a way to change the domain xml to add this
> option, is there a way to add this option? Does a better option exists
> (maybe on the guest side)?
> 
> 


Yeah, I don't think this was implemented. However, virtiofsd is running
as root:root and with no capabilities dropped. So I guess what we're
missing here is -o security_label or might as well implement the remap
as docs suggest which is much safer.

Michal

Re: Updating domains definitions via API

[Michal Prívozník]

On 5/16/22 14:52, Darragh Bailey wrote:

> So perhaps this is less a bug with the loader/nvram XML element handling
> and more a documentation bug and a possible enhancement that possibly
> the virDomainDefineXMLFlags could consider accepting a flag to verify the
> returned domain XML is equivalent as a general fix for those applications
> that would find this useful?

Yes to the first part, but no the second. Comparing XMLs is not as easy
as you would think. For instance:


  123456
  myGuest



  myGuest
  123456


The former is just an example of possible user input, the latter is how
libvirt would format it. Obviously, these XMLs are equivalent, but not
stcmp() equal.

Michal

Re: Updating domains definitions via API

[Michal Prívozník]

On 5/16/22 12:46, Darragh Bailey wrote:
> 
> On Mon 16 May 2022, 10:10 Michal Prívozník,  <mailto:mpriv...@redhat.com>> wrote:
> 
> On 5/14/22 21:23, Darragh Bailey wrote:
> >
> > On Fri, 13 May 2022 at 00:17, Darragh Bailey
> mailto:daragh.bai...@gmail.com>
> > <mailto:daragh.bai...@gmail.com <mailto:daragh.bai...@gmail.com>>>
> wrote:
> >
> > Unfortunately trying to call this via ruby-libvirt doesn't appear to
> > behave as expected. It appears that if I add an nvram element
> without a
> > loader element to the os block, the following code block will execute
> > without issue but also without changing the domain XML:
> 
> I think that's kind of expected. If you take a look how libvirt parses
> that part of XML:
> 
> 
> https://gitlab.com/libvirt/libvirt/-/blob/master/src/conf/domain_conf.c#L18257
> 
> <https://gitlab.com/libvirt/libvirt/-/blob/master/src/conf/domain_conf.c#L18257>
> 
> if no  is found then the function exits early without looking
> at  at all. It kind of makes sense - what good does nvram do
> without loader?
> 
> 
> I don't know, I realised that there was a bug in our vagrant-libvirt in
> that it should be checking for both loader and NVRAM config settings to
> be passed in if the end user was updating the machine config. And
> subsequently set both elements or neither. 
> 
> However I was not expecting the silent discarding of the XML element and
> instead to get an error saying that an nvram XML element without a
> loader XML element is invalid and for the entire request to be rejected.
> Basically that the provided XML domain definition was invalid.
> 
> Based on the previous explanation of how the define_domain_xml should
> work along with the response on the example reproducer this seems like a
> bug.
> 
> Does your reply mean this is expected behaviour? Or were you looking for
> clarification on what I expected to see?

I mean, fixing this particular problem is trivial. But what's not as
trivial is fixing ALL occurrences of this problem (where an
attribute/element is ignored because of something we've parsed earlier).
Mostly because we don't know what those are.

BTW: Have you spotted the other demonstration of this problem (in the
same function)? Yes, if fw autoselection is enabled () then  is ignored.

Anyway, there used to be an unwritten agreement with users that they get
XML of a domain they just defined to see if it contains everything they
want. This is used to be because different drivers support different
features. But I guess it's hard to argue with an unwritten rule esp. if
it's not known to everybody.

Michal

Re: Problem calling 'virsh' in a script

[Michal Prívozník]

On 5/15/22 17:48, Digimer wrote:
> Hi all,
> 
>   I've got a series of programs that monitor various things on a CentOS
> Stream 8 VM host. All of these scripts work when called directly.
> However, when I have a parent program that calls all the little programs
> in series, I found that some virsh calls hang.
> 
>   Initially, there were two scripts that were hanging repeatedly. Once
> called 'virsh net-list --all --name', so I changed it to check for
> configs in '/etc/libvirt/qemu/networks/', and that script started
> working. The other script though calls 'virsh list --all', and that
> can't be easily swapped out, so I really need to find the source of
> these hangs.
> 
>   Whenever the hang happens, about 30~45 seconds later, I see
> 'libvirtd[1643714]: Cannot recv data: Connection reset by peer'.
> 
>   I think the issue is striking other scripts that run, but this
> scenario is happening predictably and consistently right now.
> 
>   I thought it might be a concurrent connect limit or a problem with how
> many times virsh is called by a script, so I wrote a test script that
> kept calling 'virsh list --all' each second, but it was close to 100
> calls without hanging, far more that all the calls in my scripts
> combined, so I don't think that's it.
> 
> Any advice/guidance would be very much appreciated!

I wonder whether specifying the connection URI explicitly would help. I
don't know anything about your app, but if it perhaps clears some env
vars (LIBVIRT_DEFAULT_URI VIRSH_DEFAULT_CONNECT_URI) or runs under
different user than what you're testing virsh under then virsh will try
to start a session daemon. If you wan it to connect to the system URI
specify that explicitly:

  virsh -c qemu:///system list ...

Michal

Re: Updating domains definitions via API

[Michal Prívozník]

On 5/14/22 21:23, Darragh Bailey wrote:
> 
> On Fri, 13 May 2022 at 00:17, Darragh Bailey  > wrote:
> 
> Hi,
> 
> On Thu 12 May 2022, 21:34 Laine Stump,  > wrote:
> 
> The virDomainDefineXMLFlags API (and also the older/deprecated
> virDomainDefineXML API) doesn't require that the domain first be
> undefined (with one notable exception - see below[*]). If you
> define a
> domain that already exists with the same name and uuid, then the
> effect
> is to "redefine" (or "update" if you prefer) the existing domain
> of that
> name. If the domain is currently active, then the changes will take
> effect the next time the domain is shut down ("Destroy"ed in
> libvirt API
> parlance) and re-started.
> 
> 
> Unfortunately trying to call this via ruby-libvirt doesn't appear to
> behave as expected. It appears that if I add an nvram element without a
> loader element to the os block, the following code block will execute
> without issue but also without changing the domain XML:

I think that's kind of expected. If you take a look how libvirt parses
that part of XML:

https://gitlab.com/libvirt/libvirt/-/blob/master/src/conf/domain_conf.c#L18257

if no  is found then the function exits early without looking
at  at all. It kind of makes sense - what good does nvram do
without loader?

Michal

Re: Debugging hanging libvirt

[Michal Prívozník]

On 4/1/22 21:55, Tobias Hofmann (tohofman) wrote:
> Hi Daniel,
> 
>  
> 
> Thank you to Peter and you for the analysis – that was very helpful.
> 
> Indeed, it can happen that we restart the firewall during this scenario
> on our system so I think I can debug from here.
> 
>  
> 
> By the way, you mentioned that there is a libvirt version where QEMU
> process gets its own dedicated event loop. Would you mind sharing this
> libvirt version with me?

It's since libvirt-6.2.0 but IIUC you're running on unsupported CentOS 7
which doesn't have that release available.

Michal

Re: virDomainCoreDump output format

[Michal Prívozník]

On 3/23/22 14:19, Shelly Kagan wrote:
> Hi all,
> I have a small question regarding virDomainCoreDump.
> In the documentation[1] it is stated that the output is of format RAW.
> After implementing a function using this command the output file
> format that I see is ELF. I understand those formats are different, so
> is the documentation not updated? Is there a reason I will get an ELF
> format instead of RAW?
> Appreciate your help!


Hey,

I believe that what you see is expected. If you'd dump memory of any
process you'll get ELF. QEMU is no exception as guest memory is in fact
QEMU's memory too. RAW is abstraction because other hypervisors (that
are not ELF based) can provide different formats.

Michal

Re: PHP Warning: libvirt_domain_new(): Cannot get installation XML

[Michal Prívozník]

On 3/24/22 21:48, Benjamin Smith wrote:
> Despite great success at using php-libvirt with other functions, I have
> been entirely unsuccessful in getting it to create a new domain/machine.
> Any time I do, and with virtually anything I try, I get the above error
> message. 
> 
> Running PHP 7.4 (remi) on otherwise stock AlmaLinux.
> *[root@sb4 html]# cat /etc/redhat-release
> AlmaLinux release 8.5 (Arctic Sphynx
> *
> 
> *[root@sb4 html]# rpm -qa php php-fpm php-libvirt nginx
> php-7.4.28-1.el8.remi.x86_64
> nginx-1.14.1-9.module_el8.3.0+2165+af250afe.alma.x86_64
> php-fpm-7.4.28-1.el8.remi.x86_64
> php-libvirt-0.5.5-1.el8.remi.7.4.x86_64
> 12 XEON E312xx cores, 24 w/HT, 192GB RAM, etc. *
> 

Hey,

so libvirt-php is not really that well maintained. The original author
moved to different project and I used to do at least very basic
maintenance but also got side tracked too often.

Looking into the code, which lives at:

  https://gitlab.com/libvirt/libvirt-php

it's the installation_get_xml() function that's failing:


https://gitlab.com/libvirt/libvirt-php/-/blob/master/src/libvirt-php.c#L2353

and honestly, I don't have any clue why. Do you think you could attach a
debugger and step through the function to find out which line is failing?

Michal

Re: how to reliably shutdown domains ?

[Michal Prívozník]

On 3/8/22 20:25, Lentes, Bernd wrote:
> Hey guys,
> 
> i have a two-node cluster with around 20 domains. Cluster-Software is 
> pacemaker and corosync, OS is SLES 12 SP5.
> The scripts for starting/stopping the domains use virsh. Is there a way to 
> reliably shutdown the domains via virsh ?
> I'm testing around, but sometimes the domains stop, sometimes they don't, 
> sometimes it takes very long so that the cluster times out
> and fence the respective node.
> I'm using "virsh shutdown domain --mode acpi,agent" for the windows domains 
> (not reliable) and for the linux domains
> "virsh shutdown domain", also not reliable.

In general, if agent is available it's more likely to succeed than acpi,
because the shutdown is initiated from inside the guest (/sbin/shutdown
is invoked for UNIX-like systems, or Windows equivalent) while with ACPI
a guest can simply chose to ignore it. In fact, that's what libvirt does
- whenever agent method is available (i.e. --mode contains agent, or no
specific --mode was requested = libvirt is free to chose), it is preferred.

I assume you don't see any errors reported by virsh and thus "sometimes
it takes very long" [to shutdown a guest] could mean that guests are
under heavy load, e.g. they are syncing disks before shutdown, stopping
services, etc. I don't think I have a good answer for you until the root
cause if found.

Michal

Re: Number of the max supported VFs are different

[Michal Prívozník]

On 3/3/22 03:55, Yalan Zhang wrote:
> Hi there,
> 
> I have an Intel X520 network card, and I find the max supported VFs are
> different.
> Please check below outputs:
> 
> # lspci -vvv -s 04:00.0
> 04:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520
> Adapter (rev 01)
> ...
>  Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
> IOVCap: Migration-, Interrupt Message Number: 000
> IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
> IOVSta: Migration-
> ** Initial VFs: 64, Total VFs: 64**, Number of VFs: 0, Function
> Dependency Link: 00
> VF offset: 128, stride: 2, Device ID: 10ed
> Supported Page Size: 0553, System Page Size: 0001
> Region 0: Memory at 9440 (64-bit, prefetchable)
> Region 3: Memory at 9450 (64-bit, prefetchable)
> VF Migration: offset: , BIR: 0
> Kernel driver in use: ixgbe
> Kernel modules: ixgbe
> 
> # cat /sys/class/net/enp4s0f0/device/sriov_totalvfs
> 63
> 
> # echo 64 > /sys/class/net/enp4s0f0/device/sriov_numvfs
> -bash: echo: write error: Numerical result out of range
> # echo 63 > /sys/class/net/enp4s0f0/device/sriov_numvfs
> # cat /sys/class/net/enp4s0f0/device/sriov_numvfs
> 63
> 
> The lspci command says the Total VFs supported is 64, while in the file
> "sriov_totalvfs"  says it's 63.
> And the sriov_numvfs file will take precedence.
> Why are the numbers different?  Just a little curious.

I believe this comes from the driver implementation:

/*  ixgbe driver limit the max number of VFs could be enabled to
 *  63 (IXGBE_MAX_VF_FUNCTIONS - 1)
 */
#define IXGBE_MAX_VFS_DRV_LIMIT  (IXGBE_MAX_VF_FUNCTIONS - 1)

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbe/ixgbe_sriov.h#n7

Michal

Re: About RO connection

[Michal Prívozník]

On 3/2/22 03:28, Xingbo Kan wrote:
> Dear developers and users:
> Is there any standard or convention of which kind of APIs&flags could
> be invoked by RO connection? If there is any, please tell me in
> detail, thanks a lot!

Your best bet is to look at public API implementation [1] and see if
virCheckReadOnly*() is called. But in general, anything that can't
change state of a domain/network/... is considered okay for RO
connections (e.g. listing domains, getting domain XML without security
info, etc.). For everything else you'll need RW connection.

Alternatively, you may look at our RPC protocol description [2] and if
the API is described with "domain:read" then it's okay for RO connection.

1: https://gitlab.com/libvirt/libvirt/-/blob/master/src/libvirt-domain.c
2:
https://gitlab.com/libvirt/libvirt/-/blob/master/src/remote/remote_protocol.x#L3918

Michal

Re: libvirtd file descriptors leaking

[Michal Prívozník]

On 3/1/22 12:40, Дмитрий wrote:
> Any update on this issue? Does anyone have a success on reproducing that? 
> Should I upgrade/downgrade/sidegrade?
> 
> Additional info:
> 
> I am using CentOS 8 Stream, but with kernel
> 
> 5.14.15-1.el8.elrepo.x86_64 #1 SMP Tue Oct 26 11:45:20 EDT 2021 x86_64 x86_64 
> x86_64 GNU/Linux
> 

I believe this was fixed in RHEL-8, by a RHEL-only patch, because in
there glib backported a commit that broke our upstream workaround. And
since CentOS 8 Stream is behind RHEL you will get the fix eventually
(sorry I don't know the schedule). However, I don't think that the
RHEL-only patch can be merged upstream, because then it would break
distros with older glib that did not mangle it.

I'm sorry, I don't have a better answer.

Michal

Re: QEMU args generation order with libvirt

[Michal Prívozník]

On 2/28/22 08:51, M, Shivakumar wrote:
> Hello,
> 
>  
> 
> We are trying to bring up the Android VM with Libvirt, as part of this
> exercise it is advised to keep Replay Protected Memory Block (RPMB)
> device as the first virtio device in the QEMU arg list, as the secure
> storage daemon in Andriod side communicates with dev/vport0p1 for RPMB
> usage.
> 
>  
> 
> Is there any way we can make a qemu commands generation in particular
> order with Libvirt XML ?

Unfortunately no. While libvirt allows passing arbitrary cmd line
arguments (https://libvirt.org/kbase/qemu-passthrough-security.html)
these are appended only after libvirt generated part.

Michal

Re: libvirtd file descriptors leaking

[Michal Prívozník]

On 2/22/22 16:36, Дмитрий wrote:
> # lsof -p $(cat /run/libvirtd.pid) > run1
> # virt-df -d kvm3586
> # lsof -p $(cat /run/libvirtd.pid) > run2
> # diff run1 run2
> 196a197
>> libvirtd 3618992 root   54u unix 0x902dc0c58440  0t0   99330533 
>> type=STREAM
> 

I wonder whether this is the glib event loop issue I've fixed a while
ago. Symptoms look similar.


https://gitlab.com/libvirt/libvirt/-/commit/5de203f8795d96229d2663e9ea1a24fba5db38fc

Let me check if that's backported into CentOS and what versions.

Michal

Re: random but often VMs' - BUG: soft lockup

[Michal Prívozník]

On 2/22/22 09:48, lejeczek wrote:
> Hi guys
> 
> I see this often(perhaps once a day) on VMs:
> 
> ... watchdog: BUG: soft -lockup - CPU#1 stuck for xxXXxxs! [swapper/1:0]


I used to see these when I suspended the host with guests running. But
in your case, because of the "swapper" suffix I suspect that the host
has gone trashing and thus can't schedule vCPUs fast enough to appease
watchdog.

Michal

Re: Hugepages -- "Memory backend 'pc.ram' not found"

[Michal Prívozník]

On 2/21/22 17:12, Charles Polisher wrote:

Hey, please the list on CC for benefit of others, e.g. when somebody
runs into the same problem they can find the discussion in the archive.

> On 2/21/22 01:54, Michal Prívozník wrote:
> 
>> On 2/20/22 04:07, Charles Polisher wrote:
>>> Hello,
>>>
>>> After defining hugepages, as documented at
>>> https://libvirt.org/formatdomain.html#memory-backing ,
>>> when I start the guest, I get a dialogue
>>> box that says:
>>>
>>>  Error starting domain: internal error: qemu unexpectedly
>>>  closed the monitor: 2022-02-20T01:10:36.520955Z
>>>  qemu-system-x86_64: Memory backend 'pc.ram' not found
>>>  Traceback (most recent call last):
>>>        File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65,
>>> in cb_wrapper
>>>          callback(asyncjob, *args, **kwargs)
>>>        File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101,
>>> in tmpcb
>>>          callback(*args, **kwargs)
>>>    File
>>> "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57,
>>> in newfn
>>>  ret = fn(self, *args, **kwargs)
>>>    File "/usr/share/virt-manager/virtManager/object/domain.py", line
>>> 1329, in startup
>>>          self._backend.create()
>>>    File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1353,
>>> in create
>>>          raise libvirtError('virDomainCreate() failed'
>>>
>>> After backing out changes, guest starts normally.
>>> I searched online for the error message, but found nothing useful.
>>> The hypervisor is running libvirtd (libvirt) 7.8.0 and QEMU emulator
>>> version 6.1.0,
>>> both build from source. I've got plenty of hugepages available.
>>> The domain's XML definition is attached.
>> Hey, can you share your domain XML and the generated cmd line? The
>> latter should be found in /var/log/libvirt/qemu/$domain.log
>>
>> Thanks,
>> Michal
> 
> Thanks for your reply. As requested, the domain XML:
> 
>     
>   slacky-0
>   4a67eb39-9b92-8b8a-97ba-7e1250d56b07
>   slacky-0
>   elided
>   4194304
>   4194304
>   
>     
>   

This does not look correct. This is on x86_64 and there is no such size
for hugepages, only 2MiB and 1GiB.

>     
>   
>   2
>   
>     hvm
>     
>   
>   
>     
>     
>     
>   
>   
>     kvm64
>   
>   
>   destroy
>   restart
>   restart
>   
>     /usr/bin/qemu-system-x86_64
>     
>   
>   
>   
>    unit="0"/>
>     
>     
>   
>   
>   
>   
>    function="0x0"/>
>     
>     
>    function="0x7"/>
>     
>     
>   
>    function="0x0" multifunction="on"/>
>     
>     
>   
>    function="0x1"/>
>     
>     
>   
>    function="0x2"/>
>     
>     
>    function="0x1"/>
>     
>     
>    function="0x0"/>
>     
>     
>    function="0x0"/>
>     
>     
>     
>   
>   
>   
>    function="0x0"/>
>     
>     
>   
>   
>     
>   
>     
>     
>   
>   
>     
>     
>   
>     
>     
>     
>     
>   
>     
>     
>    function="0x0"/>
>     
>     
>     
>    heads="1" primary="yes"/>
>    function="0x0"/>
>     
>     
>    function="0x0"/>
>     
>   
>     
> 
> 
> And the guest log with the generated command line:
> 
>     2022-02-20 01:13:12.985+: starting up libvirt version: 7.8.0,
> qemu version: 6.1.0, kernel: 5.15.19, hostname: godzilla.peecee3.com
>     LC_ALL=C \
>     PATH=/sbin:/usr/sbin:/bin:/usr/bin \
>     HOME=/var/lib/libvirt/qemu/domain-34-slacky-0 \
> XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-34

Re: Hugepages -- "Memory backend 'pc.ram' not found"

[Michal Prívozník]

On 2/20/22 04:07, Charles Polisher wrote:
> Hello,
> 
> After defining hugepages, as documented at
> https://libvirt.org/formatdomain.html#memory-backing ,
> when I start the guest, I get a dialogue
> box that says:
> 
>     Error starting domain: internal error: qemu unexpectedly
>     closed the monitor: 2022-02-20T01:10:36.520955Z
>     qemu-system-x86_64: Memory backend 'pc.ram' not found
>     Traceback (most recent call last):
>       File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65,
> in cb_wrapper
>         callback(asyncjob, *args, **kwargs)
>       File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101,
> in tmpcb
>         callback(*args, **kwargs)
>   File
> "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57,
> in newfn
>     ret = fn(self, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/object/domain.py", line
> 1329, in startup
>         self._backend.create()
>   File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1353,
> in create
>         raise libvirtError('virDomainCreate() failed'
> 
> After backing out changes, guest starts normally.
> I searched online for the error message, but found nothing useful.
> The hypervisor is running libvirtd (libvirt) 7.8.0 and QEMU emulator
> version 6.1.0,
> both build from source. I've got plenty of hugepages available.
> The domain's XML definition is attached.

Hey, can you share your domain XML and the generated cmd line? The
latter should be found in /var/log/libvirt/qemu/$domain.log

Thanks,
Michal

Re: libvirtd daemon missing in LFS

[Michal Prívozník]

On 2/14/22 10:09, Sai Kiran Kumar Reddy wrote:
> Hi Peter,
> 
> Thanks for your inputs. I have looked at all the dependencies and built
> libvirt with the appropriate dependencies enabled. I am able to run
> virt-manager also, without any errors. I am trying to create a VM using
> virsh. I get an error saying "domain configuration does not support
> video mode qxl". Could you please let me know if it is libvirt related
> error or qemu related one?

This is QEMU related and your qemu was probably build without SPICE support:

  https://www.spice-space.org/

pass --enable-spice to QEMU ./configure script. Alternatively, Gentoo
has all these dependencies recorded and maintained. So it's easier to
install.

Michal

Re: qemu+ssh connections to a remote libvirt fail as ssh banner configured

[Michal Prívozník]

On 2/10/22 09:02, Daniel P. Berrangé wrote:
> On Thu, Feb 10, 2022 at 09:52:52AM +0800, Yalan Zhang wrote:
>> Hi there,
>>
>> I have a system configured with ssh login banner like as below:
>> # cat ~/.bashrc
>> ...
>> echo
>> "="
>> echo "== This machine is occupied by xxx for testing now. If you are
>> about to use it, contact xxx first =="
>> echo
>> "="
>>
>> It works as expected that whenever someone logs into this system by ssh,
>> he/she will see this warning message.
>> But it seems such settings will impact a virsh client connection with ssh,
>> when I try to connect the libvirt daemon on this system, it will error out :
>> # virsh -c qemu+ssh://${my_host}/system list --all
>> root@${my_host}'s password:
>> error: failed to connect to the hypervisor
>> error: packet 1027423545 bytes received from server too large, want 33554432
> 
> Libvirt is tunnelling an RPC protocol over the SSH connection.
> Your bashrc is printing this text onto the SSH conmnection and
> that corrupts the libvirt RPC protocol.
> 
> If you want to print something whjen people login use the
> /etc/motd file which is designed for this pupose, don't
> print stuff from a .bashrc.  Libvirt gives the options to
> SSH that prevent display of /etc/motd contents, so that
> its RPC protocol doesn't get corrupted.

One more thing, I wasn't able to reproduce when virt-ssh-helper was
used. But maybe I wasn't trying hard enough.

Michal

Re: error during host deploy

[Michal Prívozník]

On 1/11/22 11:13, Dana Elfassy wrote:
> In the last one I don't see a link to download the RPM, can you check?

You have to click on the arch you are interested in, for instance x86_64
is the following link:

https://koji.fedoraproject.org/koji/taskinfo?taskID=81062428

And if you don't want to download all rpms by hand you can use koji:

koji download-task 81062428

Just replace the task ID with the one you want to download. The example
will download rpms for x86_64.

Michal

Re: error during host deploy

[Michal Prívozník]

On 1/10/22 15:08, Michal Prívozník wrote:
> On 1/10/22 13:04, Dana Elfassy wrote:
>> sorry, I missed your last reply
>> I didn't reproduce it yet, but fedora rpm would be great. just in case 
>> thanks
>>
> 
> Here you go:
> 

Ooops, wrong link. This one is correct:

https://koji.fedoraproject.org/koji/taskinfo?taskID=81062331

Michal

Re: error during host deploy

[Michal Prívozník]

On 1/10/22 13:04, Dana Elfassy wrote:
> sorry, I missed your last reply
> I didn't reproduce it yet, but fedora rpm would be great. just in case 
> thanks
> 

Here you go:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1867760

Michal

Re: error during host deploy

[Michal Prívozník]

On 12/22/21 08:44, Dana Elfassy wrote:
> Thanks,
> Is it possible that you create an rpm with this fix so I can use it on
> the test runs and try to reproduce?

Oh, sure. Is fedora rpm fine?

Michal

Re: error during host deploy

[Michal Prívozník]

On 12/20/21 11:34, Dana Elfassy wrote:
> Hi,
> While running a test case of adding hosts on ovirt system tests there
> was a failure while the following command was executed:
> vdsm-tool configure --force
> 
> On libvirtd log I found this error:
> 
> 2021-12-17 00:11:41.753+: 28031: error : virNetTLSContextNew:732 :
> Unable to generate diffie-hellman parameters: Error in public key
> generation.


This is the code on that line:

  err = gnutls_dh_params_init(&ctxt->dhParams);
  if (err < 0) {
  virReportError(VIR_ERR_SYSTEM_ERROR,
 _("Unable to initialize diffie-hellman parameters: %s"),
 gnutls_strerror(err));
  goto error;
  }
  err = gnutls_dh_params_generate2(ctxt->dhParams, DH_BITS);
  if (err < 0) {
  virReportError(VIR_ERR_SYSTEM_ERROR,
 _("Unable to generate diffie-hellman parameters: %s"),
 gnutls_strerror(err));
  goto error;
  }
  
  gnutls_certificate_set_dh_params(ctxt->x509cred,
 ctxt->dhParams);


More specific, it's gnutls_dh_params_generate2() that fails. I suspect
it's because DH_BITS is defined as following:

  #define DH_BITS 2048

which might be too short for system policy. If you're able, you can try
the following patch:

diff --git i/src/rpc/virnettlscontext.c w/src/rpc/virnettlscontext.c
index 1a3dd92676..3ab9f6c4ce 100644
--- i/src/rpc/virnettlscontext.c
+++ w/src/rpc/virnettlscontext.c
@@ -717,16 +717,20 @@ static virNetTLSContext *virNetTLSContextNew(const char 
*cacert,
  * once a day, once a week or once a month. Depending on the
  * security requirements.
  */
 if (isServer) {
+unsigned int bits = 0;
+
+bits = gnutls_sec_param_to_pk_bits(GNUTLS_PK_DH, 
GNUTLS_SEC_PARAM_HIGH);
+
 err = gnutls_dh_params_init(&ctxt->dhParams);
 if (err < 0) {
 virReportError(VIR_ERR_SYSTEM_ERROR,
_("Unable to initialize diffie-hellman parameters: 
%s"),
gnutls_strerror(err));
 goto error;
 }
-err = gnutls_dh_params_generate2(ctxt->dhParams, DH_BITS);
+err = gnutls_dh_params_generate2(ctxt->dhParams, bits);
 if (err < 0) {
 virReportError(VIR_ERR_SYSTEM_ERROR,
_("Unable to generate diffie-hellman parameters: 
%s"),
gnutls_strerror(err));


If it helps, I can post it for review.

Michal

Re: Best way to install guest when it is not listed in output of osinfo-query os

[Michal Prívozník]

On 12/20/21 06:52, john doe wrote:
> Hello all,
> 
> I would like to install Debian Bullseye using virt-install, as part of
> the command I usually specify the OS used with the option '--os-variant'.
> This works great untill Buster ('debian10'), but now the guest is
> Bullseye ('debian11') and is not listed in the output of 'osinfo-query
> os' (1).
> 
> If I don't use the '--os-variant' flag I get the following warning:
> 
> "WARNING  No operating system detected, VM performance may suffer.
> Specify an OS with --os-variant for optimal results."
> 
> 
> Should I ignore the warning or what is the best way forward to install a
> Debian Bullseye guest?
> 
> Both the host and the guest are  Debian 11.
> 
> 1)  https://lists.debian.org/debian-user/2021/12/msg00742.html

Debian 11 was added into the osinfo-db roughly 4 months ago:

https://gitlab.com/libosinfo/osinfo-db/-/commit/d729bd193947cc377c5ffc054222eef12a82e414

I'm not sure who maintains the package in debian, but I'd ask them to
refresh the package. Meanwhile, as a workaround I think it's safe to use
debian10 as --os-variant.

Michal

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Michal Prívozník]

On 12/13/21 17:49, Marc Zyngier wrote:
> On Mon, 13 Dec 2021 16:06:14 +,
> Peter Maydell  wrote:
>>
>> KVM on big.little setups is a kernel-level question really; I've
>> cc'd the kvmarm list.
> 
> Thanks Peter for throwing us under the big-little bus! ;-)
> 
>>
>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo  wrote:
>>>
>>>
>>>
>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
>>>>> with upstream kernels.
>>>>>
>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>> brings quite some troubles for VMs.
>>>>>
>>>>> In short, without proper cpuset to bind the VM to either all A72 cores
>>>>> or all A55 cores, the VM will mostly fail to boot.
> 
> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> though of it makes me sick).
> 
>>>>>
>>>>> Currently the working xml is:
>>>>>
>>>>>2
>>>>>
>>>>>
>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
>>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> 
> Disclaimer: I know nothing about libvirt (and no, I don't want to
> know! ;-).
> 
> However, for things to be reliable, you need to taskset the whole QEMU
> process to the CPU type you intend to use. That's because, AFAICT,
> QEMU will snapshot the system registers outside of the vcpu threads,
> and attempt to use the result to configure the actual vcpu threads. If
> they happen to run on different CPU types, the sysregs will differ in
> incompatible ways and an error will be returned. This may or may not
> be a bug, I don't know (I see it as a feature).
> 

Oh, I had no idea that QEMU does this. Anyway, QEMU spawns a thread for
each vCPU which is perfect, because then libvirt can use CGroups to
restrict each vCPU individually onto a subset of physical CPU(s). And in
fact, libvirt has this feature for many years. I always viewed it as a
performance tuning, but TIL.

Michal

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Michal Prívozník]

On 12/14/21 01:41, Qu Wenruo wrote:
> 
> 
> On 2021/12/14 00:49, Marc Zyngier wrote:
>> On Mon, 13 Dec 2021 16:06:14 +,
>> Peter Maydell  wrote:
>>>
>>> KVM on big.little setups is a kernel-level question really; I've
>>> cc'd the kvmarm list.
>>
>> Thanks Peter for throwing us under the big-little bus! ;-)
>>
>>>
>>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo  wrote:
>>>>
>>>>
>>>>
>>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI
>>>>>> CM4,
>>>>>> with upstream kernels.
>>>>>>
>>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its
>>>>>> little.BIG
>>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>>> brings quite some troubles for VMs.
>>>>>>
>>>>>> In short, without proper cpuset to bind the VM to either all A72
>>>>>> cores
>>>>>> or all A55 cores, the VM will mostly fail to boot.
>>
>> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
>> though of it makes me sick).
>>
>>>>>>
>>>>>> Currently the working xml is:
>>>>>>
>>>>>>     2
>>>>>>     
>>>>>>
>>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM
>>>>>> will
>>>>>> mostly fail to start up due to vcpu initialization failed with
>>>>>> -EINVAL.
>>
>> Disclaimer: I know nothing about libvirt (and no, I don't want to
>> know! ;-).
>>
>> However, for things to be reliable, you need to taskset the whole QEMU
>> process to the CPU type you intend to use.
> 
> Yep, that's what I'm doing.
> 
>> That's because, AFAICT,
>> QEMU will snapshot the system registers outside of the vcpu threads,
>> and attempt to use the result to configure the actual vcpu threads. If
>> they happen to run on different CPU types, the sysregs will differ in
>> incompatible ways and an error will be returned. This may or may not
>> be a bug, I don't know (I see it as a feature).
> 
> Then this brings another question.
> 
> If we can pin each vCPU to each physical core (both little and big),
> then as long as the registers are per-vCPU based, it should be able to
> pass both big and little cores to the VM.
> 
> Yeah, I totally understand this screw up the scheduling, but that's at
> least what (some insane) users want (just like me).
> 
>>
>> If you are annoyed with this behaviour, you can always use a different
>> VMM that won't care about such difference (crosvm or kvmtool, to name
>> a few).
> 
> Sounds pretty interesting, a new world but without libvirt...
> 
>> However, the guest will be able to observe the migration from
>> one cpu type to another. This may or may not affect your guest's
>> behaviour.
> 
> Not sure if it's possible to pin each vCPU thread to each core, but let
> me try.
> 

Sure it is, for instance:











pins vCPU#0 onto host CPUs 1-4, excluding 2; vCPU#1 onto host CPUs 0-1
and so on. You can also pin emulator (QEMU) and its iothreads. It's
documented here:

https://libvirt.org/formatdomain.html#cpu-tuning

Michal

Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Michal Prívozník]

On 12/11/21 02:58, Qu Wenruo wrote:
> Hi,
> 
> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
> with upstream kernels.
> 
> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> brings quite some troubles for VMs.
> 
> In short, without proper cpuset to bind the VM to either all A72 cores
> or all A55 cores, the VM will mostly fail to boot.
> 
> Currently the working xml is:
> 
>   2
>   
> 
> But even with vcpupin, pinning each vcpu to each physical core, VM will
> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> 
> 
> This brings a problem, in theory RK3399 SoC should out-perform BCM2711
> in multi-core performance, but if a VM can only be bind to either A72 or
> A55 cores, then the performance is no longer competitive against
> BCM2711, wasting the PCIE 2.0 x4 capacity.
> 
> I guess with projects like Asahi Linux making progress, there will be
> more and more such problems.
> 
> Any clue on how to properly pass all physical CPU cores to VM for
> little.BIG setup?
> 

I have never met big.LITTLE but my understanding was that those big
cores are compatible with little ones and the only difference is that
the big ones are shut off if there's no demand (to save energy) leaving
only the little ones running.

Anyway, this is likely too high level forum and I'd ask QEMU developers:

https://www.qemu.org/support/

Michal

Re: virsh create fails with "Unable to find any firmware to satisfy 'efi'" for aarch64 guest on macOS

[Michal Prívozník]

On 11/30/21 11:40, Andrea Bolognani wrote:
> On Tue, Nov 30, 2021 at 09:54:38AM +0100, Michal Prívozník wrote:
>> On 11/30/21 06:49, Nathan Vander Wilt wrote:
>>> Ah, but it looks like the arm64 -> VIR_ARCH_AARCH64 patch
>>> is still in limbo? AFAICT that's the main issue I'm hitting now.
> 
> Unfortunately that's the case.
> 
>> Looking into virArchFromHost() I can see uname() called which is then
>> passed to virArchFromString(). In here, uname machine (which is
>> equivalent to 'uname -m' from shell) is compared against virArchData array:
>>
>> https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virarch.c#L42
>>
>> So what you are saying is that 'uname -m' reports arm64 and not aarch64?
>> If that's the case then we should revisit the patch you mention.
> 
> Correct: on Apple Silicon Macs the architecture name is reported as
> "arm64", but our code expects it to be "aarch64" because that's what
> we get on Linux.
> 
> Michal, have you actually looked at the patch mentioned earlier? If
> not, you can perhaps do a clean room implementation of the fix based
> on the information provided above and get us out of this stalemate?
> 
> It's quite a simple change, but having seen the original patch I feel
> like I couldn't possibly submit it myself and still be in the clear.
> 

Unfortunately I did. But I think the whole area can be reworked a bit so
that we would detect both arm64 and aarch64 but in a different way than
the original patch.

Michal

Re: virsh create fails with "Unable to find any firmware to satisfy 'efi'" for aarch64 guest on macOS

[Michal Prívozník]

On 11/30/21 06:49, Nathan Vander Wilt wrote:
> On Mon, Nov 29, 2021 at 2:28 AM Ján Tomko  wrote:
>> Can you try with the latest libvirt?
>> 7.10.0-rc2 was just tagged today and should be out this week:
>> https://listman.redhat.com/archives/libvirt-announce/2021-November/msg2.html
> 
> Ah, but it looks like the arm64 -> VIR_ARCH_AARCH64 patch
> (https://github.com/ihsakashi/libvirt/commit/0f062221ae23e6ea0ed5e6ba65d47395581cb143.patch)
> is still in limbo? AFAICT that's the main issue I'm hitting now.

Looking into virArchFromHost() I can see uname() called which is then
passed to virArchFromString(). In here, uname machine (which is
equivalent to 'uname -m' from shell) is compared against virArchData array:

https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virarch.c#L42

So what you are saying is that 'uname -m' reports arm64 and not aarch64?
If that's the case then we should revisit the patch you mention.

> 
>> Andrea did some fixes that are supposed to help with Apple Silicon:
>> https://gitlab.com/libvirt/libvirt/-/issues/168
> 
> Yes, I happened to find that thread a little while after posting here
> and it has lots of tips but didn't directly address my confusion
> regarding how to manually specify the files. But based on the
> "internal error: undefined hardware architecture" I'm now getting I
> think I will need some of the patches.
> 
> For now I've just wrestled through direct usage of QEMU from the
> command line as ironically this whole exercise was to try keeping
> random build chains off my main "shiny new" macOS install itself.
> Thanks for the pointers and glad these issues are gradually getting
> worked out. I think I just must still be in the early adopters on the
> M1 platform and hit some growing pains!

Yeah, unfortunately I don't have access to M1 yet so all I can give is
suggestions.

Michal

Re: virsh create fails with "Unable to find any firmware to satisfy 'efi'" for aarch64 guest on macOS

[Michal Prívozník]

On 11/30/21 06:33, Nathan Vander Wilt wrote:
> On Fri, Nov 26, 2021 at 11:08 PM Michal Prívozník  wrote:
>> Since you are providing the path to both UEFI image and varstore you can
>> drop this 'firmware="efi"' attribute. It's what's causing troubles here.
> 
> Thank you, yes!
> 
>> A short trip into not so distant past. UEFI was introduced to QEMU, so
>> libvirt came up with 

Re: migrate libvirt config between machines

[Michal Prívozník]

On 11/29/21 17:37, daggs wrote:
> Greetings,
> 
> I'd like to migrate my current libvirt config from one machine to another, 
> what is the best way to do so?
> 
> Thanks
> 

Do you mean domain/network/... XMLs or {libvirtd,qemu,...}.config files?

For the former I suggest 'virsh dumpxml $dom > dom.xml' and copying over
the file onto new machine and defining the domain again 'virsh define
dom.xml' (s/dumpxml/net-dumpxml/ and s/define/net-define/ for networks
and so on).

If you run new enough libvirt there's even 'virsh migrate --offline'
that encapsulates steps from above.

For the latter just copy over the files onto new machine.

In theory, copying over XMLs should work too, but we don't really like
seeing users touching those behind libvirt's back as 'virsh define' does
more than just saving the XML file.

Michal

Re: virsh create fails with "Unable to find any firmware to satisfy 'efi'" for aarch64 guest on macOS

[Michal Prívozník]

On 11/27/21 00:08, Nathan Vander Wilt wrote:
> I am having trouble when I `virsh create test.xml` with an aarch64
> guest on a macOS (Apple Silicon) host. I've wrestled with a variety of
> issues but the one I simply haven't been able to get past is this
> regarding the UEFI firmware:
> 
> error: Failed to create domain from test.xml
> error: operation failed: Unable to find any firmware to satisfy 'efi'
> <
> This is with `virsh --version` of 7.9.0 and `qemu-system-aarch64
> --version` of 6.1.0, both installed via the common `brew` tool from
> its main 'homebrew/core' tap. I have confirmed that the loader/nvram
> files referenced do exist.
> 
> Running `qemu-system-aarch64 -L help` outputs two lines:
> 
> /opt/homebrew/Cellar/qemu/6.1.0_1/bin/../share/qemu-firmware
> /opt/homebrew/Cellar/qemu/6.1.0_1/bin/../share/qemu
> 
> The …/share/qemu-firmware folder does NOT exist, but symlinking it to
> a …/share/qemu/firmware folder which does exist does not improve the
> situation.
> 
> I have also tried renaming the …/share/qemu/firmware folder (e.g. `mv
> firmware zzz-firmware`) as some notes I found led me to believe that
> libvirst might ignore my loader/nvram settings entirely if QEMU had
> the "new" firmware JSON configuration stuff there? But again no
> improvement….
> 
> 
> How can I debug this further? What should I try next?
> 
> thanks,
> -natevw
> 
> 
> --- the test.xml domain configuration I am trying ---
> 
> http://libvirt.org/schemas/domain/qemu/1.0";>
>   test1
>   D45FCA3E-B873-4608-A0B8-3D8529E7CFB7
>   2048
>   
>   2
>   
>   
> 
> 
>   
>   

Since you are providing the path to both UEFI image and varstore you can
drop this 'firmware="efi"' attribute. It's what's causing troubles here.

A short trip into not so distant past. UEFI was introduced to QEMU, so
libvirt came up with hvm
>  type="pflash">/opt/homebrew/share/qemu/edk2-aarch64-code.fd
>  template="/opt/homebrew/share/qemu/edk2-arm-vars.fd">/Users/me/vm_testing/vm-test1.efi.fd
> 
> 
>   
>   
> 
> 
> 
>   
>   
> /opt/homebrew/bin/qemu-system-aarch64
> 
>   
>   
>   
> 
> 
>   
>   
> 
> 
>   
> 
> 
>   /dev/urandom
> 
>   
> 
> 
> 

Michal

Re: Libvirt + Debian Live = Heart Attack

[Michal Prívozník]

On 11/24/21 16:01, Elias Mobery wrote:
> Hello Michal, thank you for the reply!
> 
> I've carefully tested everything you suggested, thanks.
> 
> I set dynamic_ownership=0 and use these hooks during the live build for
> permissions. (I googled a lot, and apparently libvirt needs the images
> to be executable too)
> 
> chown -R libvirt-qemu:kvm  /var/lib/libvirt/images
> chmod -R g+rwx /var/lib/libvirt/images

I don't think this is correct. I don't have any of my images executable
and still run VMs happily.

> 
> Booting the live debian iso everything works in virt-manager, but again,
> after clicking "run", a copy of the vm image is created in
> /run/live/overlay/rw/var/lib/libvirt/images and only then does the VM start.

So who/what creates this copy? Is this a feature of underlying FS or
what exactly? It's definitely not libvirt who creates those copies.

> 
> Either it's still being chowned or chmodded somehow, or it's something
> else, but I can't stop this copy being made.
> 
> Interestingly, when I boot the Live debian iso and then copy the images
> into /var/lib/libvirt/images from my USB stick, the VM starts
> immediately without creating any copies in the /run/live directory.
> So my guess is that maybe the squashfs could be the issue?
> 
> Editing the XML
> 
> 
>       
>     
> 
> This results in an error:
> Unsupported Configuration:
> Security driver model 'null' not available>
> Here I tried setting security_driver=none in qemu.conf but same error after.
> 
> 
>     
>   

This should have been:

  

and if you are running with SELinux you want to repeat that for
model='selinux' too.

Michal

Re: Libvirt + Debian Live = Heart Attack

[Michal Prívozník]

On 11/23/21 17:25, Elias Mobery wrote:
> 
> Hi everyone!
> 
> I've built a Debian Live ISO with packages qemu and libvirt to run a VM
> in the live environment.
> 
> The guest images are placed in  /var/lib/libvirt/images and 2GB each.
> 
> Everything works great, except for one issue.
> 
> When starting a VM, libvirt automatically issues a chown command to the
> images, changing ownership.
> 
> This results in a copy of the images being created in
> /run/live/overlay/rw/var/lib/libvirt/images
> 
> I don't want these copies to be made but can't stop it.
> 
> I've tried editing qemu.conf user/group, dynamic ownership etc. without
> any luck.
> 
> Is there a way to STOP libvirt from changing the ownership of these images?
> 
> 

Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget
to restart the daemon after you made the change).

Alternatively, you can set  either for whole domain or
individual disks, e.g. like this:

  


  

  

or for whole domain:

...


  

I'm not sure what your setup is, but if chown() is a problem then what
if guest tries to write onto its disk? Wouldn't that create a copy in
overlay?

Michal

Re: Questions on device-added and device-removed events

[Michal Prívozník]

On 11/18/21 07:37, Han Han wrote:
> Hi developers,
> When the guest OS is booting, the event for live attachment could be
> caught by libvirt, while the event for live detachment could not.
> Version:
> libvirt-7.9
> qemu-kvm-6.1
> 
> Steps:
> Terminal 1:
> ➜  ~ virsh event --loop --all
> 
> Terminal 2:
> ➜  ~ virsh start avocado-vt-vm1; qemu-img create /tmp/vdb 10M; virsh
> attach-disk avocado-vt-vm1 /tmp/vdb vdb; virsh detach-disk
> avocado-vt-vm1 vdb
> 
> The outputs from terminal 1:
> event 'agent-lifecycle' for domain 'avocado-vt-vm1': state:
> 'disconnected' reason: 'domain started'      
> event 'lifecycle' for domain 'avocado-vt-vm1': Resumed Unpaused
> event 'lifecycle' for domain 'avocado-vt-vm1': Started Booted
> event 'device-added' for domain 'avocado-vt-vm1': virtio-disk1
> event 'agent-lifecycle' for domain 'avocado-vt-vm1': state: 'connected'
> reason: 'channel event'
> 
> As you can see, the device live attachment and the device detachment are
> both executed when the guest OS is booting, but there is only the event
> 'device-added' is recorded.
> So my questions are:
> 1. Is it the expected results for the results above?
> 2. If so, what is the internal difference between 'device-added' and
> 'device-removed'?
> 3. I notice there is the DEVICE_DELETED event in qmp? Any relation
> between the 'device-removed' of libvirt and the DEVICE_DELETED of qmp?

Detaching a device requires the guest cooperation, while in contrast the
device attach does not. So I'm guessing that your guest isn't in a state
where it can satisfy the detach request.

The DEVICE_REMOVED event is emitted whenever DEVICE_DELETED event
appears on the monitor (may be delayed for a couple of microseconds,
because libvirt needs to update its internal state first).

Hope this helps.

Michal

Re: how to use vhost-user-vsock

[Michal Prívozník]

On 11/16/21 14:54, Jiatong Shen wrote:
> Hello community,
> 
>   I am looking for a way to configure vhost-user-vsock using libvirt,
> but looks like it is not written in doc. How can I use vhost-user-vsock
> using libvirt? Thank you.
> 

Libvirt has basic support for vsock:

https://libvirt.org/formatdomain.html#vsock

For instance the following:


  



will get you:

-device
'{"driver":"vhost-vsock-pci","id":"vsock0","guest-cid":4,"vhostfd":"6789","bus":"pci.0","addr":"0x7"}'

But, I don't think we have vhost-*user*-vsock implemented yet.

Michal

Re: another upgrade another vm issue

[Michal Prívozník]

On 10/28/21 8:40 PM, daggs wrote:
> Greetings,
> 
> so I've upgraded my server and yet again, one of my vm lost a functionally.
> there is no usable sound card.
> xml: https://dpaste.com/CVR5M75VH
> in vm: https://snipboard.io/aZ7Dcf.jpg
> 
> outputs:
> utils_server /home/igor # qemu-system-x86_64 --version
> QEMU emulator version 6.0.0
> Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
> utils_server /home/igor # libvirtd --version
> libvirtd (libvirt) 7.8.0
> utils_server /home/igor #
> 
> any ideas?
> 
> Thanks.
> 
> 

I suspect it's related to:

  

in the domain XML. Selecting a backend might help.

  https://libvirt.org/formatdomain.html#audio-backends

Michal

Re: VM paused during multipath iSCSI reservation

[Michal Prívozník]

On 10/13/21 2:54 PM, Vojtech Juranek wrote:
> Hi,
> I'm trying to find the root cause for BZ #1898049 [1]. When setting up 
> Windows HA 
> cluster on Windows Server VMs run on top of oVirt, Windows cluster validator 
> runs couple of tests 
> and fails during test "Validate SCSI-3 Persistent Reservation" and one of the 
> VMs of 
> the cluster is paused with IO error. Disk definition is as follows:
> 
> 
> 
>io='native'/>
>   
> 
> 
>path='/var/lib/libvirt/qemu/domain-1-Windows-2016-2/pr-helper0.sock' 
> mode='client'/>
> 
>   
>   
>   
>   
>   
>   
> 
> 
> and libvirt error I get is bellow [2].
> 
> When I try to create reservation from Windows VM manually, I get following 
> error
> (but not sure I do it whole process correctly):
> 
> .\sg_persist --out --register --param-sark=123abc e:
> QEMU   QEMU HARDDISK   2.5+
> Peripheral device type: disk
> PR out (Register): command not supported
> sg_persist failed: Illegal request, Invalid opcode
> 
> 
> Do you have any ideas what could be wrong or how to determine
> the root cause of this this issue?
> 
> Thanks in advance.
> Vojta
> 
> 
> [1] https://bugzilla.redhat.com/1898049
> [2] libvirt debug log:
> 
> 2021-10-12 11:43:25.148+: 2006427: debug : qemuMonitorEmitIOError:1243 : 
> mon=0x7fb02006a020
> 2021-10-12 11:43:25.148+: 2006427: info : virObjectRef:402 : OBJECT_REF: 
> obj=0x7fb02006a020
> 2021-10-12 11:43:25.148+: 2006427: info : virObjectRef:402 : OBJECT_REF: 
> obj=0x7fafd0130020
> 2021-10-12 11:43:25.148+: 2000208: info : virObjectRef:402 : OBJECT_REF: 
> obj=0x7fafd010d340
> 2021-10-12 11:43:25.148+: 2006427: info : virObjectNew:258 : OBJECT_NEW: 
> obj=0x7fb020082590 classname=virDomainEventIOError
> 2021-10-12 11:43:25.148+: 2000208: info : vir_object_finalize:321 : 
> OBJECT_DISPOSE: obj=0x7fb020082500
> 2021-10-12 11:43:25.148+: 2006427: info : virObjectNew:258 : OBJECT_NEW: 
> obj=0x7fb020082620 classname=virDomainEventIOError
> 2021-10-12 11:43:25.148+: 2000208: info : virObjectUnref:380 : 
> OBJECT_UNREF: obj=0x7fb020082500
> 2021-10-12 11:43:25.148+: 2006427: debug : qemuProcessHandleIOError:907 : 
> Transitioned guest Windows-2016-2 to paused state due to IO error
> 2021-10-12 11:43:25.148+: 2000208: info : virObjectUnref:380 : 
> OBJECT_UNREF: obj=0x7fafd010d340
> 2021-10-12 11:43:25.148+: 2006427: info : virObjectNew:258 : OBJECT_NEW: 
> obj=0x7fafbc1fb8c0 classname=virDomainEventLifecycle
> 2021-10-12 11:43:25.148+: 2006427: debug : virDomainLockProcessPause:204 
> : plugin=0x7fafd01272a0 dom=0x7fb01400f5e0 state=0x7fb02401d768
> 2021-10-12 11:43:25.148+: 2006427: debug : virDomainLockManagerNew:134 : 
> plugin=0x7fafd01272a0 dom=0x7fb01400f5e0 withResources=1
> 2021-10-12 11:43:25.148+: 2006427: debug : 
> virLockManagerPluginGetDriver:276 : plugin=0x7fafd01272a0
> 2021-10-12 11:43:25.148+: 2006427: debug : virLockManagerNew:300 : 
> driver=0x7fafd444a000 type=0 nparams=5 params=0x7fafd77de640 flags=0x0
> 2021-10-12 11:43:25.148+: 2006427: debug : virLockManagerLogParams:97 :   
> key=uuid type=uuid value=70eee88c-ba2c-4c6c-bd51-c2b663db27f8
> 2021-10-12 11:43:25.148+: 2006427: debug : virLockManagerLogParams:90 :   
> key=name type=string value=Windows-2016-2
> 2021-10-12 11:43:25.148+: 2006427: debug : virLockManagerLogParams:78 :   
> key=id type=uint value=1
> 2021-10-12 11:43:25.148+: 2006427: debug : virLockManagerLogParams:78 :   
> key=pid type=uint value=2006418
> 2021-10-12 11:43:25.148+: 2006427: debug : virLockManagerLogParams:93 :   
> key=uri type=cstring value=(null)
> 2021-10-12 11:43:25.148+: 2006427: debug : virDomainLockManagerNew:146 : 
> Adding leases
> 2021-10-12 11:43:25.148+: 2006427: debug : virDomainLockManagerNew:151 : 
> Adding disks
> 2021-10-12 11:43:25.149+: 2006427: debug : 
> virDomainLockManagerAddImage:90 : Add disk 
> /rhev/data-center/mnt/blockSD/7c4f09b6-9e87-436f-bda9-22d1f0b50955/images/f5d6e074-dfe9-462d-8cfd-3e14b0eb5aea/766e36b2-84a6-43e7-a48b-a5f47e669860
> 2021-10-12 11:43:25.149+: 2006427: debug : virLockManagerAddResource:326 
> : lock=0x7fafbc19e250 type=0 
> name=/rhev/data-center/mnt/blockSD/7c4f09b6-9e87-436f-bda9-22d1f0b50955/images/f5d6e074-dfe9-462d-8cfd-3e14b0eb5aea/766e36b2-84a6-43e7-a48b-a5f47e669860
>  nparams=0 params=(nil) flags=0x0
> 2021-10-12 11:43:25.149+: 2006427: debug : 
> virDomainLockManagerAddImage:90 : Add disk 
> /dev/mapper/3600a09803830447a4f244c4657616f6f
> 2021-10-12 11:43:25.149+: 2006427: debug : virLockManagerAddResource:326 
> : lock=0x7fafbc19e250 type=0 
> name=/dev/mapper/3600a09803830447a4f244c4657616f6f nparams=0 params=(nil) 
> flags=0x2
> 2021-10-12 11:43:25.149+: 2006427: debug : virLockManagerRelease:359 : 
> lock=0x7fafbc19e250 state=0x7fb02401d768 flags=0x0
> 2021-10-12 11:43:25.149+: 2006427: debug : virLockManagerFree:381 : 
> lo

Re: security driver mismatch blocks live migration..

[Michal Prívozník]

On 9/18/21 4:39 AM, Jiatong Shen wrote:
> Hello community,
> 
> I am seeing following error during performing a live migration
> 
> libvirtError: unsupported configuration: Unable to find security driver for
> model apparmor
> 
> The root cause is the destination libvirtd runs inside a container and
> apparmor profile is not enabled. Right now I am researching how to enable
> apparmor profile for a k8s pod.
> 
> But Is it possible to do a live migrate an instance and disable security
> driver at the same time?

Yes. You can feed libvirt with a new XML at the beginning of migration.
This XML will be transferred to the destination and used to start QEMU -
hence users can use it to adapt to new environment.

Long story short, you should be able to dumpx XML of the guest, remove
apparmor profile and feed it to the migration.

virsh migrate --xml $modified.xml ...

Michal

Re: virsh domifaddr domain does not show static IP

[Michal Prívozník]

On 9/10/21 4:07 AM, Kaushal Shriyan wrote:
> 

> 
> Is there a way to find out the Static IP address if the KVM Guest VM
> instance is shut off? Thanks in advance. 
> 

This strongly depends on where you configure the static IP. If it's a
libvirt managed network then you can just dump the network XML and run
fairly trivial XPATH to learn the IP address. But if you use something
else to assign IP addresses, something that's outside of libvirt's scope
then I guess what Laine wrote in the other reply is your only hope.

Michal

Re: Question The order of the output of virsh guestinfo --filesystem

[Michal Prívozník]

On 8/23/21 1:19 PM, liuyd.f...@fujitsu.com wrote:
> Hi all.
> 
> I'm curious about the output order of virsh guestinfo --filesystem.
> Is the output random?


> virsh guestinfo output.
> It is not sorted by  name or total-bytes/used-bytes

It's in whatever order the guest agent reported. And looking into the
guest agent source code it's parsing /proc/self/mountinfo. So there's no
order guaranteed.

Michal

Re: Usage of virsh commands on guest lxc container are failing

[Michal Prívozník]

On 8/4/21 8:51 AM, Yuva Raj wrote:
> Hi Team,
> 
> I am new to the Qemu/KVM, libvirt and these technologies.
> Hypervisor is running on linux kernel with libvirtd version 1.3.2.
> I am trying to spawn LXC ubuntu(21.04) container using virsh -c lxc://
> commands on the hypervisor and ubuntu container is running now.
> On the ubuntu 21.04 container, I have installed debian "libvirt-clients"
> package to use the virsh utility.
> 
> Updated the /etc/libvirt/libvirt.conf with to the hypervisor Qemu URI.
> When I run virsh commands without URI, observing below error
> root@host:~# virsh list
> error: failed to connect to the hypervisor
> error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such
> file or directory

This means that you have no daemon running, which is not surprising
because IIUC you are running virsh from within the container and it's
not configured to share /var/run/libvirt with the host.

> 
> root@host:~# virsh -c  list
>  Id   Name  State
> -
> 2  VMrunning
> 
> If I execute the virsh commands without "-c" option, then ubuntu 21.04
> container would display the VM's running as same as hypervisor.
> Could you please help to manage this on the LXC ubuntu container?

So the problem is that plain virsh without any -c argument doesn't
connect to LXC driver? By default, virsh uses UNIX sockets to connect to
daemon. This can be overridden and another transport layer can be used
(e.g. TCP), it's documented here:

  https://libvirt.org/uri.html

For UNIX sockets to work you will need to share the directory with UNIX
sockets between container and the host. Alternatively, you can try some
other transport.

Michal

Re: ERROR Couldn't find hvm kernel for Ubuntu tree.

[Michal Prívozník]

On 8/4/21 5:21 PM, Kaushal Shriyan wrote:
> Hi,
> 
> #cat /etc/redhat-release
> CentOS Linux release 7.8.2003 (Core)
> #virt-install --version
> 1.5.0
> #virt-install -n snipeitassetmanagementubuntu --ram 8096 --vcpus 2
> --virt-type kvm --os-type linux --os-variant ubuntu18.04 --graphics none
> --location '
> http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/'
> --extra-args "console=tty0 console=ttyS0,115200n8" --disk
> path=/linuxkvmguestosdisk/snipeitassetmanagementubuntu.img,size=30
> 
> I am unable to connect to the guest VM instance. Any clue and i look
> forward to hearing from you. Thanks in advance.
> 

Hey, what do you mean by 'connect'? Is it connecting to graphical
console (e.g. via virt-viewer) or it's serial console or connecting to
guest's ssh or something else?

Michal

Re: investigate locks on a domain

[Michal Prívozník]

On 7/20/21 11:17 AM, Peter Krempa wrote:
> On Tue, Jul 20, 2021 at 15:52:58 +0800, Jiatong Shen wrote:
>> Hello community,
>>
>> I am seeing following log in production,
>>
>> 2021-07-20 07:43:49.417+: 3918294: error :
>> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
>> acquire state change lock (held by qemuProcessReconnect)
>> 2021-07-20 07:44:19.424+: 3918296: warning :
>> qemuDomainObjBeginJobInternal:4933 : Cannot start job (modify, none) for
>> domain instance-074e; current job is (modify, none) owned by (3919429
>> qemuProcessReconnect, 0 ) for (2183193s, 0s)
>> 2021-07-20 07:44:19.424+: 3918296: error :
>> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
>> acquire state change lock (held by qemuProcessReconnect)
>> 2021-07-20 07:44:49.428+: 3918296: warning :
>> qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for
>> domain instance-074e; current job is (modify, none) owned by (3919429
>> qemuProcessReconnect, 0 ) for (2183223s, 0s)
>> 2021-07-20 07:44:49.428+: 3918296: error :
>> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
>> acquire state change lock (held by qemuProcessReconnect)
>> 2021-07-20 07:45:19.429+: 3918298: warning :
>> qemuDomainObjBeginJobInternal:4933 :
>>
>> I am confused about what is qemuProcessReconnect and why it acquires a
>> domain state lock..
> 
> qemuProcessReconnect is an operation that is executed in a separate
> thread which re-establishes connection to a qemu process once you
> restart libvirtd.
> 
> This usually means that the reconnection process got stuck for some
> reason. Unfortunately your log doesn't show why and unless you've got a
> debug log prior to that happening it won't be possible.
> 
> Theoretically seeing which function the thread doing the reconnect is
> stuck at could perhaps show why.

This should be pretty easy to see though. All that's needed is attaching
gdb an running backtrace. Ideally over all threads as you don't know
which one is stuck trying to reconnect. Something like the following:

gdb -q -p $(pgrep libvirtd) -ex 't a a bt' -ex 'q'

Just make sure you have debuginfo installed for libvirt otherwise the
stacktrace won't be useful.

Michal

Re: openVswitch won't get to a virtual IP

[Michal Prívozník]

On 7/14/21 3:54 PM, lejeczek wrote:
> Hi guys.
> 
> Among all the mailing lists I'm subscriber of this one I thought, would
> where chances to find 'openvswitch'+'libvirtd' experts to share thoughts
> & suggestions should be grater.
> 
> A case:
> a) all guests are KVM.
> b) I have NM managed OVS bridge and libvirtd network to/on that bridge
> c) "regular" NM bridge and network off it in libvirt
> 
> problem is - guests on OVS network can _not_ ping a virtual IP of a
> guest(s) on 'regular' bridge network, but..  can ping "actual" IP of
> that same guest(s) okey.

I'm not sure what you mean by: "actual" IP of that same guest.

But in general, if you have two bridges you also have two separate
subnets. So you want to set up routing between them. And also inspect
rules that libvirt adds to its default network, because those might
clash with your setup.

Michal

Re: DAC & glusterFS - errors

[Michal Prívozník]

On 7/5/21 11:28 PM, lejeczek wrote:
> Hi guys
> 
> I'm trying to spin up a VM whose image is on a GlusterFS vol, and I fail:
> 
> -> $ virsh create /var/lib/pacemaker/conf.d/win10Pro.xml
> error: Failed to create domain from /var/lib/pacemaker/conf.d/win10Pro.xml
> error: failed to initialize gluster connection (src=0x7f0af807b410
> priv=0x7f0af8090070): Invalid argument
> 
> In logs:
> ...
> 2021-07-05 21:13:12.950+: 141524: warning :
> virSecurityDACSetOwnership:828 : Unable to restore label on
> 'win10Pro.qcow2'. XATTRs might have been left in inconsistent state.
> internal error: child reported (status=125): failed to initialize
> gluster connection (src=0x7f0b00097030 priv=0x7f0b000956c0): Invalid
> argument
> 
> Any ideas & suggestions greatly appreciated.
> thanks, L.
> 

Hey, since this is gluster I wonder if it is related to this bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1786301

That one is a deadlock scenario, but nevertheless might be worth turning
off seclabel remembering to see if that's the root cause. Just set
remember_owner=0 in /etc/libvirt/qemu.conf, restart libvirtd and see if
the problem persists.

Michal

Re: unaccessible directory in lxcContainerResolveSymlinks

[Michal Prívozník]

On 6/23/21 11:28 AM, Priyanka Gupta wrote:
> Hi,
> 
> If I try to add only /flash (without dir subdirerctory) into the XML, it
> works fine for me.
> Does it matter what sort of permissions(user/grp) /flash/dir has? 

A-ha! It can happen that root can't access some directories even though
a regular user can. Typically, fuse mounts (unless explicitly allowed
when mounting). So it does matter, yeah.

Michal

Re: unaccessible directory in lxcContainerResolveSymlinks

[Michal Prívozník]

On 6/23/21 11:08 AM, Priyanka Gupta wrote:
> Hi Michal,
> 
> This is how snippets from my XML looks like. Full XML at the end of the
> mail.. 
>     /usr/sbin/libvirt_lxc
>     
>       
>       
>     
>     
>       
>       
>     
> 
> The issue I am facing is, my container doesnt start. Fails at mounting
> this /flash/dir  with below message.
> 
> 2021-06-09 06:52:55.548+: 1: error : lxcContainerMountFSBind:1223 :
> Failed to bind mount directory /.oldroot/flash/dir to /flash/dir: No
> such file or directory

Yeah, the lxc driver thinks that /flash/dir doesn't exist.
Unfortunately, I am unable to reproduce - when I mkdir -p /flash/dir
and add corresponding  to one of my containers it boots up
just fine. What's your libvirt version?

Michal