Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided
On 12/14/21 01:41, Qu Wenruo wrote: > > > On 2021/12/14 00:49, Marc Zyngier wrote: >> On Mon, 13 Dec 2021 16:06:14 +, >> Peter Maydell wrote: >>> >>> KVM on big.little setups is a kernel-level question really; I've >>> cc'd the kvmarm list. >> >> Thanks Peter for throwing us under the big-little bus! ;-) >> >>> >>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo wrote: On 2021/12/13 21:17, Michal Prívozník wrote: > On 12/11/21 02:58, Qu Wenruo wrote: >> Hi, >> >> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI >> CM4, >> with upstream kernels. >> >> For RPI CM4 its mostly smooth sail, but on RK3399 due to its >> little.BIG >> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it >> brings quite some troubles for VMs. >> >> In short, without proper cpuset to bind the VM to either all A72 >> cores >> or all A55 cores, the VM will mostly fail to boot. >> >> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the >> though of it makes me sick). >> >> >> Currently the working xml is: >> >> 2 >> >> >> But even with vcpupin, pinning each vcpu to each physical core, VM >> will >> mostly fail to start up due to vcpu initialization failed with >> -EINVAL. >> >> Disclaimer: I know nothing about libvirt (and no, I don't want to >> know! ;-). >> >> However, for things to be reliable, you need to taskset the whole QEMU >> process to the CPU type you intend to use. > > Yep, that's what I'm doing. > >> That's because, AFAICT, >> QEMU will snapshot the system registers outside of the vcpu threads, >> and attempt to use the result to configure the actual vcpu threads. If >> they happen to run on different CPU types, the sysregs will differ in >> incompatible ways and an error will be returned. This may or may not >> be a bug, I don't know (I see it as a feature). > > Then this brings another question. > > If we can pin each vCPU to each physical core (both little and big), > then as long as the registers are per-vCPU based, it should be able to > pass both big and little cores to the VM. > > Yeah, I totally understand this screw up the scheduling, but that's at > least what (some insane) users want (just like me). > >> >> If you are annoyed with this behaviour, you can always use a different >> VMM that won't care about such difference (crosvm or kvmtool, to name >> a few). > > Sounds pretty interesting, a new world but without libvirt... > >> However, the guest will be able to observe the migration from >> one cpu type to another. This may or may not affect your guest's >> behaviour. > > Not sure if it's possible to pin each vCPU thread to each core, but let > me try. > Sure it is, for instance: pins vCPU#0 onto host CPUs 1-4, excluding 2; vCPU#1 onto host CPUs 0-1 and so on. You can also pin emulator (QEMU) and its iothreads. It's documented here: https://libvirt.org/formatdomain.html#cpu-tuning Michal
Re: Guest vm doesn't recover after the nfs connection resume
Hi Daniel, Thanks for your reply. I tried the nfs hard mount, and got the same behavior of the soft mount. But in the /var/log/message, got nfs server recovery message which is not printed when mounting as soft mode. Dec 14 02:12:47 test-1 kernel: nfs: server ip not responding, still trying Dec 14 02:13:39 test-1 kernel: nfs: server ip not responding, timed out *Dec 14 02:14:34 test-1 kernel: nfs: server ip OK* Dec 14 02:14:34 test-1 kernel: NFS: __nfs4_reclaim_open_state: Lock reclaim failed! According to my understanding the vm boot process will not recover(the vm is still in running state, never paused) to normal until restarting the vm guest. And it is not the issue of libvirt or qemu, it is just the correct behavior with the nfs connection timeout, right? Thanks, Liang Cong On Thu, Dec 9, 2021 at 6:03 PM Daniel P. Berrangé wrote: > On Thu, Dec 09, 2021 at 05:54:15PM +0800, Liang Cong wrote: > > Dear developers： > > > > I found one issue during regular test and I could not confirm whether it > is > > a libvirt|qemu issue or it is a nfs client issue or it is not an issue, > so > > could you help to check it? > > Below is the issue reproduce steps: > > > > 1.there is a nfs server with exports file like: > > /nfs *(async,rw,no_root_squash) > > 2. host machine soft mount nfs: > > mount nfs_server_ip:/nfs /var/lib/libvirt/images/nfs -o v4,soft > > 3. start a guest vm with disk tag xml like below: > > > > > > > file='/var/lib/libvirt/images/nfs/RHEL-8.6.0-20211102.1-x86_64.qcow2' > > index='1'/> > > > > > > > > > > 4.Start the vm and during the guest vm boot, apply the iptables rule to > > drop the nfs connection to nfs server > > iptables -A OUTPUT -d nfs_server_ip -p tcp --dport 2049 -j DROP > > 5. Wait until the error log appear in /var/log/message > > kernel: nfs: server nfs_server_ip not responding, timed out > > 6. delete the iptables rule to retain the connection to nfs server > > iptables -D OUTPUT -d nfs_server_ip -p tcp --dport 2049 -j DROP > > 7. check the guest vm, found the boot process with error and can not > > recover. > > rror: ../../grub-core/disk/i386/pc/biosdisk.c:546:failure reading sector > > > > 0x7ab8 from `hd0'. > > > > error: ../../grub-core/disk/i386/pc/biosdisk.c:546:failure reading sector > > > > 0x9190 from `hd0'. > > > > error: ../../grub-core/disk/i386/pc/biosdisk.c:546:failure reading sector > > > So this shows that I/O errors have been sent from the host to the guest. > > This means two things: > > - The host has reported I/O errors to QEMU > - QEMU is confjigured to reporte I/O errors to the guest >(rerror/werror attributes for disk config) > > I expect the first point there is a result of you using 'soft' for > the NFS mount - try it again with 'hard'. > > The alternative for 'rerror/werror' is to pause the guest, allowing > the host problem to be solved whereupon you unpause the guest. > > Overall this behaviour just looks like a result of your config > choices. > > Regards, > Daniel > -- > |: https://berrange.com -o- > https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org-o- > https://www.instagram.com/dberrange :| > >
Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided
On 2021/12/14 00:49, Marc Zyngier wrote: On Mon, 13 Dec 2021 16:06:14 +, Peter Maydell wrote: KVM on big.little setups is a kernel-level question really; I've cc'd the kvmarm list. Thanks Peter for throwing us under the big-little bus! ;-) On Mon, 13 Dec 2021 at 15:02, Qu Wenruo wrote: On 2021/12/13 21:17, Michal Prívozník wrote: On 12/11/21 02:58, Qu Wenruo wrote: Hi, Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, with upstream kernels. For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it brings quite some troubles for VMs. In short, without proper cpuset to bind the VM to either all A72 cores or all A55 cores, the VM will mostly fail to boot. s/A55/A53/. There were thankfully no A72+A55 ever produced (just the though of it makes me sick). Currently the working xml is: 2 But even with vcpupin, pinning each vcpu to each physical core, VM will mostly fail to start up due to vcpu initialization failed with -EINVAL. Disclaimer: I know nothing about libvirt (and no, I don't want to know! ;-). However, for things to be reliable, you need to taskset the whole QEMU process to the CPU type you intend to use. Yep, that's what I'm doing. That's because, AFAICT, QEMU will snapshot the system registers outside of the vcpu threads, and attempt to use the result to configure the actual vcpu threads. If they happen to run on different CPU types, the sysregs will differ in incompatible ways and an error will be returned. This may or may not be a bug, I don't know (I see it as a feature). Then this brings another question. If we can pin each vCPU to each physical core (both little and big), then as long as the registers are per-vCPU based, it should be able to pass both big and little cores to the VM. Yeah, I totally understand this screw up the scheduling, but that's at least what (some insane) users want (just like me). If you are annoyed with this behaviour, you can always use a different VMM that won't care about such difference (crosvm or kvmtool, to name a few). Sounds pretty interesting, a new world but without libvirt... However, the guest will be able to observe the migration from one cpu type to another. This may or may not affect your guest's behaviour. Not sure if it's possible to pin each vCPU thread to each core, but let me try. I personally find the QEMU behaviour reasonable. KVM/arm64 make little effort to support BL virtualisation as design choice (I value my sanity), and userspace is still in control of the placement. This brings a problem, in theory RK3399 SoC should out-perform BCM2711 in multi-core performance, but if a VM can only be bind to either A72 or A55 cores, then the performance is no longer competitive against BCM2711, wasting the PCIE 2.0 x4 capacity. Vote with your money. If you too think that BL systems are utter crap, do not buy them! Or treat them as 'two systems in one', which is what I do. From that angle, this is of great value! ;-) I guess I'm setting my expectation too high for rk3399, just seeing its multi-thread perf beating RPI4 and has better IO doesn't mean it's a perfect fit for VM. Hopes rk3588 could change it. For now I guess overclocking the big core to 2.2G is what I can do to grab more performance from the board. Thanks for your detailed reason and new advices! Qu I guess with projects like Asahi Linux making progress, there will be more and more such problems. Well, not more than any other big-little system. They suffer from similar issues, plus those resulting from not fully implementing the ARM architecture. They are however more consistent in their feature set than the ARM implementations ever were. Any clue on how to properly pass all physical CPU cores to VM for little.BIG setup? I have never met big.LITTLE but my understanding was that those big cores are compatible with little ones and the only difference is that the big ones are shut off if there's no demand (to save energy) leaving only the little ones running. No. They are all notionally running. It is the scheduler that places tasks (such as a vcpu) on a 'convenient' core, where 'convenient' depends on the scheduling policy. HTH, M.
Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided
On Mon, 13 Dec 2021 16:06:14 +, Peter Maydell wrote: > > KVM on big.little setups is a kernel-level question really; I've > cc'd the kvmarm list. Thanks Peter for throwing us under the big-little bus! ;-) > > On Mon, 13 Dec 2021 at 15:02, Qu Wenruo wrote: > > > > > > > > On 2021/12/13 21:17, Michal Prívozník wrote: > > > On 12/11/21 02:58, Qu Wenruo wrote: > > >> Hi, > > >> > > >> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, > > >> with upstream kernels. > > >> > > >> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG > > >> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it > > >> brings quite some troubles for VMs. > > >> > > >> In short, without proper cpuset to bind the VM to either all A72 cores > > >> or all A55 cores, the VM will mostly fail to boot. s/A55/A53/. There were thankfully no A72+A55 ever produced (just the though of it makes me sick). > > >> > > >> Currently the working xml is: > > >> > > >>2 > > >> > > >> > > >> But even with vcpupin, pinning each vcpu to each physical core, VM will > > >> mostly fail to start up due to vcpu initialization failed with -EINVAL. Disclaimer: I know nothing about libvirt (and no, I don't want to know! ;-). However, for things to be reliable, you need to taskset the whole QEMU process to the CPU type you intend to use. That's because, AFAICT, QEMU will snapshot the system registers outside of the vcpu threads, and attempt to use the result to configure the actual vcpu threads. If they happen to run on different CPU types, the sysregs will differ in incompatible ways and an error will be returned. This may or may not be a bug, I don't know (I see it as a feature). If you are annoyed with this behaviour, you can always use a different VMM that won't care about such difference (crosvm or kvmtool, to name a few). However, the guest will be able to observe the migration from one cpu type to another. This may or may not affect your guest's behaviour. I personally find the QEMU behaviour reasonable. KVM/arm64 make little effort to support BL virtualisation as design choice (I value my sanity), and userspace is still in control of the placement. > > >> This brings a problem, in theory RK3399 SoC should out-perform BCM2711 > > >> in multi-core performance, but if a VM can only be bind to either A72 or > > >> A55 cores, then the performance is no longer competitive against > > >> BCM2711, wasting the PCIE 2.0 x4 capacity. Vote with your money. If you too think that BL systems are utter crap, do not buy them! Or treat them as 'two systems in one', which is what I do. From that angle, this is of great value! ;-) > > >> I guess with projects like Asahi Linux making progress, there will be > > >> more and more such problems. Well, not more than any other big-little system. They suffer from similar issues, plus those resulting from not fully implementing the ARM architecture. They are however more consistent in their feature set than the ARM implementations ever were. > > >> > > >> Any clue on how to properly pass all physical CPU cores to VM for > > >> little.BIG setup? > > >> > > > > > > I have never met big.LITTLE but my understanding was that those big > > > cores are compatible with little ones and the only difference is that > > > the big ones are shut off if there's no demand (to save energy) leaving > > > only the little ones running. No. They are all notionally running. It is the scheduler that places tasks (such as a vcpu) on a 'convenient' core, where 'convenient' depends on the scheduling policy. HTH, M. -- Without deviation from the norm, progress is not possible.
Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided
KVM on big.little setups is a kernel-level question really; I've cc'd the kvmarm list. On Mon, 13 Dec 2021 at 15:02, Qu Wenruo wrote: > > > > On 2021/12/13 21:17, Michal Prívozník wrote: > > On 12/11/21 02:58, Qu Wenruo wrote: > >> Hi, > >> > >> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, > >> with upstream kernels. > >> > >> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG > >> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it > >> brings quite some troubles for VMs. > >> > >> In short, without proper cpuset to bind the VM to either all A72 cores > >> or all A55 cores, the VM will mostly fail to boot. > >> > >> Currently the working xml is: > >> > >>2 > >> > >> > >> But even with vcpupin, pinning each vcpu to each physical core, VM will > >> mostly fail to start up due to vcpu initialization failed with -EINVAL. > >> > >> > >> This brings a problem, in theory RK3399 SoC should out-perform BCM2711 > >> in multi-core performance, but if a VM can only be bind to either A72 or > >> A55 cores, then the performance is no longer competitive against > >> BCM2711, wasting the PCIE 2.0 x4 capacity. > >> > >> I guess with projects like Asahi Linux making progress, there will be > >> more and more such problems. > >> > >> Any clue on how to properly pass all physical CPU cores to VM for > >> little.BIG setup? > >> > > > > I have never met big.LITTLE but my understanding was that those big > > cores are compatible with little ones and the only difference is that > > the big ones are shut off if there's no demand (to save energy) leaving > > only the little ones running. > > The big ones are not disabled AFAIK. > > And even changing the CPU model to A53 (the little ones), it still fails > to boot, thus it looks like A72 is not really able to emulate A53 cores? > > > > > Anyway, this is likely too high level forum and I'd ask QEMU developers: > > > > https://www.qemu.org/support/ > > That's indeed the case, adding qemu to the CC list. > > And I found an existing bug report: > https://bugs.linaro.org/show_bug.cgi?id=1443 (This bug tracking system was essentially abandoned years ago; the status of leftover bugs within it isn't indicative of anything.) > But I still didn't get the point why the 1:1 CPU-to-vcpu mapping still > doesn't work. -- PMM
Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided
On 2021/12/13 21:17, Michal Prívozník wrote: On 12/11/21 02:58, Qu Wenruo wrote: Hi, Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, with upstream kernels. For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it brings quite some troubles for VMs. In short, without proper cpuset to bind the VM to either all A72 cores or all A55 cores, the VM will mostly fail to boot. Currently the working xml is: 2 But even with vcpupin, pinning each vcpu to each physical core, VM will mostly fail to start up due to vcpu initialization failed with -EINVAL. This brings a problem, in theory RK3399 SoC should out-perform BCM2711 in multi-core performance, but if a VM can only be bind to either A72 or A55 cores, then the performance is no longer competitive against BCM2711, wasting the PCIE 2.0 x4 capacity. I guess with projects like Asahi Linux making progress, there will be more and more such problems. Any clue on how to properly pass all physical CPU cores to VM for little.BIG setup? I have never met big.LITTLE but my understanding was that those big cores are compatible with little ones and the only difference is that the big ones are shut off if there's no demand (to save energy) leaving only the little ones running. The big ones are not disabled AFAIK. And even changing the CPU model to A53 (the little ones), it still fails to boot, thus it looks like A72 is not really able to emulate A53 cores? Anyway, this is likely too high level forum and I'd ask QEMU developers: https://www.qemu.org/support/ That's indeed the case, adding qemu to the CC list. And I found an existing bug report: https://bugs.linaro.org/show_bug.cgi?id=1443 But I still didn't get the point why the 1:1 CPU-to-vcpu mapping still doesn't work. Thanks, Qu Michal
Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided
On 12/11/21 02:58, Qu Wenruo wrote: > Hi, > > Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4, > with upstream kernels. > > For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG > setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it > brings quite some troubles for VMs. > > In short, without proper cpuset to bind the VM to either all A72 cores > or all A55 cores, the VM will mostly fail to boot. > > Currently the working xml is: > > 2 > > > But even with vcpupin, pinning each vcpu to each physical core, VM will > mostly fail to start up due to vcpu initialization failed with -EINVAL. > > > This brings a problem, in theory RK3399 SoC should out-perform BCM2711 > in multi-core performance, but if a VM can only be bind to either A72 or > A55 cores, then the performance is no longer competitive against > BCM2711, wasting the PCIE 2.0 x4 capacity. > > I guess with projects like Asahi Linux making progress, there will be > more and more such problems. > > Any clue on how to properly pass all physical CPU cores to VM for > little.BIG setup? > I have never met big.LITTLE but my understanding was that those big cores are compatible with little ones and the only difference is that the big ones are shut off if there's no demand (to save energy) leaving only the little ones running. Anyway, this is likely too high level forum and I'd ask QEMU developers: https://www.qemu.org/support/ Michal
Re: internal error - invalid characters
On Mon, Dec 13, 2021 at 09:39:20AM +, lejeczek wrote: > > > On 07/12/2021 11:28, Ján Tomko wrote: > > On a Monday in 2021, lejeczek wrote: > > > Hi guys. > > > > > > Have you seen something like this below? > > > ... > > > internal error: The string resource has invalid characters in its > > > value > > > internal error: The provided value contains invalid characters: > > > Solarstorm SFN5162F SFP+ Server Adapter > > > > > > > This error message was only present in libvirt v7.9.0. Since v7.10.0 > > libvirt should ignore non-printable characters. > > https://bugzilla.redhat.com/show_bug.cgi?id=2022589 > > > > Jano > > > Would that "bug" be a reason why my libvirtd fails? It shouldn't have a negative impact - it just means that VPD info would not be reported against that particular host device. Everything else should be unaffected. > I have libvirtd.service started and see that in the logs, then suffices I > do: > -> $ virsh list --all > (having no VMs started) > I next thing in the logs is: > ... > internal error: The provided value contains invalid characters: Solarstorm > SFN5162F SFP+ Server Adapter > internal error: The string resource has invalid characters in its value > Stopping Virtualization daemon... > libvirtd.service: Deactivated successfully. > Stopped Virtualization daemon. > > I'm on Centos 9 with libvirt-daemon-7.9.0-1.el9.x86_64. > With .service failed I still can start a VM. > -> $ systemctl status -l libvirtd.service > ○ libvirtd.service - Virtualization daemon > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; disabled; > vendor preset: dis> > Active: inactive (dead) since Mon 2021-12-13 09:32:04 GMT; 5min ago > TriggeredBy: ○ libvirtd-tcp.socket > ● libvirtd-ro.socket > ● libvirtd-admin.socket > ● libvirtd.socket > ○ libvirtd-tls.socket > > Is it that these versions and/or Centos 9 have introduced new (different > defaults) ways to use/mange 'libvirtd', with .socket and no .service? In RHEL-9 and therefore CentOS 9 we're switching to using modular daemons, so libvirtd is no longer expected to be used. https://fedoraproject.org/wiki/Changes/LibvirtModularDaemons https://libvirt.org/daemons.html a current fresh install should setup the modular daemons automatically and leave libvirtd (and its .sockets) entirely disabled. Older installs would remain with libvirtd. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
