Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
>> > OP: what is your sysctl kern.timecounter ? >> > >> >> kern.timecounter.tick=1 >> kern.timecounter.timestepwarnings=0 >> kern.timecounter.hardware=pvclock0 >> kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) >> acpitimer0(1000) >> >> > > You might try changing this and seeing if the load changes. I tried setting kern.timecounter.hardware to the alternatives shown in timecounter.choice: They all make the situation much worse, at -smp 8 it goes from the 4-5% of pvclock0 to 8%-15% of the others.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Sat, May 27, 2023 at 04:34:27PM +, bruma wrote: > > On Sat, May 27, 2023 at 10:29:37AM +0200, Claudio Jeker wrote: > >> On Sat, May 27, 2023 at 09:16:23AM +0100, Stuart Henderson wrote: > >> > On 2023/05/27 06:36, br...@mailbox.org wrote: > >> > > > >> > > > >> > > On Sat, 27 May 2023, Mike Larkin wrote: > >> > > > >> > > > probably IPI traffic then. not sure what else to say. If a few % > >> > > > host overhead > >> > > > is too much fot you with a 16 vCPU VM, I'd suggest reducing that. > >> > > > > >> > > > What is your workload for a 16 vcpu openbsd VM anyway? > >> > > > >> > > I would like to use the OpenBSD VM as my main workstation. I also need > >> > > to > >> > > use Linux for some graphic intensive stuff, so the ideal OpenBSD on > >> > > host > >> > > with vmm for Linux is not an option unfortunately. I guess I could > >> > > accept > >> > > that CPU usage price, but of course not having to pay it would be > >> > > better. > >> > > >> > OpenBSD doesn't do brilliantly with that many CPUs yet. Things are > >> > getting better but I think you're likely to find many workloads are a > >> > bit less laggy with half that. > >> > >> Also a few % CPU on the host can be caused by the interrupts caused by the > >> clocks on every CPU. It may be that we do not select a cheap clock source > >> like TSC and the result is much more overhead on the host. > >> > >> -- > >> :wq Claudio > >> > > > > yes I did not think of that; thanks Claudio! > > > > OP: what is your sysctl kern.timecounter ? > > > > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=pvclock0 > kern.timecounter.choice=i8254(0) pvclock0(1500) acpihpet0(1000) > acpitimer0(1000) > > You might try changing this and seeing if the load changes.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On 2023/05/27 15:35, Mike Larkin wrote: > On Sat, May 27, 2023 at 10:29:37AM +0200, Claudio Jeker wrote: > > On Sat, May 27, 2023 at 09:16:23AM +0100, Stuart Henderson wrote: > > > On 2023/05/27 06:36, br...@mailbox.org wrote: > > > > > > > > > > > > On Sat, 27 May 2023, Mike Larkin wrote: > > > > > > > > > probably IPI traffic then. not sure what else to say. If a few % host > > > > > overhead > > > > > is too much fot you with a 16 vCPU VM, I'd suggest reducing that. > > > > > > > > > > What is your workload for a 16 vcpu openbsd VM anyway? > > > > > > > > I would like to use the OpenBSD VM as my main workstation. I also need > > > > to > > > > use Linux for some graphic intensive stuff, so the ideal OpenBSD on host > > > > with vmm for Linux is not an option unfortunately. I guess I could > > > > accept > > > > that CPU usage price, but of course not having to pay it would be > > > > better. > > > > > > OpenBSD doesn't do brilliantly with that many CPUs yet. Things are > > > getting better but I think you're likely to find many workloads are a > > > bit less laggy with half that. > > > > Also a few % CPU on the host can be caused by the interrupts caused by the > > clocks on every CPU. It may be that we do not select a cheap clock source > > like TSC and the result is much more overhead on the host. > > > > -- > > :wq Claudio > > > > yes I did not think of that; thanks Claudio! > > OP: what is your sysctl kern.timecounter ? on kvm I would expect pvclock to be preferred if the driver thinks it's stable, otherwise probably acpihpet. fwiw mine looks like this. $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) pvclock0(500) acpihpet0(1000) acpitimer0(1000)
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Sat, May 27, 2023 at 10:29:37AM +0200, Claudio Jeker wrote: > On Sat, May 27, 2023 at 09:16:23AM +0100, Stuart Henderson wrote: > > On 2023/05/27 06:36, br...@mailbox.org wrote: > > > > > > > > > On Sat, 27 May 2023, Mike Larkin wrote: > > > > > > > probably IPI traffic then. not sure what else to say. If a few % host > > > > overhead > > > > is too much fot you with a 16 vCPU VM, I'd suggest reducing that. > > > > > > > > What is your workload for a 16 vcpu openbsd VM anyway? > > > > > > I would like to use the OpenBSD VM as my main workstation. I also need to > > > use Linux for some graphic intensive stuff, so the ideal OpenBSD on host > > > with vmm for Linux is not an option unfortunately. I guess I could accept > > > that CPU usage price, but of course not having to pay it would be better. > > > > OpenBSD doesn't do brilliantly with that many CPUs yet. Things are > > getting better but I think you're likely to find many workloads are a > > bit less laggy with half that. > > Also a few % CPU on the host can be caused by the interrupts caused by the > clocks on every CPU. It may be that we do not select a cheap clock source > like TSC and the result is much more overhead on the host. > > -- > :wq Claudio > yes I did not think of that; thanks Claudio! OP: what is your sysctl kern.timecounter ?
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Sat, May 27, 2023 at 09:16:23AM +0100, Stuart Henderson wrote: > On 2023/05/27 06:36, br...@mailbox.org wrote: > > > > > > On Sat, 27 May 2023, Mike Larkin wrote: > > > > > probably IPI traffic then. not sure what else to say. If a few % host > > > overhead > > > is too much fot you with a 16 vCPU VM, I'd suggest reducing that. > > > > > > What is your workload for a 16 vcpu openbsd VM anyway? > > > > I would like to use the OpenBSD VM as my main workstation. I also need to > > use Linux for some graphic intensive stuff, so the ideal OpenBSD on host > > with vmm for Linux is not an option unfortunately. I guess I could accept > > that CPU usage price, but of course not having to pay it would be better. > > OpenBSD doesn't do brilliantly with that many CPUs yet. Things are > getting better but I think you're likely to find many workloads are a > bit less laggy with half that. Also a few % CPU on the host can be caused by the interrupts caused by the clocks on every CPU. It may be that we do not select a cheap clock source like TSC and the result is much more overhead on the host. -- :wq Claudio
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On 2023/05/27 06:36, br...@mailbox.org wrote: > > > On Sat, 27 May 2023, Mike Larkin wrote: > > > probably IPI traffic then. not sure what else to say. If a few % host > > overhead > > is too much fot you with a 16 vCPU VM, I'd suggest reducing that. > > > > What is your workload for a 16 vcpu openbsd VM anyway? > > I would like to use the OpenBSD VM as my main workstation. I also need to > use Linux for some graphic intensive stuff, so the ideal OpenBSD on host > with vmm for Linux is not an option unfortunately. I guess I could accept > that CPU usage price, but of course not having to pay it would be better. OpenBSD doesn't do brilliantly with that many CPUs yet. Things are getting better but I think you're likely to find many workloads are a bit less laggy with half that.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Sat, 27 May 2023, Mike Larkin wrote: probably IPI traffic then. not sure what else to say. If a few % host overhead is too much fot you with a 16 vCPU VM, I'd suggest reducing that. What is your workload for a 16 vcpu openbsd VM anyway? I would like to use the OpenBSD VM as my main workstation. I also need to use Linux for some graphic intensive stuff, so the ideal OpenBSD on host with vmm for Linux is not an option unfortunately. I guess I could accept that CPU usage price, but of course not having to pay it would be better.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Sat, May 27, 2023 at 05:07:34AM +, br...@mailbox.org wrote: > > On Sat, 27 May 2023, Mike Larkin wrote: > > > On Fri, May 26, 2023 at 08:14:23PM +0200, br...@mailbox.org wrote: > > > On 05/26/2023 8:08 PM CEST Mike Larkin wrote: > > > > > > > > > > On Fri, May 26, 2023 at 07:16:09PM +0200, br...@mailbox.org wrote: > > > > > > On 05/26/2023 6:06 PM CEST Mike Larkin wrote: > > > > > > > > > > > > perf top on the linux side to see where qemu is spending it's time? > > > > > > > > > > Sure, I ran `perf top -p $PID` with $PID being the PID of the QEMU > > > > > process and copied the screen after a few seconds. Let me know if you > > > > > intended something different: > > > > > > > > > > PerfTop: 133 irqs/sec kernel:72.9% exact: 0.0% lost: 0/0 > > > > > drop: 0/0 [4000Hz cycles], (target_pid: 9939) > > > > > -- > > > > > > > > > > 25.35% [kvm_amd] [k] svm_vcpu_run > > > > > 4.18% [kernel] [k] native_write_msr > > > > > 4.01% [kernel] [k] native_read_msr > > > > > 3.73% [kernel] [k] read_tsc > > > > > 3.64% [kvm] [k] kvm_arch_vcpu_ioctl_run > > > > > 2.21% [kvm_amd] [k] svm_vcpu_load > > > > > 1.98% [kernel] [k] ktime_get > > > > > 1.47% [kvm] [k] kvm_apic_has_interrupt > > > > > 1.40% [kernel] [k] restore_fpregs_from_fpstate > > > > > 1.29% [kvm] [k] apic_has_interrupt_for_ppr > > > > > 1.18% [kernel] [k] check_preemption_disabled > > > > > 1.10% [kernel] [k] x86_pmu_disable_all > > > > > 1.07% [kernel] [k] __srcu_read_lock > > > > > 1.07% [kernel] [k] newidle_balance > > > > > 1.03% [kvm] [k] kvm_pmu_trigger_event > > > > > 0.98% [kernel] [k] amd_pmu_addr_offset > > > > > > > > > > I tried this also on the FreeBSD VM and the irqs/sec were between 2 > > > > > and 4. > > > > > > > > > > > > > you might just be bombarded with ipis. how many vcpus? > > > > > > It should be 16, I use `-smp 16 -cpu host` > > > > > > > try with less and see if that works. > > With -smp 4 it's better although still worse than FreeBSD/Linux. In htop > OpenBSD is in the 1.3-2.6% range while the other OSes are at 0-0.7%. As I > said, the FreeBSD/Linux VMs continue to stay in the 0% even at -smp 16, > while I've seen OpenBSD idle at up to 7-8% with that. probably IPI traffic then. not sure what else to say. If a few % host overhead is too much fot you with a 16 vCPU VM, I'd suggest reducing that. What is your workload for a 16 vcpu openbsd VM anyway?
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Sat, 27 May 2023, Mike Larkin wrote: On Fri, May 26, 2023 at 08:14:23PM +0200, br...@mailbox.org wrote: On 05/26/2023 8:08 PM CEST Mike Larkin wrote: On Fri, May 26, 2023 at 07:16:09PM +0200, br...@mailbox.org wrote: On 05/26/2023 6:06 PM CEST Mike Larkin wrote: perf top on the linux side to see where qemu is spending it's time? Sure, I ran `perf top -p $PID` with $PID being the PID of the QEMU process and copied the screen after a few seconds. Let me know if you intended something different: PerfTop: 133 irqs/sec kernel:72.9% exact: 0.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 9939) -- 25.35% [kvm_amd] [k] svm_vcpu_run 4.18% [kernel] [k] native_write_msr 4.01% [kernel] [k] native_read_msr 3.73% [kernel] [k] read_tsc 3.64% [kvm] [k] kvm_arch_vcpu_ioctl_run 2.21% [kvm_amd] [k] svm_vcpu_load 1.98% [kernel] [k] ktime_get 1.47% [kvm] [k] kvm_apic_has_interrupt 1.40% [kernel] [k] restore_fpregs_from_fpstate 1.29% [kvm] [k] apic_has_interrupt_for_ppr 1.18% [kernel] [k] check_preemption_disabled 1.10% [kernel] [k] x86_pmu_disable_all 1.07% [kernel] [k] __srcu_read_lock 1.07% [kernel] [k] newidle_balance 1.03% [kvm] [k] kvm_pmu_trigger_event 0.98% [kernel] [k] amd_pmu_addr_offset I tried this also on the FreeBSD VM and the irqs/sec were between 2 and 4. you might just be bombarded with ipis. how many vcpus? It should be 16, I use `-smp 16 -cpu host` try with less and see if that works. With -smp 4 it's better although still worse than FreeBSD/Linux. In htop OpenBSD is in the 1.3-2.6% range while the other OSes are at 0-0.7%. As I said, the FreeBSD/Linux VMs continue to stay in the 0% even at -smp 16, while I've seen OpenBSD idle at up to 7-8% with that.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Fri, May 26, 2023 at 08:14:23PM +0200, br...@mailbox.org wrote: > On 05/26/2023 8:08 PM CEST Mike Larkin wrote: > > > > On Fri, May 26, 2023 at 07:16:09PM +0200, br...@mailbox.org wrote: > > > > On 05/26/2023 6:06 PM CEST Mike Larkin wrote: > > > > > > > > perf top on the linux side to see where qemu is spending it's time? > > > > > > Sure, I ran `perf top -p $PID` with $PID being the PID of the QEMU > > > process and copied the screen after a few seconds. Let me know if you > > > intended something different: > > > > > > PerfTop: 133 irqs/sec kernel:72.9% exact: 0.0% lost: 0/0 drop: > > > 0/0 [4000Hz cycles], (target_pid: 9939) > > > -- > > > > > > 25.35% [kvm_amd] [k] svm_vcpu_run > > > 4.18% [kernel] [k] native_write_msr > > > 4.01% [kernel] [k] native_read_msr > > > 3.73% [kernel] [k] read_tsc > > > 3.64% [kvm] [k] kvm_arch_vcpu_ioctl_run > > > 2.21% [kvm_amd] [k] svm_vcpu_load > > > 1.98% [kernel] [k] ktime_get > > > 1.47% [kvm] [k] kvm_apic_has_interrupt > > > 1.40% [kernel] [k] restore_fpregs_from_fpstate > > > 1.29% [kvm] [k] apic_has_interrupt_for_ppr > > > 1.18% [kernel] [k] check_preemption_disabled > > > 1.10% [kernel] [k] x86_pmu_disable_all > > > 1.07% [kernel] [k] __srcu_read_lock > > > 1.07% [kernel] [k] newidle_balance > > > 1.03% [kvm] [k] kvm_pmu_trigger_event > > > 0.98% [kernel] [k] amd_pmu_addr_offset > > > > > > I tried this also on the FreeBSD VM and the irqs/sec were between 2 and 4. > > > > > > > you might just be bombarded with ipis. how many vcpus? > > It should be 16, I use `-smp 16 -cpu host` > try with less and see if that works.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On 05/26/2023 8:08 PM CEST Mike Larkin wrote: > On Fri, May 26, 2023 at 07:16:09PM +0200, br...@mailbox.org wrote: > > > On 05/26/2023 6:06 PM CEST Mike Larkin wrote: > > > > > > perf top on the linux side to see where qemu is spending it's time? > > > > Sure, I ran `perf top -p $PID` with $PID being the PID of the QEMU process > > and copied the screen after a few seconds. Let me know if you intended > > something different: > > > > PerfTop: 133 irqs/sec kernel:72.9% exact: 0.0% lost: 0/0 drop: 0/0 > > [4000Hz cycles], (target_pid: 9939) > > -- > > > > 25.35% [kvm_amd] [k] svm_vcpu_run > > 4.18% [kernel] [k] native_write_msr > > 4.01% [kernel] [k] native_read_msr > > 3.73% [kernel] [k] read_tsc > > 3.64% [kvm] [k] kvm_arch_vcpu_ioctl_run > > 2.21% [kvm_amd] [k] svm_vcpu_load > > 1.98% [kernel] [k] ktime_get > > 1.47% [kvm] [k] kvm_apic_has_interrupt > > 1.40% [kernel] [k] restore_fpregs_from_fpstate > > 1.29% [kvm] [k] apic_has_interrupt_for_ppr > > 1.18% [kernel] [k] check_preemption_disabled > > 1.10% [kernel] [k] x86_pmu_disable_all > > 1.07% [kernel] [k] __srcu_read_lock > > 1.07% [kernel] [k] newidle_balance > > 1.03% [kvm] [k] kvm_pmu_trigger_event > > 0.98% [kernel] [k] amd_pmu_addr_offset > > > > I tried this also on the FreeBSD VM and the irqs/sec were between 2 and 4. > > > > you might just be bombarded with ipis. how many vcpus? It should be 16, I use `-smp 16 -cpu host`
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Fri, May 26, 2023 at 07:16:09PM +0200, br...@mailbox.org wrote: > > On 05/26/2023 6:06 PM CEST Mike Larkin wrote: > > > > perf top on the linux side to see where qemu is spending it's time? > > Sure, I ran `perf top -p $PID` with $PID being the PID of the QEMU process > and copied the screen after a few seconds. Let me know if you intended > something different: > > PerfTop: 133 irqs/sec kernel:72.9% exact: 0.0% lost: 0/0 drop: 0/0 > [4000Hz cycles], (target_pid: 9939) > -- > > 25.35% [kvm_amd] [k] svm_vcpu_run > 4.18% [kernel] [k] native_write_msr > 4.01% [kernel] [k] native_read_msr > 3.73% [kernel] [k] read_tsc > 3.64% [kvm] [k] kvm_arch_vcpu_ioctl_run > 2.21% [kvm_amd] [k] svm_vcpu_load > 1.98% [kernel] [k] ktime_get > 1.47% [kvm] [k] kvm_apic_has_interrupt > 1.40% [kernel] [k] restore_fpregs_from_fpstate > 1.29% [kvm] [k] apic_has_interrupt_for_ppr > 1.18% [kernel] [k] check_preemption_disabled > 1.10% [kernel] [k] x86_pmu_disable_all > 1.07% [kernel] [k] __srcu_read_lock > 1.07% [kernel] [k] newidle_balance > 1.03% [kvm] [k] kvm_pmu_trigger_event > 0.98% [kernel] [k] amd_pmu_addr_offset > > I tried this also on the FreeBSD VM and the irqs/sec were between 2 and 4. > you might just be bombarded with ipis. how many vcpus?
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
> On 05/26/2023 6:06 PM CEST Mike Larkin wrote: > > perf top on the linux side to see where qemu is spending it's time? Sure, I ran `perf top -p $PID` with $PID being the PID of the QEMU process and copied the screen after a few seconds. Let me know if you intended something different: PerfTop: 133 irqs/sec kernel:72.9% exact: 0.0% lost: 0/0 drop: 0/0 [4000Hz cycles], (target_pid: 9939) -- 25.35% [kvm_amd] [k] svm_vcpu_run 4.18% [kernel] [k] native_write_msr 4.01% [kernel] [k] native_read_msr 3.73% [kernel] [k] read_tsc 3.64% [kvm] [k] kvm_arch_vcpu_ioctl_run 2.21% [kvm_amd] [k] svm_vcpu_load 1.98% [kernel] [k] ktime_get 1.47% [kvm] [k] kvm_apic_has_interrupt 1.40% [kernel] [k] restore_fpregs_from_fpstate 1.29% [kvm] [k] apic_has_interrupt_for_ppr 1.18% [kernel] [k] check_preemption_disabled 1.10% [kernel] [k] x86_pmu_disable_all 1.07% [kernel] [k] __srcu_read_lock 1.07% [kernel] [k] newidle_balance 1.03% [kvm] [k] kvm_pmu_trigger_event 0.98% [kernel] [k] amd_pmu_addr_offset I tried this also on the FreeBSD VM and the irqs/sec were between 2 and 4.
Re: OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% idle
On Fri, May 26, 2023 at 09:11:57AM +0200, br...@mailbox.org wrote: > >Synopsis:OpenBSD in QEMU KVM: High QEMU CPU usage when OpenBSD is 100% > >idle > >Category:kernel > >Environment: > System : OpenBSD 7.3 > Details : OpenBSD 7.3 (GENERIC.MP) #1125: Sat Mar 25 10:36:29 MDT > 2023 > > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > > Architecture: OpenBSD.amd64 > Machine : amd64 > >Description: > I installed OpenBSD 7.3 inside a QEMU QCOW2 disk with QEMU > running on Linux. I ran `top -S` in the VM and idle is at 100%. > So I expect CPU usage of the host QEMU process to be 0%, instead > it ranges from 4% to 6%. I tried other VMs with the same QEMU > settings but running Linux and FreeBSD, in both cases the QEMU > process stays infact at 0% usage. I attach `vmstat -i` as was > suggested on IRC. Not sure if relevant but on FreeBSD the Total > rate given by the same command is around 50. > > >How-To-Repeat: > This is the QEMU args used to run the VM: > > qemu-system-x86_64 -enable-kvm -cpu host -smp 16 -m 2048 > -nodefaults \ -drive if=virtio,format=qcow2,file=openbsd.qcow2 > -nic user,model=virtio-net-pci,id=net0 -display curses -vga std > > > vmstat -i: > interrupt total rate > irq114/virtio0 102450 > irq98/virtio1 291362 > irq100/fdc0 10 > irq144/pckbc0 140 > irq0/clock 33566752 3177 > irq0/ipi 164241 15 > Total33770389 3196 perf top on the linux side to see where qemu is spending it's time?
