Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On 06/17/2014 11:10 AM, Brad Smith wrote: boot -c disable mpbios Because ACPI is in use which takes higher precedence over MP BIOS. You have to disable acpimadt. THANKS GUYS!! This just resolved a blocker that had for 2 years prevented me from upgrading my OpenBSD kvm guests to anything beyond 5.1! /m
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On 17/06/14 22:00, Christiano F. Haesbaert wrote: On 17 June 2014 20:47, Mikolaj Kucharski miko...@kucharski.name wrote: Do you guys think it's worth opening bug report with RedHat to get them look into this, or is the problem more on OpenBSD side? Ideally I would like to run unmodified OpenBSD kernel on my VMs. One solution is to rewrite the OpenBSD i386 interrupt handling code on the apic case to use software masking, like amd64 and the i386 pic code. Can someone translate this as prefer amd64 arch over i386 when running on top of KVM? regards, G
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On Mon, Jun 16, 2014 at 11:07:39PM +0100, Kevin Chadwick wrote: previously on this list Mikolaj Kucharski contributed: by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you I cannot find how to enable 'the old pic controller' in libvirt with qemu-kvm. Do you know by any chance how to enable it? I believe he means disabling mpbios at OpenBSD's boot or in boot.conf means KVM will automatically fall back. Virtual hosting companies like arpnetworks generally ask you to do this for OpenBSD. boot -c disable mpbios Ah, I got confused. Yes, I'm aware of this, as I've seen this on the list archives mentioned few times. I actually tested this, and I don't see any difference. See at my below tests: OpenBSD i386/virtio (default) [test12] # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 3635.270 secs (1181471 bytes/sec) 60m35.50s real 0m1.24s user54m15.15s system OpenBSD i386/virtio (disable mpbios) [test13] # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 3628.306 secs (1183739 bytes/sec) 60m28.49s real 0m1.33s user54m8.30s system On the archives I have seen recommendation to disable mpbios while machine is slow in general, however I am experiencing only slow disk I/O. I thought my problem is unrelated to mpbios. With qemu-kvm from RHEL7 on bsd.sp there is no mpbios mentioned in dmesg(8) (I didn't test bsd.mp). See dmesg output at the bottom of this email. Also starting OpenBSD with mpbios disabled via boot_config(8) ends up with: --- dmesg.txt Sat Jun 14 15:49:02 2014 +++ disable-mpbios.txt Tue Jun 17 08:15:34 2014 @@ -4,6 +4,11 @@ cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) +User Kernel Config +UKC disable mpbios +368 mpbios0 disabled +UKC quit +Continuing... mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root @@ -22,7 +27,7 @@ ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 -bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800! +bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800 pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel 82441FX rev 0x02 pcib0 at pci0 dev 1 function 0 Intel 82371SB ISA rev 0x00 @@ -32,11 +37,11 @@ uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: apic 0 int 11 piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: apic 0 int 9 iic0 at piixpm0 -iic0: addr 0x1c 0f=00 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x1d 0f=00 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x4d 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 +iic0: addr 0x1c 0f=00 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x1d 0f=00 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x4d 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 virtio0 at pci0 dev 3 function 0 Qumranet Virtio Network rev 0x00: Virtio Network Device vio0 at virtio0: address 52:54:00:12:34:70 virtio0: apic 0 int 11 Full, default dmesg: OpenBSD 5.5-current (GENERIC) #162: Tue Jun 10 21:17:31 MDT 2014 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: QEMU Virtual CPU version 1.5.3 (AuthenticAMD 686-class, 512KB L2 cache) 2.61 GHz cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 06/23/99, BIOS32 rev. 0 @ 0xfc672, SMBIOS rev. 2.4 @ 0xfdde0 (10 entries) bios0: vendor Bochs version Bochs date 01/01/2011 bios0: Red Hat KVM acpi0 at bios0: rev 0 acpi0: sleep
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On 06/17/14 10:56, Mikolaj Kucharski wrote: On Mon, Jun 16, 2014 at 11:07:39PM +0100, Kevin Chadwick wrote: previously on this list Mikolaj Kucharski contributed: by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you I cannot find how to enable 'the old pic controller' in libvirt with qemu-kvm. Do you know by any chance how to enable it? I believe he means disabling mpbios at OpenBSD's boot or in boot.conf means KVM will automatically fall back. Virtual hosting companies like arpnetworks generally ask you to do this for OpenBSD. boot -c disable mpbios Ah, I got confused. Yes, I'm aware of this, as I've seen this on the list archives mentioned few times. I actually tested this, and I don't see any difference. See at my below tests: OpenBSD i386/virtio (default) [test12] # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 3635.270 secs (1181471 bytes/sec) 60m35.50s real 0m1.24s user54m15.15s system OpenBSD i386/virtio (disable mpbios) [test13] # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 3628.306 secs (1183739 bytes/sec) 60m28.49s real 0m1.33s user54m8.30s system On the archives I have seen recommendation to disable mpbios while machine is slow in general, however I am experiencing only slow disk I/O. I thought my problem is unrelated to mpbios. With qemu-kvm from RHEL7 on bsd.sp there is no mpbios mentioned in dmesg(8) (I didn't test bsd.mp). See dmesg output at the bottom of this email. Also starting OpenBSD with mpbios disabled via boot_config(8) ends up with: --- dmesg.txt Sat Jun 14 15:49:02 2014 +++ disable-mpbios.txt Tue Jun 17 08:15:34 2014 @@ -4,6 +4,11 @@ cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) +User Kernel Config +UKC disable mpbios +368 mpbios0 disabled +UKC quit +Continuing... mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root @@ -22,7 +27,7 @@ ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 -bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800! +bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800 pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel 82441FX rev 0x02 pcib0 at pci0 dev 1 function 0 Intel 82371SB ISA rev 0x00 @@ -32,11 +37,11 @@ uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: apic 0 int 11 piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: apic 0 int 9 iic0 at piixpm0 -iic0: addr 0x1c 0f=00 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x1d 0f=00 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x4d 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 -iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 +iic0: addr 0x1c 0f=00 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x1d 0f=00 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x4d 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 +iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=8fc5 01=8fc5 02=8fc5 03=8fc5 04=8fc5 05=8fc5 06=8fc5 07=8fc5 virtio0 at pci0 dev 3 function 0 Qumranet Virtio Network rev 0x00: Virtio Network Device vio0 at virtio0: address 52:54:00:12:34:70 virtio0: apic 0 int 11 Full, default dmesg: OpenBSD 5.5-current (GENERIC) #162: Tue Jun 10 21:17:31 MDT 2014 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: QEMU Virtual CPU version 1.5.3 (AuthenticAMD 686-class, 512KB L2 cache) 2.61 GHz cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 06/23/99, BIOS32 rev. 0 @ 0xfc672, SMBIOS rev. 2.4 @ 0xfdde0 (10 entries) bios0: vendor Bochs version Bochs date 01/01/2011 bios0:
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On 17/06/14 4:56 AM, Mikolaj Kucharski wrote: On Mon, Jun 16, 2014 at 11:07:39PM +0100, Kevin Chadwick wrote: previously on this list Mikolaj Kucharski contributed: by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you I cannot find how to enable 'the old pic controller' in libvirt with qemu-kvm. Do you know by any chance how to enable it? I believe he means disabling mpbios at OpenBSD's boot or in boot.conf means KVM will automatically fall back. Virtual hosting companies like arpnetworks generally ask you to do this for OpenBSD. boot -c disable mpbios Ah, I got confused. Yes, I'm aware of this, as I've seen this on the list archives mentioned few times. I actually tested this, and I don't see any difference. See at my below tests: Because ACPI is in use which takes higher precedence over MP BIOS. You have to disable acpimadt. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On Tue, Jun 17, 2014 at 05:10:51AM -0400, Brad Smith wrote: On 17/06/14 4:56 AM, Mikolaj Kucharski wrote: On Mon, Jun 16, 2014 at 11:07:39PM +0100, Kevin Chadwick wrote: previously on this list Mikolaj Kucharski contributed: by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you I cannot find how to enable 'the old pic controller' in libvirt with qemu-kvm. Do you know by any chance how to enable it? I believe he means disabling mpbios at OpenBSD's boot or in boot.conf means KVM will automatically fall back. Virtual hosting companies like arpnetworks generally ask you to do this for OpenBSD. boot -c disable mpbios Ah, I got confused. Yes, I'm aware of this, as I've seen this on the list archives mentioned few times. I actually tested this, and I don't see any difference. See at my below tests: Because ACPI is in use which takes higher precedence over MP BIOS. You have to disable acpimadt. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. Randomly disabling parts of the kernel is likely to cause other problems. -ml
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
Mike, On Tue, Jun 17, 2014 at 10:30:23AM -0700, Mike Larkin wrote: On Tue, Jun 17, 2014 at 05:10:51AM -0400, Brad Smith wrote: Because ACPI is in use which takes higher precedence over MP BIOS. You have to disable acpimadt. Randomly disabling parts of the kernel is likely to cause other problems. I agree, but disabling mpbios and acpimadt makes a huge difference for me on qemu-kvm-1.5.3-60.el7.x86_64: OpenBSD i386/virtio (bsd.sp disable mpbios and acpimadt) # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 18.524 secs (231854084 bytes/sec) 0m18.52s real 0m0.24s user 0m14.48s system It takes now 18 seconds to run above dd(1), when previously it took 60 mintues. Thanks Christiano and Brad for the tips. Do you guys think it's worth opening bug report with RedHat to get them look into this, or is the problem more on OpenBSD side? Ideally I would like to run unmodified OpenBSD kernel on my VMs. $ diff -I'^iic0' dmesg.txt disable-mpbios-and-acpimadt.txt --- dmesg.txt Sat Jun 14 15:49:02 2014 +++ disable-mpbios-and-acpimadt.txt Tue Jun 17 13:30:46 2014 @@ -4,6 +4,13 @@ cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) +User Kernel Config +UKC disable mpbios +368 mpbios0 disabled +UKC disable acpimadt +501 acpimadt0 disabled +UKC quit +Continuing... mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root @@ -15,22 +22,20 @@ acpi0: tables DSDT FACP SSDT APIC RSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits -acpimadt0 at acpi0 addr 0xfee0: PC-AT compat -cpu0 at mainbus0: apid 0 (boot processor) -mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges -cpu0: apic clock running at 999MHz -ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 +mpbios at bios0 function 0x0 not configured bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800! +cpu0 at mainbus0: (uniprocessor) +mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel 82441FX rev 0x02 pcib0 at pci0 dev 1 function 0 Intel 82371SB ISA rev 0x00 pciide0 at pci0 dev 1 function 1 Intel 82371SB IDE rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives) -uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: apic 0 int 11 -piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: apic 0 int 9 +uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: irq 11 +piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: irq 9 iic0 at piixpm0 iic0: addr 0x1c 0f=00 words 00=d4d0 01=d4d0 02=d4d0 03=d4d0 04=d4d0 05=d4d0 06=d4d0 07=d4d0 iic0: addr 0x1d 0f=00 words 00=d4d0 01=d4d0 02=d4d0 03=d4d0 04=d4d0 05=d4d0 06=d4d0 07=d4d0 @@ -39,13 +44,13 @@ iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 virtio0 at pci0 dev 3 function 0 Qumranet Virtio Network rev 0x00: Virtio Network Device vio0 at virtio0: address 52:54:00:12:34:70 -virtio0: apic 0 int 11 +virtio0: irq 11 virtio1 at pci0 dev 4 function 0 Qumranet Virtio Storage rev 0x00: Virtio Block Device vioblk0 at virtio1 scsibus1 at vioblk0: 2 targets sd0 at scsibus1 targ 0 lun 0: VirtIO, Block Device, SCSI3 0/direct fixed sd0: 102400MB, 512 bytes/sector, 209715200 sectors -virtio1: apic 0 int 11 +virtio1: irq 11 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo -- best regards q#
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On Tue, Jun 17, 2014 at 07:47:32PM +0100, Mikolaj Kucharski wrote: Mike, On Tue, Jun 17, 2014 at 10:30:23AM -0700, Mike Larkin wrote: On Tue, Jun 17, 2014 at 05:10:51AM -0400, Brad Smith wrote: Because ACPI is in use which takes higher precedence over MP BIOS. You have to disable acpimadt. Randomly disabling parts of the kernel is likely to cause other problems. I agree, but disabling mpbios and acpimadt makes a huge difference for me on qemu-kvm-1.5.3-60.el7.x86_64: Please don't bother filing any bug reports for a system where you've done this. -ml OpenBSD i386/virtio (bsd.sp disable mpbios and acpimadt) # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 18.524 secs (231854084 bytes/sec) 0m18.52s real 0m0.24s user 0m14.48s system It takes now 18 seconds to run above dd(1), when previously it took 60 mintues. Thanks Christiano and Brad for the tips. Do you guys think it's worth opening bug report with RedHat to get them look into this, or is the problem more on OpenBSD side? Ideally I would like to run unmodified OpenBSD kernel on my VMs. $ diff -I'^iic0' dmesg.txt disable-mpbios-and-acpimadt.txt --- dmesg.txt Sat Jun 14 15:49:02 2014 +++ disable-mpbios-and-acpimadt.txt Tue Jun 17 13:30:46 2014 @@ -4,6 +4,13 @@ cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) +User Kernel Config +UKC disable mpbios +368 mpbios0 disabled +UKC disable acpimadt +501 acpimadt0 disabled +UKC quit +Continuing... mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root @@ -15,22 +22,20 @@ acpi0: tables DSDT FACP SSDT APIC RSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits -acpimadt0 at acpi0 addr 0xfee0: PC-AT compat -cpu0 at mainbus0: apid 0 (boot processor) -mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges -cpu0: apic clock running at 999MHz -ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 +mpbios at bios0 function 0x0 not configured bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800! +cpu0 at mainbus0: (uniprocessor) +mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel 82441FX rev 0x02 pcib0 at pci0 dev 1 function 0 Intel 82371SB ISA rev 0x00 pciide0 at pci0 dev 1 function 1 Intel 82371SB IDE rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives) -uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: apic 0 int 11 -piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: apic 0 int 9 +uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: irq 11 +piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: irq 9 iic0 at piixpm0 iic0: addr 0x1c 0f=00 words 00=d4d0 01=d4d0 02=d4d0 03=d4d0 04=d4d0 05=d4d0 06=d4d0 07=d4d0 iic0: addr 0x1d 0f=00 words 00=d4d0 01=d4d0 02=d4d0 03=d4d0 04=d4d0 05=d4d0 06=d4d0 07=d4d0 @@ -39,13 +44,13 @@ iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 virtio0 at pci0 dev 3 function 0 Qumranet Virtio Network rev 0x00: Virtio Network Device vio0 at virtio0: address 52:54:00:12:34:70 -virtio0: apic 0 int 11 +virtio0: irq 11 virtio1 at pci0 dev 4 function 0 Qumranet Virtio Storage rev 0x00: Virtio Block Device vioblk0 at virtio1 scsibus1 at vioblk0: 2 targets sd0 at scsibus1 targ 0 lun 0: VirtIO, Block Device, SCSI3 0/direct fixed sd0: 102400MB, 512 bytes/sector, 209715200 sectors -virtio1: apic 0 int 11 +virtio1: irq 11 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo -- best regards q#
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On 17 June 2014 20:47, Mikolaj Kucharski miko...@kucharski.name wrote: Mike, On Tue, Jun 17, 2014 at 10:30:23AM -0700, Mike Larkin wrote: On Tue, Jun 17, 2014 at 05:10:51AM -0400, Brad Smith wrote: Because ACPI is in use which takes higher precedence over MP BIOS. You have to disable acpimadt. Randomly disabling parts of the kernel is likely to cause other problems. I agree, but disabling mpbios and acpimadt makes a huge difference for me on qemu-kvm-1.5.3-60.el7.x86_64: OpenBSD i386/virtio (bsd.sp disable mpbios and acpimadt) # time dd if=/dev/zero of=/tmp/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 18.524 secs (231854084 bytes/sec) 0m18.52s real 0m0.24s user 0m14.48s system It takes now 18 seconds to run above dd(1), when previously it took 60 mintues. Thanks Christiano and Brad for the tips. Do you guys think it's worth opening bug report with RedHat to get them look into this, or is the problem more on OpenBSD side? Ideally I would like to run unmodified OpenBSD kernel on my VMs. One solution is to rewrite the OpenBSD i386 interrupt handling code on the apic case to use software masking, like amd64 and the i386 pic code. $ diff -I'^iic0' dmesg.txt disable-mpbios-and-acpimadt.txt --- dmesg.txt Sat Jun 14 15:49:02 2014 +++ disable-mpbios-and-acpimadt.txt Tue Jun 17 13:30:46 2014 @@ -4,6 +4,13 @@ cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,LONG,SSE3,CX16,LAHF,ABM,SSE4A,PERF real mem = 536367104 (511MB) avail mem = 515158016 (491MB) +User Kernel Config +UKC disable mpbios +368 mpbios0 disabled +UKC disable acpimadt +501 acpimadt0 disabled +UKC quit +Continuing... mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root @@ -15,22 +22,20 @@ acpi0: tables DSDT FACP SSDT APIC RSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits -acpimadt0 at acpi0 addr 0xfee0: PC-AT compat -cpu0 at mainbus0: apid 0 (boot processor) -mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges -cpu0: apic clock running at 999MHz -ioapic0 at mainbus0: apid 0 pa 0xfec0, version 11, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 +mpbios at bios0 function 0x0 not configured bios0: ROM list: 0xc/0x1000! 0xc1000/0xa00 0xc2000/0x2400 0xed800/0x2800! +cpu0 at mainbus0: (uniprocessor) +mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 Intel 82441FX rev 0x02 pcib0 at pci0 dev 1 function 0 Intel 82371SB ISA rev 0x00 pciide0 at pci0 dev 1 function 1 Intel 82371SB IDE rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives) -uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: apic 0 int 11 -piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: apic 0 int 9 +uhci0 at pci0 dev 1 function 2 Intel 82371SB USB rev 0x01: irq 11 +piixpm0 at pci0 dev 1 function 3 Intel 82371AB Power rev 0x03: irq 9 iic0 at piixpm0 iic0: addr 0x1c 0f=00 words 00=d4d0 01=d4d0 02=d4d0 03=d4d0 04=d4d0 05=d4d0 06=d4d0 07=d4d0 iic0: addr 0x1d 0f=00 words 00=d4d0 01=d4d0 02=d4d0 03=d4d0 04=d4d0 05=d4d0 06=d4d0 07=d4d0 @@ -39,13 +44,13 @@ iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=d1 48=d1 4a=d1 4e=d1 fc=d1 fe=d1 words 00=9d87 01=9d87 02=9d87 03=9d87 04=9d87 05=9d87 06=9d87 07=9d87 virtio0 at pci0 dev 3 function 0 Qumranet Virtio Network rev 0x00: Virtio Network Device vio0 at virtio0: address 52:54:00:12:34:70 -virtio0: apic 0 int 11 +virtio0: irq 11 virtio1 at pci0 dev 4 function 0 Qumranet Virtio Storage rev 0x00: Virtio Block Device vioblk0 at virtio1 scsibus1 at vioblk0: 2 targets sd0 at scsibus1 targ 0 lun 0: VirtIO, Block Device, SCSI3 0/direct fixed sd0: 102400MB, 512 bytes/sector, 209715200 sectors -virtio1: apic 0 int 11 +virtio1: irq 11 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo -- best regards q#
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On 16 June 2014 04:19, Mikolaj Kucharski miko...@kucharski.name wrote: Hi, Please CC me in any replies as I'm not subscribed to misc emails. My main question is, do you experience similar slow I/O on OpenBSD i386 on your Qemu/KVM installations? Are you aware of any problem with OpenBSD i386 under Qemu/KVM? Not sure should this report go to RedHat, KVM or to OpenBSD, but I've tested RHEL7rc with CentOS 6.5, NetBSD 6.1.4 installer and OpenBSD current bsd.rd and only OpenBSD i386 seems to have slow I/O problem under latest Enterprise Linux from RedHat. I've decided to report the issue here. I've tested OpenBSD guest with IDE, SCSI and VirtIO disk. SCSI disk currently don't seem to work at all with both amd64 and i386: The problem is the i386 ioapic/apic implementation and how OpenBSD uses the lapic_tpr to block incoming interrupts. Basically if you're using ioapic (and you are), OpenBSD maps the lapic_tpr to the special TPR register to block interrupts everytime you get an interrupt of raise it yourself by splraise()/spl* and so on. You don't suffer this on amd64 since the masking is done purely in software. You can also verify this statement by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you don't use ioapic, and the pic code does the mask in software, lapic_tpr is still the same variable being touched, but in this code path, it's not mapped to the cpu TPR. To have an ideia of the cost of touching the tpr: real hardware: ~25 cycles. kvm + flexpriority (cpu extension): ~2700 cycles. kvm without flextpriority: 100k+ cycles. So every interrupt you take needs to touch at least lapic_tpr twice, which before would cost ~50cycles, and now it's more than ~200kcycles. Of course these numbers are relevant to the machine I've tested, but you get an idea on how much slower it is.
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
On Mon, Jun 16, 2014 at 11:19:16AM +0200, Christiano F. Haesbaert wrote: On 16 June 2014 04:19, Mikolaj Kucharski miko...@kucharski.name wrote: My main question is, do you experience similar slow I/O on OpenBSD i386 on your Qemu/KVM installations? Are you aware of any problem with OpenBSD i386 under Qemu/KVM? Not sure should this report go to RedHat, KVM or to OpenBSD, but I've tested RHEL7rc with CentOS 6.5, NetBSD 6.1.4 installer and OpenBSD current bsd.rd and only OpenBSD i386 seems to have slow I/O problem under latest Enterprise Linux from RedHat. I've decided to report the issue here. I've tested OpenBSD guest with IDE, SCSI and VirtIO disk. SCSI disk currently don't seem to work at all with both amd64 and i386: The problem is the i386 ioapic/apic implementation and how OpenBSD uses the lapic_tpr to block incoming interrupts. Giving what you wrote, where do you think the problem should be fixed? On OpenBSD side, or on KVM side? Basically if you're using ioapic (and you are), OpenBSD maps the lapic_tpr to the special TPR register to block interrupts everytime you get an interrupt of raise it yourself by splraise()/spl* and so on. You don't suffer this on amd64 since the masking is done purely in software. You can also verify this statement by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you I cannot find how to enable 'the old pic controller' in libvirt with qemu-kvm. Do you know by any chance how to enable it? don't use ioapic, and the pic code does the mask in software, lapic_tpr is still the same variable being touched, but in this code path, it's not mapped to the cpu TPR. To have an ideia of the cost of touching the tpr: real hardware: ~25 cycles. kvm + flexpriority (cpu extension): ~2700 cycles. kvm without flextpriority: 100k+ cycles. So every interrupt you take needs to touch at least lapic_tpr twice, which before would cost ~50cycles, and now it's more than ~200kcycles. Of course these numbers are relevant to the machine I've tested, but you get an idea on how much slower it is. Thank you for the above explanation. -- best regards q#
Re: Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
previously on this list Mikolaj Kucharski contributed: by disabling mpbios on OpenBSD and falling back to the old pic controller, in this case you I cannot find how to enable 'the old pic controller' in libvirt with qemu-kvm. Do you know by any chance how to enable it? I believe he means disabling mpbios at OpenBSD's boot or in boot.conf means KVM will automatically fall back. Virtual hosting companies like arpnetworks generally ask you to do this for OpenBSD. boot -c disable mpbios checkout man boot.conf -- ___ 'Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface' (Doug McIlroy) In Other Words - Don't design like polkit or systemd ___
Very slow I/O under OpenBSD i386 on qemu-kvm from RHEL7rc
Hi, Please CC me in any replies as I'm not subscribed to misc emails. My main question is, do you experience similar slow I/O on OpenBSD i386 on your Qemu/KVM installations? Are you aware of any problem with OpenBSD i386 under Qemu/KVM? Not sure should this report go to RedHat, KVM or to OpenBSD, but I've tested RHEL7rc with CentOS 6.5, NetBSD 6.1.4 installer and OpenBSD current bsd.rd and only OpenBSD i386 seems to have slow I/O problem under latest Enterprise Linux from RedHat. I've decided to report the issue here. I've tested OpenBSD guest with IDE, SCSI and VirtIO disk. SCSI disk currently don't seem to work at all with both amd64 and i386: OpenBSD i386/scsi [test07] # time dd if=/dev/zero of=/dev/sd0c bs=4096 count=10240 sd0(vioscsi0:0:0): Check Condition (error 0x70) on opcode 0x28 SENSE KEY: Illegal Request ASC/ASCQ: ASC 0x24 ASCQ 0x00 dd: /dev/sd0c: Invalid argument 5+0 records in 4+0 records out 16384 bytes transferred in 0.093 secs (174834 bytes/sec) 0m0.25s real 0m0.00s user 0m0.15s system # time dd if=/dev/zero of=/dev/rsd0c bs=4096 count=10240 sd0(vioscsi0:0:0): Check Condition (error 0x70) on opcode 0x2a SENSE KEY: Illegal Request ASC/ASCQ: ASC 0x24 ASCQ 0x00 dd: /dev/rsd0c: Invalid argument 42+0 records in 41+0 records out 167936 bytes transferred in 0.290 secs (578587 bytes/sec) 0m0.46s real 0m0.00s user 0m0.34s system OpenBSD amd64/scsi [test08] # time dd if=/dev/zero of=/dev/sd0c bs=4096 count=10240 sd0(vioscsi0:0:0): Check Condition (error 0x70) on opcode 0x28 SENSE KEY: Illegal Request ASC/ASCQ: ASC 0x24 ASCQ 0x00 dd: /dev/sd0c: Invalid argument 5+0 records in 4+0 records out 16384 bytes transferred in 0.061 secs (265367 bytes/sec) 0m0.07s real 0m0.00s user 0m0.00s system # time dd if=/dev/zero of=/dev/rsd0c bs=4096 count=10240 sd0(vioscsi0:0:0): Check Condition (error 0x70) on opcode 0x2a SENSE KEY: Illegal Request ASC/ASCQ: ASC 0x24 ASCQ 0x00 dd: /dev/rsd0c: Invalid argument 42+0 records in 41+0 records out 167936 bytes transferred in 0.081 secs (2063603 bytes/sec) 0m0.10s real 0m0.00s user 0m0.00s system However I'm not chasing the SCSI problem, as VirtIO and IDE do work on OpenBSD. VirtIO seems to be a bit faster, so I'm focusing all my tests on VirtIO. If you look below at test02 you notice that to create 4GB zero filled file takes 42 minutes. In comparision on Scientific Linux it takes 5 minutes (also a bit slow, but not that much). On amd64 the same operation is taking seconds. +-+-+ | RHEL7 | SL6.5 | ++-+-+ | i386 | 42min | 5min | ++-+-+ | amd64 | 42sec | 28sec | ++-+-+ I did tests on OpenBSD and NetBSD installers, however I have fully installed i386 OpenBSD on RHEL7rc and it is also very slow. I'm providing some information below, but if you need any more details, please let me know. DETAILS rhel7rc rpm -q kernel qemu-kvm kernel-3.10.0-121.el7.x86_64 qemu-kvm-1.5.3-60.el7.x86_64 bsd.rd/amd64 dmesg | sed -e 2q OpenBSD 5.5-current (RAMDISK_CD) #188: Fri Jun 13 13:01:15 MDT 2014 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD bsd.rd/i386 dmesg | sed -e 2q OpenBSD 5.5-current (RAMDISK_CD) #169: Fri Jun 13 12:51:59 MDT 2014 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/RAMDISK_CD # dmesg | grep sd0 sd0 at scsibus1 targ 0 lun 0: VirtIO, Block Device, SCSI3 0/direct fixed sd0: 10240MB, 512 bytes/sector, 20971520 sectors FILESYSTEM TESTS (4GB) OpenBSD amd64/virtio (RHEL7rc) [test01] # time dd if=/dev/zero of=/mnt/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 42.753 secs (100458532 bytes/sec) 0m42.79s real 0m0.07s user 0m5.77s system OpenBSD i386/virtio (RHEL7rc) [test02] # time dd if=/dev/zero of=/mnt/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 2592.993 secs (1656374 bytes/sec) 43m13.11s real 0m0.56s user37m14.95s system OpenBSD amd64/virtio (Scientific Linux 6.5) [test03] # time dd if=/dev/zero of=/mnt/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 28.288 secs (151829547 bytes/sec) 0m28.29s real 0m0.22s user 0m6.78s system OpenBSD i386/virtio (Scientific Linux 6.5) [test04] # time dd if=/dev/zero of=/mnt/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 301.991 secs (14222149 bytes/sec) 5m2.00s real 0m0.61s user 4m15.20s system NetBSD i386/virtio (RHEL7rc) [test05] # dd if=/dev/zero of=/mnt/TEST bs=4096 count=1024x1024 1048576+0 records in 1048576+0 records out 4294967296 bytes transferred in 58.620 secs (73267951
