Re: FreeBSD PVHVM call for testing
On 05/16/13 17:43, Roger Pau Monné wrote: > Thanks for testing this on EC2, could you post the full dmesg? So I can > see the hypervisor version and if the PV timer is loaded or not. Here's what I get on a cc2.8xlarge with boot_verbose=YES: > Booting [/boot/kernel/kernel]... > -\|/-\|GDB: no debug ports present > KDB: debugger backends: ddb > KDB: current backend: ddb > SMAP type=01 base= len=0009fc00 > SMAP type=02 base=0009fc00 len=0400 > SMAP type=02 base=000e len=0002 > SMAP type=01 base=0010 len=bff0 > SMAP type=02 base=fc00 len=0400 > SMAP type=01 base=0001 len=000e6300 > Table 'FACP' at 0xfc005ee0 > Table 'APIC' at 0xfc005fe0 > APIC: Found table at 0xfc005fe0 > APIC: Using the MADT enumerator. > MADT: Found CPU APIC ID 0 ACPI ID 0: enabled > SMP: Added CPU 0 (AP) > MADT: Found CPU APIC ID 2 ACPI ID 1: enabled > SMP: Added CPU 2 (AP) > MADT: Found CPU APIC ID 4 ACPI ID 2: enabled > SMP: Added CPU 4 (AP) > MADT: Found CPU APIC ID 6 ACPI ID 3: enabled > SMP: Added CPU 6 (AP) > MADT: Found CPU APIC ID 8 ACPI ID 4: enabled > SMP: Added CPU 8 (AP) > MADT: Found CPU APIC ID 10 ACPI ID 5: enabled > SMP: Added CPU 10 (AP) > MADT: Found CPU APIC ID 12 ACPI ID 6: enabled > SMP: Added CPU 12 (AP) > MADT: Found CPU APIC ID 14 ACPI ID 7: enabled > SMP: Added CPU 14 (AP) > MADT: Found CPU APIC ID 32 ACPI ID 8: enabled > SMP: Added CPU 32 (AP) > MADT: Found CPU APIC ID 34 ACPI ID 9: enabled > SMP: Added CPU 34 (AP) > MADT: Found CPU APIC ID 36 ACPI ID 10: enabled > SMP: Added CPU 36 (AP) > MADT: Found CPU APIC ID 38 ACPI ID 11: enabled > SMP: Added CPU 38 (AP) > MADT: Found CPU APIC ID 40 ACPI ID 12: enabled > SMP: Added CPU 40 (AP) > MADT: Found CPU APIC ID 42 ACPI ID 13: enabled > SMP: Added CPU 42 (AP) > MADT: Found CPU APIC ID 44 ACPI ID 14: enabled > SMP: Added CPU 44 (AP) > MADT: Found CPU APIC ID 46 ACPI ID 15: enabled > SMP: Added CPU 46 (AP) > MADT: Found CPU APIC ID 1 ACPI ID 16: enabled > SMP: Added CPU 1 (AP) > MADT: Found CPU APIC ID 3 ACPI ID 17: enabled > SMP: Added CPU 3 (AP) > MADT: Found CPU APIC ID 5 ACPI ID 18: enabled > SMP: Added CPU 5 (AP) > MADT: Found CPU APIC ID 7 ACPI ID 19: enabled > SMP: Added CPU 7 (AP) > MADT: Found CPU APIC ID 9 ACPI ID 20: enabled > SMP: Added CPU 9 (AP) > MADT: Found CPU APIC ID 11 ACPI ID 21: enabled > SMP: Added CPU 11 (AP) > MADT: Found CPU APIC ID 13 ACPI ID 22: enabled > SMP: Added CPU 13 (AP) > MADT: Found CPU APIC ID 15 ACPI ID 23: enabled > SMP: Added CPU 15 (AP) > MADT: Found CPU APIC ID 33 ACPI ID 24: enabled > SMP: Added CPU 33 (AP) > MADT: Found CPU APIC ID 35 ACPI ID 25: enabled > SMP: Added CPU 35 (AP) > MADT: Found CPU APIC ID 37 ACPI ID 26: enabled > SMP: Added CPU 37 (AP) > MADT: Found CPU APIC ID 39 ACPI ID 27: enabled > SMP: Added CPU 39 (AP) > MADT: Found CPU APIC ID 41 ACPI ID 28: enabled > SMP: Added CPU 41 (AP) > MADT: Found CPU APIC ID 43 ACPI ID 29: enabled > SMP: Added CPU 43 (AP) > MADT: Found CPU APIC ID 45 ACPI ID 30: enabled > SMP: Added CPU 45 (AP) > MADT: Found CPU APIC ID 47 ACPI ID 31: enabled > SMP: Added CPU 47 (AP) > Copyright (c) 1992-2013 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 10.0-CURRENT #0 r+7c97e5b: Fri May 17 02:38:29 UTC 2013 > root@ip-10-148-212-216:/usr/obj/usr/src/sys/XENHVM amd64 > FreeBSD clang version 3.3 (trunk 178860) 20130405 > WARNING: WITNESS option enabled, expect reduced performance. > XEN: Hypervisor version 3.4 detected. > XEN: Disabling emulated block and network devices > Preloaded elf kernel "/boot/kernel/kernel" at 0x81912000. > Hypervisor: Origin = "XenVMMXenVMM" > Calibrating TSC clock ... TSC clock: 2593801200 Hz > CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (2593.80-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping > = 7 > > Features=0x1781fbff > > Features2=0x9c982201 > AMD Features=0x20100800 > AMD Features2=0x1 > real memory = 65011712000 (62000 MB) > Physical memory chunk(s): > 0x1000 - 0x0009bfff, 634880 bytes (155 pages) > 0x0010 - 0x001f, 1048576 bytes (256 pages) > 0x01972000 - 0xbfff, 3194544128 bytes (779918 pages) > 0x0001 - 0x000efeb95fff, 60108136448 bytes (14674838 pages) > avail memory = 60563271680 (57757 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: > INTR: Adding local APIC 1 as a target > INTR: Adding local APIC 2 as a target > INTR: Adding local APIC 3 as a target > INTR: Adding local APIC 4 as a target > INTR: Adding local APIC 5 as a target > INTR: Adding local APIC 6 as a target > INTR: Adding local APIC 7 as a target > INTR: Adding local APIC 8 as a target > INT
Re: KVM+FreeBSD+82599 10G NIC doesn't work.
Thanks a lot Jack. According to my test, SRIOV works fine! It's not the way we want, but this solution really helps. So, we'll spend more time to solve the real problem, while we can use SRIOV as a substitution. -- Best wishes. Jason Shi -- Original -- From: "Jack Vogel"; Date: Thu, May 16, 2013 04:02 PM To: "GemaEvor"; Cc: "freebsd-virtualization"; Subject: Re: KVM+FreeBSD+82599 10G NIC doesn't work. The environment that I tested in when the virtualization support was put intothe driver was using SRIOV, not pass-through, we have never validated the latter, and whenever it crops up it seems to be having problems. I can put it into my queue to look into, but I can't say how quickly I can get to it, very busy here. Jack On Thu, May 16, 2013 at 12:45 AM, GemaEvor wrote: Hi gurus: I'm working on a FreeBSD virtulizaion project under KVM. Everything works fine, except the 10G Intel 82599 SFP NIC. I'm sure there are some body else who have encountered this problem before. See: FreeBSD guest with VTD NIC not passing traffic http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806 VT-d not working for FreeBSD 9.0 guest http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/91074 But, unfortunately, none of them solved it or gave the clearly way to solve it. :< Firstly, let's make it clearly. This problem could not be reproduced with: 1. Xen hypervisor + FreeBSD guest os 2. or KVM + Linux guest os (e.g. Fedora 12) So, it must be something wrong with KVM or FreeBSD. Here is my environment: Host OS: OpenSUSE 12.2 Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 Fabrice Bellard Guest OS: FreeBSD 9.1 amd64 What I did: 1.Make the pci device of 82599SFP assignale: [CODE] 0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: QLogic, Corp. Device 00e4 Flags: fast devsel, IRQ 25 Memory at f8fe (64-bit, prefetchable) [size=128K] I/O ports at cc00 [size=32] Memory at f8fdc000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable- Count=64 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [e0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: pci-stub [/CODE] 2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file [FILE] FreeBSD9.1 4d56613e-400c-f37d-2a9a-e0332d81ca63 2097152 2097152 2 hvm destroy restart destroy /usr/bin/qemu-kvm [/FILE] 3.The device could be probed and attached by FreeBSD: [CODE] ix0: port 0xc020-0xc03f mem 0xfe00-0xfe01,0xfe02-0xfe023fff irq 11 at device 4.0 on pci0 ix0: Using MSIX interrupts with 2 vectors ix0: Ethernet address: 00:0c:bd:05:7d:4a ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 [/CODE] But no interrupt could be found: [CODE] interrupt total rate irq1: atkbd0 143 0 irq4: uart0 177 1 irq15: ata1 1593 10 cpu0:timer 7140 46 Total 9053 58 [/CODE] And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping): [CODE] ix0: flags=8802 metric 0 mtu 1500 options=401bb ether 00:0c:bd:05:7d:4a nd6 options=29 media: Ethernet autoselect status: no carrier [/CODE] 4.In the host side, kvm could see the change made by the guest on the device, but no irq: [CODE] linux-kj2u:~ # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 210: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Re: FreeBSD PVHVM call for testing
On 16/05/13 19:55, Colin Percival wrote: > On 05/13/13 11:32, Roger Pau Monné wrote: >> Right now the code is in a state where it can be tested by users, so we >> would like to encourage FreeBSD and Xen users to test it and provide >> feedback. >> >> The code is available in the following git repository, under the branch >> pvhvm_v5: >> >> http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=summary >> >> Also, I've created a wiki page that explains how to set up a FreeBSD >> PVHVM for testing: >> >> http://wiki.xen.org/wiki/Testing_FreeBSD_PVHVM > > I built a XENHVM kernel with this code on EC2, and it hanged after >> xenbusb_front0: on xenstore0 > > With a XENHVM kernel from FreeBSD HEAD the next line after that is >> xbd0: 10240MB at device/vbd/768 on xenbusb_front0 > > Any ideas? Hello Colin, Thanks for testing this on EC2, could you post the full dmesg? So I can see the hypervisor version and if the PV timer is loaded or not. Roger. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: FreeBSD PVHVM call for testing
On 05/13/13 11:32, Roger Pau Monné wrote: > Right now the code is in a state where it can be tested by users, so we > would like to encourage FreeBSD and Xen users to test it and provide > feedback. > > The code is available in the following git repository, under the branch > pvhvm_v5: > > http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=summary > > Also, I've created a wiki page that explains how to set up a FreeBSD > PVHVM for testing: > > http://wiki.xen.org/wiki/Testing_FreeBSD_PVHVM I built a XENHVM kernel with this code on EC2, and it hanged after > xenbusb_front0: on xenstore0 With a XENHVM kernel from FreeBSD HEAD the next line after that is > xbd0: 10240MB at device/vbd/768 on xenbusb_front0 Any ideas? -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
Re: KVM+FreeBSD+82599 10G NIC doesn't work.
The environment that I tested in when the virtualization support was put into the driver was using SRIOV, not pass-through, we have never validated the latter, and whenever it crops up it seems to be having problems. I can put it into my queue to look into, but I can't say how quickly I can get to it, very busy here. Jack On Thu, May 16, 2013 at 12:45 AM, GemaEvor wrote: > Hi gurus: > > > I'm working on a FreeBSD virtulizaion project under KVM. > Everything works fine, except the 10G Intel 82599 SFP NIC. > > > I'm sure there are some body else who have encountered this problem > before. See: > FreeBSD guest with VTD NIC not passing traffic > http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806 > VT-d not working for FreeBSD 9.0 guest > http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/91074 > > > But, unfortunately, none of them solved it or gave the clearly way to > solve it. :< > > > Firstly, let's make it clearly. This problem could not be reproduced with: > 1. Xen hypervisor + FreeBSD guest os > 2. or KVM + Linux guest os (e.g. Fedora 12) > So, it must be something wrong with KVM or FreeBSD. > > > Here is my environment: > Host OS: > OpenSUSE 12.2 > Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC > 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux > QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 > Fabrice Bellard > > > Guest OS: > FreeBSD 9.1 amd64 > > > What I did: > 1.Make the pci device of 82599SFP assignale: > [CODE] > 0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ > Network Connection (rev 01) > Subsystem: QLogic, Corp. Device 00e4 > Flags: fast devsel, IRQ 25 > Memory at f8fe (64-bit, prefetchable) [size=128K] > I/O ports at cc00 [size=32] > Memory at f8fdc000 (64-bit, prefetchable) [size=16K] > Capabilities: [40] Power Management version 3 > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Capabilities: [70] MSI-X: Enable- Count=64 Masked- > Capabilities: [a0] Express Endpoint, MSI 00 > Capabilities: [e0] Vital Product Data > Capabilities: [100] Advanced Error Reporting > Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00 > Capabilities: [150] Alternative Routing-ID Interpretation (ARI) > Capabilities: [160] Single Root I/O Virtualization (SR-IOV) > Kernel driver in use: pci-stub > [/CODE] > > > 2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file > [FILE] > > FreeBSD9.1 > 4d56613e-400c-f37d-2a9a-e0332d81ca63 > 2097152 > 2097152 > 2 > > hvm > > > > > > > > > > destroy > restart > destroy > > /usr/bin/qemu-kvm > > > > > > > > > > > > > > > > > > > > > [/FILE] > > > 3.The device could be probed and attached by FreeBSD: > [CODE] > ix0: port > 0xc020-0xc03f mem 0xfe00-0xfe01,0xfe02-0xfe023fff irq 11 at > device 4.0 on pci0 > ix0: Using MSIX interrupts with 2 vectors > ix0: Ethernet address: 00:0c:bd:05:7d:4a > ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 > [/CODE] > > > But no interrupt could be found: > [CODE] > interrupt total rate > irq1: atkbd0 143 0 > irq4: uart0 177 1 > irq15: ata1 1593 10 > cpu0:timer 7140 46 > Total 9053 58 > [/CODE] > > > And ix0 keeps to be "no carrier" (even if I config an IP address on it and > ping): > [CODE] > ix0: flags=8802 metric 0 mtu 1500 > > options=401bb > ether 00:0c:bd:05:7d:4a > nd6 options=29 > media: Ethernet autoselect > status: no carrier > [/CODE] > > > 4.In the host side, kvm could see the change made by the guest on the > device, but no irq: > [CODE] > linux-kj2u:~ # cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 > CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 > CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 > CPU19 CPU20 CPU21 CPU22 CPU23 > 210: 0 0 0 0 0 0 >0 0 0 0 0 0 0 > 0 0 0 0 0 0 > 0 0 0 0 0 PCI-MSI-edge > kvm::0d:00.0 > 211: 0 0 0 0 0 0 >0 0 0 0 0 0 0 > 0 0 0 0 0 0 > 0 0 0 0 0 PCI-MSI-edge > kvm::0d:00.0
KVM+FreeBSD+82599 10G NIC doesn't work.
Hi gurus: I'm working on a FreeBSD virtulizaion project under KVM. Everything works fine, except the 10G Intel 82599 SFP NIC. I'm sure there are some body else who have encountered this problem before. See: FreeBSD guest with VTD NIC not passing traffic http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806 VT-d not working for FreeBSD 9.0 guest http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/91074 But, unfortunately, none of them solved it or gave the clearly way to solve it. :< Firstly, let's make it clearly. This problem could not be reproduced with: 1. Xen hypervisor + FreeBSD guest os 2. or KVM + Linux guest os (e.g. Fedora 12) So, it must be something wrong with KVM or FreeBSD. Here is my environment: Host OS: OpenSUSE 12.2 Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 Fabrice Bellard Guest OS: FreeBSD 9.1 amd64 What I did: 1.Make the pci device of 82599SFP assignale: [CODE] 0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: QLogic, Corp. Device 00e4 Flags: fast devsel, IRQ 25 Memory at f8fe (64-bit, prefetchable) [size=128K] I/O ports at cc00 [size=32] Memory at f8fdc000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable- Count=64 Masked- Capabilities: [a0] Express Endpoint, MSI 00 Capabilities: [e0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00 Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [160] Single Root I/O Virtualization (SR-IOV) Kernel driver in use: pci-stub [/CODE] 2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file [FILE] FreeBSD9.1 4d56613e-400c-f37d-2a9a-e0332d81ca63 2097152 2097152 2 hvm destroy restart destroy /usr/bin/qemu-kvm [/FILE] 3.The device could be probed and attached by FreeBSD: [CODE] ix0: port 0xc020-0xc03f mem 0xfe00-0xfe01,0xfe02-0xfe023fff irq 11 at device 4.0 on pci0 ix0: Using MSIX interrupts with 2 vectors ix0: Ethernet address: 00:0c:bd:05:7d:4a ix0: PCI Express Bus: Speed 5.0Gb/s Width x8 [/CODE] But no interrupt could be found: [CODE] interrupt total rate irq1: atkbd0 143 0 irq4: uart0 177 1 irq15: ata1 1593 10 cpu0:timer 7140 46 Total 9053 58 [/CODE] And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping): [CODE] ix0: flags=8802 metric 0 mtu 1500 options=401bb ether 00:0c:bd:05:7d:4a nd6 options=29 media: Ethernet autoselect status: no carrier [/CODE] 4.In the host side, kvm could see the change made by the guest on the device, but no irq: [CODE] linux-kj2u:~ # cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 210: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm::0d:00.0 211: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm::0d:00.0 [/CODE] So, anyone who has the same problem or has a clue to solve it, or has some ideas to do a debug job, please help. Thanks! ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"
