Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Wed, 2009-05-06 at 11:51 +0800, Sheng Yang wrote: On Wednesday 06 May 2009 01:45:47 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. Is MSI-X can't be enabled or the device only have MSI capability? Just curious... AFAICT MSI-X is not supported on the mpt-fusion SAS device when running on the KVM host.. The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Didn't see exception in the log below... (And buried with iscsi log I can't understand. Looking forward for the help from others...) Any thing notable show in the host side? I think the target to get pSCSI work well now? Yeah, the log attached was using the SATA SSD with the IBLOCK export (the one that worked..). The exception that I was seeing from pSCSI export from the same SATA SSD from within KVM Guest running v2.6.29.2 involved a non-zero return from drivers/scsi/scsi_lib.c:scsi_execute_async() for basically all bulk struct scatterlist I/O (the control path seemed OK however). Unfortuately, scsi_execute_async() was (it has been removed for v2.6.30) really bad at returning any useful value and hardcodes a return of 'DRIVER_ERROR 24' in the error path, so I was unable to determine anything useful for what was actually causing it to fail during my initial testing.. I will keep poking at it and see if I can find anything interesting.. BTW: Maybe you can try the patch from Marcelo titled [patch 0/4] use smp_send_reschedule in vcpu_kick / assigned dev host intx race fix. Thanks for the pointer, I will have a look. Many thanks for your most valuable of time, --nab -- regards Yang, Sheng Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. --nab One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11 Call Trace: [80235fee] ? warn_slowpath+0xcb/0xe8 [80253a7c] ? generic_exec_single+0x6a/0x88 [8022acec] ? update_curr+0x67/0xeb [a0198748] ? vcpu_kick_intr+0x0/0x1 [kvm] [8020a5d8] ? __switch_to+0xb6/0x274 [8022b70a] ? __dequeue_entity+0x1b/0x2f [a01ac7e4] ?
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616 MC08 /dev/scd2 [32:0:0:0] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdb [32:0:0:1] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdc [32:0:0:2] diskLIO-ORG FILEIO 3.0 /dev/sdd [32:0:0:3] diskLIO-ORG IBLOCK 3.0 /dev/sde subjekt:~# sg_inq -i /dev/sde VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 id_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x1405 Vendor Specific Identifier: 0xa97e4ce21 Vendor Specific Identifier Extension: 0xc0711de829b000c2 [0x6001405a97e4ce21c0711de829b000c2] Designation descriptor number 2, descriptor length: 52 id_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: LIO-ORG vendor specific: IBLOCK:a97e4ce21c0711de829b000c2943d57b Designation descriptor number 3, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x1 Designation descriptor number 4, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Target port group, code_set: Binary associated with the target port Target port group: 0x0 Designation descriptor number 5, descriptor length: 8 id_type: Logical unit group, code_set: Binary associated with the addressed logical unit Logical unit group: 0x0 Designation descriptor number 6, descriptor length: 80 transport: Internet SCSI (iSCSI) id_type: SCSI name string, code_set: UTF-8
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tuesday 05 May 2009 18:43:46 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11 Call Trace: [80235fee] ? warn_slowpath+0xcb/0xe8 [80253a7c] ? generic_exec_single+0x6a/0x88 [8022acec] ? update_curr+0x67/0xeb [a0198748] ? vcpu_kick_intr+0x0/0x1 [kvm] [8020a5d8] ? __switch_to+0xb6/0x274 [8022b70a] ? __dequeue_entity+0x1b/0x2f [a01ac7e4] ? kvm_irq_delivery_to_apic+0xb3/0xf7 [kvm] [a01aa4d4] ? __apic_accept_irq+0x15a/0x173 [kvm] [a01ac883] ? kvm_set_msi+0x5b/0x60 [kvm] [80266d97] ? enable_irq+0x36/0x50 [a0195ab5] ? kvm_assigned_dev_interrupt_work_handler+0x6d/0xbc [kvm] [802449fa] ? worker_thread+0x182/0x223 [8024820b] ? autoremove_wake_function+0x0/0x2a [80244878] ? worker_thread+0x0/0x223 [80244878] ? worker_thread+0x0/0x223 [80247e72] ? kthread+0x54/0x7e [8020cb0a] ? child_rip+0xa/0x20 [804d0af5] ? _spin_lock+0x5/0x8 [80247e1e] ? kthread+0x0/0x7e [8020cb00] ? child_rip+0x0/0x20 ---[ end trace 3fbc2dd20bf89ef1 ]--- connection1:0: ping timeout of 5 secs expired, last rx 4295286327, last ping 4295285518, now 4295286768 connection1:0: detected conn error (1011) Attached are the v2.6.30-rc3 KVM host and v2.6.29.2 KVM guest dmesg output. When the 'Unbalanced enable for IRQ
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tuesday 05 May 2009 19:28:15 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW This means the fault address is too big It's got 51 bits width which is far beyond the physical address limit of current IA32e(48 bits). Don't know how you can get this... -- regards Yang, Sheng pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. --nab One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Wednesday 06 May 2009 01:45:47 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. Is MSI-X can't be enabled or the device only have MSI capability? Just curious... The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Didn't see exception in the log below... (And buried with iscsi log I can't understand. Looking forward for the help from others...) Any thing notable show in the host side? I think the target to get pSCSI work well now? BTW: Maybe you can try the patch from Marcelo titled [patch 0/4] use smp_send_reschedule in vcpu_kick / assigned dev host intx race fix. -- regards Yang, Sheng Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616 MC08 /dev/scd2 [32:0:0:0] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdb [32:0:0:1] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdc [32:0:0:2] diskLIO-ORG FILEIO 3.0 /dev/sdd [32:0:0:3] diskLIO-ORG IBLOCK 3.0 /dev/sde subjekt:~# sg_inq -i /dev/sde VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 id_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x1405 Vendor Specific Identifier: 0xa97e4ce21 Vendor Specific Identifier Extension: 0xc0711de829b000c2 [0x6001405a97e4ce21c0711de829b000c2] Designation descriptor number 2, descriptor length: 52 id_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: LIO-ORG vendor specific: IBLOCK:a97e4ce21c0711de829b000c2943d57b Designation descriptor number 3, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Relative target port, code_set:
Re: KVM x86_64 with SR-IOV..?
On Sun, 2009-05-03 at 22:28 -0700, Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X Ok, I also noticed the following output in /proc/interrupts on KVM host with dual Intel E5520 processors (16 CPUs) [r...@barret ~]# cat /proc/interrupts | grep MSI 55: 22 0 0 0 0 0 0 0 0 94 436974 0 0 0 0 0 PCI-MSI-edge eth0-rx-0 56: 27 0 0 0 0 0 0 0 0 613054 0 0 15253 0 0 0 PCI-MSI-edge eth0-tx-0 57: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0 58:521 0 0 0 5 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm_assigned_msi_device eth0 is the other e1000e port at 03:00.0 that is in use on the KVM host, and it looks like the other e1000e port at 02:00.0 has been setup to kvm_assigned_msi_device on irq 58. I also noticed the following after starting a KVM guest in host's ring buffer (not sure if this has anything to do with -pcidevice usage) kvm: 3428: cpu6 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu5 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu9 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu1 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu8 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu2 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu3 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu4 unhandled wrmsr: 0xc0010117 data 0 kvm: 3428: cpu0 unhandled wrmsr: 0xc0010117 data 0 I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements: #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) However I still see the following in the KVM guest kernel ring buffer running v2.6.30-rc in the HVM guest. [5.523790]
Re: KVM x86_64 with SR-IOV..?
On Monday 04 May 2009 12:36:04 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements: #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) However I still see the following in the KVM guest kernel ring buffer running v2.6.30-rc in the HVM guest. [5.523790] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [5.524582] e1000e :00:05.0: PCI INT A - Link[LNKA] - GSI 10 (level, high) - IRQ 10 [5.525710] e1000e :00:05.0: setting
Re: KVM x86_64 with SR-IOV..?
On Monday 04 May 2009 17:11:59 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 16:20 +0800, Sheng Yang wrote: On Monday 04 May 2009 12:36:04 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements: #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) BUG:
Re: KVM x86_64 with SR-IOV..?
On Mon, 2009-05-04 at 17:49 +0800, Sheng Yang wrote: On Monday 04 May 2009 17:11:59 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 16:20 +0800, Sheng Yang wrote: On Monday 04 May 2009 12:36:04 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements:
Re: KVM x86_64 with SR-IOV..?
Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu On Mon, May 04, 2009 at 06:40:36PM +0800, Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 17:49 +0800, Sheng Yang wrote: On Monday 04 May 2009 17:11:59 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 16:20 +0800, Sheng Yang wrote: On Monday 04 May 2009 12:36:04 Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer:
Re: KVM x86_64 with SR-IOV..?
On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. S, I enabled the debugging code in kvm-85/qemu/hw/device-assignment.c and see the PAGE aligned MMIO memory for the passed PCIe device is being released during the BUG exceptions above.. Is there something else I should be looking at..? I have pci-stub enabled, and I unbind 02:00.0 from /sys/bus/pci/drivers/e1000e/unbind successfully (just like with Xen and pciback), but I am unable to do the 'echo -n 02:00.0 /sys/bus/pci/drivers/pci-stub/bind' (it returns write error, no such device, with no dmesg output) on the KVM host running v2.6.30-rc3. Is this supposed to happen on v2.6.30-rc3 with pci-stub..? I am also using the the kvm-85 source dist kvm_intel.ko and kvm.ko kernel modules. Is there something I am missing when building kvm-85 for SR-IOV passthrough..? Also FYI, I am having to use pci=resource_alignment= because the BIOS does not PAGE_SIZE align the MMIO BARs for my multi-function devices.. Also, I tried with disabling the DMAR with the Intel IOMMU passthrough from this patch: https://lists.linux-foundation.org/pipermail/iommu/2009-April/001339.html that did not make it into v2.6.30-rc3. The patch logic was enabled but still I saw the same kvm exceptions from qemu-system-x86_64. Anyways, I am going to give it a shot with the Fedora 11 x86_64 Preview and see if it works as expected with a IOH-5520 chipset with the AMI BIOS on a Tyan S7010 with Xeon 5520s. Hopefully this is just a kvm-85 build and/or install issue I am seeing on my CentOS 5u3 install (that has a Xen PCIe passthrough setup on it as well) with v2.6.30-rc3. I will try on a fresh install on a distro with the new KVM logic and see what happens. :-) Thanks for your comments! --nab -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM x86_64 with SR-IOV..?
On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. And what do you mean by some MSI-X exceptions? Better with the log. S, I enabled the debugging code in kvm-85/qemu/hw/device-assignment.c and see the PAGE aligned MMIO memory for the passed PCIe device is being released during the BUG exceptions above.. Is there something else I should be looking at..? That part of memory should be released for trap MMIO for MSI-X table. I have pci-stub enabled, and I unbind 02:00.0 from /sys/bus/pci/drivers/e1000e/unbind successfully (just like with Xen and pciback), but I am unable to do the 'echo -n 02:00.0 /sys/bus/pci/drivers/pci-stub/bind' (it returns write error, no such device, with no dmesg output) on the KVM host running v2.6.30-rc3. Is this supposed to happen on v2.6.30-rc3 with pci-stub..? Maybe you need echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind? I am also using the the kvm-85 source dist kvm_intel.ko and kvm.ko kernel modules. Is there something I am missing when building kvm-85 for SR-IOV passthrough..? I think the first thing is to confirm that device assignment work in your environment, using on-board card. You can also refer to http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM And you can post debug_device_assignment=1 log and qemu log and the tail of dmesg as well. Thanks! -- regards Yang, Sheng Also FYI, I am having to use pci=resource_alignment= because the BIOS does not PAGE_SIZE align the MMIO BARs for my multi-function devices.. Also, I tried with disabling the DMAR with the Intel IOMMU passthrough from this patch:
Re: KVM x86_64 with SR-IOV..?
On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements: #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) However I still see the following in the KVM guest kernel ring buffer running v2.6.30-rc in the HVM guest. [5.523790] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [5.524582] e1000e :00:05.0: PCI INT A - Link[LNKA] - GSI 10 (level, high) - IRQ 10 [5.525710] e1000e :00:05.0: setting latency timer to 64 [5.526048] :00:05.0: :00:05.0: Failed to initialize MSI-X interrupts. Falling back to MSI interrupts. [5.527200]
Re: KVM x86_64 with SR-IOV..?
On Sun, 2009-05-03 at 21:36 -0700, Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements: #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) However I still see the following in the KVM guest kernel ring buffer running v2.6.30-rc in the HVM guest. [5.523790] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [5.524582] e1000e :00:05.0: PCI INT A - Link[LNKA] - GSI 10 (level, high) - IRQ 10 [
Re: KVM x86_64 with SR-IOV..?
On Sun, 2009-05-03 at 22:28 -0700, Nicholas A. Bellinger wrote: On Sun, 2009-05-03 at 21:36 -0700, Nicholas A. Bellinger wrote: On Mon, 2009-05-04 at 10:09 +0800, Sheng Yang wrote: On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote: On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote: On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. Greetings Sheng, So, I have been trying the latest kvm-85 release on a v2.6.30-rc3 checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel IOH-5520 based dual socket Nehalem board. I have enabled DMAR and Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85 after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c AFAICT.. From there, I use the freshly installed qemu-x86_64-system binary to start a Debian 5 x86_64 HVM (that previously had been moving network packets under Xen for PCIe passthrough). I see the MSI-X interrupt remapping working on the KVM host for the passed -pcidevice, and the MMIO mappings from the qemu build that I also saw while using Xen/qemu-dm built with PCI passthrough are there as well.. Hi Nicholas But while the KVM guest is booting, I see the following exception(s) from qemu-x86_64-system for one of the VFs for a multi-function PCIe device: BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) This one is mostly harmless. Ok, good to know.. :-) I try with one of the on-board e1000e ports (02:00.0) and I see the same exception along with some MSI-X exceptions from qemu-x86_64-system in KVM guest.. However, I am still able to see the e1000e and the other vxge multi-function device with lspci, but I am unable to dhcp or ping with the e1000e and VF from multi-function device fails to register the MSI-X interrupt in the guest.. Did you see the interrupt in the guest and host side? Ok, I am restarting the e1000e test with a fresh Fedora 11 install and KVM host kernel 2.6.29.1-111.fc11.x86_64. After unbinding and attaching the e1000e single-function device at 02:00.0 to pci-stub with: echo 8086 10d3 /sys/bus/pci/drivers/pci-stub/new_id echo :02:00.0 /sys/bus/pci/devices/:02:00.0/driver/unbind echo :02:00.0 /sys/bus/pci/drivers/pci-stub/bind I see the following the KVM host kernel ring buffer: e1000e :02:00.0: PCI INT A disabled pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 pci-stub :02:00.0: irq 58 for MSI/MSI-X I think you can try on- board e1000e for MSI-X first. And please ensure correlated driver have been loaded correctly. nod.. And what do you mean by some MSI-X exceptions? Better with the log. Ok, with the Fedora 11 installed qemu-kemu, I see the expected kvm_destroy_phys_mem() statements: #kvm-host qemu-kvm -m 2048 -smp 8 -pcidevice host=02:00.0 lenny64guest1-orig.img BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1) However I still see the following in the KVM guest kernel ring
Re: KVM x86_64 with SR-IOV..?
On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote: Greetings KVM folks, I wondering if any information exists for doing SR-IOV on the new VT-d capable chipsets with KVM..? From what I understand the patches for doing this with KVM are floating around, but I have been unable to find any user-level docs for actually making it all go against a upstream v2.6.30-rc3 code.. So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I am really hoping to be able to jump to KVM for single-function and and then multi-function SR-IOV. I know that the VM migration stuff for IOV in Xen is up and running, and I assume it is being worked in for KVM instance migration as well..? This part is less important (at least for me :-) than getting a stable SR-IOV setup running under the KVM hypervisor.. Does anyone have any pointers for this..? Any comments or suggestions are appreciated! Hi Nicholas The patches are not floating around now. As you know, SR-IOV for Linux have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time ago, there are several SRIOV related patches for qemu-kvm, and now they all have been checked in. And for KVM, the extra document is not necessary, for you can simple assign a VF to guest like any other devices. And how to create VF is specific for each device driver. So just create a VF then assign it to KVM guest is fine. -- regards Yang, Sheng -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
