Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Wed, 2009-05-06 at 11:51 +0800, Sheng Yang wrote: On Wednesday 06 May 2009 01:45:47 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. Is MSI-X can't be enabled or the device only have MSI capability? Just curious... AFAICT MSI-X is not supported on the mpt-fusion SAS device when running on the KVM host.. The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Didn't see exception in the log below... (And buried with iscsi log I can't understand. Looking forward for the help from others...) Any thing notable show in the host side? I think the target to get pSCSI work well now? Yeah, the log attached was using the SATA SSD with the IBLOCK export (the one that worked..). The exception that I was seeing from pSCSI export from the same SATA SSD from within KVM Guest running v2.6.29.2 involved a non-zero return from drivers/scsi/scsi_lib.c:scsi_execute_async() for basically all bulk struct scatterlist I/O (the control path seemed OK however). Unfortuately, scsi_execute_async() was (it has been removed for v2.6.30) really bad at returning any useful value and hardcodes a return of 'DRIVER_ERROR 24' in the error path, so I was unable to determine anything useful for what was actually causing it to fail during my initial testing.. I will keep poking at it and see if I can find anything interesting.. BTW: Maybe you can try the patch from Marcelo titled [patch 0/4] use smp_send_reschedule in vcpu_kick / assigned dev host intx race fix. Thanks for the pointer, I will have a look. Many thanks for your most valuable of time, --nab -- regards Yang, Sheng Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. --nab One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11 Call Trace: [80235fee] ? warn_slowpath+0xcb/0xe8 [80253a7c] ? generic_exec_single+0x6a/0x88 [8022acec] ? update_curr+0x67/0xeb [a0198748] ? vcpu_kick_intr+0x0/0x1 [kvm] [8020a5d8] ? __switch_to+0xb6/0x274 [8022b70a] ? __dequeue_entity+0x1b/0x2f [a01ac7e4] ?
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616 MC08 /dev/scd2 [32:0:0:0] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdb [32:0:0:1] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdc [32:0:0:2] diskLIO-ORG FILEIO 3.0 /dev/sdd [32:0:0:3] diskLIO-ORG IBLOCK 3.0 /dev/sde subjekt:~# sg_inq -i /dev/sde VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 id_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x1405 Vendor Specific Identifier: 0xa97e4ce21 Vendor Specific Identifier Extension: 0xc0711de829b000c2 [0x6001405a97e4ce21c0711de829b000c2] Designation descriptor number 2, descriptor length: 52 id_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: LIO-ORG vendor specific: IBLOCK:a97e4ce21c0711de829b000c2943d57b Designation descriptor number 3, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Relative target port, code_set: Binary associated with the target port Relative target port: 0x1 Designation descriptor number 4, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Target port group, code_set: Binary associated with the target port Target port group: 0x0 Designation descriptor number 5, descriptor length: 8 id_type: Logical unit group, code_set: Binary associated with the addressed logical unit Logical unit group: 0x0 Designation descriptor number 6, descriptor length: 80 transport: Internet SCSI (iSCSI) id_type: SCSI name string, code_set: UTF-8
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tuesday 05 May 2009 18:43:46 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11 Call Trace: [80235fee] ? warn_slowpath+0xcb/0xe8 [80253a7c] ? generic_exec_single+0x6a/0x88 [8022acec] ? update_curr+0x67/0xeb [a0198748] ? vcpu_kick_intr+0x0/0x1 [kvm] [8020a5d8] ? __switch_to+0xb6/0x274 [8022b70a] ? __dequeue_entity+0x1b/0x2f [a01ac7e4] ? kvm_irq_delivery_to_apic+0xb3/0xf7 [kvm] [a01aa4d4] ? __apic_accept_irq+0x15a/0x173 [kvm] [a01ac883] ? kvm_set_msi+0x5b/0x60 [kvm] [80266d97] ? enable_irq+0x36/0x50 [a0195ab5] ? kvm_assigned_dev_interrupt_work_handler+0x6d/0xbc [kvm] [802449fa] ? worker_thread+0x182/0x223 [8024820b] ? autoremove_wake_function+0x0/0x2a [80244878] ? worker_thread+0x0/0x223 [80244878] ? worker_thread+0x0/0x223 [80247e72] ? kthread+0x54/0x7e [8020cb0a] ? child_rip+0xa/0x20 [804d0af5] ? _spin_lock+0x5/0x8 [80247e1e] ? kthread+0x0/0x7e [8020cb00] ? child_rip+0x0/0x20 ---[ end trace 3fbc2dd20bf89ef1 ]--- connection1:0: ping timeout of 5 secs expired, last rx 4295286327, last ping 4295285518, now 4295286768 connection1:0: detected conn error (1011) Attached are the v2.6.30-rc3 KVM host and v2.6.29.2 KVM guest dmesg output. When the 'Unbalanced enable for IRQ
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Tuesday 05 May 2009 19:28:15 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW This means the fault address is too big It's got 51 bits width which is far beyond the physical address limit of current IA32e(48 bits). Don't know how you can get this... -- regards Yang, Sheng pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. --nab One issue I did notice while using the pci-stub method of device-assignment with same e1000 port (02:00.0) was while using an iSCSI Initiator (Open-iSCSI) on the KVM Host machine and doing sustained traffic into the LIO-Target KVM Guest on the same local KVM host to max out traffic between the other onboard e1000e port (03.00.0), I see the following: pci-stub :02:00.0: PCI INT A - GSI 17 (level, low) - IRQ 17 assign device: host bdf = 2:0:0 pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X scsi4 : iSCSI Initiator over TCP/IP scsi 4:0:0:0: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:0: Attached scsi generic sg1 type 0 scsi 4:0:0:1: Direct-Access LIO-ORG RAMDISK-DR 3.0 PQ: 0 ANSI: 5 sd 4:0:0:1: Attached scsi generic sg2 type 0 sd 4:0:0:0: [sdb] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:1: [sdc] 262144 512-byte hardware sectors: (134 MB/128 MiB) sd 4:0:0:0: [sdb] Write Protect is off sd 4:0:0:0: [sdb] Mode Sense: 2f 00 00 00 sd 4:0:0:1: [sdc] Write Protect is off sd 4:0:0:1: [sdc] Mode Sense: 2f 00 00 00 sd 4:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA sdb:6 sdc: unknown partition table sd 4:0:0:0: [sdb] Attached SCSI disk unknown partition table sd 4:0:0:1: [sdc] Attached SCSI disk [ cut here ] WARNING: at kernel/irq/manage.c:260 enable_irq+0x36/0x50() Hardware name: empty Unbalanced enable for IRQ 59 Modules linked in: ipt_REJECT xt_tcpudp bridge stp sunrpc iptable_filter ip_tables xt_state nf_conntrack ip6table_filter ip6_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi cpufreq_ondemand acpi_cpufreq freq_table ext3 jbd loop dm_multipath scsi_dh kvm_intel kvm uinput i2c_i801 firewire_ohci joydev firewire_core sg i2c_core 8250_pnp crc_itu_t e1000e 8250 serial_core rtc_cmos pcspkr serio_raw rtc_core rtc_lib button sd_mod dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod uhci_hcd ohci_hcd ehci_hcd ata_piix libata scsi_mod [last unloaded: microcode] Pid: 51, comm: events/0 Tainted: GW 2.6.30-rc3 #11
Re: KVM x86_64 with SR-IOV..? (device passthrough with LIO-Target v3.0)
On Wednesday 06 May 2009 01:45:47 Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 04:28 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 03:43 -0700, Nicholas A. Bellinger wrote: On Tue, 2009-05-05 at 09:42 +0800, Yu Zhao wrote: Hi, The VF also works in the host if the VF driver is programed properly. So it would be easier to develop the VF driver in the host and then verify the VF driver in the guest. BTW, I didn't see the SR-IOV is enabled in your dmesg, did you select the CONFIG_PCI_IOV in the kernel .config? Thanks, Yu Greetings Yu and Sheng, So the original attachment was for the v2.6.29-fc11 host kernel output, I ended up jumping to v2.6.30-rc3 (and making sure CONFIG_PCI_IOV was enabled) for KVM host with kvm-85 and now things are looking quite stable for me. So far I have been able to successfully push LIO-Target v3.0 traffic *inside* a v2.6.29.2 KVM guest via the onboard e1000e (02:00.0) port from another Linux/iSCSI Initiator machine using a Intel 1 Gb/sec port. I am running badblocks tests to iSCSI Logical Units for RAMDISK_DR and FILEIO storage objects (in the KVM Guest), and they are passing validation and I am seeing ~500 Mb/sec of throughput and very low CPU usage in the KVM guests. Ok I am seeing another issue with the e1000e port on 02:00.0..: As i start to push multiple badblocks tests RAMDISK_DR iSCSI Logical units into KVM Guest running LIO v2.6.29.2 from the external Linux/iSCSI Initiator machine, after about 100 GB of iSCSI traffic, I see the following exception in KVM host v2.6.30-rc3: DRHD: handling fault status reg 2 DMAR:[DMA Write] Request device [02:00.0] fault addr 7fc958b01 DMAR:[fault reason 04] Access beyond MGAW pci-stub :02:00.0: irq 59 for MSI/MSI-X pci-stub :02:00.0: irq 60 for MSI/MSI-X pci-stub :02:00.0: irq 61 for MSI/MSI-X I am able to restart the LIO-Target KVM Guest and the Linux/iSCSI Initiators are able to reconnect.. Wow, very cool.. Not sure if this is a bug in the target_core_mod RAMDISK_DR subsystem plugin (mapping struct iovec to internally allocated struct page) or what. I will have to look at the DMAR code to understand what this exception means.. Greetings Yu, Sheng and Co, So I have been making progress this morning.. So far, I have hooked up a LSI mpt-function PCIe SAS adapter into the KVM guest with a Sandisk SATA SSD 32 GB drive. It is using MSI interrupts (not MSI-X) and I am able to push ~70 MB/sec from a 2nd Linux/iSCSI Initiator machine (running Open-iSCSI) with the 1500 byte MTUs on e1000e ports from within the KVM guest. Is MSI-X can't be enabled or the device only have MSI capability? Just curious... The interesting thing is that I am having to use IBLOCK export (using using submit_bio(), and complete emulation of SCSI control path) for SATA SSD in order to get I/O running stable Using the pSCSI export I am getting immediate exceptions from scsi_execute_async() in the v2.6.29.2 KVM guest.. Didn't see exception in the log below... (And buried with iscsi log I can't understand. Looking forward for the help from others...) Any thing notable show in the host side? I think the target to get pSCSI work well now? BTW: Maybe you can try the patch from Marcelo titled [patch 0/4] use smp_send_reschedule in vcpu_kick / assigned dev host intx race fix. -- regards Yang, Sheng Using a 2nd SAS disk I am able to use target_core_mod/pSCSI export and push badblocks and LTP disktest traffic however.. Here is a bit about the the setup looks, *) Linux/iSCSI Initiator node accessing KVM Guest LIO-Target v3.0 storage: subjekt:~# lsscsi [6:0:0:0]diskATA ST3250820AS 3.AA /dev/sda [10:0:0:0] cd/dvd PIONEER DVD-ROM DVD-305 1.06 /dev/scd1 [18:0:0:0] cd/dvd TOSHIBA DVD/HD X807616 MC08 /dev/scd2 [32:0:0:0] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdb [32:0:0:1] diskLIO-ORG RAMDISK-DR 3.0 /dev/sdc [32:0:0:2] diskLIO-ORG FILEIO 3.0 /dev/sdd [32:0:0:3] diskLIO-ORG IBLOCK 3.0 /dev/sde subjekt:~# sg_inq -i /dev/sde VPD INQUIRY: Device Identification page Designation descriptor number 1, descriptor length: 20 id_type: NAA, code_set: Binary associated with the addressed logical unit NAA 6, IEEE Company_id: 0x1405 Vendor Specific Identifier: 0xa97e4ce21 Vendor Specific Identifier Extension: 0xc0711de829b000c2 [0x6001405a97e4ce21c0711de829b000c2] Designation descriptor number 2, descriptor length: 52 id_type: T10 vendor identification, code_set: ASCII associated with the addressed logical unit vendor id: LIO-ORG vendor specific: IBLOCK:a97e4ce21c0711de829b000c2943d57b Designation descriptor number 3, descriptor length: 8 transport: Internet SCSI (iSCSI) id_type: Relative target port, code_set:
