It seems that something with interrupt remapping is the problem... And I don´t know how to solve the problem.
But I tested the intel NVMe in passthrouged Windows 2012. There it works perfectly. Second test - intel P4600 and Samsung simultaneously: Samsung works, intel has the error. Further tests without success: - esxcli system settings kernel set --setting=iovDisableIR -v TRUE - Remapping disabled in BIOS My hardware configuration: - SC826BAC4-R920LPB - 2U - X10DRC-T4+ (BIOS-FW 3.58) - 2x intel Xeon E5-2630 v4 - BPN-SAS3-826A-N4 - AOC-SLG3-4E4T - 3x intel DC P4600 - AOC-SLG3-2M2 with 2x Samsung SM/PM961 added for tests - ESXi 6.5U1 Sunday, December 10, 2017, 7:18:24 PM, you wrote: FM> Anybody can help? FM> Removed NVMe´s for now and found this: FM> root@vsan-01:~# fmadm faulty FM> --------------- ------------------------------------ -------------- --------- FM> TIME EVENT-ID MSG-ID SEVERITY FM> --------------- ------------------------------------ -------------- --------- FM> Dec 09 20:06:26 2c3507dc-6936-6a0a-a36d-a831a64939d4 PCIEX-8000-0A Critical FM> Host : vsan-01 FM> Platform : VMware-Virtual-Platform Chassis_id : FM> VMware-56-4d-08-4b-a7-e9-d9-7a-1d-a1-5f-95-55-89-d7-23 FM> Product_sn : FM> Fault class : fault.io.pciex.device-interr FM> Affects : dev:////pci@0,0/pci15ad,7a0@18/pci8086,4702@0 FM> faulted and taken out of service FM> FRU : "MB" FM> (hc://:product-id=VMware-Virtual-Platform:server-id=vsan-01:chassis-id=VMware-56-4d-08-4b-a7-e9-d9-7a-1d-a1-5f-95-55-89-d7-23/motherboard=0) FM> faulty FM> Description : A problem was detected for a PCIEX device. FM> Refer to FM> http://illumos.org/msg/PCIEX-8000-0A for more FM> information. FM> Response : One or more device instances may be disabled FM> Impact : Loss of services provided by the device instances associated with FM> this fault FM> Action : Schedule a repair procedure to replace the affected device. Use FM> fmadm faulty to identify the device or contact your illumos FM> distribution team for support. FM> --------------- ------------------------------------ -------------- --------- FM> TIME EVENT-ID MSG-ID SEVERITY FM> --------------- ------------------------------------ -------------- --------- FM> Dec 09 20:06:26 c5accbb9-26cd-4aba-dde0-ff15b11c6823 PCIEX-8000-0A Critical FM> Host : vsan-01 FM> Platform : VMware-Virtual-Platform Chassis_id : FM> VMware-56-4d-08-4b-a7-e9-d9-7a-1d-a1-5f-95-55-89-d7-23 FM> Product_sn : FM> Fault class : fault.io.pciex.device-interr FM> Affects : dev:////pci@0,0/pci15ad,7a0@17/pci15ad,7f0@0 FM> faulted and taken out of service FM> FRU : "MB" FM> (hc://:product-id=VMware-Virtual-Platform:server-id=vsan-01:chassis-id=VMware-56-4d-08-4b-a7-e9-d9-7a-1d-a1-5f-95-55-89-d7-23/motherboard=0) FM> faulty FM> Description : A problem was detected for a PCIEX device. FM> Refer to FM> http://illumos.org/msg/PCIEX-8000-0A for more FM> information. FM> Response : One or more device instances may be disabled FM> Impact : Loss of services provided by the device instances associated with FM> this fault FM> Action : Schedule a repair procedure to replace the affected device. Use FM> fmadm faulty to identify the device or contact your illumos FM> distribution team for support. FM> --------------- ------------------------------------ -------------- --------- FM> TIME EVENT-ID MSG-ID SEVERITY FM> --------------- ------------------------------------ -------------- --------- FM> Dec 09 20:06:26 4ed4fad4-9c61-cbe2-8c71-9f7dbea60d14 PCIEX-8000-0A Critical FM> Host : vsan-01 FM> Platform : VMware-Virtual-Platform Chassis_id : FM> VMware-56-4d-08-4b-a7-e9-d9-7a-1d-a1-5f-95-55-89-d7-23 FM> Product_sn : FM> Fault class : fault.io.pciex.device-interr FM> Affects : dev:////pci@0,0/pci15ad,7a0@15,1/pci8086,4702@0 FM> faulted and taken out of service FM> FRU : "MB" FM> (hc://:product-id=VMware-Virtual-Platform:server-id=vsan-01:chassis-id=VMware-56-4d-08-4b-a7-e9-d9-7a-1d-a1-5f-95-55-89-d7-23/motherboard=0) FM> faulty FM> Description : A problem was detected for a PCIEX device. FM> Refer to FM> http://illumos.org/msg/PCIEX-8000-0A for more FM> information. FM> Response : One or more device instances may be disabled FM> Impact : Loss of services provided by the device instances associated with FM> this fault FM> Action : Schedule a repair procedure to replace the affected device. Use FM> fmadm faulty to identify the device or contact your illumos FM> distribution team for support. FM> I cleared this with: FM> root@vsan-01:~# fmadm repaired FM> dev:////pci@0,0/pci15ad,7a0@17/pci15ad,7f0@0 FM> fmadm: recorded repair to of FM> dev:////pci@0,0/pci15ad,7a0@17/pci15ad,7f0@0 FM> root@vsan-01:~# fmadm repaired FM> dev:////pci@0,0/pci15ad,7a0@15,1/pci8086,4702@0 FM> fmadm: recorded repair to of FM> dev:////pci@0,0/pci15ad,7a0@15,1/pci8086,4702@0 FM> root@vsan-01:~# fmadm repaired FM> dev:////pci@0,0/pci15ad,7a0@18/pci8086,4702@0 FM> fmadm: recorded repair to of FM> dev:////pci@0,0/pci15ad,7a0@18/pci8086,4702@0 FM> Sunday, December 10, 2017, 4:29:08 AM, you wrote: FM>> I just remembered that in another system works a Samsung SM961 with FM>> NVMe 1.2 perfectly. FM>> Anybody knows why the intel P4600 doesn´t work? Or better, anybody has FM>> a solution that the intel works? FM>> Sunday, December 10, 2017, 1:54:35 AM, you wrote: FM>>> Hi, FM>>> are there any news regarding NVMe 1.2 in ESXi-passthrough systems or FM>>> is there any trick? FM>>> I can´t see the intel P4600 as a blkdev-device (updated omnios-r151024 FM>>> on ESXi 6.5). FM>>> prtconv -v produces this: FM>>> pci8086,4702, instance #0 (retired) FM>>> Hardware properties: FM>>> name='pci-msix-capid-pointer' type=int items=1 FM>>> value=00000050 FM>>> name='pci-msi-capid-pointer' type=int items=1 FM>>> value=000000a0 FM>>> name='pcie-serial' type=byte items=8 FM>>> value=fa.1a.23.4e.41.2e.cd.55 FM>>> name='acpi-namespace' type=string items=1 FM>>> value='\_SB_.PCI0.PE41.S1F0' FM>>> name='assigned-addresses' type=int items=5 FM>>> FM>>> value=83040010.00000000.fcffc000.00000000.00004000 FM>>> name='reg' type=int items=10 FM>>> FM>>> value=00040000.00000000.00000000.00000000.00000000.03040010.00000000.00000000.00000000.00004000 FM>>> name='compatible' type=string items=13 FM>>> value='pciex8086,a54.8086.4702.0' + FM>>> 'pciex8086,a54.8086.4702' + 'pciex8086,a54.0' + 'pciex8086,a54' + FM>>> 'pciexclass,010802' + 'pciexclass,0108' + FM>>> 'pci8086,a54.8086.4702.0' + 'pci8086,a54.8086.4702' + FM>>> 'pci8086,4702' + 'pci8086,a54.0' + 'pci8086,a54' + FM>>> 'pciclass,010802' + 'pciclass,0108' FM>>> name='model' type=string items=1 FM>>> value='Unknown class of pci/pnpbios device' FM>>> name='power-consumption' type=int items=2 FM>>> value=00000001.00000001 FM>>> name='devsel-speed' type=int items=1 FM>>> value=00000000 FM>>> name='interrupts' type=int items=1 FM>>> value=00000001 FM>>> name='subsystem-vendor-id' type=int items=1 FM>>> value=00008086 FM>>> name='subsystem-id' type=int items=1 FM>>> value=00004702 FM>>> name='unit-address' type=string items=1 FM>>> value='0' FM>>> name='class-code' type=int items=1 FM>>> value=00010802 FM>>> name='revision-id' type=int items=1 FM>>> value=00000000 FM>>> name='vendor-id' type=int items=1 FM>>> value=00008086 FM>>> name='device-id' type=int items=1 FM>>> value=00000a54 FM>>> name='vendor-name' type=string items=1 FM>>> value='Intel Corporation' FM>>> name='device-name' type=string items=1 FM>>> value='unknown device' FM>>> name='subsystem-name' type=string items=1 FM>>> value='unknown subsystem' FM>>> my last try was with /kernel/drv/nvme.conf: FM>>> # FM>>> # The driver was tested only against devices supporting v1.0 of the FM>>> # NVMe specification. Uncomment this to be able to use devices conforming FM>>> # to newer specifications. FM>>> strict-version=0; FM>>> # FM>>> # The driver does currently not support any vendor specific extension to the FM>>> # specification. By default it will fault the device if it receives a vendor- FM>>> # specific command status. Uncomment this to disable this behaviour. FM>>> ignore-unknown-vendor-status=1; FM>>> # FM>>> # The maximum length of the admin queue can be overridden here (16-4096). FM>>> #admin-queue-len=256; FM>>> # FM>>> # The maximum length of the individual I/O queues can be overriden here FM>>> # (16-65536). FM>>> #io-queue-len=1024; FM>>> # FM>>> # The maximum number of outstanding asynchronous event requests can FM>>> # overridden here. FM>>> #async-event-limit=10; FM>>> # FM>>> # Enable (1) or Disable (0) the volatile write cache, if present. FM>>> # FM>>> #volatile-write-cache-enable=1; FM>>> # FM>>> # The minimum physical block size to be reported to blkdev. This value must FM>>> # be a power of 2 greater than or equal to 512. FM>>> # FM>>> #min-phys-block-size=512; FM>>> Greets FM>>> Frank FM> ------------------------------------------ FM> illumos-discuss FM> Archives: FM> https://illumos.topicbox.com/groups/discuss/discussions/Tbd539f25eed62d6a-M2ca046c4e5bf2678193f154f FM> Powered by Topicbox: https://topicbox.com -- Best regards, Frank mailto:[email protected] ------------------------------------------ illumos-discuss Archives: https://illumos.topicbox.com/groups/discuss/discussions/Tbd539f25eed62d6a-Madb6a780b4e35b0ad6c21e44 Powered by Topicbox: https://topicbox.com
