Re: [vpp-dev] found some issue in pci vfio

[Damjan Marion via Lists.Fd.Io]

Dear Ping,

I already agreed that code needs to be fixed, but the point is that looking for 
container fd is not right way to detect if vfio-pci can be used or not...

-- 
Damjan

> On 2 Jan 2019, at 02:49, Yu, Ping  wrote:
> 
> Hi, Damjan,
>  
> The issue existing in the code is container fd is initialized as “-1”, and 
> there is no code to update it before trying to read  iommu_group in pci.c, 
> and container_fd is the condition required. See below code. 
>  
>  
>   if (lvm->container_fd != -1)
> {
>   u8 *tmpstr;
>   vec_reset_length (f);
>   f = format (f, "%v/iommu_group%c", dev_dir_name, 0);
>   tmpstr = clib_sysfs_link_to_name ((char *) f);
>   if (tmpstr)
> {
>   di->iommu_group = atoi ((char *) tmpstr);
>   vec_free (tmpstr);
> }
>   <>
>  <>From: Damjan Marion [mailto:dmar...@me.com] 
> Sent: Sunday, December 30, 2018 7:10 PM
> To: Yu, Ping 
> Cc: vpp-dev@lists.fd.io
> Subject: Re: [vpp-dev] found some issue in pci vfio
>  
>  
> Fact that you can open vfio container doesn't actually mean that you can use 
> vfio-pci module on specific PCI device.
>  
> vfio-pci module can be used in 2 cases:
>  - when /sys/bus/pci/devices//iommu_group exists
>  - when /sys/module/vfio/parameters/enable_unsafe_noiommu_mode is set to Y 
> (introduced in kernel 4.4)
>  
> That code needs to be fixed, but I don't think your fix is the right one
>  
> 
> 
> On 29 Dec 2018, at 04:46, Yu, Ping  <mailto:ping...@intel.com>> wrote:
>  
> I submitted a patch to fix it. 
>  
> https://gerrit.fd.io/r/16640 <https://gerrit.fd.io/r/16640>
>  
> Ping
>  
> From: Yu, Ping 
> Sent: Saturday, December 29, 2018 10:43 AM
> To: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
> Cc: Yu, Ping mailto:ping...@intel.com>>
> Subject: found some issue in pci vfio
>  
> Hello, all
>  
> Recently I met some issue in vfio, and below is some root cause:
>  
> Before running vpp, I will use “dpdk-devbind.py” to bind the NIC to vfio-pci, 
> and then use “uio-driver auto” configure, and it once worked well, but it has 
> problem recently, so I took a look at this to resolve the problem.
> I found that vpp has some problem to “auto” detect the uio-driver to be vfio, 
> and the bug info is below:
> 1)  vlib_pci_bind_to_uio is dependent vlib_pci_get_device_info to tell 
> iommu_group
> 2)  vlib_pci_get_device_info will check whether lvm->container_fd == -1
>  
> In my case, lvm->container_fd is initialized to be “-1”, and there is no 
> chance to modify it, so in the eye of vlib_pci_bind_to_uio, iommu_group is 
> -1, then it will try to enable noiommu mode. If some kernel enabled NOIOMMU, 
> then vfio can continue work with noiommu mode, but if not, then this device 
> will be rejected due to “no VFIO” support.  (pci.c # 411. )
>  
> The solution is to drop “auto” mode, and explicitly configure “vfio-pci” in 
> the configuration, but I hope the default “auto” mode can be smarter.
>  
> Ping
>  
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#11792): https://lists.fd.io/g/vpp-dev/message/11792 
> <https://lists.fd.io/g/vpp-dev/message/11792>
> Mute This Topic: https://lists.fd.io/mt/28877871/675642 
> <https://lists.fd.io/mt/28877871/675642>
> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io>
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub 
> <https://lists.fd.io/g/vpp-dev/unsub>  [dmar...@me.com 
> <mailto:dmar...@me.com>]
> -=-=-=-=-=-=-=-=-=-=-=-
>  
> -- 
> Damjan

-- 
Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11824): https://lists.fd.io/g/vpp-dev/message/11824
Mute This Topic: https://lists.fd.io/mt/28877871/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] found some issue in pci vfio

[Yu, Ping]

Hi, Damjan,

The issue existing in the code is container fd is initialized as “-1”, and 
there is no code to update it before trying to read  iommu_group in pci.c, and 
container_fd is the condition required. See below code.


  if (lvm->container_fd != -1)
{
  u8 *tmpstr;
  vec_reset_length (f);
  f = format (f, "%v/iommu_group%c", dev_dir_name, 0);
  tmpstr = clib_sysfs_link_to_name ((char *) f);
  if (tmpstr)
{
  di->iommu_group = atoi ((char *) tmpstr);
  vec_free (tmpstr);
}

From: Damjan Marion [mailto:dmar...@me.com]
Sent: Sunday, December 30, 2018 7:10 PM
To: Yu, Ping 
Cc: vpp-dev@lists.fd.io
Subject: Re: [vpp-dev] found some issue in pci vfio


Fact that you can open vfio container doesn't actually mean that you can use 
vfio-pci module on specific PCI device.

vfio-pci module can be used in 2 cases:
 - when /sys/bus/pci/devices//iommu_group exists
 - when /sys/module/vfio/parameters/enable_unsafe_noiommu_mode is set to Y 
(introduced in kernel 4.4)

That code needs to be fixed, but I don't think your fix is the right one



On 29 Dec 2018, at 04:46, Yu, Ping 
mailto:ping...@intel.com>> wrote:

I submitted a patch to fix it.

https://gerrit.fd.io/r/16640

Ping

From: Yu, Ping
Sent: Saturday, December 29, 2018 10:43 AM
To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
Cc: Yu, Ping mailto:ping...@intel.com>>
Subject: found some issue in pci vfio

Hello, all

Recently I met some issue in vfio, and below is some root cause:

Before running vpp, I will use “dpdk-devbind.py” to bind the NIC to vfio-pci, 
and then use “uio-driver auto” configure, and it once worked well, but it has 
problem recently, so I took a look at this to resolve the problem.
I found that vpp has some problem to “auto” detect the uio-driver to be vfio, 
and the bug info is below:
1)  vlib_pci_bind_to_uio is dependent vlib_pci_get_device_info to tell 
iommu_group
2)  vlib_pci_get_device_info will check whether lvm->container_fd == -1

In my case, lvm->container_fd is initialized to be “-1”, and there is no chance 
to modify it, so in the eye of vlib_pci_bind_to_uio, iommu_group is -1, then it 
will try to enable noiommu mode. If some kernel enabled NOIOMMU, then vfio can 
continue work with noiommu mode, but if not, then this device will be rejected 
due to “no VFIO” support.  (pci.c # 411. )

The solution is to drop “auto” mode, and explicitly configure “vfio-pci” in the 
configuration, but I hope the default “auto” mode can be smarter.

Ping

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11792): https://lists.fd.io/g/vpp-dev/message/11792
Mute This Topic: https://lists.fd.io/mt/28877871/675642
Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io>
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  
[dmar...@me.com<mailto:dmar...@me.com>]
-=-=-=-=-=-=-=-=-=-=-=-

--
Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11812): https://lists.fd.io/g/vpp-dev/message/11812
Mute This Topic: https://lists.fd.io/mt/28877871/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] found some issue in pci vfio

[Damjan Marion via Lists.Fd.Io]

Fact that you can open vfio container doesn't actually mean that you can use 
vfio-pci module on specific PCI device.

vfio-pci module can be used in 2 cases:
 - when /sys/bus/pci/devices//iommu_group exists
 - when /sys/module/vfio/parameters/enable_unsafe_noiommu_mode is set to Y 
(introduced in kernel 4.4)

That code needs to be fixed, but I don't think your fix is the right one


> On 29 Dec 2018, at 04:46, Yu, Ping  wrote:
> 
> I submitted a patch to fix it. 
>  
> https://gerrit.fd.io/r/16640 
>  
> Ping
>   <>
>  <>From: Yu, Ping 
> Sent: Saturday, December 29, 2018 10:43 AM
> To: vpp-dev@lists.fd.io 
> Cc: Yu, Ping mailto:ping...@intel.com>>
> Subject: found some issue in pci vfio
>  
> Hello, all
>  
> Recently I met some issue in vfio, and below is some root cause:
>  
> Before running vpp, I will use “dpdk-devbind.py” to bind the NIC to vfio-pci, 
> and then use “uio-driver auto” configure, and it once worked well, but it has 
> problem recently, so I took a look at this to resolve the problem.
> I found that vpp has some problem to “auto” detect the uio-driver to be vfio, 
> and the bug info is below:
> 1)  vlib_pci_bind_to_uio is dependent vlib_pci_get_device_info to tell 
> iommu_group
> 2)  vlib_pci_get_device_info will check whether lvm->container_fd == -1
>  
> In my case, lvm->container_fd is initialized to be “-1”, and there is no 
> chance to modify it, so in the eye of vlib_pci_bind_to_uio, iommu_group is 
> -1, then it will try to enable noiommu mode. If some kernel enabled NOIOMMU, 
> then vfio can continue work with noiommu mode, but if not, then this device 
> will be rejected due to “no VFIO” support.  (pci.c # 411. )
>  
> The solution is to drop “auto” mode, and explicitly configure “vfio-pci” in 
> the configuration, but I hope the default “auto” mode can be smarter.
>  
> Ping
>  
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> 
> View/Reply Online (#11792): https://lists.fd.io/g/vpp-dev/message/11792 
> 
> Mute This Topic: https://lists.fd.io/mt/28877871/675642 
> 
> Group Owner: vpp-dev+ow...@lists.fd.io 
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub 
>   [dmar...@me.com 
> ]
> -=-=-=-=-=-=-=-=-=-=-=-

-- 
Damjan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11802): https://lists.fd.io/g/vpp-dev/message/11802
Mute This Topic: https://lists.fd.io/mt/28877871/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] found some issue in pci vfio

[Yu, Ping]

I submitted a patch to fix it.

https://gerrit.fd.io/r/16640

Ping

From: Yu, Ping
Sent: Saturday, December 29, 2018 10:43 AM
To: vpp-dev@lists.fd.io
Cc: Yu, Ping 
Subject: found some issue in pci vfio

Hello, all

Recently I met some issue in vfio, and below is some root cause:

Before running vpp, I will use "dpdk-devbind.py" to bind the NIC to vfio-pci, 
and then use "uio-driver auto" configure, and it once worked well, but it has 
problem recently, so I took a look at this to resolve the problem.
I found that vpp has some problem to "auto" detect the uio-driver to be vfio, 
and the bug info is below:

1)  vlib_pci_bind_to_uio is dependent vlib_pci_get_device_info to tell 
iommu_group

2)  vlib_pci_get_device_info will check whether lvm->container_fd == -1

In my case, lvm->container_fd is initialized to be "-1", and there is no chance 
to modify it, so in the eye of vlib_pci_bind_to_uio, iommu_group is -1, then it 
will try to enable noiommu mode. If some kernel enabled NOIOMMU, then vfio can 
continue work with noiommu mode, but if not, then this device will be rejected 
due to "no VFIO" support.  (pci.c # 411. )

The solution is to drop "auto" mode, and explicitly configure "vfio-pci" in the 
configuration, but I hope the default "auto" mode can be smarter.

Ping

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11792): https://lists.fd.io/g/vpp-dev/message/11792
Mute This Topic: https://lists.fd.io/mt/28877871/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] found some issue in pci vfio

[Yu, Ping]

Hello, all

Recently I met some issue in vfio, and below is some root cause:

Before running vpp, I will use "dpdk-devbind.py" to bind the NIC to vfio-pci, 
and then use "uio-driver auto" configure, and it once worked well, but it has 
problem recently, so I took a look at this to resolve the problem.
I found that vpp has some problem to "auto" detect the uio-driver to be vfio, 
and the bug info is below:

1)  vlib_pci_bind_to_uio is dependent vlib_pci_get_device_info to tell 
iommu_group

2)  vlib_pci_get_device_info will check whether lvm->container_fd == -1

In my case, lvm->container_fd is initialized to be "-1", and there is no chance 
to modify it, so in the eye of vlib_pci_bind_to_uio, iommu_group is -1, then it 
will try to enable noiommu mode. If some kernel enabled NOIOMMU, then vfio can 
continue work with noiommu mode, but if not, then this device will be rejected 
due to "no VFIO" support.  (pci.c # 411. )

The solution is to drop "auto" mode, and explicitly configure "vfio-pci" in the 
configuration, but I hope the default "auto" mode can be smarter.

Ping

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#11791): https://lists.fd.io/g/vpp-dev/message/11791
Mute This Topic: https://lists.fd.io/mt/28877871/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-