[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different 
role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
and sets the ip, queue number, etc. The user obtains the identity of
the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
of the patch

Signed-off-by: Xuan Zhuo 
---
  admin.tex | 41 -
  1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
  \hline \hline
  0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands 
supported for this group type\\
  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used 
for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
  \hline
  0x8000 - 0x & - & Reserved for future commands (possibly using a 
different structure)\\
  \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
  \field{VF Enable} refer to registers within the SR-IOV Extended
  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
  
+\subsubsection{Bind the device for member}

+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the 
virtio
If the VFs are create in a guest OS, I assume that means the user has 
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security 
issue(affects host pci).

+devices don't know about these VFs. Because each VF may be assigned a 
different role
+by the user, the virtio device can not choose one VF to bind random.
I failed to understand this, once a VF is created, it has a personality, 
e.g.,

create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device 
choose

one VF to bind?

+So only the user knows how to bind the virtio devices to the VFs.
+On the other hand, generally the virtio devices are not created by the user
+inside the guest OS. This requires some management platform to participate.
+
+So we introduce a new admin queue command to bind the VFs and the virtio
+devices.

Sorry, failed to process this. Maybe an orchestration sw layer can help?
Provision a device on demands and assign it to a guest?

Thanks

+
+\begin{lstlisting}
+struct virtio_admin_cmd_bind {
+u64 identity;
+};
+\end{lstlisting}
+
+The user got the \field{identity} from the management platform, that is not
+included by this spec.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities 
of a Virtio Device / Device groups / Group administration commands / Bind the 
device for member}
+
+VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be 
set.
+
+The \field{identity} is passed by the user. It is the identity of the virtio
+device.
+
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities 
of a Virtio Device / Device groups / Group administration commands / Bind the 
device for member}
+
+Every device MUST have one unique \field{identity} in the host.
+
+If the PF device can not find the device by the \field{identity},
+the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
+
+If the device is found by the \field{identity}, the device MUST work as the
+device of this group member specified by the \field{group_member_id}.
+
  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio 
Device / Administration Virtqueues}
  
  An administration virtqueue of an owner device is used to submit



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 3:08 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"  
wrote:


On 6/26/2023 2:22 PM, Xuan Zhuo wrote:

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different 
role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
 and sets the ip, queue number, etc. The user obtains the identity of
 the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
 of the patch

Signed-off-by: Xuan Zhuo 
---
   admin.tex | 41 -
   1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
   \hline \hline
   0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands 
supported for this group type\\
   0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands 
used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
   \hline
   0x8000 - 0x & - & Reserved for future commands (possibly using a 
different structure)\\
   \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
   \field{VF Enable} refer to registers within the SR-IOV Extended
   Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the 
virtio

If the VFs are create in a guest OS, I assume that means the user has
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security
issue(affects host pci).


No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.

OK, perhaps just say create VFs from a PF in the OS?




+devices don't know about these VFs. Because each VF may be assigned a 
different role
+by the user, the virtio device can not choose one VF to bind random.

I failed to understand this, once a VF is created, it has a personality,
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device
choose
one VF to bind?


On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a 
guest on demands.


So I am confused what the intention of this patch.


Thanks.



+So only the user knows how to bind the virtio devices to the VFs.
+On the other hand, generally the virtio devices are not created by the user
+inside the guest OS. This requires some management platform to participate.
+
+So we introduce a new admin queue command to bind the VFs and the virtio
+devices.

Sorry, failed to process this. Maybe an orchestration sw layer can help?
Provision a device on demands and assign it to a guest?

Thanks

+
+\begin{lstlisting}
+struct virtio_admin_cmd_bind {
+u64 identity;
+};
+\end{lstlisting}
+
+The user got the \field{identity} from the management platform, that is not
+included by this spec.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities 
of a Virtio Device / Device groups / Group administration commands / Bind the 
device for member}
+
+VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be 
set.
+
+The \field{identity} is passed by the user. It is the identity of the virtio
+device.
+
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities 
of a Virtio Device / Device groups / Group administration commands / Bind the 
device for member}
+
+Every device MUST have one unique \field{identity} in the host.
+
+If the PF device can not find the device by the \field{identity},
+the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
+
+If the device is found by the \field{identity}, the device MUST work as the
+device of this grou

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Mon, Jun 26, 2023 at 2:22 PM Xuan Zhuo  wrote:
>
> The VFs of the SR-IOV are created by the user inside the guest OS, so the 
> virtio
> devices don't know about these VFs. Because each VF may be assigned a 
> different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>and sets the ip, queue number, etc. The user obtains the identity of
>the network card.

Any reason we can't do this through IPC of software layers?

Thanks

> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>of the patch
>
> Signed-off-by: Xuan Zhuo 
> ---
>  admin.tex | 41 -
>  1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration 
> commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands 
> supported for this group type\\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands 
> used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member 
> \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
>  \hline
>  0x8000 - 0x & - & Reserved for future commands (possibly using a 
> different structure)\\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration 
> commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the 
> virtio
> +devices don't know about these VFs. Because each VF may be assigned a 
> different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities 
> of a Virtio Device / Device groups / Group administration commands / Bind the 
> device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST 
> be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities 
> of a Virtio Device / Device groups / Group administration commands / Bind the 
> device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio 
> Device / Administration Virtqueues}
>
>  An administration virtqueue of an owner device is used to submit
> --
> 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscr...@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
> List help: virtio-comment-h...@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


-
To unsubscribe, e-mail

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo  wrote:
>
>
> Thanks Parav for pointing it out. We may have some gaps on the case.
>
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
>
> First, the user (customer) purchased a bare metal machine.
>
> ## Bare metal machine
>
> Let me briefly explain the characteristics of a bare metal machine. It is not 
> a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to a
> DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
>
> ## DPU
>
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
>
> ## Management Platform
>
> The creation and configuration of devices is realized on the management
> platform.
>
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and 
> other
> information.
>
> ## Usage
>
> Since the user is directly on the HOST, the user can create VMs, passthrough 
> PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
>
> The reason why users use vf is that we need to use a large number of 
> virtio-net
> devices. This number reaches 1k+.
>
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device 
> when
> the user creates a vf.
>
> ## Migration
>
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated 
> to
> the DPU. At this time, our newly created vf can only be used by the vm after 
> it
> is bound to the migrated device. We do not want this vf to be a brand new
> device.
>
> ## Abstraction
>
> So, this is how I understand the process of creating vf:
>
> 1. Create a PCI VF, at this time there may be no backend virtio device, or 
> there
> is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf

3) should come before 2)?

Who is going to do 3) btw, is it the user? If yes, for example, if a
user wants another 4 queue virtio-net devices, after purchase, how
does the user know its id?

>
> In most scenarios, the first step may be enough. We can make some fine-tuning 
> on
> this default device, such as modifying its mac. In the future, we can use 
> admin
> queue to modify its msix vector and other configurations.
>
> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.

All of these could be addressed if a dynamic provisioning model is
implemented (SIOV or transport virtqueue). Trying to have a workaround
in SR-IOV might be tricky.

>
> The design of virtio itself is two layers, and virtio should allow switching 
> the
> transport layer by nature. This is our advantage.

Is it not switching the transport layer but about binding/unbinding
vitio devices to VF?

Is a new capability or similar admin cmd sufficient in this case?

struct virtio_pci_bind_cap {
struct virtio_pci_cap cap;
u16 bind; // virtio_device_id
u16 unbind; // virtio_device_id
};

Thanks

>
> ## About the identity
>
> In this patch, I used a vendor's id. The purpose of this is that I hope to
> be compatible with all devices. In the network scenario, it is actually 
> possible
> to use a mac.
>
> Perhaps, introduce a standard id for each device/driver, I think this may be
> more in line with the habit of virtio spec.
>
> Thanks.
>
> On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo  
> wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the 
> > virtio
> > devices don't know about these VFs. Because each VF may be assigned a 
> > different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >and sets the ip, queue num

[virtio-dev] RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo 
> Sent: Tuesday, June 27, 2023 4:23 AM
> 
> Thanks Parav for pointing it out. We may have some gaps on the case.
> 
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
> 
> First, the user (customer) purchased a bare metal machine.
> 
> ## Bare metal machine
> 
> Let me briefly explain the characteristics of a bare metal machine. It is not 
> a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to 
> a DPU.
> This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
> 
Each of this device can have a unique serial number or id.
Or a parent PF can have the unique serial number.

> ## DPU
> 
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
> 
> ## Management Platform
> 
> The creation and configuration of devices is realized on the management
> platform.
> 
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and
> other information.
> 
> ## Usage
> 
> Since the user is directly on the HOST, the user can create VMs, passthrough 
> PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
> 
> The reason why users use vf is that we need to use a large number of 
> virtio-net
> devices. This number reaches 1k+.
> 
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device
> when the user creates a vf.
> 
Yes, this is common practice for us as well.
A backend device needs to know the unique serial number/id of the virtio device.

> ## Migration
> 
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated 
> to
> the DPU. At this time, our newly created vf can only be used by the vm after 
> it is
> bound to the migrated device. We do not want this vf to be a brand new device.
>
Same for us too.
 
> ## Abstraction
> 
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or 
> there
> is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf
> 
> In most scenarios, the first step may be enough. We can make some fine-tuning
> on this default device, such as modifying its mac. In the future, we can use
> admin queue to modify its msix vector and other configurations.
> 
Yep.

> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.
>
This is the job of non virtio layer to do the work in backend.
 
> The design of virtio itself is two layers, and virtio should allow switching 
> the
> transport layer by nature. This is our advantage.
> 
> ## About the identity
> 
> In this patch, I used a vendor's id. The purpose of this is that I hope to be
> compatible with all devices. In the network scenario, it is actually possible 
> to
> use a mac.
>
Mac is not good as during migration there is transient where two vfs will have 
same mac.

So we should use the unique device identifier.

There are few options that I am aware of.

1. Have unique serial number for the PCI PF in the PCI VPD data and VF number 
to uniquely identify the VF of a PF.
Management system backend will know and able to identify this.
Pros:
This does not need any virtio spec involvement. Can be done today on existing 
virtio device.
Cons:
There is no single device handle to work with backend, but not a big problem.
If the REST APIs etc of the VPC are crafted properly.

2. Expose unique serial number per each VF using same above PCI VPD data.
This option #2 is good in theory, not efficient to implement as PCI level.
Pros:
a. Unique per device.
b. accessible without binding to the actual driver (vfio vs virtio for 
different use case)

3. Expose unique serial number/identifier at virtio device level over a 
virtqueue interface
Pros:
a. achieves the desired scale
b. unique per device
Cons:
a. One needs to bind to the driver to discover the unique device identifier and 
involve the device reset flow.
Not an optimal solu

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or 
> there
> is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf

I can see this making sense as a feature bit that says VFs are not
initialized by default and must first be setup through an admin command.
This will likely need to be a feature bit because it's changing
behaviour outside of admin commands.

Then, we can have:

ADMIN_SETUP VF#
ADMIN_CLEANUP VF#

I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.


Why do we need an id as a level of indirection though? What is wrong
with just using VF# directly?



-- 
MST


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin"  
wrote:
> On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > So, this is how I understand the process of creating vf:
> >
> > 1. Create a PCI VF, at this time there may be no backend virtio device, or 
> > there
> > is only a default backend. It does not fully meet our expectations.
> > 2. Create device or migrate device
> > 3. Bind the backend virtio device to the vf
>
> I can see this making sense as a feature bit that says VFs are not
> initialized by default and must first be setup through an admin command.
> This will likely need to be a feature bit because it's changing
> behaviour outside of admin commands.
>
> Then, we can have:
>
>   ADMIN_SETUP VF#
>   ADMIN_CLEANUP VF#
>
> I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.

Great!!

>
>
> Why do we need an id as a level of indirection though? What is wrong
> with just using VF# directly?

I think VF# is ok. I also need to use it. But we need an ID for virtio device
not the transport(PF, VF).

What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
to connect the virtio driver and the virtio device. Right?

The virtio device does not necessarily exist depending on PCI. For example, a
virtio device is migrated from another DPU, and it is not associated with any
PCI. What I have always wanted to say is that this device(not PCI) must have its
own ID, which has nothing to do with the transport.

Now we want to use this migrated device and connect it to the corresponding
vm (migrated from the same host). We can passthrough vf to this vm. But how do I
tell our DPU to bind this migrated device with this vf?
We can specify the VF by the VF#, but how can we specify the virtio device?
Maybe there are two migrated virtio device.

Maybe we should compare it to the case of using PF directly before. The biggest
difference here is that PF is directly hot-plugged by the DPU, so when a
user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
on the host. Now the vf is created by the user, and only the user knows how to
use each one. We cannot directly bind the device purchased or migrated by the
user to a certain vf. We need the user in the host to bind the vf(transport) to
a certain virtio device.

Thanks.






>
>
>
> --
> MST
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscr...@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
> List help: virtio-comment-h...@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo  wrote:
>
> On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang  wrote:
> > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo  
> > wrote:
> > >
> > >
> > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > >
> > > Let me introduce our case, which I think it is simple and should be easy 
> > > to
> > > understand.
> > >
> > > First, the user (customer) purchased a bare metal machine.
> > >
> > > ## Bare metal machine
> > >
> > > Let me briefly explain the characteristics of a bare metal machine. It is 
> > > not a
> > > virtual machine, it is a physical machine, and the difference between it 
> > > and a
> > > general physical machine is that its PCI is connected to a device similar 
> > > to a
> > > DPU. This DPU provides devices such as virtio-blk/net to the host through 
> > > PCI.
> > > These devices are managed by the vendor, and must be created and purchased
> > > on the vendor's management platform.
> > >
> > > ## DPU
> > >
> > > There is a software implementation in the DPU, which will respond to PCI
> > > operations. But as mentioned above, resources such as network cards must 
> > > be
> > > purchased and created before they can exist. So users can create VF, 
> > > which is
> > > just a pci-level operation, but there may not be a corresponding backend.
> > >
> > > ## Management Platform
> > >
> > > The creation and configuration of devices is realized on the management
> > > platform.
> > >
> > > After the user completed the purchase on the management platform (this is 
> > > an
> > > independent platform provided by the vendor and has nothing to do with
> > > virtio), then there will be a corresponding device implementation in the 
> > > DPU.
> > > This includes some user configurations, available bandwidth resources and 
> > > other
> > > information.
> > >
> > > ## Usage
> > >
> > > Since the user is directly on the HOST, the user can create VMs, 
> > > passthrough PF
> > > or VF into the VM. Or users can create a large number of dockers, all of 
> > > which
> > > use a separate virtio-net device for performance.
> > >
> > > The reason why users use vf is that we need to use a large number of 
> > > virtio-net
> > > devices. This number reaches 1k+.
> > >
> > > Based on this scenario, we need to bind vf to the backend device. 
> > > Because, we
> > > cannot automatically complete the creation of the virtio-net backend 
> > > device when
> > > the user creates a vf.
> > >
> > > ## Migration
> > >
> > > In addition, let's consider another scenario of migration. If a vm is 
> > > migrated
> > > from another host, of course its corresponding virtio device is also 
> > > migrated to
> > > the DPU. At this time, our newly created vf can only be used by the vm 
> > > after it
> > > is bound to the migrated device. We do not want this vf to be a brand new
> > > device.
> > >
> > > ## Abstraction
> > >
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device, 
> > > or there
> > > is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > 3) should come before 2)?
> >
> > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > user wants another 4 queue virtio-net devices, after purchase, how
> > does the user know its id?
>
> Got the id from the management platform.

So it can do the binding via that management platform which this
became a cloud vendor specific interface.

>
> >
> > >
> > > In most scenarios, the first step may be enough. We can make some 
> > > fine-tuning on
> > > this default device, such as modifying its mac. In the future, we can use 
> > > admin
> > > queue to modify its msix vector and other configurations.
> > >
> > > But we should allow, we bind a backend virtio device to a certain vf. 
> > > This is
> > > useful for live migration and virtio devices with special configurations.
> >
> > All of these could be addressed if a dynamic provisioning model is
> > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > in SR-IOV might be tricky.
>
>
> SR-IOV vf is native PCI device, this is the advancement.

The problem is that it doesn't support flexible provisioning, e.g
create and destroy a single VF.

>
>
> >
> > >
> > > The design of virtio itself is two layers, and virtio should allow 
> > > switching the
> > > transport layer by nature. This is our advantage.
> >
> > Is it not switching the transport layer but about binding/unbinding
> > vitio devices to VF?
>
> YES.
>
> >
> > Is a new capability or similar admin cmd sufficient in this case?
>
> All is ok.
>
>
> >
> > struct virtio_pci_bind_cap {
> > struct virtio_pci_cap cap;
> > u16 bind; // virtio_device_id
> > u16 unbind; // virtio_device_id
> > };
>
> You mean that the "bind" or "unbind" is writeable?

Yes.

Thank

[virtio-dev] RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo 
> Sent: Tuesday, June 27, 2023 10:21 PM

> > I can see this making sense as a feature bit that says VFs are not
> > initialized by default and must first be setup through an admin command.
> > This will likely need to be a feature bit because it's changing
> > behaviour outside of admin commands.
> >
> > Then, we can have:
> >
> > ADMIN_SETUP VF#
> > ADMIN_CLEANUP VF#
> >
> > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> 

What does this command actually do or expected do on the device?

> Great!!
> 
> >
> >
> > Why do we need an id as a level of indirection though? What is wrong
> > with just using VF# directly?
> 
> I think VF# is ok. I also need to use it. But we need an ID for virtio device 
> not the
> transport(PF, VF).
> 
> What I want to emphasize is that PCI(pf or vf) is a transport, it is only 
> used to
> connect the virtio driver and the virtio device. Right?
> 
> The virtio device does not necessarily exist depending on PCI. For example, a
> virtio device is migrated from another DPU, and it is not associated with any 
> PCI.
> What I have always wanted to say is that this device(not PCI) must have its 
> own
> ID, which has nothing to do with the transport.
> 
A virtio device can have the id visible to self and visible to group owner 
device.

> Now we want to use this migrated device and connect it to the corresponding
> vm (migrated from the same host). We can passthrough vf to this vm. But how
> do I tell our DPU to bind this migrated device with this vf?

When a virtio device state is set on a specific VF on the compute side (not on 
the dpu side),
This directly indicates to the dpu side, which virtio device is attached to 
which VF.

> We can specify the VF by the VF#, but how can we specify the virtio device?
> Maybe there are two migrated virtio device.
>
 virtio device state setting itself will contain the device identifier.

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin"  
> wrote:
> > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device, 
> > > or there
> > > is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > I can see this making sense as a feature bit that says VFs are not
> > initialized by default and must first be setup through an admin command.
> > This will likely need to be a feature bit because it's changing
> > behaviour outside of admin commands.
> >
> > Then, we can have:
> >
> > ADMIN_SETUP VF#
> > ADMIN_CLEANUP VF#
> >
> > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> 
> Great!!
> 
> >
> >
> > Why do we need an id as a level of indirection though? What is wrong
> > with just using VF# directly?
> 
> I think VF# is ok. I also need to use it. But we need an ID for virtio device
> not the transport(PF, VF).
> 
> What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> to connect the virtio driver and the virtio device. Right?
> 
> The virtio device does not necessarily exist depending on PCI. For example, a
> virtio device is migrated from another DPU, and it is not associated with any
> PCI. What I have always wanted to say is that this device(not PCI) must have 
> its
> own ID, which has nothing to do with the transport.
> 
> Now we want to use this migrated device and connect it to the corresponding
> vm (migrated from the same host). We can passthrough vf to this vm. But how 
> do I
> tell our DPU to bind this migrated device with this vf?
> We can specify the VF by the VF#, but how can we specify the virtio device?
> Maybe there are two migrated virtio device.
> 
> Maybe we should compare it to the case of using PF directly before. The 
> biggest
> difference here is that PF is directly hot-plugged by the DPU, so when a
> user(custom) buys a virtio-net device, the DPU directly inserts a new PCI 
> device
> on the host. Now the vf is created by the user, and only the user knows how to
> use each one. We cannot directly bind the device purchased or migrated by the
> user to a certain vf. We need the user in the host to bind the vf(transport) 
> to
> a certain virtio device.
> 
> Thanks.
> 
> 

Maybe I didn't have enough coffee today but I can't figure out what all
this means :( For example what does "exist depending" mean?  What does
"device is migrated" mean?  What does it mean to be "migrated from the
same host"? What is "any PCI"? E.g. I know people did vm migration
moving virtio from a hardware device to a software
implementation. Is that "not associated with any PCI" ?
What is "user(custom)"? how is "the vf is created by the user"?
what does it mean to "bind the device .. to a certain vf"?
It looks like Parav understand what's going on though, so
maybe it's my fault.

But fundamentally, the problem is that the spec patch doesn't do
anything useful by itself, it relies on some out of spec interface to
make these id values make sense. So the TC is reduced to guessing: we
could just trust you that it's useful somehow and at the same time
serves the purpose whatever it is.

But it would be better if instead of trying to explain what that does in
vague terms, you post a spec patch that allows doing whatever needs
doing for these IDs to make sense through e.g. admin commands.

-- 
MST


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Wed, Jun 28, 2023 at 02:06:32PM +0800, Xuan Zhuo wrote:
> On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang  wrote:
> > On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo  
> > wrote:
> > >
> > > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang  
> > > wrote:
> > > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo  
> > > > wrote:
> > > > >
> > > > >
> > > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > > >
> > > > > Let me introduce our case, which I think it is simple and should be 
> > > > > easy to
> > > > > understand.
> > > > >
> > > > > First, the user (customer) purchased a bare metal machine.
> > > > >
> > > > > ## Bare metal machine
> > > > >
> > > > > Let me briefly explain the characteristics of a bare metal machine. 
> > > > > It is not a
> > > > > virtual machine, it is a physical machine, and the difference between 
> > > > > it and a
> > > > > general physical machine is that its PCI is connected to a device 
> > > > > similar to a
> > > > > DPU. This DPU provides devices such as virtio-blk/net to the host 
> > > > > through PCI.
> > > > > These devices are managed by the vendor, and must be created and 
> > > > > purchased
> > > > > on the vendor's management platform.
> > > > >
> > > > > ## DPU
> > > > >
> > > > > There is a software implementation in the DPU, which will respond to 
> > > > > PCI
> > > > > operations. But as mentioned above, resources such as network cards 
> > > > > must be
> > > > > purchased and created before they can exist. So users can create VF, 
> > > > > which is
> > > > > just a pci-level operation, but there may not be a corresponding 
> > > > > backend.
> > > > >
> > > > > ## Management Platform
> > > > >
> > > > > The creation and configuration of devices is realized on the 
> > > > > management
> > > > > platform.
> > > > >
> > > > > After the user completed the purchase on the management platform 
> > > > > (this is an
> > > > > independent platform provided by the vendor and has nothing to do with
> > > > > virtio), then there will be a corresponding device implementation in 
> > > > > the DPU.
> > > > > This includes some user configurations, available bandwidth resources 
> > > > > and other
> > > > > information.
> > > > >
> > > > > ## Usage
> > > > >
> > > > > Since the user is directly on the HOST, the user can create VMs, 
> > > > > passthrough PF
> > > > > or VF into the VM. Or users can create a large number of dockers, all 
> > > > > of which
> > > > > use a separate virtio-net device for performance.
> > > > >
> > > > > The reason why users use vf is that we need to use a large number of 
> > > > > virtio-net
> > > > > devices. This number reaches 1k+.
> > > > >
> > > > > Based on this scenario, we need to bind vf to the backend device. 
> > > > > Because, we
> > > > > cannot automatically complete the creation of the virtio-net backend 
> > > > > device when
> > > > > the user creates a vf.
> > > > >
> > > > > ## Migration
> > > > >
> > > > > In addition, let's consider another scenario of migration. If a vm is 
> > > > > migrated
> > > > > from another host, of course its corresponding virtio device is also 
> > > > > migrated to
> > > > > the DPU. At this time, our newly created vf can only be used by the 
> > > > > vm after it
> > > > > is bound to the migrated device. We do not want this vf to be a brand 
> > > > > new
> > > > > device.
> > > > >
> > > > > ## Abstraction
> > > > >
> > > > > So, this is how I understand the process of creating vf:
> > > > >
> > > > > 1. Create a PCI VF, at this time there may be no backend virtio 
> > > > > device, or there
> > > > > is only a default backend. It does not fully meet our 
> > > > > expectations.
> > > > > 2. Create device or migrate device
> > > > > 3. Bind the backend virtio device to the vf
> > > >
> > > > 3) should come before 2)?
> > > >
> > > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > > user wants another 4 queue virtio-net devices, after purchase, how
> > > > does the user know its id?
> > >
> > > Got the id from the management platform.
> >
> > So it can do the binding via that management platform which this
> > became a cloud vendor specific interface.
> 
> In our scenario, this is bound by the user using this id and vf id in the os.
> 
> >
> > >
> > > >
> > > > >
> > > > > In most scenarios, the first step may be enough. We can make some 
> > > > > fine-tuning on
> > > > > this default device, such as modifying its mac. In the future, we can 
> > > > > use admin
> > > > > queue to modify its msix vector and other configurations.
> > > > >
> > > > > But we should allow, we bind a backend virtio device to a certain vf. 
> > > > > This is
> > > > > useful for live migration and virtio devices with special 
> > > > > configurations.
> > > >
> > > > All of these could be addressed if a dynamic provisioning model is
> > > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > > in SR-IOV might be tricky.
> > >
> > >

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Wed, Jun 28, 2023 at 11:55 PM Michael S. Tsirkin  wrote:
>
> On Wed, Jun 28, 2023 at 02:06:32PM +0800, Xuan Zhuo wrote:
> > On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang  wrote:
> > > On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo  
> > > wrote:
> > > >
> > > > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang  
> > > > wrote:
> > > > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo 
> > > > >  wrote:
> > > > > >
> > > > > >
> > > > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > > > >
> > > > > > Let me introduce our case, which I think it is simple and should be 
> > > > > > easy to
> > > > > > understand.
> > > > > >
> > > > > > First, the user (customer) purchased a bare metal machine.
> > > > > >
> > > > > > ## Bare metal machine
> > > > > >
> > > > > > Let me briefly explain the characteristics of a bare metal machine. 
> > > > > > It is not a
> > > > > > virtual machine, it is a physical machine, and the difference 
> > > > > > between it and a
> > > > > > general physical machine is that its PCI is connected to a device 
> > > > > > similar to a
> > > > > > DPU. This DPU provides devices such as virtio-blk/net to the host 
> > > > > > through PCI.
> > > > > > These devices are managed by the vendor, and must be created and 
> > > > > > purchased
> > > > > > on the vendor's management platform.
> > > > > >
> > > > > > ## DPU
> > > > > >
> > > > > > There is a software implementation in the DPU, which will respond 
> > > > > > to PCI
> > > > > > operations. But as mentioned above, resources such as network cards 
> > > > > > must be
> > > > > > purchased and created before they can exist. So users can create 
> > > > > > VF, which is
> > > > > > just a pci-level operation, but there may not be a corresponding 
> > > > > > backend.
> > > > > >
> > > > > > ## Management Platform
> > > > > >
> > > > > > The creation and configuration of devices is realized on the 
> > > > > > management
> > > > > > platform.
> > > > > >
> > > > > > After the user completed the purchase on the management platform 
> > > > > > (this is an
> > > > > > independent platform provided by the vendor and has nothing to do 
> > > > > > with
> > > > > > virtio), then there will be a corresponding device implementation 
> > > > > > in the DPU.
> > > > > > This includes some user configurations, available bandwidth 
> > > > > > resources and other
> > > > > > information.
> > > > > >
> > > > > > ## Usage
> > > > > >
> > > > > > Since the user is directly on the HOST, the user can create VMs, 
> > > > > > passthrough PF
> > > > > > or VF into the VM. Or users can create a large number of dockers, 
> > > > > > all of which
> > > > > > use a separate virtio-net device for performance.
> > > > > >
> > > > > > The reason why users use vf is that we need to use a large number 
> > > > > > of virtio-net
> > > > > > devices. This number reaches 1k+.
> > > > > >
> > > > > > Based on this scenario, we need to bind vf to the backend device. 
> > > > > > Because, we
> > > > > > cannot automatically complete the creation of the virtio-net 
> > > > > > backend device when
> > > > > > the user creates a vf.
> > > > > >
> > > > > > ## Migration
> > > > > >
> > > > > > In addition, let's consider another scenario of migration. If a vm 
> > > > > > is migrated
> > > > > > from another host, of course its corresponding virtio device is 
> > > > > > also migrated to
> > > > > > the DPU. At this time, our newly created vf can only be used by the 
> > > > > > vm after it
> > > > > > is bound to the migrated device. We do not want this vf to be a 
> > > > > > brand new
> > > > > > device.
> > > > > >
> > > > > > ## Abstraction
> > > > > >
> > > > > > So, this is how I understand the process of creating vf:
> > > > > >
> > > > > > 1. Create a PCI VF, at this time there may be no backend virtio 
> > > > > > device, or there
> > > > > > is only a default backend. It does not fully meet our 
> > > > > > expectations.
> > > > > > 2. Create device or migrate device
> > > > > > 3. Bind the backend virtio device to the vf
> > > > >
> > > > > 3) should come before 2)?
> > > > >
> > > > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > > > user wants another 4 queue virtio-net devices, after purchase, how
> > > > > does the user know its id?
> > > >
> > > > Got the id from the management platform.
> > >
> > > So it can do the binding via that management platform which this
> > > became a cloud vendor specific interface.
> >
> > In our scenario, this is bound by the user using this id and vf id in the 
> > os.
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > In most scenarios, the first step may be enough. We can make some 
> > > > > > fine-tuning on
> > > > > > this default device, such as modifying its mac. In the future, we 
> > > > > > can use admin
> > > > > > queue to modify its msix vector and other configurations.
> > > > > >
> > > > > > But we should allow, we bind a backend virtio device to a certain 

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 4:09 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"  
wrote:


On 6/26/2023 3:08 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different 
role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
  and sets the ip, queue number, etc. The user obtains the identity of
  the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
  of the patch

Signed-off-by: Xuan Zhuo 
---
admin.tex | 41 -
1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
\hline \hline
0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands 
supported for this group type\\
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands 
used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
\hline
0x8000 - 0x & - & Reserved for future commands (possibly using a 
different structure)\\
\hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
\field{VF Enable} refer to registers within the SR-IOV Extended
Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the 
virtio

If the VFs are create in a guest OS, I assume that means the user has
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security
issue(affects host pci).

No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.

OK, perhaps just say create VFs from a PF in the OS?

YES.





+devices don't know about these VFs. Because each VF may be assigned a 
different role
+by the user, the virtio device can not choose one VF to bind random.

I failed to understand this, once a VF is created, it has a personality,
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device
choose
one VF to bind?

On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a
guest on demands.


Yes, but we do not need to care about the guest. Because VF may only be used
in host, such as docker.

The problem is that the user (you can think of this as the orchestration
software) creates some VFs, these are only some PCI devices, which virtio
devices will work on these VFs. I think that creating a vf and creating a
virtio-net device are two different things. One is done by user in the OS, one
is done on the management platform.  So we need to bind them together.

If the VFs are created through sriov_numvfs, once created, the VF device and
its personality are determined.

PCI spec says:
All VFs associated with a PF must be the same device type as the PF, 
(e.g., the same network

device type or the same storage device type.)

So how can the creating process be splitted into separated steps?

Are we discussing something beyond the spec?

Thanks

Thanks.




So I am confused what the intention of this patch.

Thanks.



+So only the user knows how to bind the virtio devices to the VFs.
+On the other hand, generally the virtio devices are not created by the user
+inside the guest OS. This requires some management platform to participate.
+
+So we introduce a new admin queue command to bind the VFs and the virtio
+devices.

Sorry, failed to process this. Maybe an orchestration sw layer can help?
Provision a device on demands and assign it to a guest?

Thanks

+
+\begin{lstlisting}
+struct virtio_a

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Wed, 28 Jun 2023 11:41:02 -0400, "Michael S. Tsirkin"  
wrote:
> On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> > On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin"  
> > wrote:
> > > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > > So, this is how I understand the process of creating vf:
> > > >
> > > > 1. Create a PCI VF, at this time there may be no backend virtio device, 
> > > > or there
> > > > is only a default backend. It does not fully meet our expectations.
> > > > 2. Create device or migrate device
> > > > 3. Bind the backend virtio device to the vf
> > >
> > > I can see this making sense as a feature bit that says VFs are not
> > > initialized by default and must first be setup through an admin command.
> > > This will likely need to be a feature bit because it's changing
> > > behaviour outside of admin commands.
> > >
> > > Then, we can have:
> > >
> > >   ADMIN_SETUP VF#
> > >   ADMIN_CLEANUP VF#
> > >
> > > I like this because this generalizes CREATE/DESTROY that SIOV guys 
> > > proposed.
> >
> > Great!!
> >
> > >
> > >
> > > Why do we need an id as a level of indirection though? What is wrong
> > > with just using VF# directly?
> >
> > I think VF# is ok. I also need to use it. But we need an ID for virtio 
> > device
> > not the transport(PF, VF).
> >
> > What I want to emphasize is that PCI(pf or vf) is a transport, it is only 
> > used
> > to connect the virtio driver and the virtio device. Right?
> >
> > The virtio device does not necessarily exist depending on PCI. For example, 
> > a
> > virtio device is migrated from another DPU, and it is not associated with 
> > any
> > PCI. What I have always wanted to say is that this device(not PCI) must 
> > have its
> > own ID, which has nothing to do with the transport.
> >
> > Now we want to use this migrated device and connect it to the corresponding
> > vm (migrated from the same host). We can passthrough vf to this vm. But how 
> > do I
> > tell our DPU to bind this migrated device with this vf?
> > We can specify the VF by the VF#, but how can we specify the virtio device?
> > Maybe there are two migrated virtio device.
> >
> > Maybe we should compare it to the case of using PF directly before. The 
> > biggest
> > difference here is that PF is directly hot-plugged by the DPU, so when a
> > user(custom) buys a virtio-net device, the DPU directly inserts a new PCI 
> > device
> > on the host. Now the vf is created by the user, and only the user knows how 
> > to
> > use each one. We cannot directly bind the device purchased or migrated by 
> > the
> > user to a certain vf. We need the user in the host to bind the 
> > vf(transport) to
> > a certain virtio device.
> >
> > Thanks.
> >
> >
>
> Maybe I didn't have enough coffee today but I can't figure out what all
> this means :( For example what does "exist depending" mean?  What does
> "device is migrated" mean?  What does it mean to be "migrated from the
> same host"? What is "any PCI"? E.g. I know people did vm migration
> moving virtio from a hardware device to a software
> implementation. Is that "not associated with any PCI" ?
> What is "user(custom)"? how is "the vf is created by the user"?
> what does it mean to "bind the device .. to a certain vf"?
> It looks like Parav understand what's going on though, so
> maybe it's my fault.
>
> But fundamentally, the problem is that the spec patch doesn't do
> anything useful by itself, it relies on some out of spec interface to
> make these id values make sense. So the TC is reduced to guessing: we
> could just trust you that it's useful somehow and at the same time
> serves the purpose whatever it is.
>
> But it would be better if instead of trying to explain what that does in
> vague terms, you post a spec patch that allows doing whatever needs
> doing for these IDs to make sense through e.g. admin commands.

I try to describe the requirement as a common requirement.

@Parav do you think this is a requirement in your devices?

The core point here is that our virtio-net device is created by customers on the
management platform. It is not a pure virtio-net device, but also includes many
other features, such as bandwidth, such as VPC information, etc. These are not
within the scope of virtio, we cannot configure these information based on the
admin queue after creating the vf. But we can bind the device created by the
user to vf, which completes the binding of many manufacturer features to vf.

In addition to this scenario, in the case of live migration, we can migrate a vm
from one host to another host. During this process, the virtio-net devce is also
migrated from the dpu of one host to the dpu of another host . At this time, the
purpose of creating a vf is to insert the migrated virtio-net into the host. It
is impossible for us to create a vf to get a brand new device. So also bind.

The core problem of this patch is that we need to assign an id to each device.
Is this reasonable? I think this

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Mon, Jul 3, 2023 at 11:53 AM Xuan Zhuo  wrote:
>
> On Wed, 28 Jun 2023 11:41:02 -0400, "Michael S. Tsirkin"  
> wrote:
> > On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> > > On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" 
> > >  wrote:
> > > > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > > > So, this is how I understand the process of creating vf:
> > > > >
> > > > > 1. Create a PCI VF, at this time there may be no backend virtio 
> > > > > device, or there
> > > > > is only a default backend. It does not fully meet our 
> > > > > expectations.
> > > > > 2. Create device or migrate device
> > > > > 3. Bind the backend virtio device to the vf
> > > >
> > > > I can see this making sense as a feature bit that says VFs are not
> > > > initialized by default and must first be setup through an admin command.
> > > > This will likely need to be a feature bit because it's changing
> > > > behaviour outside of admin commands.
> > > >
> > > > Then, we can have:
> > > >
> > > >   ADMIN_SETUP VF#
> > > >   ADMIN_CLEANUP VF#
> > > >
> > > > I like this because this generalizes CREATE/DESTROY that SIOV guys 
> > > > proposed.
> > >
> > > Great!!
> > >
> > > >
> > > >
> > > > Why do we need an id as a level of indirection though? What is wrong
> > > > with just using VF# directly?
> > >
> > > I think VF# is ok. I also need to use it. But we need an ID for virtio 
> > > device
> > > not the transport(PF, VF).
> > >
> > > What I want to emphasize is that PCI(pf or vf) is a transport, it is only 
> > > used
> > > to connect the virtio driver and the virtio device. Right?
> > >
> > > The virtio device does not necessarily exist depending on PCI. For 
> > > example, a
> > > virtio device is migrated from another DPU, and it is not associated with 
> > > any
> > > PCI. What I have always wanted to say is that this device(not PCI) must 
> > > have its
> > > own ID, which has nothing to do with the transport.
> > >
> > > Now we want to use this migrated device and connect it to the 
> > > corresponding
> > > vm (migrated from the same host). We can passthrough vf to this vm. But 
> > > how do I
> > > tell our DPU to bind this migrated device with this vf?
> > > We can specify the VF by the VF#, but how can we specify the virtio 
> > > device?
> > > Maybe there are two migrated virtio device.
> > >
> > > Maybe we should compare it to the case of using PF directly before. The 
> > > biggest
> > > difference here is that PF is directly hot-plugged by the DPU, so when a
> > > user(custom) buys a virtio-net device, the DPU directly inserts a new PCI 
> > > device
> > > on the host. Now the vf is created by the user, and only the user knows 
> > > how to
> > > use each one. We cannot directly bind the device purchased or migrated by 
> > > the
> > > user to a certain vf. We need the user in the host to bind the 
> > > vf(transport) to
> > > a certain virtio device.
> > >
> > > Thanks.
> > >
> > >
> >
> > Maybe I didn't have enough coffee today but I can't figure out what all
> > this means :( For example what does "exist depending" mean?  What does
> > "device is migrated" mean?  What does it mean to be "migrated from the
> > same host"? What is "any PCI"? E.g. I know people did vm migration
> > moving virtio from a hardware device to a software
> > implementation. Is that "not associated with any PCI" ?
> > What is "user(custom)"? how is "the vf is created by the user"?
> > what does it mean to "bind the device .. to a certain vf"?
> > It looks like Parav understand what's going on though, so
> > maybe it's my fault.
> >
> > But fundamentally, the problem is that the spec patch doesn't do
> > anything useful by itself, it relies on some out of spec interface to
> > make these id values make sense. So the TC is reduced to guessing: we
> > could just trust you that it's useful somehow and at the same time
> > serves the purpose whatever it is.
> >
> > But it would be better if instead of trying to explain what that does in
> > vague terms, you post a spec patch that allows doing whatever needs
> > doing for these IDs to make sense through e.g. admin commands.
>
> I try to describe the requirement as a common requirement.
>
> @Parav do you think this is a requirement in your devices?
>
> The core point here is that our virtio-net device is created by customers on 
> the
> management platform. It is not a pure virtio-net device, but also includes 
> many
> other features, such as bandwidth, such as VPC information, etc. These are not
> within the scope of virtio, we cannot configure these information based on the
> admin queue after creating the vf. But we can bind the device created by the
> user to vf, which completes the binding of many manufacturer features to vf.

It sounds like your device is vDPA. Anyhow, it looks like you need
provision vf and the virtio via vendor specific way. Then we don't
need to bother it in the spec.

>
> In addition to this scenario, in the case of live mi

RE: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

Hi Xuan,

> From: Xuan Zhuo 
> Sent: Sunday, July 2, 2023 11:22 PM


> I try to describe the requirement as a common requirement.
> 
> @Parav do you think this is a requirement in your devices?
> 
> The core point here is that our virtio-net device is created by customers on 
> the
> management platform. It is not a pure virtio-net device, but also includes 
> many
> other features, such as bandwidth, such as VPC information, etc.
When these devices are on PCI transport, and if they are PF.
PCI VPD has well defined serial number, and few more fields.
This helps to cross identify the device between compute side where virtio 
device is used and other management side, who inserted this virtio PCI device 
in the compute.


>  These are not
> within the scope of virtio, we cannot configure these information based on the
> admin queue after creating the vf. But we can bind the device created by the
> user to vf, which completes the binding of many manufacturer features to vf.
> 
> In addition to this scenario, in the case of live migration, we can migrate a 
> vm
> from one host to another host. During this process, the virtio-net devce is 
> also
> migrated from the dpu of one host to the dpu of another host . At this time, 
> the
> purpose of creating a vf is to insert the migrated virtio-net into the host. 
> It is
> impossible for us to create a vf to get a brand new device. So also bind.
>
For the VF in this flow looks like below.

a. device id assignment is initiated by the compute side AQ to a VF (on 
destination)
b. management system received the command and queried some central place to get 
attributed (qos etc), that you described.

Here somehow the src side already had access to the management system in first 
place for the VF backend configuration.
And I believe same access is/should be possible on the destination side too.
So that same parameter can be configured as out of band command.

In fact for virtio device migration, a migrating sw may even need to inspect 
and figure out which is the best system to migrate to in the backend.

So out of band access is present and needed that can rely on the VF number to 
provision these things without explicit bind command.

So bind command is optional command that can help to do the work that is 
already done today out of band.

And if bind on specific system is identified, it is probably better to do this 
out of band.

Your key point is " virtio-net device is created by customers on the management 
platform".
If they are based on the VF, it can be identified uniquely by the VF number 
without explicit bind command.

For PF and VF I responded few other ideas before at [1], you probably missed it.

[1] 
https://lore.kernel.org/virtio-comment/20230628112107-mutt-send-email-...@kernel.org/T/#m99518f22871a223c4dd014bb77e712b3befd9793

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 5:16 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"  
wrote:


On 6/26/2023 4:09 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 3:08 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different 
role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
   and sets the ip, queue number, etc. The user obtains the identity of
   the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
   of the patch

Signed-off-by: Xuan Zhuo 
---
 admin.tex | 41 -
 1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
 \hline \hline
 0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands 
supported for this group type\\
 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands 
used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
 \hline
 0x8000 - 0x & - & Reserved for future commands (possibly using a 
different structure)\\
 \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
 \field{VF Enable} refer to registers within the SR-IOV Extended
 Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the 
virtio

If the VFs are create in a guest OS, I assume that means the user has
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security
issue(affects host pci).

No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.

OK, perhaps just say create VFs from a PF in the OS?

YES.



+devices don't know about these VFs. Because each VF may be assigned a 
different role
+by the user, the virtio device can not choose one VF to bind random.

I failed to understand this, once a VF is created, it has a personality,
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device
choose
one VF to bind?

On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a
guest on demands.

Yes, but we do not need to care about the guest. Because VF may only be used
in host, such as docker.

The problem is that the user (you can think of this as the orchestration
software) creates some VFs, these are only some PCI devices, which virtio
devices will work on these VFs. I think that creating a vf and creating a
virtio-net device are two different things. One is done by user in the OS, one
is done on the management platform.  So we need to bind them together.

If the VFs are created through sriov_numvfs, once created, the VF device and
its personality are determined.

PCI spec says:
All VFs associated with a PF must be the same device type as the PF,
(e.g., the same network
device type or the same storage device type.)

So how can the creating process be splitted into separated steps?

Are we discussing something beyond the spec?

NO.

The device types are same.

How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
these are managed by the management platform. On the cloud, there is an abstract
object in the backend, which contains things that are generally configured on
the management platform. It is something that users purchase.
Under the virtio standard it is similar to device.

In my understanding, we just created a pci vf, and virtio works on top of 

[virtio-dev] RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo 
> Sent: Monday, June 26, 2023 6:51 AM
> 
> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
>  wrote:
> >
> >
> > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
>  wrote:
> > >>
> > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
>  wrote:
> >  On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
>  wrote:
> > >> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > >>> The VFs of the SR-IOV are created by the user inside the guest
> > >>> OS, so the virtio devices don't know about these VFs. Because
> > >>> each VF may be assigned a different role by the user, the virtio 
> > >>> device
> can not choose one VF to bind random.
> > >>> So only the user knows how to bind the virtio devices to the VFs.
> > >>> On the other hand, generally the virtio devices are not
> > >>> created by the user inside the guest OS. This requires some
> management platform to participate.
> > >>>
> > >>> So the usage of this command:
> > >>> 1. The user purchases a virtio network card on the management
> platform,
> > >>>and sets the ip, queue number, etc. The user obtains the 
> > >>> identity
> of
> > >>>the network card.
> > >>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > >>> user binds the net crad to a VF with identity through the command
> > >>>of the patch
> > >>>
> > >>> Signed-off-by: Xuan Zhuo 
> > >>> ---
> > >>>  admin.tex | 41 -
> > >>>  1 file changed, 40 insertions(+), 1 deletion(-)
> > >>>
> > >>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > >>> 100644
> > >>> --- a/admin.tex
> > >>> +++ b/admin.tex
> > >>> @@ -115,7 +115,8 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > >>>  \hline \hline
> > >>>  0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> list of commands supported for this group type\\
> > >>>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> of commands used for this group type \\
> > >>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> virtio_admin_cmd}\\
> > >>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> one group member \\
> > >>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> virtio_admin_cmd}\\
> > >>>  \hline
> > >>>  0x8000 - 0x & - & Reserved for future commands (possibly
> using a different structure)\\
> > >>>  \hline
> > >>> @@ -429,6 +430,44 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > >>>  \field{VF Enable} refer to registers within the SR-IOV Extended
> > >>>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >>>
> > >>> +\subsubsection{Bind the device for member}
> > >>> +
> > >>> +The VFs of the SR-IOV are created by the user inside the
> > >>> +guest OS, so the virtio
> > >> If the VFs are create in a guest OS, I assume that means the
> > >> user has passthrough-ed the PF to the guest. For nested, I am
> > >> not sure whether this is a security issue(affects host pci).
> > > No care about the passthrough, we always created VFs by the PF.
> > >
> > > I should not say "inside the guest OS". I just want to say that
> > > the VF is create by the user in the OS. The devices does not know
> about it.
> >  OK, perhaps just say create VFs from a PF in the OS?
> > >>> YES.
> > >>>
> > >>>
> > >>> +devices don't know about these VFs. Because each VF may be
> > >>> +assigned a different role by the user, the virtio device can not
> choose one VF to bind random.
> > >> I failed to understand this, once a VF is created, it has a
> > >> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > >> and PF knows that.
> > >>
> > >> I am not familiar with the background, What do you mean by
> > >> virtio device choose one VF to bind?
> > > On the cloud, the nic is created by the management platform, the
> > > user can not create a new nic inside the OS.
> > >
> > > So after echo sriov_numvfs, the user just got some VFs, there is
> > > not backend virtio-net devices.
> >  I think it is not a "user" mange the VFs, the VFs usually
> >  provisioned by the orchestration software and it assign properly
> >  selected a VF to a guest on demands.
> > >>> Yes, but we do not need to care about the guest. Because VF may
> > >>> only be used in host, such as docker.
> > >>>
> > >>> The problem is that the user (you can think of this as the
> > >>> orchestration
> > >>> software) creates some VFs, these are only some PCI devices, which
> > >>> virtio devices wil

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Mon, Jun 26, 2023 at 12:19:04PM +, Parav Pandit wrote:
> 
> 
> > From: Xuan Zhuo 
> > Sent: Monday, June 26, 2023 6:51 AM
> > 
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> >  wrote:
> > >
> > >
> > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> >  wrote:
> > > >>
> > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> >  wrote:
> > >  On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> >  wrote:
> > > >> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > >>> The VFs of the SR-IOV are created by the user inside the guest
> > > >>> OS, so the virtio devices don't know about these VFs. Because
> > > >>> each VF may be assigned a different role by the user, the virtio 
> > > >>> device
> > can not choose one VF to bind random.
> > > >>> So only the user knows how to bind the virtio devices to the VFs.
> > > >>> On the other hand, generally the virtio devices are not
> > > >>> created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>
> > > >>> So the usage of this command:
> > > >>> 1. The user purchases a virtio network card on the management
> > platform,
> > > >>>and sets the ip, queue number, etc. The user obtains the 
> > > >>> identity
> > of
> > > >>>the network card.
> > > >>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > >>> user binds the net crad to a VF with identity through the command
> > > >>>of the patch
> > > >>>
> > > >>> Signed-off-by: Xuan Zhuo 
> > > >>> ---
> > > >>>  admin.tex | 41 -
> > > >>>  1 file changed, 40 insertions(+), 1 deletion(-)
> > > >>>
> > > >>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > >>> 100644
> > > >>> --- a/admin.tex
> > > >>> +++ b/admin.tex
> > > >>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>  \hline \hline
> > > >>>  0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > list of commands supported for this group type\\
> > > >>>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > of commands used for this group type \\
> > > >>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}\\
> > > >>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > one group member \\
> > > >>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}\\
> > > >>>  \hline
> > > >>>  0x8000 - 0x & - & Reserved for future commands (possibly
> > using a different structure)\\
> > > >>>  \hline
> > > >>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>  \field{VF Enable} refer to registers within the SR-IOV 
> > > >>> Extended
> > > >>>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >>>
> > > >>> +\subsubsection{Bind the device for member}
> > > >>> +
> > > >>> +The VFs of the SR-IOV are created by the user inside the
> > > >>> +guest OS, so the virtio
> > > >> If the VFs are create in a guest OS, I assume that means the
> > > >> user has passthrough-ed the PF to the guest. For nested, I am
> > > >> not sure whether this is a security issue(affects host pci).
> > > > No care about the passthrough, we always created VFs by the PF.
> > > >
> > > > I should not say "inside the guest OS". I just want to say that
> > > > the VF is create by the user in the OS. The devices does not know
> > about it.
> > >  OK, perhaps just say create VFs from a PF in the OS?
> > > >>> YES.
> > > >>>
> > > >>>
> > > >>> +devices don't know about these VFs. Because each VF may be
> > > >>> +assigned a different role by the user, the virtio device can not
> > choose one VF to bind random.
> > > >> I failed to understand this, once a VF is created, it has a
> > > >> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > >> and PF knows that.
> > > >>
> > > >> I am not familiar with the background, What do you mean by
> > > >> virtio device choose one VF to bind?
> > > > On the cloud, the nic is created by the management platform, the
> > > > user can not create a new nic inside the OS.
> > > >
> > > > So after echo sriov_numvfs, the user just got some VFs, there is
> > > > not backend virtio-net devices.
> > >  I think it is not a "user" mange the VFs, the VFs usually
> > >  provisioned by the orchestration software and it assign properly
> > >  selected a VF to a guest on demands.
> > > >>> Yes, but we do not n

[virtio-dev] RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Michael S. Tsirkin 
> Sent: Monday, June 26, 2023 8:35 AM


> > Yet to catch up on the thread, so likely I am missing something.
> >
> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs 
> > in
> the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside
> > the device
> >
> > 3. A user on the host may bind these VFs to the VF driver
> > (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of
> the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
> 
> To be frank, I am not sure binding to an ID necessarily needs to be gated on
> provisioning commands.
> What was not explained at all is what purpose does this extra level of
> indirection serve.

I didn't suggest gating binding on provisioning.
As written above _may_configure_.
So provisioning is optional.

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 6:50 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"  
wrote:


On 6/26/2023 5:16 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 4:09 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 3:08 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different 
role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
and sets the ip, queue number, etc. The user obtains the identity of
the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
of the patch

Signed-off-by: Xuan Zhuo 
---
  admin.tex | 41 -
  1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
  \hline \hline
  0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of 
commands supported for this group type\\
  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands 
used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
  \hline
  0x8000 - 0x & - & Reserved for future commands (possibly using a 
different structure)\\
  \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
  \field{VF Enable} refer to registers within the SR-IOV Extended
  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the 
virtio

If the VFs are create in a guest OS, I assume that means the user has
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security
issue(affects host pci).

No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.

OK, perhaps just say create VFs from a PF in the OS?

YES.



+devices don't know about these VFs. Because each VF may be assigned a 
different role
+by the user, the virtio device can not choose one VF to bind random.

I failed to understand this, once a VF is created, it has a personality,
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device
choose
one VF to bind?

On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a
guest on demands.

Yes, but we do not need to care about the guest. Because VF may only be used
in host, such as docker.

The problem is that the user (you can think of this as the orchestration
software) creates some VFs, these are only some PCI devices, which virtio
devices will work on these VFs. I think that creating a vf and creating a
virtio-net device are two different things. One is done by user in the OS, one
is done on the management platform.  So we need to bind them together.

If the VFs are created through sriov_numvfs, once created, the VF device and
its personality are determined.

PCI spec says:
All VFs associated with a PF must be the same device type as the PF,
(e.g., the same network
device type or the same storage device type.)

So how can the creating process be splitted into separated steps?

Are we discussing something beyond the spec?

NO.

The device types are same.

How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
these are managed by the management platform. On the cloud, there is an abstract
object in the backend, which contains things that are generally configured on
the management platform. It is something that users purchase.
Under the v

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/27/2023 4:14 PM, Xuan Zhuo wrote:

On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan"  
wrote:


On 6/26/2023 6:50 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 5:16 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 4:09 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 3:08 PM, Xuan Zhuo wrote:

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"  
wrote:

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different 
role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
 and sets the ip, queue number, etc. The user obtains the identity of
 the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
 of the patch

Signed-off-by: Xuan Zhuo 
---
   admin.tex | 41 -
   1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
   \hline \hline
   0x & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of 
commands supported for this group type\\
   0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands 
used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}\\
   \hline
   0x8000 - 0x & - & Reserved for future commands (possibly using a 
different structure)\\
   \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic 
Facilities of a Virti
   \field{VF Enable} refer to registers within the SR-IOV Extended
   Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the 
virtio

If the VFs are create in a guest OS, I assume that means the user has
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security
issue(affects host pci).

No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.

OK, perhaps just say create VFs from a PF in the OS?

YES.



+devices don't know about these VFs. Because each VF may be assigned a 
different role
+by the user, the virtio device can not choose one VF to bind random.

I failed to understand this, once a VF is created, it has a personality,
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device
choose
one VF to bind?

On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a
guest on demands.

Yes, but we do not need to care about the guest. Because VF may only be used
in host, such as docker.

The problem is that the user (you can think of this as the orchestration
software) creates some VFs, these are only some PCI devices, which virtio
devices will work on these VFs. I think that creating a vf and creating a
virtio-net device are two different things. One is done by user in the OS, one
is done on the management platform.  So we need to bind them together.

If the VFs are created through sriov_numvfs, once created, the VF device and
its personality are determined.

PCI spec says:
All VFs associated with a PF must be the same device type as the PF,
(e.g., the same network
device type or the same storage device type.)

So how can the creating process be splitted into separated steps?

Are we discussing something beyond the spec?

NO.

The device types are same.

How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
these are managed by the management platform. On the cloud, there is an abstract
object in the backend, which contain

[virtio-dev] RE: RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo 
> Sent: Monday, June 26, 2023 8:33 AM

[..]

> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs 
> > in
> the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside
> > the device
> >
> > 3. A user on the host may bind these VFs to the VF driver
> > (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of
> the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
> 
> 
> YES
> 
> My problem is that the nic is created by user on the management platform.  Or
> the device is migrated from other host.
>
Do you mean the management platform != the owner PF device?

 
> So we need to bind the device(created by user) to the vf between step #2 and
> #3.
>
We likely have some terminology issues.
Bind the device to the vf?

At OS level, there are
a. virtio PCI PF owner device
b. virtio PCI VF member device

c. some OS level device (such as net/blk, vfio, vdpa created by attaching #b to 
the driver)

d. An optional representor device on the DPU side (defined and managed by the 
vendor, outside the virtio spec as device implemtion).
Is this #d you call management platform?
If so, yes, orchestration may need to consult management platform, but this is 
optional.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org