[virtio-dev] Re: [virtio-comment] [PATCH v10] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

在 2023/3/7 上午6:57, Michael S. Tsirkin 写道:

On Thu, Mar 02, 2023 at 11:36:18AM +, David Edmondson wrote:

+for an enabled transmit/receive 
virtqueue whose number is \field{vqn}.

Should this now be "whose index is \field{vqn}"?

Ugh.  I guess we'll have to fix the number/index mess in the spec
first. Parav you said you are looking into it?




# Where virtqueue number and virtqueue index are used.
  1. In the Virtqueue Configuration Section, use the virtqueue index: 
"Write the **virtqueue index** (first queue is 0) to queue_select."

  2. Both descriptions are used separately in the Notification Section.
  2.1 Here vqn is called virtqueue index:
    "When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, 
the driver sends an available buffer notification
  to the device by writing the **16-bit virtqueue index** 
of this virtqueue to the Queue Notify address.

  ...

  le32 {
 vqn: 16;
 next_off : 15;
 next_wrap: 1;
 };
 ...

 If VIRTIO_F_NOTIFICATION_DATA has not been negotiated, the 
driver MUST use the queue_notify_data value instead of the **virtqueue 
index**."


  2.2 Here vqn is called virtqueue number:
    "When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, 
the Notification data contains the **Virtqueue number**.

    ...

    be32 {
    vqn: 16;
    next_off : 15;
    next_wrap: 1;
 };
    ...

    When VIRTIO_F_NOTIFICATION_DATA has not been negotiated, 
this notification involves sending the **virtqueue number** to the 
device (method depending on the transport).

    ...

    vqn -- **VQ number** to be notified."

# 0-based index and 0-based number are used respectively in the RSS Section:
1. "Field unclassified_queue contains the **0-based index** of the 
receive virtqueue to place unclassified packets in. Index 0 corresponds 
to receiveq1."
2. "use the result as the index in the indirection table to get 
**0-based number** of destination receiveq (value of 0 corresponds to 
receiveq1)."




\field{vqn} has been called '0-based virtqueue index' or '0-based 
virtqueue number',

I think both seem to be friendly to readers, so what are your options?

Thanks.






-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: [virtio-dev] RE: [PATCH v10] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

在 2023/3/9 上午6:30, Parav Pandit 写道:

From: virtio-dev@lists.oasis-open.org  On
Behalf Of Heng Qi
Sent: Thursday, March 2, 2023 10:27 PM


I remember we discussed that instead of mentioning each individual field,

better to describe the whole structure being read-only or write-only.

Consider the following scenarios:
1. A read-only field of the structure virtio_net_ctrl_coal is extended for
CTRL_NOTF_COAL_RX/TX_SET to get some extra information

A set command cannot extend the struct virtio_net_ctrl_coal, particularly for 
read-only and partially for write-only.
This would mean that for the tiny number of bytes, an additional dma descriptor 
is to be allocated with read/write-only permissions.
It would be inefficient for the driver to do so for the SET command to have vqn 
as write-only, reserved as read-only, rest fields as write-only dma attributes.

As I think more, I think the whole set command fields should be read-only for 
device. Better to describe it this way instead of splitting individual fields.
This way driver can just do a single DMA allocation with read-only attributes 
for set cmd.

Get command doesn’t have any choice today the way CVQ is structured to it lives 
with the limitation.


I think it is reasonable and will be revised in the next version.




Looks good, however you have well covered in the device normative

statements.

So possibly it can be removed from here.

I tend to keep this, as we have done in the past, we can have normative
descriptions and the corresponding non-normative descriptions.


Ok. but please revisit if the description can be simpler text than the 
normative lines.


Ok.

Thanks.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/8 下午10:39, Michael S. Tsirkin 写道:

On Wed, Mar 01, 2023 at 10:56:31AM +0800, Heng Qi wrote:


在 2023/2/28 下午7:16, Michael S. Tsirkin 写道:

On Sat, Feb 18, 2023 at 10:37:15PM +0800, Heng Qi wrote:

If the tunnel is used to encapsulate the packets, the hash calculated
using the outer header of the receive packets is always fixed for the
same flow packets, i.e. they will be steered to the same receive queue.

Wait a second. How is this true? Does not everyone stick the
inner header hash in the outer source port to solve this?

Yes, you are right. That's what we did before the inner header hash, but it
has a performance penalty, which I'll explain below.


For example geneve spec says:

 it is necessary for entropy from encapsulated packets to be
 exposed in the tunnel header.  The most common technique for this is
 to use the UDP source port

The end point of the tunnel called the gateway (with DPDK on top of it).

1. When there is no inner header hash, entropy can be inserted into the udp
src port of the outer header of the tunnel,
and then the tunnel packet is handed over to the host. The host needs to
take out a part of the CPUs to parse the outer headers (but not drop them)
to calculate the inner hash for the inner payloads,
and then use the inner
hash to forward them to another part of the CPUs that are responsible for
processing.

I don't get this part. Leave inner hashes to the guest inside the
tunnel, why is your host doing this?


Assuming that the same flow includes a unidirectional flow a->b, or a 
bidirectional flow a->b and b->a,

such flow may be out of order when processed by the gateway(DPDK):

1. In unidirectional mode, if the same flow is switched to another 
gateway for some reason, resulting in different outer IP address,
    then this flow may be processed by different CPUs after reaching 
the host if there is no inner hash. So after the host receives the
    flow, first use the forwarding CPUs to parse the inner hash, and 
then use the hash to ensure that the flow is processed by the

    same CPU.
2. In bidirectional mode, a->b flow may go to gateway 1, and b->a flow 
may go to gateway 2. In order to ensure that the same flow is
    processed by the same CPU, we still need the forwarding CPUs to 
parse the real inner hash(here, the hash key needs to be replaced with a 
symmetric hash key).





1). During this process, the CPUs on the host is divided into two parts, one
part is used as a forwarding node to parse the outer header,
  and the CPU utilization is low. Another part handles packets.

Some overhead is clearly involved in *sending* packets -
to calculate the hash and stick it in the port number.
This is, however, a separate problem and if you want to
solve it then my suggestion would be to teach the *transmit*
side about GRE offloads, so it can fill the source port in the card.


2). The entropy of the source udp src port is not enough, that is, the queue
is not widely distributed.

how isn't it enough? 16 bit is enough to cover all vqs ...


A 5-tuple brings more entropy than a single port, doesn't it? In fact, 
the inner hash of the physical network card used by
the business team is indeed better than the udp port number of the outer 
header we modify now, but they did not give me the data.



2. When there is an inner header hash, the gateway will directly help parse
the outer header, and use the inner 5 tuples to calculate the inner hash.
The tunneled packet is then handed over to the host.
1) All the CPUs of the host are used to process data packets, and there is
no need to use some CPUs to forward and parse the outer header.

You really have to parse the outer header anyway,
otherwise there's no tunneling.
Unless you want to teach virtio to implement tunneling
in hardware, which is something I'd find it easier to
get behind.


There is no need to parse the outer header twice, because we use shared 
memory.



2) The entropy of the original quintuple is sufficient, and the queue is
widely distributed.

It's exactly the same entropy, why would it be better? In fact you
are taking out the outer hash entropy making things worse.


I don't get the point, why the entropy of the inner 5-tuple and the 
outer tunnel header is the same,

multiple streams have the same outer header.

Thanks.



Thanks.

same goes for vxlan did not check further.

so what is the problem?  and which tunnel types actually suffer from the
problem?



This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscr...@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
List help: virtio-comment-h...@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/

Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/2/28 下午7:16, Michael S. Tsirkin 写道:

On Sat, Feb 18, 2023 at 10:37:15PM +0800, Heng Qi wrote:

If the tunnel is used to encapsulate the packets, the hash calculated
using the outer header of the receive packets is always fixed for the
same flow packets, i.e. they will be steered to the same receive queue.

Wait a second. How is this true? Does not everyone stick the
inner header hash in the outer source port to solve this?
For example geneve spec says:

it is necessary for entropy from encapsulated packets to be
exposed in the tunnel header.  The most common technique for this is
to use the UDP source port

same goes for vxlan did not check further.

so what is the problem?  and which tunnel types actually suffer from the
problem?



Inner hash can at least hash tunnel flows without outer transport 
headers like GRE to multiple queues,

which is beneficial to us.

For tunnel flows with outer transport headers like VXLAN, although they 
can hash flows to different queues
by setting different outer udp port, this does not conflict with inner 
hash. Inner hashing can also be used for this purpose.


For the same flow, packets in the receiving and sending directions may 
pass through different tunnels respectively, which cause
the same flow to be hashed to different queues. In this case, we have to 
calculate a symmetric hash (can be called an inner symmetric hash, which 
is a type of inner hash.)
through the inner header, so that the same flow can be hashed to the 
same queue.


Symmetric hashing can ignore the order of the 5-tuples to calculate the 
hash, that is, the hash values ​​calculated by (a1, a2, a3, a4) and (a2, 
a1, a4, a3) respectively are the same.


Thanks.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/10 上午3:36, Michael S. Tsirkin 写道:

On Thu, Mar 09, 2023 at 12:55:02PM +0800, Heng Qi wrote:


在 2023/3/8 下午10:39, Michael S. Tsirkin 写道:

On Wed, Mar 01, 2023 at 10:56:31AM +0800, Heng Qi wrote:

在 2023/2/28 下午7:16, Michael S. Tsirkin 写道:

On Sat, Feb 18, 2023 at 10:37:15PM +0800, Heng Qi wrote:

If the tunnel is used to encapsulate the packets, the hash calculated
using the outer header of the receive packets is always fixed for the
same flow packets, i.e. they will be steered to the same receive queue.

Wait a second. How is this true? Does not everyone stick the
inner header hash in the outer source port to solve this?

Yes, you are right. That's what we did before the inner header hash, but it
has a performance penalty, which I'll explain below.


For example geneve spec says:

  it is necessary for entropy from encapsulated packets to be
  exposed in the tunnel header.  The most common technique for this is
  to use the UDP source port

The end point of the tunnel called the gateway (with DPDK on top of it).

1. When there is no inner header hash, entropy can be inserted into the udp
src port of the outer header of the tunnel,
and then the tunnel packet is handed over to the host. The host needs to
take out a part of the CPUs to parse the outer headers (but not drop them)
to calculate the inner hash for the inner payloads,
and then use the inner
hash to forward them to another part of the CPUs that are responsible for
processing.

I don't get this part. Leave inner hashes to the guest inside the
tunnel, why is your host doing this?



Let's simplify some details and take a fresh look at two different 
scenarios: VXLAN and GENEVE (Scenario1) and GRE (Scenario2).


1. In Scenario1, we can improve the processing performance of the same 
flow by implementing inner symmetric hashing.


This is because even though client1 and client2 communicate 
bidirectionally through the same flow, their data may pass


through and be encapsulated by different tunnels, resulting in the same 
flow being hashed to different queues and processed by different CPUs.


To ensure consistency and optimized processing, we need to parse out the 
inner header and compute a symmetric hash on it using a special rss key.


Sorry for not mentioning the inner symmetric hash before, in order to 
prevent the introduction of more concepts, but it is indeed a kind of 
inner hash.


2. In Scenario2 with GRE, the lack of outer transport headers means that 
flows between multiple communication pairs encapsulated by the same tunnel


will all be hashed to the same queue. To address this, we need to 
implement inner hashing to improve the performance of RSS. By parsing 
and calculating


the inner hash, different flows can be hashed to different queues.

Thanks.




Assuming that the same flow includes a unidirectional flow a->b, or a
bidirectional flow a->b and b->a,
such flow may be out of order when processed by the gateway(DPDK):

1. In unidirectional mode, if the same flow is switched to another gateway
for some reason, resulting in different outer IP address,
     then this flow may be processed by different CPUs after reaching the
host if there is no inner hash. So after the host receives the
     flow, first use the forwarding CPUs to parse the inner hash, and then
use the hash to ensure that the flow is processed by the
     same CPU.
2. In bidirectional mode, a->b flow may go to gateway 1, and b->a flow may
go to gateway 2. In order to ensure that the same flow is
     processed by the same CPU, we still need the forwarding CPUs to parse
the real inner hash(here, the hash key needs to be replaced with a symmetric
hash key).

Oh intersting. What are those gateways, how come there's expectation
that you can change their addresses and topology
completely seamlessly without any reordering whatsoever?
Isn't network topology change kind of guaranteed to change ordering
sometimes?



1). During this process, the CPUs on the host is divided into two parts, one
part is used as a forwarding node to parse the outer header,
   and the CPU utilization is low. Another part handles packets.

Some overhead is clearly involved in *sending* packets -
to calculate the hash and stick it in the port number.
This is, however, a separate problem and if you want to
solve it then my suggestion would be to teach the *transmit*
side about GRE offloads, so it can fill the source port in the card.


2). The entropy of the source udp src port is not enough, that is, the queue
is not widely distributed.

how isn't it enough? 16 bit is enough to cover all vqs ...

A 5-tuple brings more entropy than a single port, doesn't it?

But you don't need more for RSS, the indirection table is not
that large.


In fact, the
inner hash of the physical network card used by
the business team is indeed better than the udp port number of the outer
header we modify now, but they did not give m

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/15 下午7:58, Michael S. Tsirkin 写道:

On Sat, Mar 11, 2023 at 11:23:08AM +0800, Heng Qi wrote:



在 2023/3/10 上午3:36, Michael S. Tsirkin 写道:

On Thu, Mar 09, 2023 at 12:55:02PM +0800, Heng Qi wrote:

在 2023/3/8 下午10:39, Michael S. Tsirkin 写道:

On Wed, Mar 01, 2023 at 10:56:31AM +0800, Heng Qi wrote:

在 2023/2/28 下午7:16, Michael S. Tsirkin 写道:

On Sat, Feb 18, 2023 at 10:37:15PM +0800, Heng Qi wrote:

If the tunnel is used to encapsulate the packets, the hash calculated
using the outer header of the receive packets is always fixed for the
same flow packets, i.e. they will be steered to the same receive queue.

Wait a second. How is this true? Does not everyone stick the
inner header hash in the outer source port to solve this?

Yes, you are right. That's what we did before the inner header hash, but it
has a performance penalty, which I'll explain below.


For example geneve spec says:

   it is necessary for entropy from encapsulated packets to be
   exposed in the tunnel header.  The most common technique for this is
   to use the UDP source port

The end point of the tunnel called the gateway (with DPDK on top of it).

1. When there is no inner header hash, entropy can be inserted into the udp
src port of the outer header of the tunnel,
and then the tunnel packet is handed over to the host. The host needs to
take out a part of the CPUs to parse the outer headers (but not drop them)
to calculate the inner hash for the inner payloads,
and then use the inner
hash to forward them to another part of the CPUs that are responsible for
processing.

I don't get this part. Leave inner hashes to the guest inside the
tunnel, why is your host doing this?


Let's simplify some details and take a fresh look at two different
scenarios: VXLAN and GENEVE (Scenario1) and GRE (Scenario2).

1. In Scenario1, we can improve the processing performance of the same flow
by implementing inner symmetric hashing.

This is because even though client1 and client2 communicate bidirectionally
through the same flow, their data may pass

through and be encapsulated by different tunnels, resulting in the same flow
being hashed to different queues and processed by different CPUs.

To ensure consistency and optimized processing, we need to parse out the
inner header and compute a symmetric hash on it using a special rss key.

Sorry for not mentioning the inner symmetric hash before, in order to
prevent the introduction of more concepts, but it is indeed a kind of inner
hash.

If parts of a flow go through different tunnels won't this cause
reordering at the network level? Why is it so important to prevent it at
the nic then?  Or, since you are stressing symmetric hash, are you
talking about TX and RX side going through different tunnels?


Yes, the directions client1->client2 and client2->client1 may go through 
different tunnels.
Using inner symmetric hashing can satisfy the same CPU to process two 
directions of the same flow to improve performance.






2. In Scenario2 with GRE, the lack of outer transport headers means that
flows between multiple communication pairs encapsulated by the same tunnel

will all be hashed to the same queue. To address this, we need to implement
inner hashing to improve the performance of RSS. By parsing and calculating

the inner hash, different flows can be hashed to different queues.

Thanks.



Well 2 is at least inexact, there's flowID there. It's just 8 bit


We use the most basic GRE header fields (not NVGRE), not even optional 
fields.
There is also no flow id in the GRE header, should you be referring to 
NVGRE?


Thanks.


so not sufficient if there are more than 512 queues. Still 512 queues
is quite a lot. Are you trying to solve for configurations with
more than 512 queues then?



Assuming that the same flow includes a unidirectional flow a->b, or a
bidirectional flow a->b and b->a,
such flow may be out of order when processed by the gateway(DPDK):

1. In unidirectional mode, if the same flow is switched to another gateway
for some reason, resulting in different outer IP address,
      then this flow may be processed by different CPUs after reaching the
host if there is no inner hash. So after the host receives the
      flow, first use the forwarding CPUs to parse the inner hash, and then
use the hash to ensure that the flow is processed by the
      same CPU.
2. In bidirectional mode, a->b flow may go to gateway 1, and b->a flow may
go to gateway 2. In order to ensure that the same flow is
      processed by the same CPU, we still need the forwarding CPUs to parse
the real inner hash(here, the hash key needs to be replaced with a symmetric
hash key).

Oh intersting. What are those gateways, how come there's expectation
that you can change their addresses and topology
completely seamlessly without any reordering whatsoever?
Isn't network topology change kind of guaranteed to change ordering
sometimes?

[virtio-dev] Re: [PATCH v10] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/15 上午11:23, Parav Pandit 写道:



On 3/6/2023 10:48 AM, Heng Qi wrote:


  +\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash

May be to say inner packet header hash..
This make its little more clear about "which header" that you 
explained in the commit log.




Sure, I'll add this.


+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications 
coalescing.
    \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 
packets.
@@ -139,6 +142,7 @@ \subsubsection{Feature bit 
requirements}\label{sec:Device Types / Network Device

  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.

  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.
I think this should also say that HASH_TUNNEL requires either of the 
F_RSS or F_HASH_REPORT.

Because without it HASH_TUNNEL is not useful.


F_HASH_TUNNEL indicates that the hash should be calculated using the 
inner packet header, even without F_RSS or F_HASH_REPORT,
we can continue to use the hash value in scenarios such as RPS or ebpf 
programs.



Right?
if no, than my below comments are meaningless.


I think it's fine to let F_HASH_TUNNEL rely on F_RSS or _F_HASH_REPORT 
as those are probably important scenarios where inner packet header hash 
is used.






  \end{description}
    \subsubsection{Legacy Interface: Feature bits}\label{sec:Device 
Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -198,20 +202,27 @@ \subsection{Device configuration 
layout}\label{sec:Device Types / Network Device

  u8 rss_max_key_size;
  le16 rss_max_indirection_table_length;
  le32 supported_hash_types;
+    le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
-The following field, \field{rss_max_key_size} only exists if 
VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+The following field, \field{rss_max_key_size} only exists if 
VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or 
VIRTIO_NET_F_HASH_TUNNEL is set.

  It specifies the maximum supported length of RSS key in bytes.

I think rss_max_key_size field dependency should be only of the 
existing feature bits F_RSS and F_HASH_REPORT.
This is because those are the bits really deciding to consider 
rss_max_key_size.


  The following field, \field{rss_max_indirection_table_length} only 
exists if VIRTIO_NET_F_RSS is set.
  It specifies the maximum number of 16-bit entries in RSS 
indirection table.
    The next field, \field{supported_hash_types} only exists if the 
device supports hash calculation,

-i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+i.e. if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or 
VIRTIO_NET_F_HASH_TUNNEL is set.



Same as above.

  Field \field{supported_hash_types} contains the bitmask of 
supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / 
Processing of Incoming Packets / Hash calculation for incoming 
packets / Supported/enabled hash types} for details of supported hash 
types.
  +The next field, \field{supported_tunnel_hash_types} only exists if 
the device
+supports inner hash calculation, i.e. if VIRTIO_NET_F_HASH_TUNNEL is 
set.

+
Above line "the next field .." can be just same as "Device supports 
inner packet header hash calculation, i.e..."
This is because here, the term "header" is missed, which is present in 
the definition of feature bit 52.


I'll rephrase the description to make the whole text more consistent.



    The device MUST set \field{rss_max_key_size} to at least 40, if 
it offers

-VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
+VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL.
This needs change if the above first comment about rss_max_key_size is 
right.


  The device MUST set \field{rss_max_indirection_table_length} to at 
least 128, if it offers

  VIRTIO_NET_F_RSS.
@@ -843,11 +854,13 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network

  \begin{itemize}
  \item The feature VIRTIO_NET_F_RSS was negotiated. The device uses 
the hash to determine the receive virtqueue to place incoming packets.
  \item The feature VIRTIO_NET_F_HASH_REPORT was negotiated. The 
device reports the hash value and the hash type with the packet.
+\item The feature VIRTIO_NET_F_HASH_TUNNEL was negotiated. The 
device supports inner hash calculation.

  \end{itemize}


inner packet header hash ..


Ok.



+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated, the 
encapsulation
+hash type below indicates that the hash is calculated over the inner 
header of

+the encapsulated packet:
+Hash type applicable for inner payload of the gre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE (1 << 1)
+\end{lst

[virtio-dev] Re: [virtio-comment] Re: [PATCH v10] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/15 下午8:10, Michael S. Tsirkin 写道:

On Tue, Mar 14, 2023 at 11:23:55PM -0400, Parav Pandit wrote:

If not, for now it may be better to skip vxlan and nvegre as they inherently
have unique outer header UDP src port based on the inner header.

So what's left, GRE?  GRE is actually different, in that it's not IP at
all.


I do not think so, I mentioned that VXLAN and GENEVE need inner 
symmetric hashing, and we need this.


And we know inner hashing doesn't conflict with other ways of adding 
entropy.


Thanks.


So if we are talking about GRE, hash is indeed not calculated at all at
the moment, right?  And I would say a natural first step for GRE is
actually adding a hash type that will support this protocol.

How about doing that? It seems like this should be a small step
and completely uncontroversial.





-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

On Wed, Mar 15, 2023 at 10:57:40AM -0400, Michael S. Tsirkin wrote:
> On Wed, Mar 15, 2023 at 08:55:45PM +0800, Heng Qi wrote:
> > 
> > 
> > 在 2023/3/15 下午7:58, Michael S. Tsirkin 写道:
> > > On Sat, Mar 11, 2023 at 11:23:08AM +0800, Heng Qi wrote:
> > > > 
> > > > 
> > > > 在 2023/3/10 上午3:36, Michael S. Tsirkin 写道:
> > > > > On Thu, Mar 09, 2023 at 12:55:02PM +0800, Heng Qi wrote:
> > > > > > 在 2023/3/8 下午10:39, Michael S. Tsirkin 写道:
> > > > > > > On Wed, Mar 01, 2023 at 10:56:31AM +0800, Heng Qi wrote:
> > > > > > > > 在 2023/2/28 下午7:16, Michael S. Tsirkin 写道:
> > > > > > > > > On Sat, Feb 18, 2023 at 10:37:15PM +0800, Heng Qi wrote:
> > > > > > > > > > If the tunnel is used to encapsulate the packets, the hash 
> > > > > > > > > > calculated
> > > > > > > > > > using the outer header of the receive packets is always 
> > > > > > > > > > fixed for the
> > > > > > > > > > same flow packets, i.e. they will be steered to the same 
> > > > > > > > > > receive queue.
> > > > > > > > > Wait a second. How is this true? Does not everyone stick the
> > > > > > > > > inner header hash in the outer source port to solve this?
> > > > > > > > Yes, you are right. That's what we did before the inner header 
> > > > > > > > hash, but it
> > > > > > > > has a performance penalty, which I'll explain below.
> > > > > > > > 
> > > > > > > > > For example geneve spec says:
> > > > > > > > > 
> > > > > > > > >it is necessary for entropy from encapsulated packets 
> > > > > > > > > to be
> > > > > > > > >exposed in the tunnel header.  The most common 
> > > > > > > > > technique for this is
> > > > > > > > >to use the UDP source port
> > > > > > > > The end point of the tunnel called the gateway (with DPDK on 
> > > > > > > > top of it).
> > > > > > > > 
> > > > > > > > 1. When there is no inner header hash, entropy can be inserted 
> > > > > > > > into the udp
> > > > > > > > src port of the outer header of the tunnel,
> > > > > > > > and then the tunnel packet is handed over to the host. The host 
> > > > > > > > needs to
> > > > > > > > take out a part of the CPUs to parse the outer headers (but not 
> > > > > > > > drop them)
> > > > > > > > to calculate the inner hash for the inner payloads,
> > > > > > > > and then use the inner
> > > > > > > > hash to forward them to another part of the CPUs that are 
> > > > > > > > responsible for
> > > > > > > > processing.
> > > > > > > I don't get this part. Leave inner hashes to the guest inside the
> > > > > > > tunnel, why is your host doing this?
> > > > 
> > > > Let's simplify some details and take a fresh look at two different
> > > > scenarios: VXLAN and GENEVE (Scenario1) and GRE (Scenario2).
> > > > 
> > > > 1. In Scenario1, we can improve the processing performance of the same 
> > > > flow
> > > > by implementing inner symmetric hashing.
> > > > 
> > > > This is because even though client1 and client2 communicate 
> > > > bidirectionally
> > > > through the same flow, their data may pass
> > > > 
> > > > through and be encapsulated by different tunnels, resulting in the same 
> > > > flow
> > > > being hashed to different queues and processed by different CPUs.
> > > > 
> > > > To ensure consistency and optimized processing, we need to parse out the
> > > > inner header and compute a symmetric hash on it using a special rss key.
> > > > 
> > > > Sorry for not mentioning the inner symmetric hash before, in order to
> > > > prevent the introduction of more concepts, but it is indeed a kind of 
> > > > inner
> > > > hash.
> > > If parts of a flow go through 

[virtio-dev] Re: [PATCH v10] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/15 下午11:09, Michael S. Tsirkin 写道:

On Wed, Mar 15, 2023 at 09:19:43PM +0800, Heng Qi wrote:

Any encapsulation technology that includes UDP/L4 header likely do not
prefer based on the inner header. This is because the outer header src
port entropy is added based on the inner header.

I was not able to follow the discussion in v9 that you had with Michael.
Did you conclude if this is needed for vxlan too?

If not, for now it may be better to skip vxlan and nvegre as they
inherently have unique outer header UDP src port based on the inner
header.

Symmetric hashing ignores the order of the five-tuples when calculating the
hash, that is, using (a1,a2,p1,p2) and (a2,a1,p2,p1) respectively can
calculate the same hash.
There is a scenario that the two directions client1->client2 and
client2->client1 of the same flow may pass through different tunnels.
In order to allow the data in two directions to be processed by the same
CPU, we need to calculate a symmetric hash based on the inner packet header.
Sorry I didn't mention this earlier just to avoid introducing the concept of
symmetric hashing.

But the hash is already there in the port. Is it then maybe just
the question of ignoring the IP addresses when hashing?


We do not ignore the IP address, because after the tunnel sends the 
packets to the processing host,
the processing host will parse the outer headers, and then use the inner 
symmetric hash to hand over the
packets of the same flow to the same cpu for processing (for the network 
topology, please check my latest reply thread).


Thanks.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v10] virtio-net: support inner header hash

[Heng Qi]

On Wed, Mar 15, 2023 at 07:06:53PM -0400, Parav Pandit wrote:
> 
> 
> On 3/15/2023 9:19 AM, Heng Qi wrote:
> >
> >
> >在 2023/3/15 上午11:23, Parav Pandit 写道:
> >>
> >>
> >>On 3/6/2023 10:48 AM, Heng Qi wrote:
> >>
> [..]
> >>>  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> >>>+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.
> >>I think this should also say that HASH_TUNNEL requires either of
> >>the F_RSS or F_HASH_REPORT.
> >>Because without it HASH_TUNNEL is not useful.
> >
> >F_HASH_TUNNEL indicates that the hash should be calculated using
> >the inner packet header, even without F_RSS or F_HASH_REPORT,
> >we can continue to use the hash value in scenarios such as RPS or
> >ebpf programs.
> Yes.
> Even for rps or ebpf programs, F_HASH_TUNNEL is fine.
> When such feature arrives in future, it above line will have OR
> condition for RPS feature bit.
> 
> >
> >
> >I think it's fine to let F_HASH_TUNNEL rely on F_RSS or
> >_F_HASH_REPORT as those are probably important scenarios where
> >inner packet header hash is used.
> Yes.
> 
> >>If not, for now it may be better to skip vxlan and nvegre as
> >>they inherently have unique outer header UDP src port based on
> >>the inner header.
> >
> >Symmetric hashing ignores the order of the five-tuples when
> >calculating the hash, that is, using (a1,a2,p1,p2) and
> >(a2,a1,p2,p1) respectively can calculate the same hash.
> >There is a scenario that the two directions client1->client2 and
> >client2->client1 of the same flow may pass through different
> >tunnels.
> >In order to allow the data in two directions to be processed by
> >the same CPU, we need to calculate a symmetric hash based on the
> >inner packet header.
> I am lost in two directions and two clients above.
> When you say two directions, do you mean tx and rx?
> and do symmetric hashing between tx and rx between two end points
> within single tunnel?
>

A rough description of this scene is as follows:
Client1 sends packets to client2, and client2 sends packets to client1 
respectively.
This is called two directions, and they are in the same flow.
The packets in the two directions may reach the processing host through
different tunnels. In this scenario, we need to hash these packets to the
same queue for the same cpu to process, so inner symmetric hashing is required.


client1   client2
| |
|  __ |
+->| tunnels |<---+
   |-|
  |  |
  |  |
  |  |
  v  v
+-+
| processing host |
+-+


> >>>+\field{hash_tunnel_types} is set to
> >>>VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the
> >>>+unencapsulated packets.
> >>>+
> I missed this before. unencapsulated is not a term.
> s/unencapsulated packets/Non encapsulated packets or non tunneled packets
> 

Yes, you are right!

Thanks.

> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscr...@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
> List help: virtio-comment-h...@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v10] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/16 上午7:24, Parav Pandit 写道:


On 3/15/2023 8:10 AM, Michael S. Tsirkin wrote:

On Tue, Mar 14, 2023 at 11:23:55PM -0400, Parav Pandit wrote:
If not, for now it may be better to skip vxlan and nvegre as they 
inherently

have unique outer header UDP src port based on the inner header.


So what's left, GRE?  GRE is actually different, in that it's not IP at
all.

Sorry, I wrongly wrote nvegre above.

IPoIP, GRE and NVGRE are left.

vxlan and geneve has the udp src entropy.




Not sure I understand "its not IP at all".

GRE has outer IP header + GRE header with the key to identify the flow.
The key is effectively the hash for the flow.


So if we are talking about GRE, hash is indeed not calculated at all at
the moment, right? 
Hash of the outer IP header of the src and dst IP can be still 
calculated currently for GRE when the optional key is not present.



And I would say a natural first step for GRE is
actually adding a hash type that will support this protocol.

For GRE and NVGRE GRE_header.key as the flow/hash identifier should 
work without inner header hash.
Older version of the GRE doesn't have key, so inner header hash is 
useful.


Yes, indeed.
The old GRE does not have keys such as flow id. Even with the new GRE, 
we have no chance to use optional fields,
because we have connected with various operators, and the most basic 
fields can get the best compatibility.


Thanks.




How about doing that? It seems like this should be a small step
and completely uncontroversial.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] RE: [virtio-comment] RE: [PATCH v13] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

在 2023/3/23 上午1:02, Parav Pandit 写道:

From: Michael S. Tsirkin 
Sent: Wednesday, March 22, 2023 12:53 PM

On Wed, Mar 22, 2023 at 04:49:58PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, March 22, 2023 12:47 PM

I agree with Cornelia here. Yes if devices do not want to trust
drivers then they will validate input but what exactly happens then is

currently up to device.

If we want to try and specify devices in all cases of out of spec
input that's a big project, certainly doable but I would rather not
connect it to this, rather boutique, feature.

Both of your and Cornelia's comment is abstract to me.
We cannot change past.

But we can make sure things are consistent. Currently we don't describe device
behaviour if driver is out of spec and I see 0 reasons to start doing it with
coalescing commands specifically.


For the new command of interest here, hen driver supplied incorrect values,

the device will return error.

It might be easier for device to just set NEEDS_RESET and stop responding.

This approach of treating all errors as a fatal category is completely the 
opposite of making the device and driver resilient to (recoverable) errors.
We shouldn't go this route.
Different discussion...


For
a hypervisor implementation that's often better than returning error since
device state is then preserved making things easier to debug.


How to implement is upto the device to figure out.


what to do is also up to the device.

Previously error code as not returned hence new command cannot return the error 
code is going backward.

Returning the failure code is a way to indicate that the driver had a 
recoverable error.


I agree with you. Part of the specification [1] covered something we're 
talking about, e.g. if an untrusted driver sends a disabled vq, the 
device returns an error:


[1] +The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and 
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the 
designated virtqueue is disabled.


Maybe we should modify [1] to:

"The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and 
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the 
designated \field{vqn} is not the virtqueue number of an enabled 
transmit or receive virtqueue."



Thanks!





-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v11] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/23 上午11:13, Parav Pandit 写道:

From: Michael S. Tsirkin 
Sent: Wednesday, March 22, 2023 12:42 PM
  

Yes. But my point is this. Some flows can be IPv4 others IPv6.
Do you see a way to have a key that will result in a symmetrical hash for both
IPv4 and IPv6? Can you give an example please?


Heng,

Is that the requirement to have two completely different flows (ipv, ipv6) to 
steer to a single RQ?


Michael should be talking about whether there is a symmetric key that 
can serve both IPv4 and IPv6, so that they can respectively achieve the 
purpose of symmetric hashing.
I am not an expert in hashing, but this article [1] deduces a symmetric 
hash key, and I think it should be possible to deduce a specific key to 
meet this requirement.


Or we should support XOR hashing.

Thanks.


The requirement, what I understood, is between two directions for a flow to 
result in a single hash value.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v11] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/23 上午11:58, Heng Qi 写道:



在 2023/3/23 上午11:13, Parav Pandit 写道:

From: Michael S. Tsirkin 
Sent: Wednesday, March 22, 2023 12:42 PM
Yes. But my point is this. Some flows can be IPv4 others IPv6.
Do you see a way to have a key that will result in a symmetrical 
hash for both

IPv4 and IPv6? Can you give an example please?


Heng,

Is that the requirement to have two completely different flows (ipv, 
ipv6) to steer to a single RQ?


Michael should be talking about whether there is a symmetric key that 
can serve both IPv4 and IPv6, so that they can respectively achieve 
the purpose of symmetric hashing.
I am not an expert in hashing, but this article [1] deduces a 
symmetric hash key, and I think it should be possible to deduce a 
specific key to meet this requirement.


Sorry for the lost link:
[1] https://www.ndsl.kaist.edu/~kyoungsoo/papers/TR-symRSS.pdf



Or we should support XOR hashing.

Thanks.

The requirement, what I understood, is between two directions for a 
flow to result in a single hash value.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/21 上午3:45, Michael S. Tsirkin 写道:

On Thu, Mar 16, 2023 at 09:17:26PM +0800, Heng Qi wrote:

On Wed, Mar 15, 2023 at 10:57:40AM -0400, Michael S. Tsirkin wrote:

On Wed, Mar 15, 2023 at 08:55:45PM +0800, Heng Qi wrote:


在 2023/3/15 下午7:58, Michael S. Tsirkin 写道:

On Sat, Mar 11, 2023 at 11:23:08AM +0800, Heng Qi wrote:


在 2023/3/10 上午3:36, Michael S. Tsirkin 写道:

On Thu, Mar 09, 2023 at 12:55:02PM +0800, Heng Qi wrote:

在 2023/3/8 下午10:39, Michael S. Tsirkin 写道:

On Wed, Mar 01, 2023 at 10:56:31AM +0800, Heng Qi wrote:

在 2023/2/28 下午7:16, Michael S. Tsirkin 写道:

On Sat, Feb 18, 2023 at 10:37:15PM +0800, Heng Qi wrote:

If the tunnel is used to encapsulate the packets, the hash calculated
using the outer header of the receive packets is always fixed for the
same flow packets, i.e. they will be steered to the same receive queue.

Wait a second. How is this true? Does not everyone stick the
inner header hash in the outer source port to solve this?

Yes, you are right. That's what we did before the inner header hash, but it
has a performance penalty, which I'll explain below.


For example geneve spec says:

it is necessary for entropy from encapsulated packets to be
exposed in the tunnel header.  The most common technique for this is
to use the UDP source port

The end point of the tunnel called the gateway (with DPDK on top of it).

1. When there is no inner header hash, entropy can be inserted into the udp
src port of the outer header of the tunnel,
and then the tunnel packet is handed over to the host. The host needs to
take out a part of the CPUs to parse the outer headers (but not drop them)
to calculate the inner hash for the inner payloads,
and then use the inner
hash to forward them to another part of the CPUs that are responsible for
processing.

I don't get this part. Leave inner hashes to the guest inside the
tunnel, why is your host doing this?

Let's simplify some details and take a fresh look at two different
scenarios: VXLAN and GENEVE (Scenario1) and GRE (Scenario2).

1. In Scenario1, we can improve the processing performance of the same flow
by implementing inner symmetric hashing.

This is because even though client1 and client2 communicate bidirectionally
through the same flow, their data may pass

through and be encapsulated by different tunnels, resulting in the same flow
being hashed to different queues and processed by different CPUs.

To ensure consistency and optimized processing, we need to parse out the
inner header and compute a symmetric hash on it using a special rss key.

Sorry for not mentioning the inner symmetric hash before, in order to
prevent the introduction of more concepts, but it is indeed a kind of inner
hash.

If parts of a flow go through different tunnels won't this cause
reordering at the network level? Why is it so important to prevent it at
the nic then?  Or, since you are stressing symmetric hash, are you
talking about TX and RX side going through different tunnels?

Yes, the directions client1->client2 and client2->client1 may go through
different tunnels.
Using inner symmetric hashing can satisfy the same CPU to process two
directions of the same flow to improve performance.

Well sure but ... are you just doing forwarding or inner processing too?

When there is an inner hash, there is no forwarding anymore.


If forwarding why do you care about matching TX and RX queues? If e2e

In fact, we are just matching on the same rx queue. The network topology
is roughly as follows. The processing host will receive the packets
sent from client1 and client2 respectively, then make some action judgments,
and return them to client2 and client1 respectively.

client1   client2
| |
|  __ |
+->| tunnel |<+
   ||
  |  |
  |  |
  |  |
  v  v
+-+
| processing host |
+-+

Thanks.

monotoring host would be a better term


Sure.

I'm so sorry I didn't realize I missed this until I checked my emails. 😮 :(





processing can't you just store the incoming hash in the flow and reuse
on TX? This is what Linux is doing...






2. In Scenario2 with GRE, the lack of outer transport headers means that
flows between multiple communication pairs encapsulated by the same tunnel

will all be hashed to the same queue. To address this, we need to implement
inner hashing to improve the performance of RSS. By parsing and calculating

the inner hash, different flows can be hashed to different queues.

Thanks.



Well 2 is at least inexact, there's flowID there. It's just 8 bit

We use the most basic GRE header fields (not NVGRE), not even optional
fields.
There is also no flow id in the GRE header, should you be referring to
NVGRE?

Thanks.


so not sufficient if there are more than 512 queues

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/3/21 上午3:48, Michael S. Tsirkin 写道:

On Wed, Mar 15, 2023 at 08:55:45PM +0800, Heng Qi wrote:

We use the most basic GRE header fields (not NVGRE), not even optional
fields.

I'd say yes, the most convincing usecase is with legacy GRE.


Yes. But we still have a strong need for VXLAN and GENEVE to do 
symmetric hashing. Please consider this.



Given that, do you need the rest of protocols there?


I would say that I checked the current tunneling protocols used for 
overlay networks and their respective RFC versions compared to each other.


They are:

1. GRE_rfc2784 :This protocol is only specified for IPv4 and used as 
either the payload or delivery protocol.

    link : https://datatracker.ietf.org/doc/rfc2784/

2. GRE_rfc2890: This protocol describes extensions by which two fields, 
Key and Sequence Number, can be optionally carried in the GRE Header.

    link: https://www.rfc-editor.org/rfc/rfc2890

3. GRE_rfc7676: IPv6 Support for Generic Routing Encapsulation (GRE). 
This protocol is specified for IPv6 and used as either the payload or 
delivery protocol.
    Note that this does not change the GRE header format or any 
behaviors specified by RFC 2784 or RFC 2890.

    link: https://datatracker.ietf.org/doc/rfc7676/

4. GRE-in-UDP: GRE-in-UDP Encapsulation. This specifies a method of 
encapsulating network protocol packets within GRE and UDP headers.
    This GRE-in-UDP encapsulation allows the UDP source port field to 
be used as an entropy field. This protocol is specified for IPv4 and 
IPv6, and used as either the payload or delivery protocol.

    link: https://www.rfc-editor.org/rfc/rfc8086

5. VXLAN: Virtual eXtensible Local Area Network.
    link: https://datatracker.ietf.org/doc/rfc7348/

6. VXLAN-GPE: Generic Protocol Extension for VXLAN. This protocol 
describes extending Virtual eXtensible Local Area Network (VXLAN) via 
changes to the VXLAN header.

    link: https://www.ietf.org/archive/id/draft-ietf-nvo3-vxlan-gpe-12.txt

7. GENEVE: Generic Network Virtualization Encapsulation.
    link: https://datatracker.ietf.org/doc/rfc8926/

8. IPIP: IP Encapsulation within IP.
    link: https://www.rfc-editor.org/rfc/rfc2003

9. NVGRE: Network Virtualization Using Generic Routing Encapsulation
    link: https://www.rfc-editor.org/rfc/rfc7637.html

10. STT: Stateless Transport Tunneling. STT is particularly useful when 
some tunnel endpoints are in end-systems, as it utilizes the 
capabilities of the network interface card to improve performance.

  link: https://www.ietf.org/archive/id/draft-davie-stt-08.txt

Among them, GRE_rfc2784, VXLAN and GENEVE are our internal requirements 
for inner header hashing.

GRE_rfc2784 requires RSS hashing to different queues.
For the monitoring scenario I mentioned, VXLAN or GRE_rfc2890 also needs 
to use inner symmetric hashing.


I know you mean to want this feature to only support GRE_rfc2784, since 
it's the most convincing for RSS.

But RSS hashes packets to different queues for different streams.
For the same flow, it needs to hash it to the same queue.
So this doesn't distort the role of RSS, and I believe that for modern 
protocols like VXLAN and others, inner symmetric hashing is still a 
common requirement for other vendors using virtio devices.


So, can we make this feature support all the protocols I have checked 
above, so that vendors can choose to support the protocols they want. 
And this can avoid the addition of new tunnel protocols

in the near future as much as possible.

Do you think it's ok?

Again: I'm so sorry I didn't realize I missed this until I checked my 
emails. 🙁😮



We can start with just legacy GRE (think about including IPv6 or not).
Given how narrow this usecase is, I'd be fine with focusing
just on this, and addressing more protocols down the road
with something programmable like BPF. WDYT?




-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v9] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/8 下午6:29, Michael S. Tsirkin 写道:

On Thu, Mar 30, 2023 at 08:37:21PM +0800, Heng Qi wrote:


在 2023/3/21 上午3:48, Michael S. Tsirkin 写道:

On Wed, Mar 15, 2023 at 08:55:45PM +0800, Heng Qi wrote:

We use the most basic GRE header fields (not NVGRE), not even optional
fields.

I'd say yes, the most convincing usecase is with legacy GRE.

Yes. But we still have a strong need for VXLAN and GENEVE to do symmetric
hashing. Please consider this.

Using a specific key seems fragile though in that a different one is
needed for e.g. ipv4 and ipv6.  An issue with VXLAN and GENEVE, yes?


Yes.


Will support for XOR hashing address this sufficiently or is that not
acceptable to you? Or alternatively a modified Toeplitz, e.g. this


This is a very good opinion, I will want to follow up on this work and I 
have expressed in other threads.


Thanks.


https://inbox.dpdk.org/dev/20190731123040.gg4...@6wind.com/
suggests Mellanox supports that. WDYT?




-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH v12] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/12 上午5:03, Parav Pandit 写道:



On 4/3/2023 12:58 AM, Heng Qi wrote:

To achieve this, the device can calculate a suitable hash based on 
the inner headers
of this flow, for example using the Toeplitz combined with a 
symmetric hash key.


I am not sure you need symmetric hash key. Toeplitz with symmetric 
hashing without the symmetric key is possible too.


So just mentioning it as a 'combined with symmetric hashing' is enough.


Yes, as discussed with Michael, we will also support XOR hashing or 
Toeplitz symmetric hashing or even both, I'm thinking of starting a 
draft in a separate thread to let us have initial discussions.


The statement here I will modify.



  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device 
Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -198,6 +202,7 @@ \subsection{Device configuration 
layout}\label{sec:Device Types / Network Device

  u8 rss_max_key_size;
  le16 rss_max_indirection_table_length;
  le32 supported_hash_types;
+    le32 supported_tunnel_hash_types;
  };
Given that a set command is added via cvq, it make sense to also do 
symetrric work to get it via a cvq.
This is also similar to the latest work for notification coalescing 
for VQ where get and set done using single channel = cvq.


Only set is given to match the existing RSS/HASH configuration methods. 
But we should really look ahead as you suggest.




Granted that RSS and other fields are done differently, but it was bit 
in the past.


Yes.



With that no need to define two fields at two different places in 
config area and also in cvq.


Just the new opcode is needed for GET and things will be fine.


Right.



+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports 
inner header hash and
+the driver can configure the inner header hash calculation for 
encapsulated packets \ref{sec:Device Types / Network Device / Device 
OperatiHn / Processing of Incoming Packets / Hash calculation for 
incoming packets / Tunnel/Encapsulated packet}
+by issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG from the 
class VIRTIO_NET_CTRL_MQ.
+The command sets \field{hash_tunnel_types} in the structure 
virtio_net_hash_tunnel_config.

+
+struct virtio_net_hash_tunnel_config {
+    le32 hash_tunnel_types;
+};
+

VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG_SET
and
VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG_GET


Will do.



+Filed \field{hash_tunnel_types} contains a bitmask of supported hash 
tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation 
/ Processing of Incoming Packets / Hash calculation for incoming 
packets / Supported/enabled hash tunnel types}.

+
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / 
Processing of Incoming Packets / Hash calculation for incoming 
packets / Tunnel/Encapsulated packet}

+
+A tunnel packet is encapsulated from the original packet based on 
the tunneling
+protocol (only a single level of encapsulation is currently 
supported). The
+encapsulated packet contains an outer header and an inner header, 
and the device

+calculates the hash over either the inner header or the outer header.
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received 
encapsulated packet's
+outer header matches one of the supported \field{hash_tunnel_types}, 
the hash

+of the inner header is calculated.

s/one of the supported/one of the configured/
Because support comes from GET or config space area; out of which a 
smaller or equal subset of tunnel types are configured.


Yes, configured is obviously more accurate than supported.



+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types / 
Network Device / Device Operation / Control Virtqueue / Inner Header 
Hash}

+
+The device MUST calculate the outer header hash if the received 
encapsulated packet has an encapsulation type not in 
\field{supported_tunnel_hash_types}.

+

Since the configured set can be smaller, a better reword is:
The device MUST calculate the hash from the outer header if the 
received encapsulated packet type is not matching from hash_tunnel_types.


Will modify.



+The device MUST respond to the VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG 
command with VIRTIO_NET_ERR if the device
+received an unrecognized or unsupported VIRTIO_NET_HASH_TUNNEL_TYPE_ 
flag.

+
+Upon reset, the device MUST initialize \field{hash_tunnel_type} to 0.
+
+\drivernormative{\subparagraph}{Inner Header Hash}{Device Types / 
Network Device / Device Operation / Control Virtqueue / Inner Header 
Hash}

+
+The driver MUST have negotiated the feature VIRTIO_NET_F_HASH_TUNNEL 
when issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.

+
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags that 
are not supported by the device.

+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / 
Processing of Incoming Packets 

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v12] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/14 上午5:43, Michael S. Tsirkin 写道:

On Thu, Apr 13, 2023 at 07:03:26PM +0800, Heng Qi wrote:

   For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive
queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to
become full, resulting in packet loss.


We have many places that can lead to packet dropping. For example, the
automatic steering is best effort. I tend to avoid mentioning things
like this.

Ok. And Michael what do you think about this?


I think this text did not do a great job explaining the
security aspect. Here's a better, shorter explanation:

It is often an expectation of users that a tunnel isolates the external
network from the internal one. By completely ignoring entropy in the
external header and replacing it with entropy from the internal header,
for hash calculations, this expectation might be violated to a certain
extent, depending on how the hash is used. When the hash use is limited
to RSS queue selection, the effect will likely be limited to ability of
users inside the tunnel to cause packet drops in multiple queues (as
opposed to a single queue without the feature).


Sure. Will do  in the v13.






+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the
receive queue from filling up.
+\item If the QoS is unavailable, the driver can set
\field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+  to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Perform appropriate QoS before packets consume the receive
buffers of the receive queues.
+\end{itemize}
+
+The limitations mentioned above exist with/without the inner header
hash.


This conflicts with the tile "Tunnel QoS limitation" which readers may
think it happens only for tunnel.

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".


My fault.😅



This advice is somewhat bogus though.

The point I keep trying to make is that this:

Choose a hash key that can avoid queue collisions.

is impossible with the feature and possible without.


I don't think so, the outer headers also has corresponding entropy for 
different streams.


Thanks.


This was the whole reason I asked for a security
considerations sections.



Thanks!


Thanks


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscr...@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
List help: virtio-comment-h...@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v12] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/14 上午11:10, Jason Wang 写道:

On Fri, Apr 14, 2023 at 5:46 AM Michael S. Tsirkin  wrote:

On Thu, Apr 13, 2023 at 07:03:26PM +0800, Heng Qi wrote:

   For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive
queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to
become full, resulting in packet loss.


We have many places that can lead to packet dropping. For example, the
automatic steering is best effort. I tend to avoid mentioning things
like this.

Ok. And Michael what do you think about this?


I think this text did not do a great job explaining the
security aspect. Here's a better, shorter explanation:

 It is often an expectation of users that a tunnel isolates the external
 network from the internal one. By completely ignoring entropy in the
 external header and replacing it with entropy from the internal header,
 for hash calculations, this expectation might be violated to a certain
 extent, depending on how the hash is used. When the hash use is limited
 to RSS queue selection, the effect will likely be limited to ability of
 users inside the tunnel to cause packet drops in multiple queues (as
 opposed to a single queue without the feature).

And this is only for GRE-in-UDP? This makes me think if we should add
GRE support for the outer header like:

https://docs.napatech.com/r/Feature-Set-N-ANL9/Hash-Key-Type-10-3-Tuple-GREv0


I think this is for tunneling protocols with specific flow fields, such 
as GRE:key filed, NVGRE:FLOWID filed.


This requires us to make a requirement when calculating the hash of the 
tunnels when F_TUNNEL_HASH is not negotiated. It's a new work.


Thanks.



Thanks





+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the
receive queue from filling up.
+\item If the QoS is unavailable, the driver can set
\field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+  to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Perform appropriate QoS before packets consume the receive
buffers of the receive queues.
+\end{itemize}
+
+The limitations mentioned above exist with/without the inner header
hash.


This conflicts with the tile "Tunnel QoS limitation" which readers may
think it happens only for tunnel.

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".

This advice is somewhat bogus though.

The point I keep trying to make is that this:

 Choose a hash key that can avoid queue collisions.

is impossible with the feature and possible without.
This was the whole reason I asked for a security
considerations sections.

--
MST



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/26 上午4:28, Parav Pandit 写道:



On 4/23/2023 3:35 AM, Heng Qi wrote:
    \subsubsection{Legacy Interface: Feature bits}\label{sec:Device 
Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -198,6 +202,7 @@ \subsection{Device configuration 
layout}\label{sec:Device Types / Network Device

  u8 rss_max_key_size;
  le16 rss_max_indirection_table_length;
  le32 supported_hash_types;
+    le32 supported_tunnel_hash_types;
  };
In v12 I was asking this to move to above field from the config area 
to the GET command in comment [1] as,


"With that no need to define two fields at two different places in 
config area and also in cvq."


I'm not sure if this is sufficiently motivated, RSS also has 
supports_hash_types in config space.


We don't actually need cvq and config to sync on 
supported_tunnel_hash_types, since it doesn't need to change (meaning 
supported_tunnel_hash_types doesn't send configuration change 
notifications).




I am sorry if that was not clear enough.

[1] 
https://lore.kernel.org/virtio-dev/569cbaf9-f1fb-0e1f-a2ef-b1d7cd7db...@nvidia.com/



  \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / 
Processing of Incoming Packets / Hash calculation for incoming 
packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]}, 
\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.

  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4  (1 << 0)
@@ -980,6 +993,152 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / 
Processing of Incoming Packets / Hash calculation for incoming 
packets / IPv6 packets without extension header}).

  \end{itemize}
  +\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation / 
Processing of Incoming Packets / Inner Header Hash}

+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports 
inner header hash and the driver can send
+commands VIRTIO_NET_CTRL_TUNNEL_HASH_SET and 
VIRTIO_NET_CTRL_TUNNEL_HASH_GET for the inner header hash configuration.

+
+struct virtio_net_hash_tunnel_config {

Please move field from the config struct to here. Both are RO fields.

le32 supported_hash_tunnel_types;

+    le32 hash_tunnel_types;
+};
+
+#define VIRTIO_NET_CTRL_TUNNEL_HASH 7
+ #define VIRTIO_NET_CTRL_TUNNEL_HASH_SET 0
+ #define VIRTIO_NET_CTRL_TUNNEL_HASH_GET 1
+
+Filed \field{hash_tunnel_types} contains a bitmask of configured 
hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation 
/ Processing of Incoming Packets / Hash calculation for incoming 
packets / Supported/enabled hash tunnel types}.

+
+The class VIRTIO_NET_CTRL_TUNNEL_HASH has the following commands:
+\begin{itemize}
+\item VIRTIO_NET_CTRL_TUNNEL_HASH_SET: set the 
\field{hash_tunnel_types} to configure the inner header hash 
calculation for the device.
+\item VIRTIO_NET_CTRL_TUNNEL_HASH_GET: get the 
\field{hash_tunnel_types} from the device.

+\end{itemize}
+
+For the command VIRTIO_NET_CTRL_TUNNEL_HASH_SET, the structure 
virtio_net_hash_tunnel_config is write-only for the driver.
+For the command VIRTIO_NET_CTRL_TUNNEL_HASH_GET, the structure 
virtio_net_hash_tunnel_config is read-only for the driver.

+
You need to split the structures to two, one for get and one for set 
in above description as get and set contains different fields.

+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type 
is not included in \field{hash_tunnel_types},
+the hash of the outer header is calculated for the received 
encapsulated packet.

+
+
+For scenarios with sufficient external entropy or no internal 
hashing requirements, inner header hash may not be needed:
+A tunnel is often expected to isolate the external network from the 
internal one. By completely ignoring entropy
+in the external header and replacing it with entropy from the 
internal header, for hash calculations, this expectation

You wanted to say inner here like rest of the places.

s/internal header/inner header


I want to make the 'external' and 'internal' correspond, but avoid the 
internal header, and use a unified 'inner header' is also reasonable.:)





+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags that 
are not supported by the device.

Multiple flags so,

s/flags that are/flags which are/


Will fix.

Thanks!


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] RE: [virtio-comment] RE: [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/26 下午10:24, Parav Pandit 写道:



From: Heng Qi 
Sent: Wednesday, April 26, 2023 10:04 AM
Yes, but that seems like a tiny cost, and the cvq command-related structure is
much simpler.

Current structure size is 24 bytes.
This size becomes multiplier with device count scale to be always available and 
rarely changes.

As we add new features such device capabilities grow making the multiplier 
bigger.
For example
a. flow steering capabilities (how many flows, what mask, supported protocols, 
generic options)
b. hds capabilities
c. counter capabilities (histogram based, which error counters supported, etc)
d. which new type of tx vq improvements supported.
e. hw gro context count supported

May be more..

Depending on the container/VM size certain capabilities may change from device 
to device.
Hence it is hard to deduplicate them at device level.


This makes sense. In general, we should be careful about adding things 
to the device space unless the benefit is non-trivial.


Thanks.



Therefore, ability to query them over a non_always_available transport is 
preferred choice from the device.

A driver may choose to cache it if its being frequently accessed or ask device 
when needed.
Even when it's cached by driver, it is coming from the component that doesn’t 
have transport level timeouts associated with it.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:

On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:

This does not mean that every device needs to implement and support all of
these, they can choose to support some protocols they want.

I add these because we have scale application scenarios for modern protocols
VXLAN-GPE/GENEVE:

+\item In scenarios where the same flow passing through different tunnels is 
expected to be received in the same queue,
+  warm caches, lessing locking, etc. are optimized to obtain receiving 
performance.


Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a little crossover.

Thanks.

But VXLAN-GPE/GENEVE can use source port for entropy.

It is recommended that the UDP source port number
 be calculated using a hash of fields from the inner packet

That is best because
it allows end to end control and is protocol agnostic.


Yes. I agree with this, I don't think we have an argument on this point 
right now.:)


For VXLAN-GPE/GENEVE or other modern tunneling protocols, we have to 
deal with

scenarios where the same flow passes through different tunnels.

Having them hashed to the same rx queue, is hard to do via outer headers.


All that is missing is symmetric Toepliz and all is well?


The scenarios above or in the commit log also require inner headers.


Thanks.






-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin wrote:
> On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:
> > 
> > 
> > 在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:
> > > On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:
> > > > This does not mean that every device needs to implement and support all 
> > > > of
> > > > these, they can choose to support some protocols they want.
> > > > 
> > > > I add these because we have scale application scenarios for modern 
> > > > protocols
> > > > VXLAN-GPE/GENEVE:
> > > > 
> > > > +\item In scenarios where the same flow passing through different 
> > > > tunnels is expected to be received in the same queue,
> > > > +  warm caches, lessing locking, etc. are optimized to obtain 
> > > > receiving performance.
> > > > 
> > > > 
> > > > Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a little 
> > > > crossover.
> > > > 
> > > > Thanks.
> > > But VXLAN-GPE/GENEVE can use source port for entropy.
> > > 
> > >   It is recommended that the UDP source port number
> > >be calculated using a hash of fields from the inner packet
> > > 
> > > That is best because
> > > it allows end to end control and is protocol agnostic.
> > 
> > Yes. I agree with this, I don't think we have an argument on this point
> > right now.:)
> > 
> > For VXLAN-GPE/GENEVE or other modern tunneling protocols, we have to deal
> > with
> > scenarios where the same flow passes through different tunnels.
> > 
> > Having them hashed to the same rx queue, is hard to do via outer headers.
> > > All that is missing is symmetric Toepliz and all is well?
> > 
> > The scenarios above or in the commit log also require inner headers.
> 
> Hmm I am not sure I get it 100%.
> Could you show an example with inner header hash in the port #,
> hash is symmetric, and you still have trouble?
> 
> 
> It kinds of sounds like not enough entropy is not the problem
> at this point.

Sorry for the late reply. :)

For modern tunneling protocols, yes.

> You now want to drop everything from the header
> except the UDP source port. Is that a fair summary?
> 

For example, for the same flow passing through different VXLAN tunnels,
packets in this flow have the same inner header and different outer
headers. Sometimes these packets of the flow need to be hashed to the
same rxq, then we can use the inner header as the hash input.

Thanks!

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

在 2023/5/5 下午10:56, Michael S. Tsirkin 写道:

On Fri, May 05, 2023 at 09:51:15PM +0800, Heng Qi wrote:

On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin wrote:

On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:


在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:

On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:

This does not mean that every device needs to implement and support all of
these, they can choose to support some protocols they want.

I add these because we have scale application scenarios for modern protocols
VXLAN-GPE/GENEVE:

+\item In scenarios where the same flow passing through different tunnels is 
expected to be received in the same queue,
+  warm caches, lessing locking, etc. are optimized to obtain receiving 
performance.


Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a little crossover.

Thanks.

But VXLAN-GPE/GENEVE can use source port for entropy.

It is recommended that the UDP source port number
 be calculated using a hash of fields from the inner packet

That is best because
it allows end to end control and is protocol agnostic.

Yes. I agree with this, I don't think we have an argument on this point
right now.:)

For VXLAN-GPE/GENEVE or other modern tunneling protocols, we have to deal
with
scenarios where the same flow passes through different tunnels.

Having them hashed to the same rx queue, is hard to do via outer headers.

All that is missing is symmetric Toepliz and all is well?

The scenarios above or in the commit log also require inner headers.

Hmm I am not sure I get it 100%.
Could you show an example with inner header hash in the port #,
hash is symmetric, and you still have trouble?


It kinds of sounds like not enough entropy is not the problem
at this point.

Sorry for the late reply. :)

For modern tunneling protocols, yes.


You now want to drop everything from the header
except the UDP source port. Is that a fair summary?


For example, for the same flow passing through different VXLAN tunnels,
packets in this flow have the same inner header and different outer
headers. Sometimes these packets of the flow need to be hashed to the
same rxq, then we can use the inner header as the hash input.

Thanks!

So, they will have the same source port yes?


Yes. The outer source port can be calculated using the 5-tuple of the 
original packet,
and the outer ports are the same but the outer IPs are different after 
different directions of the same flow pass through different tunnels.



Any way to use that


We use it in monitoring, firewall and other scenarios.


so we don't depend on a specific protocol?


Yes, selected tunneling protocols can be used in this scenario like this.

Thanks.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

在 2023/5/9 下午11:15, Michael S. Tsirkin 写道:

On Tue, May 09, 2023 at 10:22:19PM +0800, Heng Qi wrote:


在 2023/5/5 下午10:56, Michael S. Tsirkin 写道:

On Fri, May 05, 2023 at 09:51:15PM +0800, Heng Qi wrote:

On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin wrote:

On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:

在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:

On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:

This does not mean that every device needs to implement and support all of
these, they can choose to support some protocols they want.

I add these because we have scale application scenarios for modern protocols
VXLAN-GPE/GENEVE:

+\item In scenarios where the same flow passing through different tunnels is 
expected to be received in the same queue,
+  warm caches, lessing locking, etc. are optimized to obtain receiving 
performance.


Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a little crossover.

Thanks.

But VXLAN-GPE/GENEVE can use source port for entropy.

It is recommended that the UDP source port number
 be calculated using a hash of fields from the inner packet

That is best because
it allows end to end control and is protocol agnostic.

Yes. I agree with this, I don't think we have an argument on this point
right now.:)

For VXLAN-GPE/GENEVE or other modern tunneling protocols, we have to deal
with
scenarios where the same flow passes through different tunnels.

Having them hashed to the same rx queue, is hard to do via outer headers.

All that is missing is symmetric Toepliz and all is well?

The scenarios above or in the commit log also require inner headers.

Hmm I am not sure I get it 100%.
Could you show an example with inner header hash in the port #,
hash is symmetric, and you still have trouble?


It kinds of sounds like not enough entropy is not the problem
at this point.

Sorry for the late reply. :)

For modern tunneling protocols, yes.


You now want to drop everything from the header
except the UDP source port. Is that a fair summary?


For example, for the same flow passing through different VXLAN tunnels,
packets in this flow have the same inner header and different outer
headers. Sometimes these packets of the flow need to be hashed to the
same rxq, then we can use the inner header as the hash input.

Thanks!

So, they will have the same source port yes?

Yes. The outer source port can be calculated using the 5-tuple of the
original packet,
and the outer ports are the same but the outer IPs are different after
different directions of the same flow pass through different tunnels.

Any way to use that

We use it in monitoring, firewall and other scenarios.


so we don't depend on a specific protocol?

Yes, selected tunneling protocols can be used in this scenario like this.

Thanks.


No, the question was - can we generalize this somehow then?
For example, a flag to ignore source IP when hashing?
Or maybe just for UDP packets?


1. I think the common solution is based on the inner header, so that 
GRE/IPIP tunnels can also enjoy inner symmetric hashing.


2. The VXLAN spec does not show that the outer source port in both 
directions of the same flow must be the same [1]
(although the outer source port is calculated based on the consistent 
hash in the kernel. The consistent hash will sort the five-tuple before 
calculating hashing),
but it is best not to assume that consistent hashing is used in all 
VXLAN implementations. The GENEVE spec uses "SHOUlD"[2].


3. How should we generalize? The device uses a feature to advertise all 
the tunnel types it supports, and hashes these tunnel types using the 
outer source port,
and then we still have to give the specific tunneling protocols 
supported by the device, just like we do now.


[1] "Source Port: It is recommended that the UDP source port number be 
calculated using a hash of fields from the inner packet -- one example
being a hash of the inner Ethernet frame's headers. This is to enable a 
level of entropy for the ECMP/load-balancing of the VM-to-VM traffic across
the VXLAN overlay. When calculating the UDP source port number in this 
manner, it is RECOMMENDED that the value be in the dynamic/private

port range 49152-65535 [RFC6335] "

[2] "Source Port: A source port selected by the originating tunnel 
endpoint. This source port SHOULD be the same for all packets belonging to a
single encapsulated flow to prevent reordering due to the use of 
different paths. To encourage an even distribution of flows across 
multiple links,
the source port SHOULD be calculated using a hash of the encapsulated 
packet headers using, for example, a traditional 5-tuple. Since the port
represents a flow identifier rather than a true UDP connection, the 
entire 16-bit range MAY be used to maximize entropy. In addition to 
setting the
source port, for IPv6, the flow label MAY also be used for providing 
entropy. For an example of using the IP

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

On Thu, May 11, 2023 at 02:22:12AM -0400, Michael S. Tsirkin wrote:
> On Wed, May 10, 2023 at 05:15:37PM +0800, Heng Qi wrote:
> > 
> > 
> > 在 2023/5/9 下午11:15, Michael S. Tsirkin 写道:
> > > On Tue, May 09, 2023 at 10:22:19PM +0800, Heng Qi wrote:
> > > > 
> > > > 在 2023/5/5 下午10:56, Michael S. Tsirkin 写道:
> > > > > On Fri, May 05, 2023 at 09:51:15PM +0800, Heng Qi wrote:
> > > > > > On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:
> > > > > > > > 在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:
> > > > > > > > > On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:
> > > > > > > > > > This does not mean that every device needs to implement and 
> > > > > > > > > > support all of
> > > > > > > > > > these, they can choose to support some protocols they want.
> > > > > > > > > > 
> > > > > > > > > > I add these because we have scale application scenarios for 
> > > > > > > > > > modern protocols
> > > > > > > > > > VXLAN-GPE/GENEVE:
> > > > > > > > > > 
> > > > > > > > > > +\item In scenarios where the same flow passing through 
> > > > > > > > > > different tunnels is expected to be received in the same 
> > > > > > > > > > queue,
> > > > > > > > > > +  warm caches, lessing locking, etc. are optimized to 
> > > > > > > > > > obtain receiving performance.
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a 
> > > > > > > > > > little crossover.
> > > > > > > > > > 
> > > > > > > > > > Thanks.
> > > > > > > > > But VXLAN-GPE/GENEVE can use source port for entropy.
> > > > > > > > > 
> > > > > > > > >   It is recommended that the UDP source port number
> > > > > > > > >be calculated using a hash of fields from the inner 
> > > > > > > > > packet
> > > > > > > > > 
> > > > > > > > > That is best because
> > > > > > > > > it allows end to end control and is protocol agnostic.
> > > > > > > > Yes. I agree with this, I don't think we have an argument on 
> > > > > > > > this point
> > > > > > > > right now.:)
> > > > > > > > 
> > > > > > > > For VXLAN-GPE/GENEVE or other modern tunneling protocols, we 
> > > > > > > > have to deal
> > > > > > > > with
> > > > > > > > scenarios where the same flow passes through different tunnels.
> > > > > > > > 
> > > > > > > > Having them hashed to the same rx queue, is hard to do via 
> > > > > > > > outer headers.
> > > > > > > > > All that is missing is symmetric Toepliz and all is well?
> > > > > > > > The scenarios above or in the commit log also require inner 
> > > > > > > > headers.
> > > > > > > Hmm I am not sure I get it 100%.
> > > > > > > Could you show an example with inner header hash in the port #,
> > > > > > > hash is symmetric, and you still have trouble?
> > > > > > > 
> > > > > > > 
> > > > > > > It kinds of sounds like not enough entropy is not the problem
> > > > > > > at this point.
> > > > > > Sorry for the late reply. :)
> > > > > > 
> > > > > > For modern tunneling protocols, yes.
> > > > > > 
> > > > > > > You now want to drop everything from the header
> > > > > > > except the UDP source port. Is that a fair summary?
> > > > > > > 
> > > > > > For example, for the same flow passing through different VXLAN 
> > > > > > tunnels,
> > > > > > packets in this flow have the same inner header

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

On Fri, May 12, 2023 at 02:54:34AM -0400, Michael S. Tsirkin wrote:
> On Fri, May 12, 2023 at 02:00:19PM +0800, Heng Qi wrote:
> > On Thu, May 11, 2023 at 02:22:12AM -0400, Michael S. Tsirkin wrote:
> > > On Wed, May 10, 2023 at 05:15:37PM +0800, Heng Qi wrote:
> > > > 
> > > > 
> > > > 在 2023/5/9 下午11:15, Michael S. Tsirkin 写道:
> > > > > On Tue, May 09, 2023 at 10:22:19PM +0800, Heng Qi wrote:
> > > > > > 
> > > > > > 在 2023/5/5 下午10:56, Michael S. Tsirkin 写道:
> > > > > > > On Fri, May 05, 2023 at 09:51:15PM +0800, Heng Qi wrote:
> > > > > > > > On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin 
> > > > > > > > wrote:
> > > > > > > > > On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:
> > > > > > > > > > 在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:
> > > > > > > > > > > On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:
> > > > > > > > > > > > This does not mean that every device needs to implement 
> > > > > > > > > > > > and support all of
> > > > > > > > > > > > these, they can choose to support some protocols they 
> > > > > > > > > > > > want.
> > > > > > > > > > > > 
> > > > > > > > > > > > I add these because we have scale application scenarios 
> > > > > > > > > > > > for modern protocols
> > > > > > > > > > > > VXLAN-GPE/GENEVE:
> > > > > > > > > > > > 
> > > > > > > > > > > > +\item In scenarios where the same flow passing through 
> > > > > > > > > > > > different tunnels is expected to be received in the 
> > > > > > > > > > > > same queue,
> > > > > > > > > > > > +  warm caches, lessing locking, etc. are optimized 
> > > > > > > > > > > > to obtain receiving performance.
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has 
> > > > > > > > > > > > a little crossover.
> > > > > > > > > > > > 
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > But VXLAN-GPE/GENEVE can use source port for entropy.
> > > > > > > > > > > 
> > > > > > > > > > >   It is recommended that the UDP source port number
> > > > > > > > > > >be calculated using a hash of fields from the inner 
> > > > > > > > > > > packet
> > > > > > > > > > > 
> > > > > > > > > > > That is best because
> > > > > > > > > > > it allows end to end control and is protocol agnostic.
> > > > > > > > > > Yes. I agree with this, I don't think we have an argument 
> > > > > > > > > > on this point
> > > > > > > > > > right now.:)
> > > > > > > > > > 
> > > > > > > > > > For VXLAN-GPE/GENEVE or other modern tunneling protocols, 
> > > > > > > > > > we have to deal
> > > > > > > > > > with
> > > > > > > > > > scenarios where the same flow passes through different 
> > > > > > > > > > tunnels.
> > > > > > > > > > 
> > > > > > > > > > Having them hashed to the same rx queue, is hard to do via 
> > > > > > > > > > outer headers.
> > > > > > > > > > > All that is missing is symmetric Toepliz and all is well?
> > > > > > > > > > The scenarios above or in the commit log also require inner 
> > > > > > > > > > headers.
> > > > > > > > > Hmm I am not sure I get it 100%.
> > > > > > > > > Could you show an example with inner header hash in the port 
> > > > > > > > > #,
> > > > > &g

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v13] virtio-net: support inner header hash

[Heng Qi]

在 2023/5/12 下午7:27, Michael S. Tsirkin 写道:

On Fri, May 12, 2023 at 03:23:46PM +0800, Heng Qi wrote:

On Fri, May 12, 2023 at 02:54:34AM -0400, Michael S. Tsirkin wrote:

On Fri, May 12, 2023 at 02:00:19PM +0800, Heng Qi wrote:

On Thu, May 11, 2023 at 02:22:12AM -0400, Michael S. Tsirkin wrote:

On Wed, May 10, 2023 at 05:15:37PM +0800, Heng Qi wrote:


在 2023/5/9 下午11:15, Michael S. Tsirkin 写道:

On Tue, May 09, 2023 at 10:22:19PM +0800, Heng Qi wrote:

在 2023/5/5 下午10:56, Michael S. Tsirkin 写道:

On Fri, May 05, 2023 at 09:51:15PM +0800, Heng Qi wrote:

On Thu, Apr 27, 2023 at 01:13:29PM -0400, Michael S. Tsirkin wrote:

On Thu, Apr 27, 2023 at 10:28:29AM +0800, Heng Qi wrote:

在 2023/4/26 下午10:48, Michael S. Tsirkin 写道:

On Wed, Apr 26, 2023 at 10:14:30PM +0800, Heng Qi wrote:

This does not mean that every device needs to implement and support all of
these, they can choose to support some protocols they want.

I add these because we have scale application scenarios for modern protocols
VXLAN-GPE/GENEVE:

+\item In scenarios where the same flow passing through different tunnels is 
expected to be received in the same queue,
+  warm caches, lessing locking, etc. are optimized to obtain receiving 
performance.


Maybe the legacy GRE, VXLAN-GPE and GENEVE? But it has a little crossover.

Thanks.

But VXLAN-GPE/GENEVE can use source port for entropy.

It is recommended that the UDP source port number
 be calculated using a hash of fields from the inner packet

That is best because
it allows end to end control and is protocol agnostic.

Yes. I agree with this, I don't think we have an argument on this point
right now.:)

For VXLAN-GPE/GENEVE or other modern tunneling protocols, we have to deal
with
scenarios where the same flow passes through different tunnels.

Having them hashed to the same rx queue, is hard to do via outer headers.

All that is missing is symmetric Toepliz and all is well?

The scenarios above or in the commit log also require inner headers.

Hmm I am not sure I get it 100%.
Could you show an example with inner header hash in the port #,
hash is symmetric, and you still have trouble?


It kinds of sounds like not enough entropy is not the problem
at this point.

Sorry for the late reply. :)

For modern tunneling protocols, yes.


You now want to drop everything from the header
except the UDP source port. Is that a fair summary?


For example, for the same flow passing through different VXLAN tunnels,
packets in this flow have the same inner header and different outer
headers. Sometimes these packets of the flow need to be hashed to the
same rxq, then we can use the inner header as the hash input.

Thanks!

So, they will have the same source port yes?

Yes. The outer source port can be calculated using the 5-tuple of the
original packet,
and the outer ports are the same but the outer IPs are different after
different directions of the same flow pass through different tunnels.

Any way to use that

We use it in monitoring, firewall and other scenarios.


so we don't depend on a specific protocol?

Yes, selected tunneling protocols can be used in this scenario like this.

Thanks.


No, the question was - can we generalize this somehow then?
For example, a flag to ignore source IP when hashing?
Or maybe just for UDP packets?

1. I think the common solution is based on the inner header, so that
GRE/IPIP tunnels can also enjoy inner symmetric hashing.

2. The VXLAN spec does not show that the outer source port in both
directions of the same flow must be the same [1]
(although the outer source port is calculated based on the consistent hash
in the kernel. The consistent hash will sort the five-tuple before
calculating hashing),
but it is best not to assume that consistent hashing is used in all VXLAN
implementations.

I agree, best not to assume if it's not in the spec.
The requirement to hash two sides to same queue might
not be necessary for everyone though, right?

The outer source port is also not reliable when it needs to be hashed to
the same queue, but the inner header identifies a flow reliably and
universally.


The GENEVE spec uses "SHOUlD"[2].

What about other tunnels? Could you summarize please?

Sure.

The VXLAN spec[1] does not show that the outer source port in both
directions of the same flow must be the same.

VXLAN-GPE[2]("SHOULD")/GENEVE[3]("SHOULD")/GRE-in-UDP[4.1]/STT[5]
recommend that the outer source port of the same flow be calculated
based on the inner header hash and set to the same.

But the udp source port of GRE-in-UDP may be used in a scenario similar
to NAPT [4.2], where the udp source port is no longer used for entropy,
but for identifying different internal hosts. So using udp source port
does not identify the same stream. This is why using the inner header is
more general, since information about the original stream can reliably
identify a flow.

[1] "Source Port: It is recommended tha

[virtio-dev] [Proposal] Relationship between XDP and rx-csum in virtio-net

[Heng Qi]

Currently, the VIRTIO_NET_F_GUEST_CSUM(NETIF_F_RXCSUM) feature of the virtio-net
driver conflicts with the loading of the XDP program, which is caused by the
problem described in [1][2], that is, XDP may cause errors in partial 
csumed-related
fields and resulting in packet dropping. rx CHECKSUM_PARTIAL mainly exists in 
the
virtualized environment, and its purpose is to save computing resource overhead.

The *goal* of this proposal is to enable the coexistence of XDP and 
VIRTIO_NET_F_GUEST_CSUM.

1. We need to understand why the device driver receives the rx CHECKSUM_PARTIAL 
packet.

Drivers related to the virtualized environment, such as 
virtio-net/veth/loopback,
etc., may receive partial csumed packets.

When the tx device finds that the destination rx device of the packet is
located on the same host, it is clear that the packet may not pass through
the physical link, so the tx device sends the packet with csum_{start, offset}
directly to the rx side to save computational resources without computing a 
fully csum
(depends on the specific implementation, some virtio-net backend devices are 
known to
behave like this currently). From [3], the stack trusts such packets.

However, veth still has NETIF_F_RXCSUM turned on when loading XDP. This may 
cause
packet dropping as [1][2] stated. But currently the veth community does not 
seem to
have reported such problems, can we guess that the coexistence of XDP and
rx CHECKSUM_PARTIAL has less negative impact?

2. About rx CHECKSUM_UNECESSARY:

We have just seen that in a virtualized environment a packet may flow between 
the
same host, so not computing the complete csum for the packet saves some cpu 
resources.

The purpose of the checksum is to verify that packets passing through the
physical link are correct. Of course, it is also reasonable to do a fully csum 
for
packets of the virtualized environment, which is exactly what we need.

rx CHECKSUM_UNECESSARY indicates that the packet has been fully checked,
that is, it is a credible packet. If such a packet is modified by the XDP 
program,
the user should recalculate the correct checksum using bpf_csum_diff() and
bpf_{l3,l4}_csum_replace().

Therefore, for those drivers(physical nic drivers?), such as atlantic/bnxt/mlx,
etc., XDP and NETIF_F_RXCSUM coexist, because their packets will be fully 
checked
at the tx side.

AWS's ena driver is also designed to be in this fully checksum mode
(we also mentioned below that a feature bit can be provided for virtio-net,
telling the sender that a fully checksum must be calculated to implement similar
behavior to other drivers), although it is in a virtualized environment.

3. To sum up:

It seems that only virtio-net sets XDP and VIRTIO_NET_F_GUEST_CSUM as mutually
exclusive, which may cause the following problems:

When XDP loads,

1) For packets that are fully checked by the sender, packets are marked as 
CHECKSUM_UNECESSARY
by the rx csum hw offloading.

virtio-net driver needs additional CPU resources to compute the checksum for 
any packet.

When testing with the following command in Aliyun ECS:
qperf dst_ip -lp 8989 -m 64K -t 20 tcp_bw
(mtu = 1500, dev layer GRO is on)

The csum-related overhead we tested on X86 is 11.7%, and on ARM is 15.8%.

2)
One of the main functions of the XDP prog is to be used as a monitoring and
firewall, etc., which means that the XDP prog may not modify the packet.
This is applicable to both rx CHECKSUM_PARTIAL and rx CHECKSUM_UNECESSARY,
but we ignore the rx csum hw offloading capability for both cases.

4. How we try to solve:

1) Add a feature bit to the virtio specification to tell the sender that a fully
csumed packet must be sent. Then XDP can coexist with VIRTIO_NET_F_GUEST_CSUM 
when this
feature bit is negotiated. (similar to ENA behavior)

2) Modify the current virtio-net driver

No longer filter the VIRTIO_NET_F_GUEST_CSUM feature in virtnet_xdp_set().
Then we can immediately get the ability from VIRTIO_NET_F_GUEST_CSUM and enjoy 
the software
CPU resources saved by rx csum hw offloading.
(This method is a bit rude)

5. Ending 

This is a proposal and does not represent a formal solution. Looking forward to 
feedback
from the community and exploring a possible/common solution to the problem 
described in
this proposal.

6. Quote

[1] 18ba58e1c234

virtio-net: fail XDP set if guest csum is negotiated

We don't support partial csumed packet since its metadata will be lost
or incorrect during XDP processing. So fail the XDP set if guest_csum
feature is negotiated.

[2] e59ff2c49ae1

virtio-net: disable guest csum during XDP set

We don't disable VIRTIO_NET_F_GUEST_CSUM if XDP was set. This means we
can receive partial csumed packets with metadata kept in the
vnet_hdr. This may have several side effects:

- It could be overridden by header adjustment, thus is might be not
  correct after XDP processing.
- There's no way to pass such metadata information through
  XDP_REDI

[virtio-dev] Re: [Proposal] Relationship between XDP and rx-csum in virtio-net

[Heng Qi]

On Mon, May 22, 2023 at 03:10:05PM -0400, Michael S. Tsirkin wrote:
> On Mon, May 22, 2023 at 08:12:00PM +0800, Heng Qi wrote:
> > 1) Add a feature bit to the virtio specification to tell the sender that a 
> > fully
> > csumed packet must be sent.
> 
> Who is the sender in this picture? The driver?

The device or the driver.

When the device is hw, the sender is more likely to be a device.
When the device is sw, the sender can be a device or a driver.

But in general, this feature is inclined to constrain the behavior of the 
device and
the driver from the receiving side.

For example: 
VIRTIO_NET_F_UNNECESSARY_CSUM : The driver tells the device that you must send 
me a fully csumed packet.

Then the specific implementation can be

(1) the sender sends a fully csumed packet;
(2) the receiver receives a CHECKSUM_PARTIAL packet, and the device helps 
calculate the fully csum
(because the two parties in the communication are located on the same host, 
the packet is trusted.).

In summary, if VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the driver will no 
longer receive any packets marked CHECKSUM_PARTIAL.

Thanks.

> 
> -- 
> MST

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [Proposal] Relationship between XDP and rx-csum in virtio-net

[Heng Qi]

On Tue, May 23, 2023 at 03:15:37AM -0400, Michael S. Tsirkin wrote:
> On Tue, May 23, 2023 at 10:41:18AM +0800, Heng Qi wrote:
> > On Mon, May 22, 2023 at 03:10:05PM -0400, Michael S. Tsirkin wrote:
> > > On Mon, May 22, 2023 at 08:12:00PM +0800, Heng Qi wrote:
> > > > 1) Add a feature bit to the virtio specification to tell the sender 
> > > > that a fully
> > > > csumed packet must be sent.
> > > 
> > > Who is the sender in this picture? The driver?
> > 
> > The device or the driver.
> > 
> > When the device is hw, the sender is more likely to be a device.
> > When the device is sw, the sender can be a device or a driver.
> >
> > But in general, this feature is inclined to constrain the behavior of the 
> > device and
> > the driver from the receiving side.
> 
> Based on above I am guessing you are talking about driver getting
> packets from device, I wish you used terms from virtio spec.

Yes, I'm going to use the terminology of the virtio spec.

> 
> > For example: 
> > VIRTIO_NET_F_UNNECESSARY_CSUM : The driver tells the device that you must 
> > send me a fully csumed packet.
> > 
> > Then the specific implementation can be
> > 
> > (1) the sender sends a fully csumed packet;
> > (2) the receiver receives a CHECKSUM_PARTIAL packet, and the device helps 
> > calculate the fully csum
> > (because the two parties in the communication are located on the same 
> > host, the packet is trusted.).
> > 
> > In summary, if VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the driver will 
> > no longer receive any packets marked CHECKSUM_PARTIAL.
> > 
> > Thanks.
> 
> This is what clearing VIRTIO_NET_F_GUEST_CSUM does.

Yes, but with VIRTIO_NET_F_GUEST_CSUM cleared, although the device can
receive a fully checksummed packet, we can no longer enjoy
the device's ability to validate the packet checksum. That is, the value
of \field{flags} in the virtio_net_hdr structure is set to 0, which means
that the packet received by the driver will not be marked as
VIRTIO_NET_HDR_F_DATA_VALID.

So, we need a feature bit (let's say VIRTIO_NET_F_UNNECESSARY_CSUM).
If VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the device must give the
driver a fully checksummed packet, and the packet is validated by the
device with \field{flags} set to VIRTIO_NET_HDR_F_DATA_VALID.

> 
> I feel you are trying to say that clearing VIRTIO_NET_F_GUEST_CSUM
> disables all offloads but you want to keep some of them?
> 

No, what I mean is that a feature VIRTIO_NET_F_UNNECESSARY_CSUM is needed
in addition to VIRTIO_NET_F_GUEST_CSUM, if both features are negotiated,
then the driver may always receive packets marked as
VIRTIO_NET_HDR_F_DATA_VALID, which means that we can now load XDP at the
same time.

> Again please use virtio terminology not Linux. to help you out,
> in current linux, VIRTIO_NET_HDR_F_NEEDS_CSUM and VIRTIO_NET_HDR_F_DATA_VALID
> will set CHECKSUM_PARTIAL and CHECKSUM_UNNECESSARY respectively.
> 

Sure. Will do as you suggested.

Thanks.

> 
> -- 
> MST
> 
> 
> -
> To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [Proposal] Relationship between XDP and rx-csum in virtio-net

[Heng Qi]

On Tue, May 23, 2023 at 09:30:28AM -0400, Michael S. Tsirkin wrote:
> On Tue, May 23, 2023 at 05:18:20PM +0800, Heng Qi wrote:
> > On Tue, May 23, 2023 at 03:15:37AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, May 23, 2023 at 10:41:18AM +0800, Heng Qi wrote:
> > > > On Mon, May 22, 2023 at 03:10:05PM -0400, Michael S. Tsirkin wrote:
> > > > > On Mon, May 22, 2023 at 08:12:00PM +0800, Heng Qi wrote:
> > > > > > 1) Add a feature bit to the virtio specification to tell the sender 
> > > > > > that a fully
> > > > > > csumed packet must be sent.
> > > > > 
> > > > > Who is the sender in this picture? The driver?
> > > > 
> > > > The device or the driver.
> > > > 
> > > > When the device is hw, the sender is more likely to be a device.
> > > > When the device is sw, the sender can be a device or a driver.
> > > >
> > > > But in general, this feature is inclined to constrain the behavior of 
> > > > the device and
> > > > the driver from the receiving side.
> > > 
> > > Based on above I am guessing you are talking about driver getting
> > > packets from device, I wish you used terms from virtio spec.
> > 
> > Yes, I'm going to use the terminology of the virtio spec.
> > 
> > > 
> > > > For example: 
> > > > VIRTIO_NET_F_UNNECESSARY_CSUM : The driver tells the device that you 
> > > > must send me a fully csumed packet.
> > > > 
> > > > Then the specific implementation can be
> > > > 
> > > > (1) the sender sends a fully csumed packet;
> > > > (2) the receiver receives a CHECKSUM_PARTIAL packet, and the device 
> > > > helps calculate the fully csum
> > > > (because the two parties in the communication are located on the 
> > > > same host, the packet is trusted.).
> > > > 
> > > > In summary, if VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the driver 
> > > > will no longer receive any packets marked CHECKSUM_PARTIAL.
> > > > 
> > > > Thanks.
> > > 
> > > This is what clearing VIRTIO_NET_F_GUEST_CSUM does.
> > 
> > Yes, but with VIRTIO_NET_F_GUEST_CSUM cleared, although the device can
> > receive a fully checksummed packet, we can no longer enjoy
> > the device's ability to validate the packet checksum. That is, the value
> > of \field{flags} in the virtio_net_hdr structure is set to 0, which means
> > that the packet received by the driver will not be marked as
> > VIRTIO_NET_HDR_F_DATA_VALID.
> > 
> > So, we need a feature bit (let's say VIRTIO_NET_F_UNNECESSARY_CSUM).
> > If VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the device must give the
> > driver a fully checksummed packet, and the packet is validated by the
> > device with \field{flags} set to VIRTIO_NET_HDR_F_DATA_VALID.
> > 
> > > 
> > > I feel you are trying to say that clearing VIRTIO_NET_F_GUEST_CSUM
> > > disables all offloads but you want to keep some of them?
> > > 
> > 
> > No, what I mean is that a feature VIRTIO_NET_F_UNNECESSARY_CSUM is needed
> > in addition to VIRTIO_NET_F_GUEST_CSUM, if both features are negotiated,
> > then the driver may always receive packets marked as
> > VIRTIO_NET_HDR_F_DATA_VALID, which means that we can now load XDP at the
> > same time.
> 
> Makes no sense to me. VIRTIO_NET_F_GUEST_CSUM set already allows
> VIRTIO_NET_HDR_F_DATA_VALID:

We need to focus on what happens when the XDP program is loaded:

The current virtio-net needs to turn off the VIRTIO_NET_F_GUEST_CSUM
feature when loading XDP. The main reason for doing this is because
VIRTIO_NET_F_GUEST_CSUM allows to receive packets marked as
VIRTIO_NET_HDR_F_NEEDS_CSUM. Such packets are not compatible with
XDP programs, because we cannot guarantee that the csum_{start, offset}
fields are correct after XDP modifies the packets.

So in order for the driver to continue to enjoy the device's ability to
validate packets while XDP is loaded, we need a new feature to tell the
device not to deliver packets marked as VIRTIO_NET_HDR_F_NEEDS_CSUM (in
other words, the device can only deliver packets marked as
VIRTIO_NET_HDR_F_DATA_VALID).

Thanks.

> \item If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the
>   VIRTIO_NET_HDR_F_NEEDS_CSUM bit in \field{flags} can be
>   set: if so, the packet checksum at offset \field{csum_offset}
>   from \field{csum_start} and any preceding checksums
>   have been validated.  The checksum on the packet is incomplete and
>   if bit VIR

Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [Proposal] Relationship between XDP and rx-csum in virtio-net

[Heng Qi]

On Wed, May 24, 2023 at 02:07:14AM -0400, Michael S. Tsirkin wrote:
> On Tue, May 23, 2023 at 09:51:44PM +0800, Heng Qi wrote:
> > On Tue, May 23, 2023 at 09:30:28AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, May 23, 2023 at 05:18:20PM +0800, Heng Qi wrote:
> > > > On Tue, May 23, 2023 at 03:15:37AM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, May 23, 2023 at 10:41:18AM +0800, Heng Qi wrote:
> > > > > > On Mon, May 22, 2023 at 03:10:05PM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Mon, May 22, 2023 at 08:12:00PM +0800, Heng Qi wrote:
> > > > > > > > 1) Add a feature bit to the virtio specification to tell the 
> > > > > > > > sender that a fully
> > > > > > > > csumed packet must be sent.
> > > > > > > 
> > > > > > > Who is the sender in this picture? The driver?
> > > > > > 
> > > > > > The device or the driver.
> > > > > > 
> > > > > > When the device is hw, the sender is more likely to be a device.
> > > > > > When the device is sw, the sender can be a device or a driver.
> > > > > >
> > > > > > But in general, this feature is inclined to constrain the behavior 
> > > > > > of the device and
> > > > > > the driver from the receiving side.
> > > > > 
> > > > > Based on above I am guessing you are talking about driver getting
> > > > > packets from device, I wish you used terms from virtio spec.
> > > > 
> > > > Yes, I'm going to use the terminology of the virtio spec.
> > > > 
> > > > > 
> > > > > > For example: 
> > > > > > VIRTIO_NET_F_UNNECESSARY_CSUM : The driver tells the device that 
> > > > > > you must send me a fully csumed packet.
> > > > > > 
> > > > > > Then the specific implementation can be
> > > > > > 
> > > > > > (1) the sender sends a fully csumed packet;
> > > > > > (2) the receiver receives a CHECKSUM_PARTIAL packet, and the device 
> > > > > > helps calculate the fully csum
> > > > > > (because the two parties in the communication are located on 
> > > > > > the same host, the packet is trusted.).
> > > > > > 
> > > > > > In summary, if VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the 
> > > > > > driver will no longer receive any packets marked CHECKSUM_PARTIAL.
> > > > > > 
> > > > > > Thanks.
> > > > > 
> > > > > This is what clearing VIRTIO_NET_F_GUEST_CSUM does.
> > > > 
> > > > Yes, but with VIRTIO_NET_F_GUEST_CSUM cleared, although the device can
> > > > receive a fully checksummed packet, we can no longer enjoy
> > > > the device's ability to validate the packet checksum. That is, the value
> > > > of \field{flags} in the virtio_net_hdr structure is set to 0, which 
> > > > means
> > > > that the packet received by the driver will not be marked as
> > > > VIRTIO_NET_HDR_F_DATA_VALID.
> > > > 
> > > > So, we need a feature bit (let's say VIRTIO_NET_F_UNNECESSARY_CSUM).
> > > > If VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the device must give the
> > > > driver a fully checksummed packet, and the packet is validated by the
> > > > device with \field{flags} set to VIRTIO_NET_HDR_F_DATA_VALID.
> > > > 
> > > > > 
> > > > > I feel you are trying to say that clearing VIRTIO_NET_F_GUEST_CSUM
> > > > > disables all offloads but you want to keep some of them?
> > > > > 
> > > > 
> > > > No, what I mean is that a feature VIRTIO_NET_F_UNNECESSARY_CSUM is 
> > > > needed
> > > > in addition to VIRTIO_NET_F_GUEST_CSUM, if both features are negotiated,
> > > > then the driver may always receive packets marked as
> > > > VIRTIO_NET_HDR_F_DATA_VALID, which means that we can now load XDP at the
> > > > same time.
> > > 
> > > Makes no sense to me. VIRTIO_NET_F_GUEST_CSUM set already allows
> > > VIRTIO_NET_HDR_F_DATA_VALID:
> > 
> > We need to focus on what happens when the XDP program is loaded:
> > 
> > The current virtio-net needs to turn off the VIRTIO_NET_F_GUEST_CSUM
> > feature when loading XDP. The main reason for doing 

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [Proposal] Relationship between XDP and rx-csum in virtio-net

[Heng Qi]

On Tue, May 30, 2023 at 03:33:22PM -0400, Michael S. Tsirkin wrote:
> On Wed, May 24, 2023 at 04:12:46PM +0800, Heng Qi wrote:
> > On Wed, May 24, 2023 at 02:07:14AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, May 23, 2023 at 09:51:44PM +0800, Heng Qi wrote:
> > > > On Tue, May 23, 2023 at 09:30:28AM -0400, Michael S. Tsirkin wrote:
> > > > > On Tue, May 23, 2023 at 05:18:20PM +0800, Heng Qi wrote:
> > > > > > On Tue, May 23, 2023 at 03:15:37AM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Tue, May 23, 2023 at 10:41:18AM +0800, Heng Qi wrote:
> > > > > > > > On Mon, May 22, 2023 at 03:10:05PM -0400, Michael S. Tsirkin 
> > > > > > > > wrote:
> > > > > > > > > On Mon, May 22, 2023 at 08:12:00PM +0800, Heng Qi wrote:
> > > > > > > > > > 1) Add a feature bit to the virtio specification to tell 
> > > > > > > > > > the sender that a fully
> > > > > > > > > > csumed packet must be sent.
> > > > > > > > > 
> > > > > > > > > Who is the sender in this picture? The driver?
> > > > > > > > 
> > > > > > > > The device or the driver.
> > > > > > > > 
> > > > > > > > When the device is hw, the sender is more likely to be a device.
> > > > > > > > When the device is sw, the sender can be a device or a driver.
> > > > > > > >
> > > > > > > > But in general, this feature is inclined to constrain the 
> > > > > > > > behavior of the device and
> > > > > > > > the driver from the receiving side.
> > > > > > > 
> > > > > > > Based on above I am guessing you are talking about driver getting
> > > > > > > packets from device, I wish you used terms from virtio spec.
> > > > > > 
> > > > > > Yes, I'm going to use the terminology of the virtio spec.
> > > > > > 
> > > > > > > 
> > > > > > > > For example: 
> > > > > > > > VIRTIO_NET_F_UNNECESSARY_CSUM : The driver tells the device 
> > > > > > > > that you must send me a fully csumed packet.
> > > > > > > > 
> > > > > > > > Then the specific implementation can be
> > > > > > > > 
> > > > > > > > (1) the sender sends a fully csumed packet;
> > > > > > > > (2) the receiver receives a CHECKSUM_PARTIAL packet, and the 
> > > > > > > > device helps calculate the fully csum
> > > > > > > > (because the two parties in the communication are located 
> > > > > > > > on the same host, the packet is trusted.).
> > > > > > > > 
> > > > > > > > In summary, if VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the 
> > > > > > > > driver will no longer receive any packets marked 
> > > > > > > > CHECKSUM_PARTIAL.
> > > > > > > > 
> > > > > > > > Thanks.
> > > > > > > 
> > > > > > > This is what clearing VIRTIO_NET_F_GUEST_CSUM does.
> > > > > > 
> > > > > > Yes, but with VIRTIO_NET_F_GUEST_CSUM cleared, although the device 
> > > > > > can
> > > > > > receive a fully checksummed packet, we can no longer enjoy
> > > > > > the device's ability to validate the packet checksum. That is, the 
> > > > > > value
> > > > > > of \field{flags} in the virtio_net_hdr structure is set to 0, which 
> > > > > > means
> > > > > > that the packet received by the driver will not be marked as
> > > > > > VIRTIO_NET_HDR_F_DATA_VALID.
> > > > > > 
> > > > > > So, we need a feature bit (let's say VIRTIO_NET_F_UNNECESSARY_CSUM).
> > > > > > If VIRTIO_NET_F_UNNECESSARY_CSUM is negotiated, the device must 
> > > > > > give the
> > > > > > driver a fully checksummed packet, and the packet is validated by 
> > > > > > the
> > > > > > device with \field{flags} set to VIRTIO_NET_HDR_F_DATA_VALID.
> > > > > > 
> > > > > > > 
> > > > > > > I fee

Re: [virtio-dev] Re: [virtio-comment] [PATCH v17] virtio-net: support inner header hash

[Heng Qi]

On Tue, Jun 20, 2023 at 08:06:16AM -0400, Michael S. Tsirkin wrote:
> On Mon, Jun 12, 2023 at 04:09:20PM +0800, Heng Qi wrote:
> > 1. Currently, a received encapsulated packet has an outer and an inner 
> > header, but
> > the virtio device is unable to calculate the hash for the inner header. The 
> > same
> > flow can traverse through different tunnels, resulting in the encapsulated
> > packets being spread across multiple receive queues (refer to the figure 
> > below).
> > However, in certain scenarios, we may need to direct these encapsulated 
> > packets of
> > the same flow to a single receive queue. This facilitates the processing
> > of the flow by the same CPU to improve performance (warm caches, less 
> > locking, etc.).
> > 
> >client1client2
> >   |+---+ |
> >   +--->|tunnels|<+
> >+---+
> >   |  |
> >   v  v
> >   +-+
> >   | monitoring host |
> >   +-+
> > 
> > To achieve this, the device can calculate a symmetric hash based on the 
> > inner headers
> > of the same flow.
> > 
> > 2. For legacy systems, they may lack entropy fields which modern protocols 
> > have in
> > the outer header, resulting in multiple flows with the same outer header but
> > different inner headers being directed to the same receive queue. This 
> > results in
> > poor receive performance.
> > 
> > To address this limitation, inner header hash can be used to enable the 
> > device to advertise
> > the capability to calculate the hash for the inner packet, regaining better 
> > receive performance.
> > 
> > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/173
> > 
> > Signed-off-by: Heng Qi 
> > Reviewed-by: Xuan Zhuo 
> > Reviewed-by: Parav Pandit 
> 
> 
> Looks good but small rewording suggestions:
> 
> > ---
> > v16->v17:
> > 1. Some small rewrites. @Parav Pandit
> > 2. Add Parav's Reviewed-by tag (Thanks!).
> > 
> > v15->v16:
> > 1. Remove the hash_option. In order to delimit the inner header hash 
> > and RSS
> >configuration, the ability to configure the outer src udp port hash 
> > is given
> >to RSS. This is orthogonal to inner header hash, which will be done 
> > in the
> >RSS capability extension topic (considered as an RSS extension 
> > together
> >with the symmetric toeplitz hash algorithm, etc.). @Parav Pandit 
> > @Michael S . Tsirkin
> 
> Hmm maybe. I'd like the TC see this patch though, they are designed to
> work together. Can we see at least a draft? I have been burned

I've drawn up a simple draft on hashing based on outer src port,
but not yet on toeplitz symmetric hashing, I'll send it out in about 4
weeks, but please don't delay this work. You have my word.

> many times by contributors just addressing what they
> care about, promising to follow through with other work and
> never bothering.

I got it.

> 
> > 2. Fix a 'field' typo. @Parav Pandit
> > 
> > v14->v15:
> > 1. Add tunnel hash option suggested by @Michael S . Tsirkin
> > 2. Adjust some descriptions.
> > 
> > v13->v14:
> > 1. Move supported_hash_tunnel_types from config space into cvq command. 
> > @Parav Pandit
> > 2. Rebase to master branch.
> > 3. Some minor modifications.
> > 
> > v12->v13:
> > 1. Add a GET command for hash_tunnel_types. @Parav Pandit
> > 2. Add tunneling protocol explanation. @Jason Wang
> > 3. Add comments on some usage scenarios for inner hash.
> > 
> > v11->v12:
> > 1. Add a command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
> > 2. Refine the commit log. @Michael S . Tsirkin
> > 3. Add some tunnel types.
> > 
> > v10->v11:
> > 1. Revise commit log for clarity for readers.
> > 2. Some modifications to avoid undefined terms. @Parav Pandit
> > 3. Change VIRTIO_NET_F_HASH_TUNNEL dependency. @Parav Pandit
> > 4. Add the normative statements. @Parav Pandit
> > 
> > v9->v10:
> > 1. Removed hash_report_tunnel related information. @Parav Pandit
> > 2. Re-describe the limitations of QoS for tunneling.
> > 3. Some clarification.
> > 
> > v8->v9:
> > 1. Merge hash_re

Re: [virtio-dev] RE: [virtio-comment] Re: [PATCH v18] virtio-net: support inner header hash

[Heng Qi]

在 2023/6/22 上午4:52, Parav Pandit 写道:

From: Michael S. Tsirkin 
Sent: Wednesday, June 21, 2023 4:38 PM

And the field is RO so no memory cost to exposing it in all VFs.

Two structures do not bring the asymmetry.
Accessing current and enabled fields via two different mechanism is bringing

the asymmetry.

I guess it's a matter of taste, but it is clearly more consistent with other 
hash
things, to which it's very similar.


This is consistent with new commands we define including notification 
coalescing whose GET is not coming config space.


Yes.

  

Nah, config space is too convenient when we can live with its limitations. I 
don't
thin kwe prefer not to keep growing it.
For some things such as this one it's perfect.


Fields are different between different devices.


For example, for migration driver might want to validate that two devices have
same capability. doing it without dma is nicer.


A migration driver for real world scenario, will almost have to use the dma for 
amount of data it needs to exchange.


Another example, future admin transport will have ability to provision devices
by supplying their config space.
This will include this capability automatically, if instead we hide it in a 
command
we need to do extra custom work.


So we do not prefer to keep growing the config space anymore, hence
GET is the right approach to me.

Heh I know you hate config space. Let it go, stop wasting time arguing about the
same thing on every turn and instead help define admin transport to solve it

This was discussed many times, a driver to have a direct (non-intercepted by 
owner device) channel to device.
If you mean this non-intercepted channel as admin transport, fine.
If you mean this is intercepted and it is going over admin cmd, then it is of 
no use for all future interfaces.

We discussed this in thread with you and Jason.
I provided concrete example with size and device provisioning math too and 
other example of multi-physical address VQ.
So transporting register by register over some admin transport is sub-optimal.


Parav, your implementation prefers two separate struct versions and 
doesn't let supported_hash_tunnel_types expand in configuration space. I 
remember this.
I agree that we don't want to jump back and forth, especially as there 
are practical reasons and 5 version jumps to get 
supported_hash_tunnel_types back into the config space.


The original intention of Michael's proposal to merge structures in v18 
should be that two separate structures will cause asynchrony.
I don't think so, the driver can cache enabled hash_tunnel_types every 
SET command. Or after the SET command the driver *SHOULD* use the GET 
command again, which is the workaround.


Thanks.






-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v18] virtio-net: support inner header hash

[Heng Qi]

在 2023/6/22 下午8:32, Parav Pandit 写道:



From: Michael S. Tsirkin 
Sent: Thursday, June 22, 2023 2:23 AM

On Wed, Jun 21, 2023 at 08:52:04PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, June 21, 2023 4:38 PM

And the field is RO so no memory cost to exposing it in all VFs.

Two structures do not bring the asymmetry.
Accessing current and enabled fields via two different mechanism
is bringing

the asymmetry.

I guess it's a matter of taste, but it is clearly more consistent
with other hash things, to which it's very similar.


This is consistent with new commands we define including notification

coalescing whose GET is not coming config space.

But there GET just reports the current state. Not the read only capability. So
there would be cost per VF to keep it in config space.
This one is RO no cost per VF. Let's make it convenient?


And each VF can have different value hence requires per VF storage in the 
device.
  

Nah, config space is too convenient when we can live with its
limitations. I don't thin kwe prefer not to keep growing it.
For some things such as this one it's perfect.


Fields are different between different devices.

Not sure what's the implication?

Implication is device needs to store this in always available on-chip memory 
which is not good.


For example, for migration driver might want to validate that two
devices have same capability. doing it without dma is nicer.


A migration driver for real world scenario, will almost have to use the dma for

amount of data it needs to exchange.

Not migration itself, provisioning.


Provisioning driver usually do not attach to the member device directly.
This requires device reset, followed by reaching _DRIVER stage, querying 
features etc and config area.
And unbinding it and second reset by member driver. Ugh.

Provisioning driver also needs to get the state or capabilities even when 
member driver is already attached.
So config space is not much a gain either.
  

Another example, future admin transport will have ability to
provision devices by supplying their config space.
This will include this capability automatically, if instead we hide
it in a command we need to do extra custom work.


So we do not prefer to keep growing the config space anymore,
hence GET is the right approach to me.

Heh I know you hate config space. Let it go, stop wasting time
arguing about the same thing on every turn and instead help define
admin transport to solve it

This was discussed many times, a driver to have a direct (non-intercepted by

owner device) channel to device.

If you mean this non-intercepted channel as admin transport, fine.

we can do that, sure.


If you mean this is intercepted and it is going over admin cmd, then it is of no

use for all future interfaces.

We discussed this in thread with you and Jason.
I provided concrete example with size and device provisioning math too and

other example of multi-physical address VQ.

So transporting register by register over some admin transport is sub-optimal.


Not register by register, we can send all of config space as long as it's RO. 
This
field is.


It is RO in context of one member device, but every VF can have different value.
The device will never know if one will use new cmdvq to access or some old 
driver will use without it.
And hence, it always needs to provision it on onchip memory for backward 
compatibility.


Yes, I think we also have to consider upcoming
    1. device counters (e.g. supported_device_counter),
    2. receive flow filters (e.g. supported_flow_types, 
supported_max_entries),

    3. header splits (e.g. supported_split_types) etc.
Continuous expansion of the configuration space needs to be careful.



Instead of decision point being RO vs RW,
any new fields via cmdvq and existing fields stays in cfg space, give 
predictable behavior to size the member devices in the system.
Once the cmdvq is available, we can get rid of GET command used in this version 
for new future features.
Till that arrives, GET command is the efficient way.


Yes, I agree.

Thanks.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: [PATCH v19] virtio-net: support inner header hash

[Heng Qi]

在 2023/6/29 上午9:56, Parav Pandit 写道:



From: Michael S. Tsirkin 
Sent: Wednesday, June 28, 2023 3:45 PM

Maybe I get it. You want to use the new features as a carrot to
force drivers to implement DMA? You suspect they will ignore the
spec requirement just because things seem to work?


Right because it is not a must normative.

Well SHOULD also does not mean "ok to just ignore".

This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.


RECOMMENDED and SHOULD forces the device to support MMIO, which is not good.
So rather a good design is device tells the starting offset for the extended 
config space.
And extended config space MUST be accessed using a DMA.
With this sw can have infinite size MMIO and hw device forces DMA based on its 
implementation of where to start DMA from.
This also gives the ability to maintain current config as MMIO for backward 
compatibility.
  



There's some logic here, for sure. you just might be right.

However, surely we can discuss this small tweak in 1.4 timeframe?

Sure, if we prefer the DMA approach I don't have a problem in adding

temporary one field to config space.

I propose to add a line to the spec " Device Configuration Space"
section, something like,

Note: Any new device configuration space fields additional MUST consider

accessing such fields via a DMA interface.

And this will guide the new patches of what to do instead of last moment

rush.

Yea, except again I'd probably make it a SHOULD: e.g. I can see how switching to
MMIO might be an option for qemu helping us debug DMA issues.


There are too many queues whose debugging is needed and MMIO likely not the way 
to debug.
  

The time to discuss this detail would be around when proposal for the DMA
access to config space is on list though: I feel this SHOULD vs MUST is a small
enough detail.


 From implementation POV it is certainly critical and good step forward to 
optimize virtio interface.
  

Going back to inner hash. If we move supported_tunnels back to config space,
do you feel we still need GET or just drop it? I note we do not have GET for
either hash or rss config.


For hash and rss config, debugging is missing. :)
Yes, we can drop the GET after switching supported_tunnels to struct 
virtio_net_hash_config.


Great! Glad to hear this!

  

And if we no longer have GET is there still a reason for a separate command as
opposed to a field in virtio_net_hash_config?
I know this was done in v11 but there it was misaligned.
We went with a command because we needed it for supported_tunnels but
now that is no longer the case and there are reserved words in
virtio_net_hash_config ...

Let me know how you feel it about that, not critical for me.

struct virtio_net_hash_config reserved is fine.


+1.

Inner header hash is orthogonal to RSS, and it's fine to have its own 
structure and commands.
There is no need to send additional RSS fields when we configure inner 
header hash.


Thanks.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: [PATCH v19] virtio-net: support inner header hash

[Heng Qi]

On Thu, Jun 29, 2023 at 01:56:34AM +, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin 
> > Sent: Wednesday, June 28, 2023 3:45 PM
> > > > Maybe I get it. You want to use the new features as a carrot to
> > > > force drivers to implement DMA? You suspect they will ignore the
> > > > spec requirement just because things seem to work?
> > > >
> > > Right because it is not a must normative.
> > 
> > Well SHOULD also does not mean "ok to just ignore".
> > 
> > This word, or the adjective "RECOMMENDED", mean that there
> >may exist valid reasons in particular circumstances to ignore a
> >particular item, but the full implications must be understood and
> >carefully weighed before choosing a different course.
> >
> RECOMMENDED and SHOULD forces the device to support MMIO, which is not good.
> So rather a good design is device tells the starting offset for the extended 
> config space.
> And extended config space MUST be accessed using a DMA.
> With this sw can have infinite size MMIO and hw device forces DMA based on 
> its implementation of where to start DMA from.
> This also gives the ability to maintain current config as MMIO for backward 
> compatibility.
>  
> > 
> > 
> > > > There's some logic here, for sure. you just might be right.
> > > >
> > > > However, surely we can discuss this small tweak in 1.4 timeframe?
> > >
> > > Sure, if we prefer the DMA approach I don't have a problem in adding
> > temporary one field to config space.
> > >
> > > I propose to add a line to the spec " Device Configuration Space"
> > > section, something like,
> > >
> > > Note: Any new device configuration space fields additional MUST consider
> > accessing such fields via a DMA interface.
> > >
> > > And this will guide the new patches of what to do instead of last moment
> > rush.
> > 
> > Yea, except again I'd probably make it a SHOULD: e.g. I can see how 
> > switching to
> > MMIO might be an option for qemu helping us debug DMA issues.
> >
> There are too many queues whose debugging is needed and MMIO likely not the 
> way to debug.
>  
> > The time to discuss this detail would be around when proposal for the DMA
> > access to config space is on list though: I feel this SHOULD vs MUST is a 
> > small
> > enough detail.
> >
> From implementation POV it is certainly critical and good step forward to 
> optimize virtio interface.
>  
> > Going back to inner hash. If we move supported_tunnels back to config space,
> > do you feel we still need GET or just drop it? I note we do not have GET for
> > either hash or rss config.
> >
> For hash and rss config, debugging is missing. :)
> Yes, we can drop the GET after switching supported_tunnels to struct 
> virtio_net_hash_config.
>  

I would like to make sure if we're aligned. The new version should contain the 
following:
1. The supported_tunnel_types are placed in the device config space;
2. Reserve the following structure:

 struct virtnet_hash_tunnel {
le32 enabled_tunnel_types;
 };

3. Reserve the SET command for enabled_tunnel_types and remove the GET command 
for enabled_tunnel_types.

If there is no problem, I will modify it accordingly.

Thanks!

> > And if we no longer have GET is there still a reason for a separate command 
> > as
> > opposed to a field in virtio_net_hash_config?
> > I know this was done in v11 but there it was misaligned.
> > We went with a command because we needed it for supported_tunnels but
> > now that is no longer the case and there are reserved words in
> > virtio_net_hash_config ...
> > 
> > Let me know how you feel it about that, not critical for me.
> 
> struct virtio_net_hash_config reserved is fine.

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] RE: [PATCH v19] virtio-net: support inner header hash

[Heng Qi]

在 2023/6/29 下午7:48, Michael S. Tsirkin 写道:

On Thu, Jun 29, 2023 at 10:05:09AM +0800, Heng Qi wrote:


在 2023/6/29 上午9:56, Parav Pandit 写道:

From: Michael S. Tsirkin 
Sent: Wednesday, June 28, 2023 3:45 PM

Maybe I get it. You want to use the new features as a carrot to
force drivers to implement DMA? You suspect they will ignore the
spec requirement just because things seem to work?


Right because it is not a must normative.

Well SHOULD also does not mean "ok to just ignore".

This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.


RECOMMENDED and SHOULD forces the device to support MMIO, which is not good.
So rather a good design is device tells the starting offset for the extended 
config space.
And extended config space MUST be accessed using a DMA.
With this sw can have infinite size MMIO and hw device forces DMA based on its 
implementation of where to start DMA from.
This also gives the ability to maintain current config as MMIO for backward 
compatibility.

There's some logic here, for sure. you just might be right.

However, surely we can discuss this small tweak in 1.4 timeframe?

Sure, if we prefer the DMA approach I don't have a problem in adding

temporary one field to config space.

I propose to add a line to the spec " Device Configuration Space"
section, something like,

Note: Any new device configuration space fields additional MUST consider

accessing such fields via a DMA interface.

And this will guide the new patches of what to do instead of last moment

rush.

Yea, except again I'd probably make it a SHOULD: e.g. I can see how switching to
MMIO might be an option for qemu helping us debug DMA issues.


There are too many queues whose debugging is needed and MMIO likely not the way 
to debug.

The time to discuss this detail would be around when proposal for the DMA
access to config space is on list though: I feel this SHOULD vs MUST is a small
enough detail.


  From implementation POV it is certainly critical and good step forward to 
optimize virtio interface.

Going back to inner hash. If we move supported_tunnels back to config space,
do you feel we still need GET or just drop it? I note we do not have GET for
either hash or rss config.


For hash and rss config, debugging is missing. :)
Yes, we can drop the GET after switching supported_tunnels to struct 
virtio_net_hash_config.

Great! Glad to hear this!


And if we no longer have GET is there still a reason for a separate command as
opposed to a field in virtio_net_hash_config?
I know this was done in v11 but there it was misaligned.
We went with a command because we needed it for supported_tunnels but
now that is no longer the case and there are reserved words in
virtio_net_hash_config ...

Let me know how you feel it about that, not critical for me.

struct virtio_net_hash_config reserved is fine.

+1.

Inner header hash is orthogonal to RSS, and it's fine to have its own
structure and commands.
There is no need to send additional RSS fields when we configure inner
header hash.

Thanks.

Not RSS, hash calculations. It's not critical, but I note that
practically you said you will enable this with symmetric hash
so it makes sense to me to send this in the same command


This works for me.

Thanks.


with the key.

Not critical though if there's opposition.




-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] RE: [virtio-comment] RE: [PATCH v19] virtio-net: support inner header hash

[Heng Qi]

On Thu, Jun 29, 2023 at 04:59:28PM +, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin 
> > Sent: Thursday, June 29, 2023 7:48 AM
> 
> 
> > > > struct virtio_net_hash_config reserved is fine.
> > >
> > > +1.
> > >
> > > Inner header hash is orthogonal to RSS, and it's fine to have its own
> > > structure and commands.
> > > There is no need to send additional RSS fields when we configure inner
> > > header hash.
> > >
> > > Thanks.
> > 
> > Not RSS, hash calculations. It's not critical, but I note that practically 
> > you said
> > you will enable this with symmetric hash so it makes sense to me to send 
> > this in
> > the same command with the key.
> > 
> 
> In the v19, we have,
> 
> +\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along with 
> VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
> 
> So it is done along with rss, so in same struct as rss config is fine.

Do you mean having both virtio_net_rss_config and virtio_net_hash_config
have enabled_hash_types?
Like this:

struct virtio_net_rss_config {
 le32 hash_types;
 le16 indirection_table_mask;
 struct rss_rq_id unclassified_queue;
 struct rss_rq_id indirection_table[indirection_table_length];
 le16 max_tx_vq;
 u8 hash_key_length;
 u8 hash_key_data[hash_key_length];
+le32 enabled_tunnel_types; 
};

struct virtio_net_hash_config {
 le32 hash_types;
-le16 reserved[4];
+le32 enabled_tunnel_types;
+le16 reserved[2];
 u8 hash_key_length;
 u8 hash_key_data[hash_key_length];
};


If yes, this should have been discussed in v10 [1] before, enabled_tunnel_types 
in virtio_net_rss_config
will follow the variable length field and cause misalignment.

If we let the inner header hash reuse the virtio_net_hash_config structure, it 
can work, but the only disadvantage
is that the configuration of the inner header hash and *RSS*(not hash 
calculations) becomes somewhat coupled.
Just imagine:
If the driver and the device negotiated VIRTIO_NET_F_HASH_TUNNEL and 
VIRTIO_NET_F_RSS, but did not negotiate VIRTIO_NET_F_HASH_REPORT,
1. then if we only want to configure the inner header hash (such as 
enabled_tunnel_types), it is good for us to send
virtio_net_hash_config alone;
2. but then if we want to configure the inner header hash and RSS (such as 
indirection table), we need to send all
virtio_net_rss_config and virtio_net_hash_config once, because 
virtio_net_rss_config now does not carry enabled_tunnel_types
due to misalignment.

So, I think the following structure will make it clearer to configure inner 
header hash and RSS/hash calculation.
But in any case, if we still propose to reuse virtio_net_hash_config proposal, 
I am ok, no objection:

1. The supported_tunnel_types are placed in the device config space;

2.
Reserve the following structure:

  struct virtnet_hash_tunnel {
le32 enabled_tunnel_types;
  };

3. Reserve the SET command for enabled_tunnel_types and remove the GET
command for enabled_tunnel_types.

[1] https://lists.oasis-open.org/archives/virtio-dev/202303/msg00317.html

Thanks a lot!

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH 0/4] Short document fixes to inner hash feature

[Heng Qi]

在 2023/7/13 下午7:33, Michael S. Tsirkin 写道:

On Thu, Jul 13, 2023 at 11:12:13AM +0200, Cornelia Huck wrote:

On Thu, Jul 13 2023, Parav Pandit  wrote:


This short patches fixes the editing errors that stops the pdf and html
generation.

These 3 fixes are for the patch [1] for the github issue [2].

[1] https://lists.oasis-open.org/archives/virtio-comment/202307/msg00024.html
[2] https://github.com/oasis-tcs/virtio-spec/issues/173

Patch summary:
patch-1 place C code under listing
patch-2 avoid hyphen and extra braces
patch-3 use table as hyperlink do not work well in C code listing
patch-4 refer 'advice' as 'note'

Patch 1 to 3 appears to be must in the testing.
Patch 4 is not a fix and can be done later if it requires discussion.

Parav Pandit (4):
   virtio-net: Place C code under listing
   virtio-net: Avoid hyphen and extra braces
   virtio-net: Use table to describe inner hash to rfc mapping
   virtio-net: Use note instead of advice

  device-types/net/description.tex | 45 ++--
  introduction.tex | 15 +--
  2 files changed, 38 insertions(+), 22 deletions(-)


FTR, this is the diff I have locally (I had missed one underscore in the
references yesterday...); maybe we can make the intra-reference links in
introdcution.tex a bit nicer, but otherwise, this should be the minimal
change to make this build:

Perfect. Seems like clearly an editorial fix.

Heng Qi, in the future I'd like to ask you to please build the
spec and review the resulting PDF and HTML, before posting.


Yes, I will!






diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index 206020de567d..76585b0dd6d3 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -1024,12 +1024,14 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the driver can send the 
command
  VIRTIO_NET_CTRL_HASH_TUNNEL_SET to configure the calculation of the inner 
header hash.
  
+\begin{lstlisting}

  struct virtnet_hash_tunnel {
  le32 enabled_tunnel_types;
  };
  
  #define VIRTIO_NET_CTRL_HASH_TUNNEL 7

   #define VIRTIO_NET_CTRL_HASH_TUNNEL_SET 0
+\end{lstlisting}
  
  Field \field{enabled_tunnel_types} contains the bitmask of encapsulation types enabled for inner header hash.

  See \ref{sec:Device Types / Network Device / Device Operation / Processing of 
Incoming Packets /
@@ -1063,16 +1065,16 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  Hash calculation for incoming packets / Encapsulation types supported/enabled 
for inner header hash}
  
  Encapsulation types applicable for inner header hash:

-\begin{lstlisting}
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784(1 << 0) /* 
\hyperref[intro:gre_rfc2784]{[GRE_rfc2784]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890(1 << 1) /* 
\hyperref[intro:gre_rfc2890]{[GRE_rfc2890]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676(1 << 2) /* 
\hyperref[intro:gre_rfc7676]{[GRE_rfc7676]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP (1 << 3) /* 
\hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN   (1 << 4) /* 
\hyperref[intro:vxlan]{[VXLAN]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 5) /* 
\hyperref[intro:vxlan_gpe]{[VXLAN-GPE]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE  (1 << 6) /* 
\hyperref[intro:geneve]{[GENEVE]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP(1 << 7) /* 
\hyperref[intro:ipip]{[IPIP]} */
-#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE   (1 << 8) /* 
\hyperref[intro:nvgre]{[NVGRE]} */
+\begin{lstlisting}[escapechar=|]
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784(1 << 0) /* 
|\hyperref[intro:rfc2784]{[RFC2784]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890(1 << 1) /* 
|\hyperref[intro:rfc2890]{[RFC2890]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676(1 << 2) /* 
|\hyperref[intro:rfc7676]{[RFC7676]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP (1 << 3) /* 
|\hyperref[intro:rfc8086]{[GRE-in-UDP]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN   (1 << 4) /* 
|\hyperref[intro:vxlan]{[VXLAN]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 5) /* 
|\hyperref[intro:vxlan-gpe]{[VXLAN-GPE]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE  (1 << 6) /* 
|\hyperref[intro:geneve]{[GENEVE]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP(1 << 7) /* 
|\hyperref[intro:ipip]{[IPIP]}| */
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE   (1 << 8) /* 
|\hyperref[intro:nvgre]{[NVGRE]}| */
  \end{lstlisting}
  
  \subparagraph{Advice}

diff --git a/introduction.tex b/introduction.tex
index 81f07a4fee19..6f10a94b6fde 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -102,18 +102,18 @@ \section{N

[virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

Hi all!

I would like to ask if anyone has any comments on this version, if so 
please let me know!
If not, I will collect Michael's comments and publish a new version next 
Monday.


Since Christmas is coming, I think this feature may be in danger of 
following the pace of
our hw version releases, so I sincerely request that you please review 
it as soon as possible.


Thanks!

在 2023/12/12 下午5:30, Heng Qi 写道:



在 2023/12/12 下午5:23, Heng Qi 写道:



在 2023/12/12 下午4:44, Michael S. Tsirkin 写道:

On Tue, Dec 12, 2023 at 11:28:21AM +0800, Heng Qi wrote:


在 2023/12/12 上午12:35, Michael S. Tsirkin 写道:

On Mon, Dec 11, 2023 at 05:11:59PM +0800, Heng Qi wrote:
virtio-net works in a virtualized system and is somewhat 
different from

physical nics. One of the differences is that to save virtio device
resources, rx may receive partially checksummed packets. However, 
XDP may

cause partially checksummed packets to be dropped.
So XDP loading currently conflicts with the feature 
VIRTIO_NET_F_GUEST_CSUM.


This patch lets the device to supply fully checksummed packets to 
the driver.
Then XDP can coexist with VIRTIO_NET_F_GUEST_CSUM to enjoy the 
benefits of

device validation checksum.

In addition, implementation of some performant devices always do 
not generate
partially checksummed packets, but the standard driver still need 
to clear

VIRTIO_NET_F_GUEST_CSUM when XDP is there.
A new feature VIRTIO_NET_F_GUEST_FULLY_CSUM is added to solve the 
above

situation, which provides the driver with configurable offload.
If the offload is enabled, then the device must deliver fully
checksummed packets to the driver and may validate the checksum.

Use case example:
If VIRTIO_NET_F_GUEST_FULLY_CSUM is negotiated and the offload is 
enabled,
after XDP processes a fully checksummed packet, the 
VIRTIO_NET_HDR_F_DATA_VALID bit
is retained if the device has validated its checksum, resulting 
in the guest
not needing to validate the checksum again. This is useful for 
guests:

    1. Bring the driver advantages such as cpu savings.
    2. For devices that do not generate partially checksummed 
packets themselves,
   XDP can be loaded in the driver without modifying the 
hardware behavior.


Several solutions have been discussed in the previous proposal[1].
After historical discussion, we have tried the method proposed by 
Jason[2],
but some complex scenarios and challenges are difficult to deal 
with.

We now return to the method suggested in [1].

[1] 
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00291.html 

[2] 
https://lore.kernel.org/all/20230628030506.2213-1-hen...@linux.alibaba.com/


Signed-off-by: Heng Qi 
Reviewed-by: Xuan Zhuo 
---
v4->v5:
- Remove the modification to the GUEST_CSUM.
- The description of this feature has been reorganized for 
greater clarity.


v3->v4:
- Streamline some repetitive descriptions. @Jason
- Add how features should work, when to be enabled, and overhead. 
@Jason @Michael


v2->v3:
- Add a section named "Driver Handles Fully Checksummed Packets"
    and more descriptions. @Michael

v1->v2:
- Modify full checksum functionality as a configurable offload
    that is initially turned off. @Jason

   device-types/net/description.tex    | 74 
+++--

   device-types/net/device-conformance.tex |  1 +
   device-types/net/driver-conformance.tex |  1 +
   introduction.tex    |  3 +
   4 files changed, 73 insertions(+), 6 deletions(-)

diff --git a/device-types/net/description.tex 
b/device-types/net/description.tex

index aff5e08..ab6c13d 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -122,6 +122,9 @@ \subsection{Feature bits}\label{sec:Device 
Types / Network Device / Feature bits

   device with the same MAC address.
   \item[VIRTIO_NET_F_SPEED_DUPLEX(63)] Device reports speed and 
duplex.

+
+\item[VIRTIO_NET_F_GUEST_FULLY_CSUM (64)] Device delivers fully 
checksummed packets

+    to the driver and may validate the checksum.
   \end{description}

I propose
VIRTIO_NET_F_GUEST_CSUM_COMPLETE
instead.

Can I ask here if *complete* in VIRTIO_NET_F_GUEST_CSUM_COMPLETE and
CHECKSUM_COMPLETE mean the same thing?

If so, it seems that it's no longer the same as the description of 
this

patch.

Oh. I thought it is. Then I guess I misunderstand what this patch is
supposed to be doing, again.


Here's some context:

From the perspective of the Linux kernel, the GUEST_CSUM feature is 
negotiated to support
(1)  CHECKSUM_NONE, (2) CHECKSUM_UNNECESSARY, (3) CHECKSUM_PARTIAL, 
which
respectively correspond to (1) the device does not validate the 
packet checksum (may not have
the ability to validate some protocols or does not recognize the 
packet); (2) the device has verified
the data packet, then sets DATA_VALID bit in flags; (3) In order to 
save device resources, VMs
on the same host deliver partially checksummed packets, and 
NEEDS_CSUM bit is set in flags.

[virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/18 上午11:10, Jason Wang 写道:

On Fri, Dec 15, 2023 at 5:51 PM Heng Qi  wrote:

Hi all!

I would like to ask if anyone has any comments on this version, if so
please let me know!
If not, I will collect Michael's comments and publish a new version next
Monday.

I have a dumb question. (And sorry if I asked it before)

Looking at the spec and code. It looks to me DATA_VALID could be set
without GUEST_CSUM.


I don't see that in the spec.
Am I missing something? [1][2]

[1] If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the 
VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has 
validated the packet checksum. In case of multiple encapsulated 
protocols, one level of checksums has been validated.
Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features 
*enable receive checksum*, large receive offload and ECN support which 
are the input equivalents of the transmit checksum, transmit 
segmentation *offloading* and ECN features, as described in 5.1.6.2.


[2] If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST set 
flags to zero* and SHOULD supply a fully checksummed packet to the driver.


I think the reason why the feature bit is not checked in the code is 
because the check is omitted because it is on a per-packet basis,
just like the reason why supported_valid_types is not needed as 
discussed in the v4 version threads. It is not unnecessary.


Thanks!



If yes, why do we need to bother here? If we disable GUEST_CSUM, the
packet will contain checksum. And if the device sets DATA_VALID, it
means the checksum is validated.

Thanks




Since Christmas is coming, I think this feature may be in danger of
following the pace of
our hw version releases, so I sincerely request that you please review
it as soon as possible.

Thanks!

在 2023/12/12 下午5:30, Heng Qi 写道:


在 2023/12/12 下午5:23, Heng Qi 写道:


在 2023/12/12 下午4:44, Michael S. Tsirkin 写道:

On Tue, Dec 12, 2023 at 11:28:21AM +0800, Heng Qi wrote:

在 2023/12/12 上午12:35, Michael S. Tsirkin 写道:

On Mon, Dec 11, 2023 at 05:11:59PM +0800, Heng Qi wrote:

virtio-net works in a virtualized system and is somewhat
different from
physical nics. One of the differences is that to save virtio device
resources, rx may receive partially checksummed packets. However,
XDP may
cause partially checksummed packets to be dropped.
So XDP loading currently conflicts with the feature
VIRTIO_NET_F_GUEST_CSUM.

This patch lets the device to supply fully checksummed packets to
the driver.
Then XDP can coexist with VIRTIO_NET_F_GUEST_CSUM to enjoy the
benefits of
device validation checksum.

In addition, implementation of some performant devices always do
not generate
partially checksummed packets, but the standard driver still need
to clear
VIRTIO_NET_F_GUEST_CSUM when XDP is there.
A new feature VIRTIO_NET_F_GUEST_FULLY_CSUM is added to solve the
above
situation, which provides the driver with configurable offload.
If the offload is enabled, then the device must deliver fully
checksummed packets to the driver and may validate the checksum.

Use case example:
If VIRTIO_NET_F_GUEST_FULLY_CSUM is negotiated and the offload is
enabled,
after XDP processes a fully checksummed packet, the
VIRTIO_NET_HDR_F_DATA_VALID bit
is retained if the device has validated its checksum, resulting
in the guest
not needing to validate the checksum again. This is useful for
guests:
 1. Bring the driver advantages such as cpu savings.
 2. For devices that do not generate partially checksummed
packets themselves,
XDP can be loaded in the driver without modifying the
hardware behavior.

Several solutions have been discussed in the previous proposal[1].
After historical discussion, we have tried the method proposed by
Jason[2],
but some complex scenarios and challenges are difficult to deal
with.
We now return to the method suggested in [1].

[1]
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00291.html

[2]
https://lore.kernel.org/all/20230628030506.2213-1-hen...@linux.alibaba.com/

Signed-off-by: Heng Qi 
Reviewed-by: Xuan Zhuo 
---
v4->v5:
- Remove the modification to the GUEST_CSUM.
- The description of this feature has been reorganized for
greater clarity.

v3->v4:
- Streamline some repetitive descriptions. @Jason
- Add how features should work, when to be enabled, and overhead.
@Jason @Michael

v2->v3:
- Add a section named "Driver Handles Fully Checksummed Packets"
 and more descriptions. @Michael

v1->v2:
- Modify full checksum functionality as a configurable offload
 that is initially turned off. @Jason

device-types/net/description.tex| 74
+++--
device-types/net/device-conformance.tex |  1 +
device-types/net/driver-conformance.tex |  1 +
introduction.tex|  3 +
4 files changed, 73 insertions(+), 6 deletions(-)

diff --git a/device-types/net/description.tex
b/device-types/net/description.tex
index aff5

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/19 下午3:53, Jason Wang 写道:

On Mon, Dec 18, 2023 at 12:54 PM Heng Qi  wrote:



在 2023/12/18 上午11:10, Jason Wang 写道:

On Fri, Dec 15, 2023 at 5:51 PM Heng Qi  wrote:

Hi all!

I would like to ask if anyone has any comments on this version, if so
please let me know!
If not, I will collect Michael's comments and publish a new version next
Monday.

I have a dumb question. (And sorry if I asked it before)

Looking at the spec and code. It looks to me DATA_VALID could be set
without GUEST_CSUM.

I don't see that in the spec.
Am I missing something? [1][2]

[1] If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the
VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has
validated the packet checksum. In case of multiple encapsulated
protocols, one level of checksums has been validated.
Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features
*enable receive checksum*, large receive offload and ECN support which
are the input equivalents of the transmit checksum, transmit
segmentation *offloading* and ECN features, as described in 5.1.6.2.

[2] If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST set
flags to zero* and SHOULD supply a fully checksummed packet to the driver.

So this is kind of ambiguous and seems not what I wanted when I wrote
the code for DATA_VALID in 2011.


Hi Jason, please see below.



NEEDS_CSUM maps to CHECKSUM_PARTIAL which means the packet checksum is
correct.


Yes. This mapping is because the PARTIAL checksum usually does not go 
through the physical wire,

so it is considered safe, and the checksum does not need to be verified.


So spec had

"""
If neither VIRTIO_NET_HDR_F_NEEDS_CSUM nor VIRTIO_NET_HDR_F_DATA_VALID
is set, the driver MUST NOT rely on the packet checksum being correct.
"""


Yes. The checksum of a packet without NEEDS_CSUM or has not been 
verified (DATA_VALID set) is unreliable.

This patch doesn't break that.



For DATA_VALID, it maps to CHECKSUM_UNNECESSARY which is mutually
exclusive with CHECKSUM_PARTAIL.


Yes. Both cannot be set or appear at the same time.


And this is what Linux did right now:

For tun_put_user():

 if (skb->ip_summed == CHECKSUM_PARTIAL) {
 ...
 } else if (has_data_valid &&
skb->ip_summed == CHECKSUM_UNNECESSARY) {
hdr->flags = VIRTIO_NET_HDR_F_DATA_VALID;
 } /* else everything is zero */

This CHECKSUM_UNNECESSARY will work even if GUEST_CSUM is disabled if
I was not wrong.


I think you are talking about this commit: 
10a8d94a95742bb15b4e617ee9884bb4381362be


But in fact, as your commit log says, I think this is a hack. Host nics 
does not fall into the scope of virtio spec?





And in receive_buf():

 if (hdr->hdr.flags & VIRTIO_NET_HDR_F_DATA_VALID)
 skb->ip_summed = CHECKSUM_UNNECESSARY;

I think we can fix this by safely removing "*MUST set flags to zero*"
in [2] from the spec.


Sorry. I cannot follow this view.

1. First of all, VIRTIO_NET_F_GUEST_CSUM (partial csum is not considered 
now, because we have no dispute about it) does represent the device's 
ability to calculate and verify checksums.
Its ability to handle partial checksums (NEEDS_CSUM) is just a special 
processing of virtio, the Linux kernel never had a netdev feature for 
partial checksum handling.


  1.1 VIRTIO_NET_F_GUEST_{TSO4, TSO6, USO4} etc. depend on 
VIRTIO_NET_F_GUEST_CSUM.
    The reason for being relied upon is not that they are related 
to NEEDS_CSUM, but that the device needs to recalculate and verify the 
checksum of the packets when merging the packets.

    See netdev_fix_features:
   if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
 dev->features |= NETIF_F_RXCSUM;
  - netdev_fix_features ->
   if (!(features & NETIF_F_RXCSUM)) {
 /* NETIF_F_GRO_HW implies doing RXCSUM since every packet
  * successfully merged by hardware must also have the
  * checksum verified by hardware. If the user does not
  * want to enable RXCSUM, logically, we should disable 
GRO_HW.

  */
 if (features & NETIF_F_GRO_HW) {
 netdev_dbg(dev, "Dropping NETIF_F_GRO_HW since 
no RXCSUM feature.\n");

 features &= ~NETIF_F_GRO_HW;
 }
 }

  1.2 See NETIF_F_RXCSUM_BIT    /* Receive checksumming offload */
 Most device drivers use NETIF_RX_CSUM to indicate device checksum 
capabilities,
 and the corresponding offload can be dynamically switched on and 
off by user tools such as ethtool.


2. The implementation of vhost-user, large-scale commercial virtio 
device that I know of, and other devices are
completely designed and implemented in accordance with virtio 1.0 and 
later. They are comply with 

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/20 下午1:48, Jason Wang 写道:

On Wed, Dec 20, 2023 at 12:07 AM Heng Qi  wrote:



在 2023/12/19 下午3:53, Jason Wang 写道:

On Mon, Dec 18, 2023 at 12:54 PM Heng Qi  wrote:


在 2023/12/18 上午11:10, Jason Wang 写道:

On Fri, Dec 15, 2023 at 5:51 PM Heng Qi  wrote:

Hi all!

I would like to ask if anyone has any comments on this version, if so
please let me know!
If not, I will collect Michael's comments and publish a new version next
Monday.

I have a dumb question. (And sorry if I asked it before)

Looking at the spec and code. It looks to me DATA_VALID could be set
without GUEST_CSUM.

I don't see that in the spec.
Am I missing something? [1][2]

[1] If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the
VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has
validated the packet checksum. In case of multiple encapsulated
protocols, one level of checksums has been validated.
Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features
*enable receive checksum*, large receive offload and ECN support which
are the input equivalents of the transmit checksum, transmit
segmentation *offloading* and ECN features, as described in 5.1.6.2.

[2] If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST set
flags to zero* and SHOULD supply a fully checksummed packet to the driver.

So this is kind of ambiguous and seems not what I wanted when I wrote
the code for DATA_VALID in 2011.

Hi Jason, please see below.


NEEDS_CSUM maps to CHECKSUM_PARTIAL which means the packet checksum is
correct.

Yes. This mapping is because the PARTIAL checksum usually does not go
through the physical wire,
so it is considered safe, and the checksum does not need to be verified.


So spec had

"""
If neither VIRTIO_NET_HDR_F_NEEDS_CSUM nor VIRTIO_NET_HDR_F_DATA_VALID
is set, the driver MUST NOT rely on the packet checksum being correct.
"""

Yes. The checksum of a packet without NEEDS_CSUM or has not been
verified (DATA_VALID set) is unreliable.
This patch doesn't break that.


For DATA_VALID, it maps to CHECKSUM_UNNECESSARY which is mutually
exclusive with CHECKSUM_PARTAIL.

Yes. Both cannot be set or appear at the same time.

So setting both DATA_VALID and NEEDS_CSUM seems ambiguous.

NEEDS_CSUM: the data is correct but the packet doesn't contain checksum


This is not containing checksum, the pseudo header checksum is saved in 
the checksum field of the transport header.



DATA_VALID: the checksum has been validated, this implies the packet
contains a checksum


I'm not sure if both are set at the same time, and even if set, 
CHECKSUM_PARTIAL will still work when forwarded.

But why are we discussing this?




And this is what Linux did right now:

For tun_put_user():

  if (skb->ip_summed == CHECKSUM_PARTIAL) {
  ...
  } else if (has_data_valid &&
 skb->ip_summed == CHECKSUM_UNNECESSARY) {
 hdr->flags = VIRTIO_NET_HDR_F_DATA_VALID;
  } /* else everything is zero */

This CHECKSUM_UNNECESSARY will work even if GUEST_CSUM is disabled if
I was not wrong.

I think you are talking about this commit:
10a8d94a95742bb15b4e617ee9884bb4381362be

But in fact, as your commit log says, I think this is a hack.

It's not, see below.


Host nics
does not fall into the scope of virtio spec?

Seems not, a lot of NIC produces CHECKSUM_UNNECESSARY, I don't see how
virtio-net differs in this case.




And in receive_buf():

  if (hdr->hdr.flags & VIRTIO_NET_HDR_F_DATA_VALID)
  skb->ip_summed = CHECKSUM_UNNECESSARY;

I think we can fix this by safely removing "*MUST set flags to zero*"
in [2] from the spec.

Sorry. I cannot follow this view.

1. First of all, VIRTIO_NET_F_GUEST_CSUM (partial csum is not considered
now, because we have no dispute about it) does represent the device's
ability to calculate and verify checksums.
Its ability to handle partial checksums (NEEDS_CSUM) is just a special
processing of virtio, the Linux kernel never had a netdev feature for
partial checksum handling.

1.1 VIRTIO_NET_F_GUEST_{TSO4, TSO6, USO4} etc. depend on
VIRTIO_NET_F_GUEST_CSUM.
  The reason for being relied upon is not that they are related
to NEEDS_CSUM, but that the device needs to recalculate and verify the
checksum of the packets when merging the packets.
  See netdev_fix_features:
 if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_CSUM))
   dev->features |= NETIF_F_RXCSUM;
- netdev_fix_features ->
 if (!(features & NETIF_F_RXCSUM)) {
   /* NETIF_F_GRO_HW implies doing RXCSUM since every packet
* successfully merged by hardware must also have the
* checksum verified by hardware. If the user does not
* want to enable RXCSUM, logically, we should disable
GRO_HW.
  

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/20 下午2:59, Jason Wang 写道:

On Wed, Dec 20, 2023 at 2:30 PM Heng Qi  wrote:



在 2023/12/20 下午1:48, Jason Wang 写道:

On Wed, Dec 20, 2023 at 12:07 AM Heng Qi  wrote:


在 2023/12/19 下午3:53, Jason Wang 写道:

On Mon, Dec 18, 2023 at 12:54 PM Heng Qi  wrote:

在 2023/12/18 上午11:10, Jason Wang 写道:

On Fri, Dec 15, 2023 at 5:51 PM Heng Qi  wrote:

Hi all!

I would like to ask if anyone has any comments on this version, if so
please let me know!
If not, I will collect Michael's comments and publish a new version next
Monday.

I have a dumb question. (And sorry if I asked it before)

Looking at the spec and code. It looks to me DATA_VALID could be set
without GUEST_CSUM.

I don't see that in the spec.
Am I missing something? [1][2]

[1] If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the
VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has
validated the packet checksum. In case of multiple encapsulated
protocols, one level of checksums has been validated.
Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features
*enable receive checksum*, large receive offload and ECN support which
are the input equivalents of the transmit checksum, transmit
segmentation *offloading* and ECN features, as described in 5.1.6.2.

[2] If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST set
flags to zero* and SHOULD supply a fully checksummed packet to the driver.

So this is kind of ambiguous and seems not what I wanted when I wrote
the code for DATA_VALID in 2011.

Hi Jason, please see below.


NEEDS_CSUM maps to CHECKSUM_PARTIAL which means the packet checksum is
correct.

Yes. This mapping is because the PARTIAL checksum usually does not go
through the physical wire,
so it is considered safe, and the checksum does not need to be verified.


So spec had

"""
If neither VIRTIO_NET_HDR_F_NEEDS_CSUM nor VIRTIO_NET_HDR_F_DATA_VALID
is set, the driver MUST NOT rely on the packet checksum being correct.
"""

Yes. The checksum of a packet without NEEDS_CSUM or has not been
verified (DATA_VALID set) is unreliable.
This patch doesn't break that.


For DATA_VALID, it maps to CHECKSUM_UNNECESSARY which is mutually
exclusive with CHECKSUM_PARTAIL.

Yes. Both cannot be set or appear at the same time.

So setting both DATA_VALID and NEEDS_CSUM seems ambiguous.

NEEDS_CSUM: the data is correct but the packet doesn't contain checksum

This is not containing checksum, the pseudo header checksum is saved in
the checksum field of the transport header.

I have a hard time understanding this. But yes, basically I meant the
checksum is partial. So the device can't do validation.


If the rx device does receive a partially checksummed packet, but the 
driver requires a fullly
checksummed packet, then the rx device can help to calculate the full 
checksum for packets.





DATA_VALID: the checksum has been validated, this implies the packet
contains a checksum

I'm not sure if both are set at the same time, and even if set,
CHECKSUM_PARTIAL will still work when forwarded.
But why are we discussing this?

I don't get this question.

As a reviewer, I have the right to raise any issue I spot. This is how
the community works.


Sorry I wasn't questioning your question, and I think you captured the 
concerns very well from a nic perspective.




It is intended to reply to the past discussion

1) like your above statement "Both cannot be set or appear at the same time."
2) the example in Linux where CHECKSUM_UNNECESSARY and
CHECKSUM_PARTIAL are mutually exclusive.


And this is what Linux did right now:

For tun_put_user():

   if (skb->ip_summed == CHECKSUM_PARTIAL) {
   ...
   } else if (has_data_valid &&
  skb->ip_summed == CHECKSUM_UNNECESSARY) {
  hdr->flags = VIRTIO_NET_HDR_F_DATA_VALID;
   } /* else everything is zero */

This CHECKSUM_UNNECESSARY will work even if GUEST_CSUM is disabled if
I was not wrong.

I think you are talking about this commit:
10a8d94a95742bb15b4e617ee9884bb4381362be

But in fact, as your commit log says, I think this is a hack.

It's not, see below.


Host nics
does not fall into the scope of virtio spec?

Seems not, a lot of NIC produces CHECKSUM_UNNECESSARY, I don't see how
virtio-net differs in this case.


And in receive_buf():

   if (hdr->hdr.flags & VIRTIO_NET_HDR_F_DATA_VALID)
   skb->ip_summed = CHECKSUM_UNNECESSARY;

I think we can fix this by safely removing "*MUST set flags to zero*"
in [2] from the spec.

Sorry. I cannot follow this view.

1. First of all, VIRTIO_NET_F_GUEST_CSUM (partial csum is not considered
now, because we have no dispute about it) does represent the device's
ability to calculate and verify checksums.
Its ability to handle partial checksums (NEEDS_CSUM) is just a special
processing of virtio, the L

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/20 下午3:35, Michael S. Tsirkin 写道:

On Wed, Dec 20, 2023 at 02:30:01PM +0800, Heng Qi wrote:

But why are we discussing this?

I think basically at this point everyone is confused about what
the feature does. right now we have packets
with
#define VIRTIO_NET_HDR_F_NEEDS_CSUM 1   -> partial
#define VIRTIO_NET_HDR_F_DATA_VALID 2   -> unnecessary
and packets without either  -> none

if both 1 and 2 are set then linux uses VIRTIO_NET_HDR_F_NEEDS_CSUM but
I am not sure it's not a mistake. Maybe it does not matter.

What does this new thing do? So far all we have is "XDP will turn it on"
which is not really sufficient. I assumed it somehow replaces
partial with complete. That would make sense for many reasons,
for example the checksum fields in the header can be reused
for other purposes. But maybe not?



Hello Jaosn and Michael. I've summarized our discussion so far, so check 
it out below. Thank you very much!


From the nic perspective, I think Jason's statement is correct, the 
nic's checksum capability and setting DATA_VALID in flags
should not be determined by GUEST_CSUM feature. As long as the rx 
checksum offload is turned on, DATA_VALID
should be set. (Though we now bind GUEST_CSUM negotiation with rx 
checksum offload.)


Therefore, we need to pay attention to the information of rx checksum 
offload. Please check it out:


Devices that comply with the below description are said to be existing 
devices:
    "If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST* 
set flags to zero and SHOULD supply a fully checksummed packet to the 
driver."


As suggested by Jason, devices that comply with the below description 
are said to be new devices:
    "If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MAY* set 
flags to zero and SHOULD supply a fully checksummed packet to the driver."



1. Rx checksum offload is turned on
GUEST_CSUM feature is not negotiated. (now it is only used to indicate 
whether the driver can handle partially checksummed packets)

   a. Existing devices continue to set flags to 0;
   b. New devices may validate the packets and have flags set to 
DATA_VALID;

   c. Migration.
   Migration of existing devices continues to check GUEST_CSUM 
feature and rx checksum offload;

   Migration of new devices only check rx checksum offload;
   Without updating the existing migration management and control 
system, existing devices cannot be migrated to new devices, and new 
devices cannot be migrated to existing devices.
   d. How offload should be controlled now needs attention. Should 
CTRL_GUEST_OFFLOADS still issue GUEST_CSUM feature bit to control the rx 
checksum offload?


2. The new FULLY_CSUM feature must disable NEEDS_CSUM.
The device may set DATA_VALID regardless of whether FULLY_CSUM or 
GUEST_CSUM is negotiated.
   a. Rx fully checksum offload is still controlled by 
CTRL_GUEST_OFFLOADS carrying GUEST_FULLY_CSUM.
   b. When the rx device receives a partially checksummed packet, it 
should calculate the checksum and delivering a fully checksummed packet 
to the driver.



So now, if we modify the existing spec as Jason suggested, I think it's OK.
But we need to find out how to control rx checksum offload. WDYT?

Thanks!







-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/20 下午2:59, Jason Wang 写道:

On Wed, Dec 20, 2023 at 2:30 PM Heng Qi  wrote:



在 2023/12/20 下午1:48, Jason Wang 写道:

On Wed, Dec 20, 2023 at 12:07 AM Heng Qi  wrote:


在 2023/12/19 下午3:53, Jason Wang 写道:

On Mon, Dec 18, 2023 at 12:54 PM Heng Qi  wrote:

在 2023/12/18 上午11:10, Jason Wang 写道:

On Fri, Dec 15, 2023 at 5:51 PM Heng Qi  wrote:

Hi all!

I would like to ask if anyone has any comments on this version, if so
please let me know!
If not, I will collect Michael's comments and publish a new version next
Monday.

I have a dumb question. (And sorry if I asked it before)

Looking at the spec and code. It looks to me DATA_VALID could be set
without GUEST_CSUM.

I don't see that in the spec.
Am I missing something? [1][2]

[1] If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the
VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has
validated the packet checksum. In case of multiple encapsulated
protocols, one level of checksums has been validated.
Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features
*enable receive checksum*, large receive offload and ECN support which
are the input equivalents of the transmit checksum, transmit
segmentation *offloading* and ECN features, as described in 5.1.6.2.

[2] If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST set
flags to zero* and SHOULD supply a fully checksummed packet to the driver.

So this is kind of ambiguous and seems not what I wanted when I wrote
the code for DATA_VALID in 2011.

Hi Jason, please see below.


NEEDS_CSUM maps to CHECKSUM_PARTIAL which means the packet checksum is
correct.

Yes. This mapping is because the PARTIAL checksum usually does not go
through the physical wire,
so it is considered safe, and the checksum does not need to be verified.


So spec had

"""
If neither VIRTIO_NET_HDR_F_NEEDS_CSUM nor VIRTIO_NET_HDR_F_DATA_VALID
is set, the driver MUST NOT rely on the packet checksum being correct.
"""

Yes. The checksum of a packet without NEEDS_CSUM or has not been
verified (DATA_VALID set) is unreliable.
This patch doesn't break that.


For DATA_VALID, it maps to CHECKSUM_UNNECESSARY which is mutually
exclusive with CHECKSUM_PARTAIL.

Yes. Both cannot be set or appear at the same time.

So setting both DATA_VALID and NEEDS_CSUM seems ambiguous.

NEEDS_CSUM: the data is correct but the packet doesn't contain checksum

This is not containing checksum, the pseudo header checksum is saved in
the checksum field of the transport header.

I have a hard time understanding this. But yes, basically I meant the
checksum is partial. So the device can't do validation.


DATA_VALID: the checksum has been validated, this implies the packet
contains a checksum

I'm not sure if both are set at the same time, and even if set,
CHECKSUM_PARTIAL will still work when forwarded.
But why are we discussing this?

I don't get this question.

As a reviewer, I have the right to raise any issue I spot. This is how
the community works.

It is intended to reply to the past discussion

1) like your above statement "Both cannot be set or appear at the same time."
2) the example in Linux where CHECKSUM_UNNECESSARY and
CHECKSUM_PARTIAL are mutually exclusive.


And this is what Linux did right now:

For tun_put_user():

   if (skb->ip_summed == CHECKSUM_PARTIAL) {
   ...
   } else if (has_data_valid &&
  skb->ip_summed == CHECKSUM_UNNECESSARY) {
  hdr->flags = VIRTIO_NET_HDR_F_DATA_VALID;
   } /* else everything is zero */

This CHECKSUM_UNNECESSARY will work even if GUEST_CSUM is disabled if
I was not wrong.

I think you are talking about this commit:
10a8d94a95742bb15b4e617ee9884bb4381362be

But in fact, as your commit log says, I think this is a hack.

It's not, see below.


Host nics
does not fall into the scope of virtio spec?

Seems not, a lot of NIC produces CHECKSUM_UNNECESSARY, I don't see how
virtio-net differs in this case.


And in receive_buf():

   if (hdr->hdr.flags & VIRTIO_NET_HDR_F_DATA_VALID)
   skb->ip_summed = CHECKSUM_UNNECESSARY;

I think we can fix this by safely removing "*MUST set flags to zero*"
in [2] from the spec.

Sorry. I cannot follow this view.

1. First of all, VIRTIO_NET_F_GUEST_CSUM (partial csum is not considered
now, because we have no dispute about it) does represent the device's
ability to calculate and verify checksums.
Its ability to handle partial checksums (NEEDS_CSUM) is just a special
processing of virtio, the Linux kernel never had a netdev feature for
partial checksum handling.

 1.1 VIRTIO_NET_F_GUEST_{TSO4, TSO6, USO4} etc. depend on
VIRTIO_NET_F_GUEST_CSUM.
   The reason for being relied upon is not that they are related
to NEEDS_CSUM, but that the device needs to recalculate and verify the
checksum of the packe

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/21 上午9:34, Jason Wang 写道:

On Wed, Dec 20, 2023 at 3:42 PM Heng Qi  wrote:



在 2023/12/20 下午2:59, Jason Wang 写道:

On Wed, Dec 20, 2023 at 2:30 PM Heng Qi  wrote:


在 2023/12/20 下午1:48, Jason Wang 写道:

On Wed, Dec 20, 2023 at 12:07 AM Heng Qi  wrote:

在 2023/12/19 下午3:53, Jason Wang 写道:

On Mon, Dec 18, 2023 at 12:54 PM Heng Qi  wrote:

在 2023/12/18 上午11:10, Jason Wang 写道:

On Fri, Dec 15, 2023 at 5:51 PM Heng Qi  wrote:

Hi all!

I would like to ask if anyone has any comments on this version, if so
please let me know!
If not, I will collect Michael's comments and publish a new version next
Monday.

I have a dumb question. (And sorry if I asked it before)

Looking at the spec and code. It looks to me DATA_VALID could be set
without GUEST_CSUM.

I don't see that in the spec.
Am I missing something? [1][2]

[1] If the VIRTIO_NET_F_GUEST_CSUM feature was negotiated, the
VIRTIO_NET_HDR_F_DATA_VALID bit in flags can be set: if so, device has
validated the packet checksum. In case of multiple encapsulated
protocols, one level of checksums has been validated.
Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN features
*enable receive checksum*, large receive offload and ECN support which
are the input equivalents of the transmit checksum, transmit
segmentation *offloading* and ECN features, as described in 5.1.6.2.

[2] If VIRTIO_NET_F_GUEST_CSUM is not negotiated, the device *MUST set
flags to zero* and SHOULD supply a fully checksummed packet to the driver.

So this is kind of ambiguous and seems not what I wanted when I wrote
the code for DATA_VALID in 2011.

Hi Jason, please see below.


NEEDS_CSUM maps to CHECKSUM_PARTIAL which means the packet checksum is
correct.

Yes. This mapping is because the PARTIAL checksum usually does not go
through the physical wire,
so it is considered safe, and the checksum does not need to be verified.


So spec had

"""
If neither VIRTIO_NET_HDR_F_NEEDS_CSUM nor VIRTIO_NET_HDR_F_DATA_VALID
is set, the driver MUST NOT rely on the packet checksum being correct.
"""

Yes. The checksum of a packet without NEEDS_CSUM or has not been
verified (DATA_VALID set) is unreliable.
This patch doesn't break that.


For DATA_VALID, it maps to CHECKSUM_UNNECESSARY which is mutually
exclusive with CHECKSUM_PARTAIL.

Yes. Both cannot be set or appear at the same time.

So setting both DATA_VALID and NEEDS_CSUM seems ambiguous.

NEEDS_CSUM: the data is correct but the packet doesn't contain checksum

This is not containing checksum, the pseudo header checksum is saved in
the checksum field of the transport header.

I have a hard time understanding this. But yes, basically I meant the
checksum is partial. So the device can't do validation.

If the rx device does receive a partially checksummed packet, but the
driver requires a fullly
checksummed packet, then the rx device can help to calculate the full
checksum for packets.

So this can only happen for virtual devices as hardware devices can't
receive partial csum packets.


YES. It should be.




DATA_VALID: the checksum has been validated, this implies the packet
contains a checksum

I'm not sure if both are set at the same time, and even if set,
CHECKSUM_PARTIAL will still work when forwarded.
But why are we discussing this?

I don't get this question.

As a reviewer, I have the right to raise any issue I spot. This is how
the community works.

Sorry I wasn't questioning your question, and I think you captured the
concerns very well from a nic perspective.

I see, thanks. I want to offer help indeed.


Thanks very much!




It is intended to reply to the past discussion

1) like your above statement "Both cannot be set or appear at the same time."
2) the example in Linux where CHECKSUM_UNNECESSARY and
CHECKSUM_PARTIAL are mutually exclusive.


And this is what Linux did right now:

For tun_put_user():

if (skb->ip_summed == CHECKSUM_PARTIAL) {
...
} else if (has_data_valid &&
   skb->ip_summed == CHECKSUM_UNNECESSARY) {
   hdr->flags = VIRTIO_NET_HDR_F_DATA_VALID;
} /* else everything is zero */

This CHECKSUM_UNNECESSARY will work even if GUEST_CSUM is disabled if
I was not wrong.

I think you are talking about this commit:
10a8d94a95742bb15b4e617ee9884bb4381362be

But in fact, as your commit log says, I think this is a hack.

It's not, see below.


Host nics
does not fall into the scope of virtio spec?

Seems not, a lot of NIC produces CHECKSUM_UNNECESSARY, I don't see how
virtio-net differs in this case.


And in receive_buf():

if (hdr->hdr.flags & VIRTIO_NET_HDR_F_DATA_VALID)
skb->ip_summed = CHECKSUM_UNNECESSARY;

I think we can fix this by safely removing "*MUST set flags to zero*"
in [2] from the spec.

Sorry. I cannot follow this view.

1. First o

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v5] virtio-net: device does not deliver partially checksummed packet and may validate the checksum

[Heng Qi]

在 2023/12/21 上午9:34, Jason Wang 写道:

On Wed, Dec 20, 2023 at 3:35 PM Michael S. Tsirkin  wrote:

On Wed, Dec 20, 2023 at 02:30:01PM +0800, Heng Qi wrote:

But why are we discussing this?

I think basically at this point everyone is confused about what
the feature does. right now we have packets
with
#define VIRTIO_NET_HDR_F_NEEDS_CSUM 1   -> partial
#define VIRTIO_NET_HDR_F_DATA_VALID 2   -> unnecessary
and packets without either  -> none

if both 1 and 2 are set then linux uses VIRTIO_NET_HDR_F_NEEDS_CSUM but
I am not sure it's not a mistake. Maybe it does not matter.

What does this new thing do? So far all we have is "XDP will turn it on"
which is not really sufficient. I assumed it somehow replaces
partial with complete.

It looks not? CHECKSUM_COMPLETE is less optimal than
CHECKSUM_UNNCESSARY as validation is still needed.

If I understand correctly, this new thing wants DATA_VALID only.


Disable NEEDS_CSUM or calculate fully checksummed packets to fully 
checksummed packets (how this is done does not matter).
The driver will only receive two types of packets: CHECKSUM_NONE and 
DATA_VALID (CHECKSUM_UNNECESSARY).


Thanks！



Thanks




That would make sense for many reasons,
for example the checksum fields in the header can be reused
for other purposes. But maybe not?


--
MST




-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] RE: [virtio-dev] RE: [virtio-comment] [PATCH v2] virtio-net: support setting coalescing params for multiple vqs

[Heng Qi]

在 2024/1/23 下午3:15, Michael S. Tsirkin 写道:

On Tue, Jan 23, 2024 at 05:55:02AM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Monday, January 22, 2024 1:06 PM

On Mon, Jan 22, 2024 at 05:03:38AM +, Parav Pandit wrote:

The right test on Linux to do without rtnl lock which is anyway
ugly and

wrong semantic to use blocking the whole netdev stack.

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?


I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

I feel the issue is at the conceptual level.

Not for requests which are initiated by the kernel stack (non user initiated).

So how is this different? Is it basically just because
tweaking coalescing in unexpected ways is considered mostly
harmless?


DIM sends configurations frequently, which is try best.




Yes some drivers will take a command
and just queue it for execution later, but this means that errors can not be
propagated back at all. Imagine device with mac
0x123 in promisc mode. Now commands:

1- program MAC 0xabcdef
2- disable promisc mode


User initiated commands error can be propagated when the command completes.
Enqueuing command is at the different bottom level in the driver.


If command 1 fails but 2 proceeds then packets with MAC 0xabc will be
dropped.

Any attempts to batch arbitrary commands will have this issue - be it at driver
or device level.


There is no suggestion to batch arbitrary commands from the driver side.
The suggestion is to batch VQs notification coalescing from the driver side.


So, here's my question: what exactly is the guest behaviour that is driving this
work? Is it with a linux guest?

At least looks to me yes based on the partial patches which are taking rtnl 
lock on netdim's worker callbacks.


which commands does userspace issue that we
need to send multiple vq coalescing commands?

None.


  If all you want is to send
same config to all VQs then why not just use
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as opposed to
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET ?

Only kernel stack initiated VQ notification coalescing changes.
Since every VQ has different values, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET is not 
sufficient.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] RE: [virtio-comment] RE: [virtio-dev] RE: [virtio-comment] [PATCH v2] virtio-net: support setting coalescing params for multiple vqs

[Heng Qi]

在 2024/1/22 下午1:03, Parav Pandit 写道:



From: Heng Qi 
Sent: Monday, January 22, 2024 8:27 AM

在 2024/1/20 下午5:59, Parav Pandit 写道:

From: Heng Qi 
Sent: Wednesday, January 17, 2024 10:22 AM

在 2024/1/15 下午9:21, Parav Pandit 写道:

From: virtio-comm...@lists.oasis-open.org
 On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the
coalescing parameters for a vq, it needs to kick the device and
wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does
not use

the queue as "queue".

Such driver limitation should be removed in the driver. It does not
qualify

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the
final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep

for several micro to milliseconds.

The user command cannot return before the final result is obtained.
And wait is not the problem this patch solves.


By not holding the rtnl lock, rest of the context that needs to enqueue the

request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.


Inside the virtnet_rx_dim_work() there should be rtnl lock call.
A virtio_device level lock to be used for cvq. :)


In addition, I have made batching and asynchronousization of the netdim
command, you can refer to this patch:
https://lore.kernel.org/all/1705410693-118895-4-git-send-email-
hen...@linux.alibaba.com/


In the listed above driver patch the motivation "to optimize the
CPU overhead of the DIM worker caused by the guest being busy
waiting for the command response result."

Is not right.
Because guest is still busy waiting.


There is no busy wait for guests, see get_cvq_work().


Without batching, due to rtnl lock every VQ command is serialized as one 
outstanding command at a time in virtnet_rx_dim_work().
Due to this device is unable to take benefit of DMA batching at large scale.


Adding dim commands is now asynchronous, and the device will receive 
batches of commands.


  

This will enable driver to enqueue multiple cvq commands without
waiting

for previous one.

This will also enable device to find natural coalescing done on
multiple

commands.

When batch user commands occur, ensuring synchronization is a concern.


The following path is observed: 1. Driver kicks the device; 2.
After the device receives the kick, CPU scheduling occurs and DMA
multiple buffers multiple times; 3. The device completes processing
and replies

with a response.

When large-queue devices issue multiple requests and kick the
device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a
cpu that is N times faster than the device side cpu, the device may
find natural

coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command.
Even if processed in a batch way on device side, the number of kicks
and the number of backend DMAs has not been reduced.

Driver notifications are PCI writes so it should not hamper device side,

which can ignore them when they do not bother about it.

Driver notifications need to be processed by the DPU, which interferes with
the CPU on the DPU.


I was asking, if there is anyway to disable for your DPU to ignore these 
notifications while previous one is pending?
 From your above description, it seems there isn’t.


While the device is processing the request, additional kicks are ignored.

Thanks.




Backend DMAs should be reduced by avoiding the LIFO pattern followed by

the splitq driver.

Placing the descriptors contiguously like packedq reduces amount of DMA

naturally.

splitq is widely used, migrating to packedq is not that easy, especially when
there are many components and hardware involved.

I am not suggesting to packed_VQ.
I am suggesting fixing the driver to not do LIFO on descriptors for splitq.
In other words, using contiguous set of descriptors on splitq will improve the 
splitq for DMA.
This will allow using splitq more efficiently for dma as short-term solution 
for DMA until more efficient queues are defined.


The second predicable DMA to avoid is having 8Bytes of data inline in the

descriptor, instead of 16B indirection and extra dma.

Looking forward to working inline!
But I think this does not conflict with batch work, and combining the two will
be more beneficial.


It does not conflict. However, batching for large number of queues may not use 
the inline as the data bytes may not fit in the inline.
  

For multiple DMAs, we need to way to send 8 bytes of data without 16

bytes of indirection via a descriptor.

This is what we discussed a while back to do in txq and Stefan
suggested to

generalize for more queues, which is also a good idea.

Yes, this sounds good.


This the next item to focus as soon as flow 

[virtio-dev] [PATCH] virtio_net: support split header

[Heng Qi]

From: Xuan Zhuo 

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo 
Signed-off-by: Heng Qi 
Reviewed-by: Kangjie Xu 
---

v6:
1. Fix some syntax issues. @Cornelia Huck
2. Clarify some paragraphs. @Cornelia Huck
3. Determine the device what to do if it does not perform header split on a 
packet.

v5:
1. Determine when hdr_len is credible in the process of rx
2. Clean up the use of buffers and descriptors
3. Clarify the meaning of used lenght if the first descriptor is skipped in 
the case of merge

v4:
1. fix typo @Cornelia Huck @Jason Wang
2. do not split header for IP fragmentation packet. @Jason Wang

v3:
1. Fix some syntax issues
2. Fix some terminology issues
3. It is not unified with ip alignment, so ip alignment is not included
4. Make it clear that the device must support four types, in the case of
successful negotiation.

 conformance.tex |   2 +
 content.tex | 111 ++--
 2 files changed, 109 insertions(+), 4 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..bd0f463 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..74c36fe 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
 channel.
 
+\item[VIRTIO_NET_F_SPLIT_HEADER (52)] Device supports splitting the protocol
+header and the payload.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
 #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
 #define VIRTIO_NET_HDR_F_DATA_VALID2
 #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_HEADER  8
 u8 flags;
 #define VIRTIO_NET_HDR_GSO_NONE0
 #define VIRTIO_NET_HDR_GSO_TCPV4   1
@@ -3799,9 +3804,10 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
 not set VIRTIO_NET_HDR_F_RSC_INFO bit in \field{flags}.
 
 If one of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have
-been negotiated, the device SHOULD set \field{hdr_len} to a value

[virtio-dev] Re: [PATCH] virtio_net: support split header

[Heng Qi]

The content of the mail just sent is about "[PATCH v6] virtio_net: 
support split header".


在 2022/8/1 下午2:59, Heng Qi 写道:

From: Xuan Zhuo 

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo 
Signed-off-by: Heng Qi 
Reviewed-by: Kangjie Xu 
---

v6:
 1. Fix some syntax issues. @Cornelia Huck
 2. Clarify some paragraphs. @Cornelia Huck
 3. Determine the device what to do if it does not perform header split on 
a packet.

v5:
 1. Determine when hdr_len is credible in the process of rx
 2. Clean up the use of buffers and descriptors
 3. Clarify the meaning of used lenght if the first descriptor is skipped 
in the case of merge

v4:
 1. fix typo @Cornelia Huck @Jason Wang
 2. do not split header for IP fragmentation packet. @Jason Wang

v3:
 1. Fix some syntax issues
 2. Fix some terminology issues
 3. It is not unified with ip alignment, so ip alignment is not included
 4. Make it clear that the device must support four types, in the case of
 successful negotiation.

  conformance.tex |   2 +
  content.tex | 111 ++--
  2 files changed, 109 insertions(+), 4 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..bd0f463 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}

@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}

diff --git a/content.tex b/content.tex
index e863709..74c36fe 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.
  
+\item[VIRTIO_NET_F_SPLIT_HEADER (52)] Device supports splitting the protocol

+header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
  
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
  
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_HEADER  8
  u8 flags;
  #define VIRTIO_NET_HDR_GSO_NONE0
  #define VIRTIO_NET_HDR_GSO_TCPV4   1
@@ -3799,9 +3804,10 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  not set VIRTIO_NET_HDR_F

Re: [virtio-dev] Re: [PATCH] virtio_net: support split header

[Heng Qi]

在 2022/8/4 下午2:27, Jason Wang 写道:

On Mon, Aug 1, 2022 at 2:59 PM Heng Qi  wrote:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---

v6:
 1. Fix some syntax issues. @Cornelia Huck
 2. Clarify some paragraphs. @Cornelia Huck
 3. Determine the device what to do if it does not perform header split on 
a packet.

v5:
 1. Determine when hdr_len is credible in the process of rx
 2. Clean up the use of buffers and descriptors
 3. Clarify the meaning of used lenght if the first descriptor is skipped 
in the case of merge

v4:
 1. fix typo @Cornelia Huck @Jason Wang
 2. do not split header for IP fragmentation packet. @Jason Wang

v3:
 1. Fix some syntax issues
 2. Fix some terminology issues
 3. It is not unified with ip alignment, so ip alignment is not included
 4. Make it clear that the device must support four types, in the case of
 successful negotiation.

  conformance.tex |   2 +
  content.tex | 111 ++--
  2 files changed, 109 insertions(+), 4 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..bd0f463 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
  \end{itemize}

  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
  \end{itemize}

  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..74c36fe 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.

+\item[VIRTIO_NET_F_SPLIT_HEADER (52)] Device supports splitting the protocol
+header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.

  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}

  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_HEADER  8
  u8 flags;
  #define VIRTIO_NET_HDR_GSO_NONE0
  #define VIRTIO_NET_HDR_GSO_TCPV4   1
@@ -3799,9 +3804,10 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  not set VIRTIO_NET_HDR_F_RSC_INFO bit in \field{flags}.

  If one of the VIRTIO_NET_F_GUES

Re: [virtio-dev] Re: [PATCH] virtio_net: support split header

[Heng Qi]

在 2022/8/4 下午9:50, Cornelia Huck 写道:

On Thu, Aug 04 2022, Heng Qi  wrote:


在 2022/8/4 下午2:27, Jason Wang 写道:

On Mon, Aug 1, 2022 at 2:59 PM Heng Qi   wrote:

@@ -3820,9 +3826,13 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
   driver MUST NOT use the \field{csum_start} and \field{csum_offset}.

   If one of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have
-been negotiated, the driver MAY use \field{hdr_len} only as a hint about the
+been negotiated and the VIRTIO_NET_HDR_F_SPLIT_HEADER bit in \field{flags}
+is not set, the driver MAY use \field{hdr_len} only as a hint about the
   transport header size.
-The driver MUST NOT rely on \field{hdr_len} to be correct.
+
+If the VIRTIO_NET_HDR_F_SPLIT_HEADER bit in \field{flags} is not set, the 
driver
+MUST NOT rely on \field{hdr_len} to be correct.

I think we should keep the above description as-is. For whatever case,
the driver must not trust the metadata set by the device and must
perform necessary sanity tests on them.


My idea is to keep the current description as it is,
but to emphasize in the next version:
"If the VIRTIO_NET_HDR_F_SPLIT_HEADER bit in \field{flags} is set,
the driver MAY treat the \field{hdr_len} as the length of the
protocol header inside the first descriptor."


Just to be clear, you suggest using

"If one of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have
been negotiated, the driver MAY use \field{hdr_len} only as a hint about the
transport header size.

The driver MUST NOT rely on \field{hdr_len} to be correct.

If the VIRTIO_NET_HDR_F_SPLIT_HEADER bit in \field{flags} is set,
the driver MAY treat the \field{hdr_len} as the length of the
protocol header inside the first descriptor."


Yes. I will use the above description to make it clearer in the next version.




(Maybe "...the driver MAY use \field{hdr_len} as a hint about the length
of the protocol header..."? It's still not reliable, right?)


\field{hdr_len} is unreliable when VIRTIO_NET_F_SPLIT_HEADER is not negotiated.


If VIRTIO_NET_F_SPLIT_HEADER is negotiated, "split header" MAY perform the split
from the IP layer, so the protocol header and the transport header are 
different.

so I think the "...the driver MAY use \field{hdr_len} only as a hint about the
transport header size..." paragraph can be left as-is.

Re: [virtio-dev] Re: [PATCH] virtio_net: support split header

[Heng Qi]

在 2022/8/9 下午5:18, Jason Wang 写道:

On Thu, Aug 4, 2022 at 8:48 PM Heng Qi  wrote:


在 2022/8/4 下午2:27, Jason Wang 写道:

On Mon, Aug 1, 2022 at 2:59 PM Heng Qi  wrote:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---

v6:
 1. Fix some syntax issues. @Cornelia Huck
 2. Clarify some paragraphs. @Cornelia Huck
 3. Determine the device what to do if it does not perform header split on 
a packet.

v5:
 1. Determine when hdr_len is credible in the process of rx
 2. Clean up the use of buffers and descriptors
 3. Clarify the meaning of used lenght if the first descriptor is skipped 
in the case of merge

v4:
 1. fix typo @Cornelia Huck @Jason Wang
 2. do not split header for IP fragmentation packet. @Jason Wang

v3:
 1. Fix some syntax issues
 2. Fix some terminology issues
 3. It is not unified with ip alignment, so ip alignment is not included
 4. Make it clear that the device must support four types, in the case of
 successful negotiation.

  conformance.tex |   2 +
  content.tex | 111 ++--
  2 files changed, 109 insertions(+), 4 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..bd0f463 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
  \end{itemize}

  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Header}
  \end{itemize}

  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..74c36fe 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.

+\item[VIRTIO_NET_F_SPLIT_HEADER (52)] Device supports splitting the protocol
+header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.

  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}

  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_HEADER  8
  u8 flags;
  #define VIRTIO_NET_HDR_GSO_NONE0
  #define VIRTIO_NET_HDR_GSO_TCPV4   1
@@ -3799,9 +3804,10 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  n

[virtio-dev] [PATCH v7] virtio_net: support split header

[Heng Qi]

From: Xuan Zhuo 

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo 
Signed-off-by: Heng Qi 
Reviewed-by: Kangjie Xu 
---
v7:
1. Fix some presentation issues.
2. Use "split transport header". @Jason Wang
3. Clarify some paragraphs. @Cornelia Huck
4. determine the device what to do if it does not perform header split 
on a packet.

v6:
1. Fix some syntax issues. @Cornelia Huck
2. Clarify some paragraphs. @Cornelia Huck
3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
1. Determine when hdr_len is credible in the process of rx
2. Clean up the use of buffers and descriptors
3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
1. fix typo @Cornelia Huck @Jason Wang
2. do not split header for IP fragmentation packet. @Jason Wang

v3:
1. Fix some syntax issues
2. Fix some terminology issues
3. It is not unified with ip alignment, so ip alignment is not included
4. Make it clear that the device must support four types, in the case 
of successful negotiation.

 conformance.tex |   2 ++
 content.tex | 102 
 2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
 \end{itemize}
 
 \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
 \end{itemize}
 
 \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
 channel.
 
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting
+the transport header and the payload.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
 #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
 #define VIRTIO_NET_HDR_F_DATA_VALID2
 #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_TRANSPORT_HEADER  8
 u8 flags;
 #define VIRTIO_NET_HDR_GSO_NONE0
 #define VIRTIO_N

Re: [virtio-dev] [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/8/25 下午10:22, Cornelia Huck 写道:

On Tue, Aug 16 2022, Heng Qi  wrote:


From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
1. Fix some presentation issues.
2. Use "split transport header". @Jason Wang
3. Clarify some paragraphs. @Cornelia Huck
4. determine the device what to do if it does not perform header split 
on a packet.

v6:
1. Fix some syntax issues. @Cornelia Huck
2. Clarify some paragraphs. @Cornelia Huck
3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
1. Determine when hdr_len is credible in the process of rx
2. Clean up the use of buffers and descriptors
3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
1. fix typo @Cornelia Huck @Jason Wang
2. do not split header for IP fragmentation packet. @Jason Wang

v3:
1. Fix some syntax issues
2. Fix some terminology issues
3. It is not unified with ip alignment, so ip alignment is not included
4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

I do not have any further comments on the change, let's see what the
networking folks think.


Okay. Thanks for your review.



[Do we require patches to be posted to virtio-comment, or is virtio-dev
enough? I'm a bit unsure right now.]


We can then consider posting patches to virtio-comment.

Re: [virtio-dev] [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/8/16 下午5:34, Heng Qi 写道:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
1. Fix some presentation issues.
2. Use "split transport header". @Jason Wang
3. Clarify some paragraphs. @Cornelia Huck
4. determine the device what to do if it does not perform header split 
on a packet.

v6:
1. Fix some syntax issues. @Cornelia Huck
2. Clarify some paragraphs. @Cornelia Huck
3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
1. Determine when hdr_len is credible in the process of rx
2. Clean up the use of buffers and descriptors
3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
1. fix typo @Cornelia Huck @Jason Wang
2. do not split header for IP fragmentation packet. @Jason Wang

v3:
1. Fix some syntax issues
2. Fix some terminology issues
3. It is not unified with ip alignment, so ip alignment is not included
4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}

@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}

diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.
  
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting

+the transport header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
  
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
  
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_TRANSPORT_HEADER  8
  u8 

Re: [virtio-dev] [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/8/16 下午5:34, Heng Qi 写道:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
1. Fix some presentation issues.
2. Use "split transport header". @Jason Wang
3. Clarify some paragraphs. @Cornelia Huck
4. determine the device what to do if it does not perform header split 
on a packet.

v6:
1. Fix some syntax issues. @Cornelia Huck
2. Clarify some paragraphs. @Cornelia Huck
3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
1. Determine when hdr_len is credible in the process of rx
2. Clean up the use of buffers and descriptors
3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
1. fix typo @Cornelia Huck @Jason Wang
2. do not split header for IP fragmentation packet. @Jason Wang

v3:
1. Fix some syntax issues
2. Fix some terminology issues
3. It is not unified with ip alignment, so ip alignment is not included
4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}

@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}

diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.
  
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting

+the transport header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
  
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
  
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_TRANSPORT_HEADER  8
  u8 

[virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/9/2 下午2:21, Jason Wang 写道:

On Tue, Aug 16, 2022 at 5:35 PM Heng Qi  wrote:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
 1. Fix some presentation issues.
 2. Use "split transport header". @Jason Wang
 3. Clarify some paragraphs. @Cornelia Huck
 4. determine the device what to do if it does not perform header split 
on a packet.

v6:
 1. Fix some syntax issues. @Cornelia Huck
 2. Clarify some paragraphs. @Cornelia Huck
 3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
 1. Determine when hdr_len is credible in the process of rx
 2. Clean up the use of buffers and descriptors
 3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
 1. fix typo @Cornelia Huck @Jason Wang
 2. do not split header for IP fragmentation packet. @Jason Wang

v3:
 1. Fix some syntax issues
 2. Fix some terminology issues
 3. It is not unified with ip alignment, so ip alignment is not included
 4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}

  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}

  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.

+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting
+the transport header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.

  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}

  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO

[virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/9/2 下午2:21, Jason Wang 写道:

On Tue, Aug 16, 2022 at 5:35 PM Heng Qi  wrote:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
 1. Fix some presentation issues.
 2. Use "split transport header". @Jason Wang
 3. Clarify some paragraphs. @Cornelia Huck
 4. determine the device what to do if it does not perform header split 
on a packet.

v6:
 1. Fix some syntax issues. @Cornelia Huck
 2. Clarify some paragraphs. @Cornelia Huck
 3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
 1. Determine when hdr_len is credible in the process of rx
 2. Clean up the use of buffers and descriptors
 3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
 1. fix typo @Cornelia Huck @Jason Wang
 2. do not split header for IP fragmentation packet. @Jason Wang

v3:
 1. Fix some syntax issues
 2. Fix some terminology issues
 3. It is not unified with ip alignment, so ip alignment is not included
 4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}

  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}

  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.

+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting
+the transport header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.

  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}

  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO

[virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/9/2 下午2:41, Michael S. Tsirkin 写道:

On Fri, Sep 02, 2022 at 02:21:04PM +0800, Jason Wang wrote:

On Tue, Aug 16, 2022 at 5:35 PM Heng Qi  wrote:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
 1. Fix some presentation issues.
 2. Use "split transport header". @Jason Wang
 3. Clarify some paragraphs. @Cornelia Huck
 4. determine the device what to do if it does not perform header split 
on a packet.

v6:
 1. Fix some syntax issues. @Cornelia Huck
 2. Clarify some paragraphs. @Cornelia Huck
 3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
 1. Determine when hdr_len is credible in the process of rx
 2. Clean up the use of buffers and descriptors
 3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
 1. fix typo @Cornelia Huck @Jason Wang
 2. do not split header for IP fragmentation packet. @Jason Wang

v3:
 1. Fix some syntax issues
 2. Fix some terminology issues
 3. It is not unified with ip alignment, so ip alignment is not included
 4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}

  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / 
Driver Conformance / Block Driver Conformance}
@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}

  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / 
Device Conformance / Block Device Conformance}
diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.

+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting
+the transport header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.

  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}

  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F

[virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/9/5 下午3:52, Xuan Zhuo 写道:

On Sun, 4 Sep 2022 16:31:59 -0400, "Michael S. Tsirkin"  wrote:

On Fri, Sep 02, 2022 at 04:58:16PM +0800, Heng Qi wrote:

When VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER is negotiated,
the driver requires that the buffers submitted to receiveq
MUST be composed of at least two descriptors,
which means that each buffer the device gets is a descriptor chain,
even if the device does not split the header for some packets.

To store packet in the descriptor chain without header splitting
by the device, the device MUST start with the first descriptor of
the descriptor chain to store the packet, and MUST NOT set the
VIRTIO_NET_HDR_F_SPLIT_TRANSPORT_HEADER bit in \field{flags}.

Thanks.

Descriptor chains will hurt performance badly.

I understand the reasons for the performance impact here are:
1. Two buffers are used
2. One buffer occupies two descs

This is the same as my understanding in the case of mergeable. We also need to
pack the packets into two buffers, and a packet will eventually occupy two
descs.



How about simply making this feature depend on mergeable buffers?
Then we have a separate buffer for the header and
this works cleanly.


Under mergeable, each buffer is independent, and the split header requires two
unequal descs.

If we implement it based on mergeable, then consider the scenario of tcp
zerocopy, when we fill receive vq, each buffer is an separate page, and if we 
use an
separate buffer to save the header, then this is a waste, we may
have to copy at the driver layer.

@Qi Do you think there will be other problems with this approach?

Thanks.


When we think about specs, we shouldn't be too distracted by the implementation.

But when we did think about this, suppose the driver fills by page based on
mergeable mode. in order to use the xdp program, the driver usually takes
the beginning of a single page as the headroom, and fills the rest of the page
into the virtqueue. Therefore, the empty buffer obtained by the
device is always smaller than a page when we implement split header
based on this mode, that is, the data load finally obtained by the driver
is offset from the beginning of the page. This does not enjoy the benefits of 
zero copy.

At the same time, since the header is always only more than 100 bytes,
the page occupied by the header is a waste of the buffer.

Thanks.

Re: [virtio-dev] [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/8/16 下午5:34, Heng Qi 写道:

From: Xuan Zhuo

The purpose of this feature is to split the header and the payload of
the packet.

|receive buffer|
|   0th descriptor | 1th descriptor|
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|   payload |

We can use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be
independently in a page, which is very beneficial for the zerocopy
implemented by the upper layer.

Signed-off-by: Xuan Zhuo
Signed-off-by: Heng Qi
Reviewed-by: Kangjie Xu
---
v7:
1. Fix some presentation issues.
2. Use "split transport header". @Jason Wang
3. Clarify some paragraphs. @Cornelia Huck
4. determine the device what to do if it does not perform header split 
on a packet.

v6:
1. Fix some syntax issues. @Cornelia Huck
2. Clarify some paragraphs. @Cornelia Huck
3. Determine the device what to do if it does not perform header split 
on a packet.

v5:
1. Determine when hdr_len is credible in the process of rx
2. Clean up the use of buffers and descriptors
3. Clarify the meaning of used lenght if the first descriptor is 
skipped in the case of merge

v4:
1. fix typo @Cornelia Huck @Jason Wang
2. do not split header for IP fragmentation packet. @Jason Wang

v3:
1. Fix some syntax issues
2. Fix some terminology issues
3. It is not unified with ip alignment, so ip alignment is not included
4. Make it clear that the device must support four types, in the case 
of successful negotiation.

  conformance.tex |   2 ++
  content.tex | 102 
  2 files changed, 104 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 2b86fc6..4e2b82e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -150,6 +150,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Driver Conformance}\label{sec:Conformance / Driver Conformance / Block Driver Conformance}

@@ -415,6 +416,7 @@ \section{Conformance Targets}\label{sec:Conformance / 
Conformance Targets}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / 
Control Virtqueue / Split Transport Header}
  \end{itemize}
  
  \conformance{\subsection}{Block Device Conformance}\label{sec:Conformance / Device Conformance / Block Device Conformance}

diff --git a/content.tex b/content.tex
index e863709..5676da9 100644
--- a/content.tex
+++ b/content.tex
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / 
Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
  channel.
  
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER (52)] Device supports splitting

+the transport header and the payload.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
  
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device 
Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
  
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -3371,6 +3375,7 @@ \subsection{Device Operation}\label{sec:Device Types / 
Network Device / Device O
  #define VIRTIO_NET_HDR_F_NEEDS_CSUM1
  #define VIRTIO_NET_HDR_F_DATA_VALID2
  #define VIRTIO_NET_HDR_F_RSC_INFO  4
+#define VIRTIO_NET_HDR_F_SPLIT_TRANSPORT_HEADER  8
  u8 

[virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/9/5 上午4:27, Michael S. Tsirkin 写道:

On Fri, Sep 02, 2022 at 03:36:25PM +0800, Heng Qi wrote:

We need to clarify that the purpose of header splitting is to make all payloads
can be independently in a page, which is beneficial for the zerocopy
implemented by the upper layer.

absolutely, pls add motivation.


If the driver does not enforce that the buffers submitted to the receiveq MUST
be composed of at least two descriptors, then header splitting will become 
meaningless,
or the VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER feature should not be negotiated at 
this time.


Thanks.




This seems very narrow and unecessarily wasteful of descriptors.
What is wrong in this:

.. 

seems to achieve the goal of data in a separate page without
using extra descriptors.

thus my proposal to replace the requirement of a separate
descriptor with an offset of data from beginning of
buffer that driver sets.



We have carefully considered your suggestion.

We refer to spec v7 and earlier as scheme A for short. Review scheme A 
below:


| receive buffer |

| 0th descriptor | 1th descriptor |

| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->| payload |

We use a buffer plus a separate page when allocating the receive

buffer. In this way, we can ensure that all payloads can be

independently in a page, which is very beneficial for the zerocopy

implemented by the upper layer.

scheme A better solves the problem of headroom, tailroom and memory 
waste, but as you said, this solution relies on descriptor chain.


Our rethinking approach is no longer based on or using descriptor chain.

We refer to your proposed offset-based scheme as scheme B:

As you suggested, scheme B gives the device a buffer, using offset to 
indicate where to place the payload like this:


.. 

But how to apply for this buffer? Since we want the payload to be placed 
on a separate page, the method we consider is to directly apply to the 
driver for two pages of contiguous memory.


Then the beginning of this contiguous memory is used to store the 
headroom, and the contiguous memory after the headroom is directly 
handed over to the device. similar to the following:


<-- receive buffer(2 pages) 
->


<<-- first page 
--->< second page -->>


<header>>


Based on your previous suggestion, we also considered another new scheme C.

This scheme is implemented based on mergeable buffer, filling a separate 
page each time.


If the split header is negotiated and the packet can be successfully 
split by the device, the device needs to find at least two buffers, 
namely two pages, one for the virtio-net header and transport header, 
and the other for the data payload. Like the following:


| receive buffer1(page) | receive buffer2 (page) |

| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->| payload |

At the same time, if XDP is considered, then the device needs to add 
headroom at the beginning of receive buffer1 when receiving packets, so 
that the driver can process programs similar to XDP. In order to solve 
this problem, can scheme C introduce an offset, which requires the 
device to write data from the offset position to receive buffer1, like 
the following:


| receive buffer (page) | receive buffer (page) |

| <-- offset(hold) --> | virtnet hdr | mac | ip hdr | tcp hdr|<-- hold 
-->| payload |


Then we simply compare the advantages and disadvantages of scheme A(spec 
v7), scheme B (offset buffer(2 pages)) and scheme C (based on mergeable 
buffer):


1. desc chain:

- A depends on desciptor chain; - B, C do not depend on desciptor chain.

2. page alloc

- B fills two consecutive pages, which causes a great waste of memory 
for small packages such as arp; - C fills a single page, slightly better 
than B.


3. Memory waste:

- The memory waste of scheme A is mainly the 0th descriptor that is 
skipped by the device; - When scheme B and scheme C successfully split 
the header, there is a huge waste of the first page, but the first page 
can be quickly released by copying.


4. headroom

- The headrooms of plan A and plan B are reserved; - Scheme C requires 
the driver to set off to let the device skip off when using receive buffer1.


5. tailroom

- When splitting the header, skb usually needs to store each independent 
page in the non-linear data area based on shinfo. - The tailroom of 
scheme A is reserved by itself; - Scheme B requires the driver to set 
the reserved padding area for the first receive buffer(2 pages) to use 
shinfo when the split header is not successfully executed; - Scheme C 
requires the driver to set max_len for the first receive buffer(page).



Which plan do you prefer?

---

Thanks.


-
To unsubscribe, e-mail: virtio-dev-unsubsc

Re: [virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

On Sun, Sep 04, 2022 at 04:27:38PM -0400, Michael S. Tsirkin wrote:
> On Fri, Sep 02, 2022 at 03:36:25PM +0800, Heng Qi wrote:
> > We need to clarify that the purpose of header splitting is to make all 
> > payloads
> > can be independently in a page, which is beneficial for the zerocopy
> > implemented by the upper layer.
> 
> absolutely, pls add motivation.
> 
> > If the driver does not enforce that the buffers submitted to the receiveq 
> > MUST
> > be composed of at least two descriptors, then header splitting will become 
> > meaningless,
> > or the VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER feature should not be negotiated 
> > at this time.
> > 
> > 
> > Thanks.
> > 
> > 
> 
> 
> This seems very narrow and unecessarily wasteful of descriptors.
> What is wrong in this:
> 
> .. 
> 
> seems to achieve the goal of data in a separate page without
> using extra descriptors.
> 
> thus my proposal to replace the requirement of a separate
> descriptor with an offset of data from beginning of
> buffer that driver sets.
>


We have carefully considered your suggestion. 

Let's summarize the schemes we've considered before and now.

1. Scheme A ( refer to spec v7 )

We refer to spec v7 and earlier as scheme A for short. Review scheme A below: 
| receive buffer| 
|  0th descriptor  | 1th descriptor | 
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|  payload   | 

We use a buffer plus a separate page when allocating the receive
buffer. In this way, we can ensure that all payloads can be put
independently in a page, which is very beneficial for the zerocopy 
implemented by the upper layer. 

Scheme A better solves the problem of headroom, tailroom and
memory waste, but as you said, this solution relies on descriptor chain. 

2. Scheme B ( refer to your suggestion )

Our rethinking approach is no longer based on descriptor chain.
 
We refer to your proposed offset-based scheme as scheme B.
As you suggested, scheme B gives the device a buffer, using offset to
indicate where to place the payload. Like this: 

..  

But how to apply for this buffer?
Since we want the payload to be placed on a separate page, the method
we consider is to directly alloc two pages from driver of contiguous memory. 

Then the beginning of this contiguous memory is used to store the headroom,
and the contiguous memory after the headroom is directly handed over to the 
device.
Similar to the following: 

[-- receive buffer(2 pages) --] 
[<first page ---><-- second page >] 
[<-> ..<   payload >] 
   ^^
   ||
   |pointer to device
   |
   |
   Driver reserved, the later part is filled

3. Scheme C (this sheme we have sent to you on September 7th, maybe you miss 
it.)

Based on your previous suggestion, we also considered another new scheme C. 
This scheme is implemented based on mergeable buffer, filling a separate page 
each time. 

If the split header is negotiated and the packet can be successfully split by 
the device,
the device needs to find at least two buffers, namely two pages, one for the 
virtio-net header
and transport header, and the other for the payload. Like the following: 

|   receive buffer1(page)  | receive buffer2 (page) 
  | 
| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->| payload
  | 

At the same time, if XDP is considered, then the device needs to add headroom 
at the
beginning of receive buffer1 when receiving packets, so that the driver can 
process
programs similar to XDP. 

In order to solve this problem, scheme C introduce an offset which requires
the device to write data from the offset to receive buffer1, like the 
following: 

|   receive buffer (page) | 
receive buffer (page) | 
| <-- offset(hold) --> | virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->| 
payload   | 
^
|
pointer to device
   


4. Summarize

Then we simply compare the advantages and disadvantages of scheme A(spec v7),
scheme B (offset buffer(2 pages)) and scheme C (based on mergeable buffer): 

1). descriptor chain: 
 i)  A depends on desciptor chain;
 ii) B, C do not depend on desciptor chain. 

2). page alloc 
 i) B fills with two consecutive pages, which causes a great waste of 
memory
for small packages such as arp;
 ii) C fills with a single page, slightly better than B. 

3). Memory waste: 
 i) The memory waste of scheme A is mainly the 0th descriptor
that is skipped by the device;
 ii) When sch

Re: [virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

On Fri, Sep 09, 2022 at 08:47:57PM +0800, Xuan Zhuo wrote:
> 
> hi
>Qi also sent another same email today. Due to some email client problems,
>this email has some confusion in the format, so we can discuss
>under another one.
> 
>https://lists.oasis-open.org/archives/virtio-dev/202209/msg00066.html
> 

Yes. Due to some formatting issues with the mail client, I resent this new email
which may be in a clear style for your review.

Do you have more questions about the contents of this new email?

Looking forward to your comments.

Thanks.

> 
> -
> To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v7] virtio_net: support split header

[Heng Qi]

在 2022/9/28 上午5:35, Michael S. Tsirkin 写道:

On Wed, Sep 14, 2022 at 11:34:43AM +0800, Jason Wang wrote:

在 2022/9/9 20:38, Xuan Zhuo 写道:

On Fri, 9 Sep 2022 07:15:02 -0400, "Michael S. Tsirkin"  wrote:

On Fri, Sep 09, 2022 at 03:41:54PM +0800, Heng Qi wrote:

在 2022/9/5 上午4:27, Michael S. Tsirkin 写道:

On Fri, Sep 02, 2022 at 03:36:25PM +0800, Heng Qi wrote:

We need to clarify that the purpose of header splitting is to make all payloads
can be independently in a page, which is beneficial for the zerocopy
implemented by the upper layer.

absolutely, pls add motivation.


If the driver does not enforce that the buffers submitted to the receiveq MUST
be composed of at least two descriptors, then header splitting will become 
meaningless,
or the VIRTIO_NET_F_SPLIT_TRANSPORT_HEADER feature should not be negotiated at 
this time.


Thanks.



This seems very narrow and unecessarily wasteful of descriptors.
What is wrong in this:

.. 

seems to achieve the goal of data in a separate page without
using extra descriptors.

thus my proposal to replace the requirement of a separate
descriptor with an offset of data from beginning of
buffer that driver sets.



We have carefully considered your suggestion.

We refer to spec v7 and earlier as scheme A for short. Review scheme A
below:

| receive buffer |

| 0th descriptor | 1th descriptor |

| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->| payload |

We use a buffer plus a separate page when allocating the receive

buffer. In this way, we can ensure that all payloads can be

independently in a page, which is very beneficial for the zerocopy

implemented by the upper layer.

scheme A better solves the problem of headroom, tailroom and memory waste,
but as you said, this solution relies on descriptor chain.

Our rethinking approach is no longer based on or using descriptor chain.

We refer to your proposed offset-based scheme as scheme B:

As you suggested, scheme B gives the device a buffer, using offset to
indicate where to place the payload like this:

.. 

But how to apply for this buffer? Since we want the payload to be placed on
a separate page, the method we consider is to directly apply to the driver
for two pages of contiguous memory.

Then the beginning of this contiguous memory is used to store the headroom,
and the contiguous memory after the headroom is directly handed over to the
device. similar to the following:

<-- receive buffer(2 pages)
->

<<-- first page
--->< second page -->>

<>

Based on your previous suggestion, we also considered another new scheme C.

This scheme is implemented based on mergeable buffer, filling a separate
page each time.

If the split header is negotiated and the packet can be successfully split
by the device, the device needs to find at least two buffers, namely two
pages, one for the virtio-net header and transport header, and the other for
the data payload. Like the following:

| receive buffer1(page) | receive buffer2 (page) |

| virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->| payload |

At the same time, if XDP is considered, then the device needs to add
headroom at the beginning of receive buffer1 when receiving packets, so that
the driver can process programs similar to XDP. In order to solve this
problem, can scheme C introduce an offset, which requires the device to
write data from the offset position to receive buffer1, like the following:

| receive buffer (page) | receive buffer (page) |

| <-- offset(hold) --> | virtnet hdr | mac | ip hdr | tcp hdr|<-- hold -->|
payload |

And in fact, B and C both use an offset now, right?

B: offset is used to get the position to place the payload.
C: The offset is used to reserve some space for the device, which the driver can
 use as headroom.

 In order to make the payload page-aligned, we can only hand over the entire
 page to the device, so we cannot reserve some headroom in advance.


For C, it might be better to do some tweak since mergeable buffer doesn't
forbid using a descriptor chain as a single buffer.

So if it's a descriptor chain we got back the method A by placing the
payload in a dedicated buffer. If it's not placing the payload in an
adjacent buffer.

Thanks

Let's find a way so devices do not care how descriptors are laid out.


Hi, I'm not sure if you're replying to the wrong email (now in v7 thread),
if not, I'm guessing you mean we continue to discuss a way for devices
to not care how descriptors are laid out based on split header v7,
and no longer consider the new options B and C?


Thanks.




Then we simply compare the advantages and disadvantages of scheme A(spec
v7), scheme B (offset buffer(2 pages)) and scheme C (based on mergeable
buffer):

1. desc chain:

- A

Re: [virtio-dev] [PATCH v8] virtio_net: support for split transport header

[Heng Qi]

On Sat, Oct 08, 2022 at 12:37:45PM +0800, Jason Wang wrote:
> On Thu, Sep 29, 2022 at 3:04 PM Michael S. Tsirkin  wrote:
> >
> > On Thu, Sep 29, 2022 at 09:48:33AM +0800, Jason Wang wrote:
> > > On Wed, Sep 28, 2022 at 9:39 PM Michael S. Tsirkin  
> > > wrote:
> > > >
> > > > On Mon, Sep 26, 2022 at 04:06:17PM +0800, Jason Wang wrote:
> > > > > > Jason I think the issue with previous proposals is that they 
> > > > > > conflict
> > > > > > with VIRTIO_F_ANY_LAYOUT. We have repeatedly found that giving the
> > > > > > driver flexibility in arranging the packet in memory is benefitial.
> > > > >
> > > > >
> > > > > Yes, but I didn't found how it can conflict the any_layout. Device 
> > > > > can just
> > > > > to not split the header when the layout doesn't fit for header 
> > > > > splitting.
> > > > > (And this seems the case even if we're using buffers).
> > > >
> > > > Well spec says:
> > > >
> > > > indicates to both the device and the driver that no
> > > > assumptions were made about framing.
> > > >
> > > > if device assumes that descriptor boundaries are where
> > > > driver wants packet to be stored that is clearly
> > > > an assumption.
> > >
> > > Yes but what I want to say is, the device can choose to not split the
> > > packet if the framing doesn't fit. Does it still comply with the above
> > > description?
> > >
> > > Thanks
> >
> > The point of ANY_LAYOUT is to give drivers maximum flexibility.
> > For example, if driver wants to split the header at some specific
> > offset this is already possible without extra functionality.
> 
> I'm not sure how this would work without the support from the device.
> This probably can only work if:
> 
> 1) the driver know what kind of packet it can receive
> 2) protocol have fixed length of the header
> 
> This is probably not true consider:
> 
> 1) TCP and UDP have different header length
> 2) IPv6 has an variable length of the header
> 
> 
> >
> > Let's keep it that way.
> >
> > Now, let's formulate what are some of the problems with the current way.
> >
> >
> >
> > A- mergeable buffers is even more flexible, since a single packet
> >   is built up of multiple buffers. And in theory device can
> >   choose arbitrary set of buffers to store a packet.
> >   So you could supply a small buffer for headers followed by a bigger
> >   one for payload, in theory even without any changes.
> >   Problem 1: However since this is not how devices currently operate,
> >   a feature bit would be helpful.
> 
> How do we know the bigger buffer is sufficient for the packet? If we
> try to allocate 64K (not sufficient for the future even) it breaks the
> effort of the mergeable buffer:
> 
> header buffer #1
> payload buffer #1
> header buffer #2
> payload buffer #2
> 
> Is the device expected to
> 
> 1) fill payload in header buffer #2, this breaks the effort that we
> want to make payload page aligned
> 2) skip header buffer #2, in this case, the device assumes the framing
> when it breaks any layout
> 
> >
> >   Problem 2: Also, in the past we found it useful to be able to figure out 
> > whether
> >   packet fits in a single buffer without looking at the header.
> >   For this reason, we have this text:
> >
> > If a receive packet is spread over multiple buffers, the device
> > MUST use all buffers but the last (i.e. the first 
> > \field{num_buffers} -
> > 1 buffers) completely up to the full length of each buffer
> > supplied by the driver.
> >
> >   if we want to keep this optimization and allow using a separate
> >   buffer for headers, then I think we could rely on the feature bit
> >   from Problem 1 and just make an exception for the first buffer.
> >   Also num_buffers is then always >= 2, maybe state this to avoid
> >   confusion.
> >
> >
> >
> >
> >
> > B- without mergeable, there's no flexibility. In particular, there can
> > not be uninitialized space between header and data.
> 
> I had two questions
> 
> 1) why is this not a problem of mergeable? There's no guarantee that
> the header is just the length of what the driver allocates for header
> buffer anyhow
> 
> E.g the header length could be smaller than the header buffer, the
> device still needs to skip part of the space in the header buffer.
> 
> 2) it should be the responsibility of the driver to handle the
> uninitialized space, it should do anything that is necessary for
> security, more below
> 


We've talked a bit more about split header so far, but there still seem to
be some issues, so let's recap.

一、 Method Discussion Review

In order to adapt to the Eric's tcp receive interface to achieve zero copy,
header and payload are required to be stored separately, and the payload is
stored in a paged alignment way. Therefore, we have discussed several options
for split header as follows:

1: method A ( depend on the descriptor chain )
| receive buffer| 
|  0th descriptor

[virtio-dev] Re: [PATCH 1/2] virtio_net: fix syntax errors

[Heng Qi]

在 2022/11/9 下午9:23, Michael S. Tsirkin 写道:

On Wed, Nov 09, 2022 at 07:35:14PM +0800, Heng Qi wrote:

Please ignore this email.

Thanks.

ok. if 2/2 is still relevant pls post it separately.


I sent a new patch set with a cover-letter, please see in 
https://lists.oasis-open.org/archives/virtio-dev/202211/msg00025.html.





-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH v3] virtio_net: support inner header hash

[Heng Qi]

在 2022/12/16 下午8:49, Michael S. Tsirkin 写道:

On Mon, Dec 05, 2022 at 02:36:39PM +0800, Heng Qi wrote:

@@ -4005,6 +4159,24 @@ \subsubsection{Processing of Incoming 
Packets}\label{sec:Device Types / Network
  #define VIRTIO_NET_HASH_REPORT_UDPv6_EX9
  \end{lstlisting}
  
+If \field{hash_report} differs from VIRTIO_NET_HASH_REPORT_NONE,

+\field{hash_report_tunnel} can report the type of the tunnel-encapsulated
+packet to the driver over the inner header hash calculation.
+Possible values that the device can report in \field{hash_report_tunnel}
+are defined below:
+
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_REPORT_GRE 1
+#define VIRTIO_NET_HASH_REPORT_VXLAN   2
+#define VIRTIO_NET_HASH_REPORT_GENEVE  3
+\end{lstlisting}
+
+The values VIRTIO_NET_HASH_REPORT_GRE, VIRTIO_NET_HASH_REPORT_VXLAN and
+VIRTIO_NET_HASH_REPORT_GENEVE correspond to VIRTIO_NET_HASH_TYPE_GRE_INNER,
+VIRTIO_NET_HASH_TYPE_VXLAN_INNER and VIRTIO_NET_HASH_TYPE_GENEVE_INNER bits
+of supported hash types defined in respectively
+\ref{sec:Device Types / Network Device / Device Operation / Processing of 
Incoming Packets / Hash calculation for incoming packets / Supported/enabled 
hash types}.
+
  \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / 
Device Operation / Control Virtqueue}
  
  The driver uses the control virtqueue (if VIRTIO_NET_F_CTRL_VQ is

If the new feature flag is negotiated, we need to spell out more clearly what 
are the rules
for packets that are not encapsulated. Is hash_report_tunnel set to 0


Yes, we should. When the _TUNNEL feature is negotiated, for 
non-encapsulated packets we set \field{hash_report_tunnel} to 0.



then? Another comment is that we keep repeating GRE/VXLAN/GENEVE too
many times. Let's add a paragraph defining a concept e.g. a "tunnel" or
"tunneled packets", explaining how they are handled at a high level,
and then just refer to the tunnel everywhere.
Let's also add external references to specifications documenting the
relevant tunnel types.


Ok, I'll try to make this clear.

Thanks.





--
2.19.1.6.gb485710b



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] [PATCH] virtio_net: support low and high rate of notification coalescing sets

[Heng Qi]

在 2022/12/21 下午7:48, Alvaro Karsz 写道:

Hi,


I want to know which one is better than NetDim(Coalesce Adaptive) in driver.

I know Heng Qi's work in this.

Thanks


Why choose? we can have both.
ethtool can handle both pkt_rate_low/pkt_rate_high and
use_adaptive_rx_coalesce/use_adaptive_tx_coalesce.

The adaptive algorithm can even use this feature to set low and high
coalescing sets.


Hi, all.


NetDIM is currently a mature library in the kernel. It uses the number 
of bytes, PPS and interrupt rate


as samples to make an action, and it is performed independently in tx or 
rx direction. Also, There will


be an extra worker to help us send the configuration based on the 
control queue to avoid interrupting
the softirq. Although the method of pkt_rate_{low, high} params does not 
conflict with dim, I seem


to have some doubts that the colasce parameters of the rx and tx 
directions are determined at the


same time based on pkt_rate alone, will this be a problem?


Thanks.



Alvaro



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v7] virtio-net: support inner header hash

[Heng Qi]

On Tue, Jan 10, 2023 at 12:57:38AM -0500, Michael S. Tsirkin wrote:
> On Tue, Jan 10, 2023 at 12:25:02AM -0500, Michael S. Tsirkin wrote:
> > > This will give extra pressure on the management stack, e.g it requires
> > > the device to have an out of spec way for introspection.
> > > 
> > > Thanks
> > 
> > As I tried to explain this is already the case. Feature bits do not
> > describe device capabilities fully, some of them are in config space.
> 
> To be precise, this does not necessarily require introspection, but
> it does require management control over config space
> such as supported hash types just like it has control over feature bits.
> E.g. QEMU currently seems to hard-code these to
> #define VIRTIO_NET_RSS_SUPPORTED_HASHES (VIRTIO_NET_RSS_HASH_TYPE_IPv4 | \
>  VIRTIO_NET_RSS_HASH_TYPE_TCPv4 | \
>  VIRTIO_NET_RSS_HASH_TYPE_UDPv4 | \
>  VIRTIO_NET_RSS_HASH_TYPE_IPv6 | \
>  VIRTIO_NET_RSS_HASH_TYPE_TCPv6 | \
>  VIRTIO_NET_RSS_HASH_TYPE_UDPv6 | \
>  VIRTIO_NET_RSS_HASH_TYPE_IP_EX | \
>  VIRTIO_NET_RSS_HASH_TYPE_TCP_EX | \
>  VIRTIO_NET_RSS_HASH_TYPE_UDP_EX)
> 
> but there's no reason not to give management control over these.

Yes, QEMU has requirements for live migration: the PCI config space will be
checked in get_pci_config_device(), and if src and dst are inconsistent, it
will prompt that the live migration failed. 
In fact, this is also done within our group. Live migration requires that
the two VMs have the same rss configuration, otherwise the migration will fail.

Therefore, it seems that we can regularize the description of 
VIRTIO_NET_F_HASH_TUNNEL into
"[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash for 
tunnel-encapsulated packets.",
and use different hash_types to help the migration determine whether it can 
succeed.

Thanks.

> 
> -- 
> MST

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v7] virtio-net: support inner header hash

[Heng Qi]

在 2023/1/9 下午7:39, Michael S. Tsirkin 写道:

Btw this "are defined below" all over the place is just contributing
to making the spec unnecesarily verbose. Simple "are:" will do.


Sure. I'll fix it in the next version.

Thanks.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] Re: [PATCH v7] virtio-net: support inner header hash

[Heng Qi]

在 2023/1/10 下午3:26, Heng Qi 写道:

On Tue, Jan 10, 2023 at 12:57:38AM -0500, Michael S. Tsirkin wrote:

On Tue, Jan 10, 2023 at 12:25:02AM -0500, Michael S. Tsirkin wrote:

This will give extra pressure on the management stack, e.g it requires
the device to have an out of spec way for introspection.

Thanks

As I tried to explain this is already the case. Feature bits do not
describe device capabilities fully, some of them are in config space.

To be precise, this does not necessarily require introspection, but
it does require management control over config space
such as supported hash types just like it has control over feature bits.
E.g. QEMU currently seems to hard-code these to
#define VIRTIO_NET_RSS_SUPPORTED_HASHES (VIRTIO_NET_RSS_HASH_TYPE_IPv4 | \
  VIRTIO_NET_RSS_HASH_TYPE_TCPv4 | \
  VIRTIO_NET_RSS_HASH_TYPE_UDPv4 | \
  VIRTIO_NET_RSS_HASH_TYPE_IPv6 | \
  VIRTIO_NET_RSS_HASH_TYPE_TCPv6 | \
  VIRTIO_NET_RSS_HASH_TYPE_UDPv6 | \
  VIRTIO_NET_RSS_HASH_TYPE_IP_EX | \
  VIRTIO_NET_RSS_HASH_TYPE_TCP_EX | \
  VIRTIO_NET_RSS_HASH_TYPE_UDP_EX)

but there's no reason not to give management control over these.

Yes, QEMU has requirements for live migration: the PCI config space will be
checked in get_pci_config_device(), and if src and dst are inconsistent, it
will prompt that the live migration failed.


To be clearer, I mean \filed{supported_hash_types} in structure 
virtio_net_config.


Thanks.


In fact, this is also done within our group. Live migration requires that
the two VMs have the same rss configuration, otherwise the migration will fail.

Therefore, it seems that we can regularize the description of 
VIRTIO_NET_F_HASH_TUNNEL into
"[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash for 
tunnel-encapsulated packets.",
and use different hash_types to help the migration determine whether it can 
succeed.

Thanks.


--
MST

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscr...@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
List help: virtio-comment-h...@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: [virtio-dev] [PATCH v7] virtio-net: support inner header hash

[Heng Qi]

On Wed, Jan 18, 2023 at 11:45:39PM +, Parav Pandit wrote:
> 
> 
> > From: virtio-dev@lists.oasis-open.org  
> > Sent: Wednesday, January 4, 2023 2:14 AM
> 
> > If the tunnel is used to encapsulate the packets, the hash calculated using 
> > the
> > outer header of the receive packets is always fixed for the same flow 
> > packets,
> > i.e. they will be steered to the same receive queue.
> 
> > +\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner
> > +header hash for GRE, VXLAN and GENEVE tunnel-encapsulated packets.
> > +
> A device may not support all 3 at the same time.
> Please remove mentioning tunneling protocols description from here.
> Just say device support inner header hash ...

Sorry for the late reply due to vacation.

Good idea, Michael suggested doing the same. But we also discussed this issue:
Early, we used a feature bit to force devices to support GRE and VXLAN (see
https://lists.oasis-open.org/archives/virtio-dev/202211/msg00183.html ).
Later Jason suggested to use VIRTIO_NET_F_HASH_TUNNEL instead of 
VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER (see
https://lists.oasis-open.org/archives/virtio-dev/202212/msg00014.html ). Now 
Michael proposes to remove this list (see
https://lists.oasis-open.org/archives/virtio-dev/202301/msg00079.html ), 
because the migration is based on the feature
bit and hash types to determine whether the live migration is successful.
> 
> An additional bit map somewhere else should say supported hash over different 
> tunneling types.
> 

Yes, we use \field{supported_hash_types} to declare supported hash types.

> [...]
> 
> > +The device calculates the hash on the inner IPv4 packet of an
> > +encapsulated packet according to 'Enabled hash types' bitmask as follows:
> > +\begin{itemize}
> > +  \item If VIRTIO_NET_HASH_TYPE_TCPv4 is set and the encapsulated packet
> > has an inner
> > +   TCPv4 header, the hash is calculated over the following fields:
> > +\begin{itemsize}
> > +  \item inner Source IP address
> > +  \item inner Destination IP address
> > +  \item inner Source TCP port
> > +  \item inner Destination TCP port
> > +\end{itemsize}
> > +  \item Else if VIRTIO_NET_HASH_TYPE_UDPv4 is set and the encapsulated
> > packet has an
> > +   inner UDPv4 header, the hash is calculated over the following fields:
> > +\begin{itemsize}
> > +  \item inner Source IP address
> > +  \item inner Destination IP address
> > +  \item inner Source UDP port
> > +  \item inner Destination UDP port
> > +\end{itemize}
> > +  \item Else if VIRTIO_NET_HASH_TYPE_IPv4 is set, the hash is calculated 
> > over
> > the
> > +   following fields:
> > +\begin{itemsize}
> > +  \item inner Source IP address
> > +  \item inner Destination IP address
> > +\end{itemsize}
> > +  \item Else the device does not calculate the hash \end{itemize}
> > +
> > +The device calculates the hash on the inner IPv6 packet without an
> > +extension header of an encapsulated packet according to 'Enabled hash 
> > types'
> > bitmask as follows:
> > +\begin{itemize}
> > +  \item If VIRTIO_NET_HASH_TYPE_TCPv6 is set and the encapsulated packet
> > has an inner
> > +   TCPv6 header, the hash is calculated over the following fields:
> > +\begin{itemsize}
> > +  \item inner Source IPv6 address
> > +  \item inner Destination IPv6 address
> > +  \item inner Source TCP port
> > +  \item inner Destination TCP port
> > +\end{itemsize}
> > +  \item Else if VIRTIO_NET_HASH_TYPE_UDPv6 is set and the encapsulated
> > packet has an
> > +   inner UDPv6 header, the hash is calculated over the following fields:
> > +\begin{itemsize}
> > +  \item inner Source IPv6 address
> > +  \item inner Destination IPv6 address
> > +  \item inner Source UDP port
> > +  \item inner Destination UDP port
> > +\end{itemize}
> > +  \item Else if VIRTIO_NET_HASH_TYPE_IPv6 is set, the hash is calculated 
> > over
> > the
> > +   following fields:
> > +\begin{itemsize}
> > +  \item inner Source IPv6 address
> > +  \item inner Destination IPv6 address
> > +\end{itemsize}
> > +  \item Else the device does not calculate the hash \end{itemize}
> > +
> > +The device calculates the hash on the inner IPv6 packet with an
> > +extension header of an encapsulated packet according to 'Enabled hash 
> > types'
> > bitmask as follows:
> > +\begin{itemsize}
> > +  \item If VIRTIO_NET_HASH_TYPE_TCP_EX is set and the encapsulated packet
> > has an inner
> > +   TCPv6 header, the hash is calculated over the following fields:
> > +\begin{itemize}
> > +  \item Home address from the home address option in the inner IPv6
> > destination
> > +  options header. If the inner extension header is not 
> > present, use the
> > +  inner Source IPv6 address.
> > +  \item I

[virtio-dev] Re: [virtio-comment] RE: [virtio-dev] [PATCH v7] virtio-net: support inner header hash

[Heng Qi]

在 2023/2/2 上午11:55, Parav Pandit 写道:

From: virtio-comm...@lists.oasis-open.org  On Behalf Of Michael S. Tsirkin
Sent: Wednesday, February 1, 2023 1:57 AM

Also, this patch is adding two functionalities.
1. Inner header hash calculation of existing already defined hash
types 2. outer header hash for new type for GRE,VXLAN,GENEVE.
#1 should be in 1st patch.
#2 should be in 2nd patch.
This is better to review.

Parav, you come to this discussion pretty late. Asking to split up the patch 
when
it's v1/v2 is ok. Asking after others have already reviewed v6 is not you are
making review easier for yourself but re-review harder for others who already
have a mind map of the patch.

In this case unless we really want to enable these separately (and frankly I 
don't
see a good reason to) then splitting it up makes review more confusing.


No. There is no need to enable it separately.
It was hard to parse new inner type decoding addition which has close to zero 
relation to outer headers.

As you say, it has some history, I don't have strong opinion to split.
But going forward in subsequent work, it is better to see logical changes in 
multiple patches.


It seems that we don't need to emphasize the outer header hash, which is the
same behavior as usual when the inner header hash is not raised.

Thanks.



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] RE: [virtio-dev] [PATCH] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature

[Heng Qi]

在 2023/2/7 上午5:53, Parav Pandit 写道:



From: virtio-dev@lists.oasis-open.org  On
Behalf Of Michael S. Tsirkin

On Mon, Feb 06, 2023 at 07:13:43PM +, Parav Pandit wrote:



From: virtio-dev@lists.oasis-open.org
 On Behalf Of Alvaro Karsz

This patch makes several improvements to the notification coalescing
feature,
including:

- Consolidating virtio_net_ctrl_coal_tx and virtio_net_ctrl_coal_rx
   into a single struct, virtio_net_ctrl_coal, as they are identical.
- Emphasizing that the coalescing commands are best-effort.
- Defining the behavior of coalescing with regards to delivering
   notifications when a change occur.


Patch needs to do one thing at a time.
Please split above into three patches.


Signed-off-by: Alvaro Karsz 
---
  device-types/net/description.tex | 40
++--
  1 file changed, 22 insertions(+), 18 deletions(-)

diff --git a/device-types/net/description.tex
b/device-types/net/description.tex
index 1741c79..2a98411 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -1514,15 +1514,12 @@ \subsubsection{Control
Virtqueue}\label{sec:Device Types / Network Device / Devi  If the
VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can  send
control commands for dynamically changing the coalescing parameters.

-\begin{lstlisting}
-struct virtio_net_ctrl_coal_rx {
-le32 rx_max_packets;
-le32 rx_usecs;
-};
+Note: In general, these commands are best-effort: A device could
+send a
notification even if it is not supposed to.


Please remove this note from this patch.
Instead of Note, we need to describe this device requirements description.
We better need to describe the motivation for it.
We may want to say there may be jitter in notification, but device should not

be sending when it is not supposed to.

It's explicitly allowed:

split-ring.tex:The driver MUST handle spurious notifications from the device.
split-ring.tex:The device MUST handle spurious notifications from the driver.


The intent is the guide to device implementation to have less spurious 
interrupts.
Best effort wording says that device is free to implement timer of any 
granularity of choice, which kind of defeats the purpose of IM.

So, both can handle/generate spurious notifications, but it shouldn't be best 
line guidance.




I also have more description to add in this area with regards to GSO and LRO.
I make humble suggestion that we draft is jointly in separate patch combining

these clarifications.

-struct virtio_net_ctrl_coal_tx {
-le32 tx_max_packets;
-le32 tx_usecs;
+\begin{lstlisting}
+struct virtio_net_ctrl_coal {
+le32 max_packets;
+le32 usecs;
  };


This is one good change to go as separate patch.


  #define VIRTIO_NET_CTRL_NOTF_COAL 6 @@ -1532,25 +1529,25 @@
\subsubsection{Control Virtqueue}\label{sec:Device Types / Network
Device / Devi

  Coalescing parameters:
  \begin{itemize}
-\item \field{rx_usecs}: Maximum number of usecs to delay a RX

notification.

-\item \field{tx_usecs}: Maximum number of usecs to delay a TX

notification.

-\item \field{rx_max_packets}: Maximum number of packets to receive
before a RX notification.
-\item \field{tx_max_packets}: Maximum number of packets to send
before a TX notification.
+\item \field{usecs} for RX: Maximum number of usecs to delay a RX
notification.
+\item \field{usecs} for TX: Maximum number of usecs to delay a TX
notification.
+\item \field{max_packets} for RX: Maximum number of packets to
+receive
before a RX notification.
+\item \field{max_packets} for TX: Maximum number of packets to send
+before
a TX notification.
  \end{itemize}


s/for Rx/For receive virtqueue
s/for Tx/For transmit virtqueue

Which virtqueue? It says TX/RX pretty consistently in this text.
Changing to receive virtqueue/transmit virtqueue would be a big change and
frankly for a very modest gain in readability.
Rather maybe just say RX/TX where we describe virtqueue.


We are describing rest of the previous sections as transmitq, receiveq.
Would like to keep this section consistent with rest of the Network device 
section, and not just notification coalescing section.


  The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
  \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs}
and \field{tx_max_packets} parameters.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs}
and \field{rx_max_packets} parameters.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{usecs} and
\field{max_packets} parameters for TX.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{usecs} and
\field{max_packets} parameters for RX.
  \end{enumerate}

  \subparagraph{RX Notifications}\label{sec:Device Types / Network
Device / Device Operation / Control Virtqueue / Notifications
Coalescing / RX Notifications}

  If, for example:
  \begin{itemize}
-\item \field{rx_usecs} = 10.
-\item \field{rx_max_packets} = 15.
+\item \field{usecs} = 10.
+\item \field{max_packets} = 15.
  

Re: [virtio-dev] Re: [virtio-comment] [PATCH] virtio-net: support per-queue coalescing moderation

[Heng Qi]

在 2023/2/8 下午6:10, Michael S. Tsirkin 写道:

On Wed, Feb 08, 2023 at 09:57:54AM +0800, Heng Qi wrote:

I think it's a good idea to do this on top of Alvaro's patch
unifying these two structures.

I saw Alvaro's patch, but it doesn't seem to be stable yet, is there a good
way for me to
unify the two structures, since a patch should only do one thing.

Problem is you were trying to change these existing structures too
so the patches conflicted. However I think at this point
we are in agreement on a new command with a new structure.
In this case there won't be a conflict as you are
not touching existing commands so no need for you
to depend on Alvaro's patch.


I get it.

Thanks.






-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] [PATCH] virtio-net: support per-queue coalescing moderation

[Heng Qi]

在 2023/2/9 上午1:53, Alvaro Karsz 写道:

  > > From: Michael S. Tsirkin 

Sent: Wednesday, February 8, 2023 9:48 AM

On Wed, Feb 08, 2023 at 02:44:37PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:43 AM

On Wed, Feb 08, 2023 at 02:37:55PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:18 AM

On Wed, Feb 08, 2023 at 07:30:34PM +0800, Heng Qi wrote:

I see two options.
1. Just have per VQ params. Software has the full knowledge
of in which it is

operating, and state remains at software level.

This effectively achieves both the mode.

2. Have a mode cmd,
Mode = (a) per device or (b) per VQ (c) disable After the
mode is set, driver can set per device or per VQ.

I find this more clear.

Thanks.


Rereading this I think I misunderstood the proposal.
Now we are burning memory on maintaining mode, and this
information is duplicated.


It is not maintained in the pci resident memory, so it doesn't hurt.


I'd say let's just add a new command COAL_QUEUE_SET with vqn as

parameter.

Existing commands are simply defined as a shortcut to running
COAL_QUEUE_SET on all tx/rx queues respectively.

Latest command dictates the parameters. To disable just set
everything to 0 (btw we should make this explicit in the spec,
but it can be

guessed from:

Upon reset, a device MUST initialize all coalescing parameters to 0.
)


Switching between the modes (per q vs per device) implicitly is
ambiguous

and it only means device may need to iterate.

hmm i feel it's only ambiguous because i failed to explain in well.


This state is either better maintained in sw by always having per
vq or have

clearly defined mode of what device should do.

Per Q is very common even for several years old devices.
Last time I counted, there were at least 15 such devices supporting it.

So actual usage wise, I practically see that most implementations
will end up

with per vq mode.

I like to hear from Heng or Alvaro if they see any use of per device.


Right so given this, most devices will be in per queue mode all the
time. why do you want a mode then? just keep per queue.
existing commands are kept around for compat but internally just
translate to per-queue.

Since the space is not released, do we need to keep the compat?

It's been accepted for half a year so we can't say for sure no one built this.

That is likely but we should have the ability to have the Errata/ECN to correct 
it, specially for unrelease spec.


The way I propose is just a bit of firmware on device that scans all queues and
copies same parameters everywhere.

This scanning loop in sw appears cheaper to me than some embedded fw.
But is not a lot of concern.


Seems easier than worrying about this,
and we get disabling coalescing for free which you wanted. With an extra mode
its extra logic in the device fast path. Maybe it's cheap on hardware side but 
in
software it's an extra branch, not free.

Most performant data path wouldn't implement and read the extra mode.
It is always fw that is going to program same value, or per queue valued or 
disable value in each Q regardless whichever way we craft the CVQ cmd.

The sequence that bothers me is below.
1. driver set global params
2. few minutes later, now driver set param for Q=1

On this command, a device need to decide:
Should Q = 2 to N
(a) either work with previous globals, or
(b) because per Q was set for one queue, they rest of the queues implicitly 
disable it.

If it is (b),
When a command on Q object =1 is issued, it affects other Q objects. <- This I 
want to avoid.
A cmd that modifies the object, should only modify that object.

If it is (a), it is mixed mode operation, which is ambiguous definition.

A better semantic is to define such change at device level and no extra cost in 
the data path.

I think that (a) is the way to go.
I don't think that we should work with operation modes at all.


I agree to keep the current global settings (VIRTIO_NET_F_NOTF_COAL and 
its corresponding commands),
because our hardware team has limited resources for the control queue, 
and they don't want to send a separate

cmd for each queue when send a global setting cmd.

Then adding a VIRTIO_NET_F_PERQUEUE_NOTF_COAL or 
VIRTIO_NET_F_VQ_NOTF_COAL feature bit and new commands for per-queue or 
VQ setting looks better to me.

In my opinion:

We should have 2 features:
VIRTIO_NET_F_PERQUEUE_NOTF_COAL and VIRTIO_NET_F_NOTF_COAL.

VIRTIO_NET_F_PERQUEUE_NOTF_COAL sets per queue parameters, and
VIRTIO_NET_F_NOTF_COAL sets parameters for all queues.

VIRTIO_NET_F_NOTF_COAL has 2 commands:
 VIRTIO_NET_CTRL_NOTF_COAL_RX_SET
 VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

VIRTIO_NET_F_PERQUEUE_NOTF_COAL has 2 commands:
 VIRTIO_NET_CTRL_NOTF_COAL_PER_QUEUE_TX_SET
 VIRTIO_NET_CTRL_NOTF_COAL_PER_QUEUE_RX_SET

We can see VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as a virtio level shortcut
for sett

Re: [virtio-dev] Re: [virtio-comment] [PATCH] virtio-net: support per-queue coalescing moderation

[Heng Qi]

在 2023/2/9 上午6:35, Alvaro Karsz 写道:

From: Alvaro Karsz 
Sent: Wednesday, February 8, 2023 4:56 PM

Alvaro,
Do you know if any software used it? Can you get some real data?

I implemented this feature in our DPU, so at least 1 vendor is using this 
feature

But which software (virtio net driver) in which OS is using this?

Sorry, I'm not sure I understand your question.

The feature is implemented in the linux kernel
https://github.com/torvalds/linux/commit/699b045a8e43bd1063db4795be685bfd659649dc
So we'll always have kernel versions accepting this feature, if offered.


(I will add support for the per vq command of course).
I really don't know about other vendors..

You are suggesting to reserve the command and feature bit for safety, so, if we
reserve them, why not just use them? What do we lose here?


If it is used by some unknown software, only that sw breaks by using non 
release spec.
If we use it by changing the definition, it may break that unknown sw.
If we know there is no known sw, we are better of with redefinition (by adding 
vqn, and by removing tx,rx from it).


Not having this feature/command even complicates things now that we are
talking about removing the RX and TX from the per vq command, how do you
change parameters to all TX queues? to all RX queues? we'll need 2 special
indexes, so we now need le32 to hold the queue index.

No need for special index.
How does a driver disable all queues or reset all queues? -> One by one.
So if user want to change for all TXQ, sw can do it one by one by iterating TXQ 
vqns.

Yes, but resetting the queues doesn't require a control command.
If a server has 64K queues, and a user wants to set all coalescing
parameters to X (maybe with ethtool), it will generate 64K control
commands..


At least our hardware design doesn't expect that.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH] virtio-net: support per-queue coalescing moderation

[Heng Qi]

在 2023/2/9 上午4:52, Michael S. Tsirkin 写道:

On Wed, Feb 08, 2023 at 07:53:09PM +0200, Alvaro Karsz wrote:

  > > From: Michael S. Tsirkin 

Sent: Wednesday, February 8, 2023 9:48 AM

On Wed, Feb 08, 2023 at 02:44:37PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:43 AM

On Wed, Feb 08, 2023 at 02:37:55PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:18 AM

On Wed, Feb 08, 2023 at 07:30:34PM +0800, Heng Qi wrote:

I see two options.
1. Just have per VQ params. Software has the full knowledge
of in which it is

operating, and state remains at software level.

This effectively achieves both the mode.

2. Have a mode cmd,
Mode = (a) per device or (b) per VQ (c) disable After the
mode is set, driver can set per device or per VQ.

I find this more clear.

Thanks.


Rereading this I think I misunderstood the proposal.
Now we are burning memory on maintaining mode, and this
information is duplicated.


It is not maintained in the pci resident memory, so it doesn't hurt.


I'd say let's just add a new command COAL_QUEUE_SET with vqn as

parameter.

Existing commands are simply defined as a shortcut to running
COAL_QUEUE_SET on all tx/rx queues respectively.

Latest command dictates the parameters. To disable just set
everything to 0 (btw we should make this explicit in the spec,
but it can be

guessed from:

Upon reset, a device MUST initialize all coalescing parameters to 0.
)


Switching between the modes (per q vs per device) implicitly is
ambiguous

and it only means device may need to iterate.

hmm i feel it's only ambiguous because i failed to explain in well.


This state is either better maintained in sw by always having per
vq or have

clearly defined mode of what device should do.

Per Q is very common even for several years old devices.
Last time I counted, there were at least 15 such devices supporting it.

So actual usage wise, I practically see that most implementations
will end up

with per vq mode.

I like to hear from Heng or Alvaro if they see any use of per device.


Right so given this, most devices will be in per queue mode all the
time. why do you want a mode then? just keep per queue.
existing commands are kept around for compat but internally just
translate to per-queue.

Since the space is not released, do we need to keep the compat?

It's been accepted for half a year so we can't say for sure no one built this.

That is likely but we should have the ability to have the Errata/ECN to correct 
it, specially for unrelease spec.


The way I propose is just a bit of firmware on device that scans all queues and
copies same parameters everywhere.

This scanning loop in sw appears cheaper to me than some embedded fw.
But is not a lot of concern.


Seems easier than worrying about this,
and we get disabling coalescing for free which you wanted. With an extra mode
its extra logic in the device fast path. Maybe it's cheap on hardware side but 
in
software it's an extra branch, not free.

Most performant data path wouldn't implement and read the extra mode.
It is always fw that is going to program same value, or per queue valued or 
disable value in each Q regardless whichever way we craft the CVQ cmd.

The sequence that bothers me is below.
1. driver set global params
2. few minutes later, now driver set param for Q=1

On this command, a device need to decide:
Should Q = 2 to N
(a) either work with previous globals, or
(b) because per Q was set for one queue, they rest of the queues implicitly 
disable it.

If it is (b),
When a command on Q object =1 is issued, it affects other Q objects. <- This I 
want to avoid.
A cmd that modifies the object, should only modify that object.

If it is (a), it is mixed mode operation, which is ambiguous definition.

A better semantic is to define such change at device level and no extra cost in 
the data path.

I think that (a) is the way to go.
I don't think that we should work with operation modes at all.

In my opinion:

We should have 2 features:
VIRTIO_NET_F_PERQUEUE_NOTF_COAL and VIRTIO_NET_F_NOTF_COAL.

VIRTIO_NET_F_PERQUEUE_NOTF_COAL sets per queue parameters, and
VIRTIO_NET_F_NOTF_COAL sets parameters for all queues.

VIRTIO_NET_F_NOTF_COAL has 2 commands:
 VIRTIO_NET_CTRL_NOTF_COAL_RX_SET
 VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

VIRTIO_NET_F_PERQUEUE_NOTF_COAL has 2 commands:
 VIRTIO_NET_CTRL_NOTF_COAL_PER_QUEUE_TX_SET
 VIRTIO_NET_CTRL_NOTF_COAL_PER_QUEUE_RX_SET

We can see VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as a virtio level shortcut
for setting all queues with one command, exactly as intended with
rx_qid= 0x, and without breaking devices following the current
spec.

The device's FW can decide if it stores parameters received with
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET in a global set, or if it iterates
through all queues, but IMO the best way it to iterate through all
queues.

Seems like

Re: [virtio-dev] RE: [virtio-comment] [PATCH] virtio-net: support per-queue coalescing moderation

[Heng Qi]

在 2023/2/8 下午11:04, Parav Pandit 写道:



From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:48 AM

On Wed, Feb 08, 2023 at 02:44:37PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:43 AM

On Wed, Feb 08, 2023 at 02:37:55PM +, Parav Pandit wrote:

From: Michael S. Tsirkin 
Sent: Wednesday, February 8, 2023 9:18 AM

On Wed, Feb 08, 2023 at 07:30:34PM +0800, Heng Qi wrote:

I see two options.
1. Just have per VQ params. Software has the full knowledge
of in which it is

operating, and state remains at software level.

This effectively achieves both the mode.

2. Have a mode cmd,
Mode = (a) per device or (b) per VQ (c) disable After the
mode is set, driver can set per device or per VQ.

I find this more clear.

Thanks.


Rereading this I think I misunderstood the proposal.
Now we are burning memory on maintaining mode, and this
information is duplicated.


It is not maintained in the pci resident memory, so it doesn't hurt.


I'd say let's just add a new command COAL_QUEUE_SET with vqn as

parameter.

Existing commands are simply defined as a shortcut to running
COAL_QUEUE_SET on all tx/rx queues respectively.

Latest command dictates the parameters. To disable just set
everything to 0 (btw we should make this explicit in the spec,
but it can be

guessed from:

Upon reset, a device MUST initialize all coalescing parameters to 0.
)


Switching between the modes (per q vs per device) implicitly is
ambiguous

and it only means device may need to iterate.

hmm i feel it's only ambiguous because i failed to explain in well.


This state is either better maintained in sw by always having per
vq or have

clearly defined mode of what device should do.

Per Q is very common even for several years old devices.
Last time I counted, there were at least 15 such devices supporting it.

So actual usage wise, I practically see that most implementations
will end up

with per vq mode.

I like to hear from Heng or Alvaro if they see any use of per device.


Right so given this, most devices will be in per queue mode all the
time. why do you want a mode then? just keep per queue.
existing commands are kept around for compat but internally just
translate to per-queue.

Since the space is not released, do we need to keep the compat?

It's been accepted for half a year so we can't say for sure no one built this.

That is likely but we should have the ability to have the Errata/ECN to correct 
it, specially for unrelease spec.


The way I propose is just a bit of firmware on device that scans all queues and
copies same parameters everywhere.

This scanning loop in sw appears cheaper to me than some embedded fw.
But is not a lot of concern.


Seems easier than worrying about this,
and we get disabling coalescing for free which you wanted. With an extra mode
its extra logic in the device fast path. Maybe it's cheap on hardware side but 
in
software it's an extra branch, not free.

Most performant data path wouldn't implement and read the extra mode.
It is always fw that is going to program same value, or per queue valued or 
disable value in each Q regardless whichever way we craft the CVQ cmd.

The sequence that bothers me is below.
1. driver set global params
2. few minutes later, now driver set param for Q=1

On this command, a device need to decide:
Should Q = 2 to N
(a) either work with previous globals, or
(b) because per Q was set for one queue, they rest of the queues implicitly 
disable it.

If it is (b),
When a command on Q object =1 is issued, it affects other Q objects. <- This I 
want to avoid.
A cmd that modifies the object, should only modify that object.

If it is (a), it is mixed mode operation, which is ambiguous definition.


I think it should be a. I think we should blur the concept of mode. 
There seems to be no mode here.
From the perspective of the device, it only needs to distinguish 
commands and do what it should do.


Thanks.



A better semantic is to define such change at device level and no extra cost in 
the data path.

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v2] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

在 2023/2/10 下午6:16, Alvaro Karsz 写道:

So, should we remove VIRTIO_NET_F_CTRL_VQ here, or fix VIRTIO_NET_F_HOST_ECN?

Ah good point.
But I think  VIRTIO_NET_F_VQ_NOTF_COAL should not depend on 
VIRTIO_NET_F_NOTF_COAL.
This way devices can drop the all-rx/all-tx commands if they want to.

We need to confirm this. If we make VIRTIO_NET_F_VQ_NOTF_COAL
independent of VIRTIO_NET_F_NOTF_COAL,
do we need to give vqn a special value so that the driver can also have
the fast path of sending all queues with global settings via
VIRTIO_NET_F_VQ_NOTF_COAL?

IMO we don't need a special vqn value.
A device that can modify all the vqs should offer VIRTIO_NET_F_NOTF_COAL.


That's clear, thanks for the quick reply.
Have a great weekend!


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v2] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

在 2023/2/11 下午4:45, Alvaro Karsz 写道:

Please add short description something like,

When the driver prefers to use per virtqueue notifications coalescing, and if 
queue group (transmit or receive) level notification coalescing is enabled, 
driver SHOULD first disable device level notification coalescing.
Or it should be,


I disagree here.
IMO "queue group level notification coalescing" is not something to
enable or disable, but a shortcut to set all TX/RX queues at once.
Why should the spec force a driver to "disable device level
notification coalescing" (I assume you mean send a
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET command with zeros)?
What if the driver sends a VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET
command, and then a single queue traffic increases? why should it zero
the parameters to all other queues?


Hi, Alvaro! Thanks for your reply!

I think Parav refers more to the scene where ethool sets parameters at 
the queue group level and
the scene where netdim sets parameters for a single queue. In this 
scenario, netdim should really
determine the coalescing parameters of the device, and the parameters at 
the queue group level set
by ethtool should be ignored (many drivers are designed this way, such 
as mlx), that is, we need to give netdim the right,
because it It has the ability to dynamically adjust parameters. 
(However, I think this friendly constraint is also possible in driver 
implementation.)


Of course, if we consider setting coalescing parameters at the queue 
group level and single queue level separately through ethtool,

then as you said, we should not set any priority for them.

Back to reality, I think the function of ethtool to set single queue 
parameters may come later, which is thankless for users because of netdim.


Therefore, if our specification tends to be practical, we can add 
Parav's proposal, and if our specification tends to be more general, 
then hand over

the constraints to the driver implementation. what do you think?


I think that this should be discussed in the driver implementation
stage, not in the spec.


Virtqueue level notifications coalescing, and device level notifications can be 
enabled together.
When both of them are enabled, per virtqueue notifications coalescing take 
priority over queue group level.

How do you enable  Virtqueue level notifications coalescing? Why are
they different entities?
I don't think that we should have priorities, but the last command
should be the one that dictates the coalescing parameters.

For example, let's look at vq0 (RX):
Device receives VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, vq0 should change
the parameters accordingly (all RX vqs should do the same).
Then device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with vqn = 0,
vq0 changes the parameters accordingly (all RX vqs are still using the
"old" parameters)
Then device receives VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, vq0 changes the
parameters accordingly (all RX vqs should do the same).


Yes I see what you mean, thanks for the clear example. This should be 
the second scenario I described above,

let's discuss how to solve it in the above reply.

Thanks! :)



This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscr...@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscr...@lists.oasis-open.org
List help: virtio-comment-h...@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/



-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH v2] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

On Sat, Feb 11, 2023 at 01:47:16PM +, Parav Pandit wrote:
> 
> 
> > From: Alvaro Karsz 
> > Sent: Saturday, February 11, 2023 3:45 AM
> > 
> > > Please add short description something like,
> > >
> > > When the driver prefers to use per virtqueue notifications coalescing, 
> > > and if
> > queue group (transmit or receive) level notification coalescing is enabled, 
> > driver
> > SHOULD first disable device level notification coalescing.
> > > Or it should be,
> > >
> > 
> > I disagree here.
> > IMO "queue group level notification coalescing" is not something to enable 
> > or
> > disable, but a shortcut to set all TX/RX queues at once.
> That short cut is the enable/disablement.
> 
> > Why should the spec force a driver to "disable device level notification
> > coalescing" (I assume you mean send a
> > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET command with zeros)?
> Yes. Because to have well defined behavior when sw configured both one after 
> the another.
> 
> > What if the driver sends a VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET
> > command, and then a single queue traffic increases? why should it zero the
> > parameters to all other queues?
> That is short transition when driver is switching over to per queue mode.
> This is fine to have short glitch.
> 
> > I think that this should be discussed in the driver implementation stage, 
> > not in
> > the spec.
> > 
> There should be a clear guidance on how device should behave when both per q 
> and per device are configured.
> 
> > > Virtqueue level notifications coalescing, and device level notifications 
> > > can be
> > enabled together.
> > > When both of them are enabled, per virtqueue notifications coalescing take
> > priority over queue group level.
> > 
> > How do you enable  Virtqueue level notifications coalescing? Why are they
> > different entities?
> Using the new command that has vqn in it.
> 
> > I don't think that we should have priorities, but the last command should 
> > be the
> > one that dictates the coalescing parameters.
> > 
> Priority is applicable when driver has issued both the commands. Per tx/rx, 
> and per vqn.
> 
> > For example, let's look at vq0 (RX):
> > Device receives VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, vq0 should change
> > the parameters accordingly (all RX vqs should do the same).
> > Then device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with vqn = 0,
> > vq0 changes the parameters accordingly (all RX vqs are still using the "old"
> > parameters) Then device receives VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, vq0
> > changes the parameters accordingly (all RX vqs should do the same).
> In this example, per VQ were overridden with per device.
> Yes, so the last one is applicable, so priority of last one applies.
> 
> We continue to refuse to add the mode, and hence need to supply these 
> description of both the sequence on how device should behave.
> 
> Sequence_1:
> 1. tx/rx group level
> 2. per vqn level
> When #2 is done, VQ's whose per vq level is configured, follows vqn, rest of 
> the VQs follow #1.
> 
> Sequence_2:
> 1. per vqn
> 2. tx/rx group level
> When #2 is done, group level overrides the per vqn parameters. 

Adding examples of the two command sequences is due, I will do so in the next 
release. :)

Thanks.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [PATCH v2] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

On Sat, Feb 11, 2023 at 06:13:55PM +0200, Alvaro Karsz wrote:
> I think that I wasn't clear enough.
> 
> I'm not saying that we should not define in the spec how to handle a
> situation when a device receives both  RX_SET and VQ_SET (or a driver
> sends both).
> I'm saying that I don't think that the driver should handle the
> situation the way you described it:
> 
> > When the driver prefers to use per virtqueue notifications coalescing, and 
> > if queue group (transmit or receive) level notification coalescing is 
> > enabled, driver SHOULD first disable device level notification coalescing.
> > Or it should be,
> >
> > Virtqueue level notifications coalescing, and device level notifications 
> > can be enabled together.
> > When both of them are enabled, per virtqueue notifications coalescing take 
> > priority over queue group level.
> 
> This implies that we have 2 modes and have priorities.
> 
> I think that if we want to refer to this situation in the spec, it
> should be something like:
> "A Device should use the last coalescing parameters received for a
> virtqueue, regardless of the command used to deliver the parameters."

Your suggestion is good, the per-device command and the per-queue command need 
some examples and behavior definitions,
I will add them to avoid some misunderstandings.

Thanks.

> (just an example to make the point).

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH v2] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

On Sun, Feb 12, 2023 at 04:35:37AM -0500, Michael S. Tsirkin wrote:
> On Sat, Feb 11, 2023 at 01:47:16PM +, Parav Pandit wrote:
> > 
> > 
> > > From: Alvaro Karsz 
> > > Sent: Saturday, February 11, 2023 3:45 AM
> > > 
> > > > Please add short description something like,
> > > >
> > > > When the driver prefers to use per virtqueue notifications coalescing, 
> > > > and if
> > > queue group (transmit or receive) level notification coalescing is 
> > > enabled, driver
> > > SHOULD first disable device level notification coalescing.
> > > > Or it should be,
> > > >
> > > 
> > > I disagree here.
> > > IMO "queue group level notification coalescing" is not something to 
> > > enable or
> > > disable, but a shortcut to set all TX/RX queues at once.
> > That short cut is the enable/disablement.
> > 
> > > Why should the spec force a driver to "disable device level notification
> > > coalescing" (I assume you mean send a
> > > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET command with zeros)?
> > Yes. Because to have well defined behavior when sw configured both one 
> > after the another.
> > 
> > > What if the driver sends a VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET
> > > command, and then a single queue traffic increases? why should it zero the
> > > parameters to all other queues?
> > That is short transition when driver is switching over to per queue mode.
> > This is fine to have short glitch.
> > 
> > > I think that this should be discussed in the driver implementation stage, 
> > > not in
> > > the spec.
> > > 
> > There should be a clear guidance on how device should behave when both per 
> > q and per device are configured.
> > 
> > > > Virtqueue level notifications coalescing, and device level 
> > > > notifications can be
> > > enabled together.
> > > > When both of them are enabled, per virtqueue notifications coalescing 
> > > > take
> > > priority over queue group level.
> > > 
> > > How do you enable  Virtqueue level notifications coalescing? Why are they
> > > different entities?
> > Using the new command that has vqn in it.
> > 
> > > I don't think that we should have priorities, but the last command should 
> > > be the
> > > one that dictates the coalescing parameters.
> > > 
> > Priority is applicable when driver has issued both the commands. Per tx/rx, 
> > and per vqn.
> > 
> > > For example, let's look at vq0 (RX):
> > > Device receives VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, vq0 should change
> > > the parameters accordingly (all RX vqs should do the same).
> > > Then device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with vqn = 0,
> > > vq0 changes the parameters accordingly (all RX vqs are still using the 
> > > "old"
> > > parameters) Then device receives VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, vq0
> > > changes the parameters accordingly (all RX vqs should do the same).
> > In this example, per VQ were overridden with per device.
> > Yes, so the last one is applicable, so priority of last one applies.
> > 
> > We continue to refuse to add the mode, and hence need to supply these 
> > description of both the sequence on how device should behave.
> > 
> > Sequence_1:
> > 1. tx/rx group level
> > 2. per vqn level
> > When #2 is done, VQ's whose per vq level is configured, follows vqn, rest 
> > of the VQs follow #1.
> > 
> > Sequence_2:
> > 1. per vqn
> > 2. tx/rx group level
> > When #2 is done, group level overrides the per vqn parameters. 
> 
> Since there's apparently some room for misunderstanding, I think adding these 
> examples can't hurt.

Ok, I'll handle this.

> I would also be more specific and just use specific numbers in the
> example, to avoid any ambiguity.

Agree.

Thanks.

> 
> -- 
> MST

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [PATCH v4] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature

[Heng Qi]

在 2023/2/16 下午3:32, Alvaro Karsz 写道:

This patch makes several improvements to the notification coalescing
feature, including:

- Consolidating virtio_net_ctrl_coal_tx and virtio_net_ctrl_coal_rx
   into a single struct, virtio_net_ctrl_coal, as they are identical.
- Emphasizing that the coalescing commands are best-effort.
- Defining the behavior of coalescing with regards to delivering
   notifications when a change occur.
- Stating that the commands should apply to all the receive/transmit
   virtqueues.
- Stating that every receive/transmit virtqueue should count it's own
   packets.
- A new intro explaining the entire coalescing operation.

Signed-off-by: Alvaro Karsz 
---
v2:
- Add the last 2 points to the patch.
- Rephrase the "commands are best-effort" note.
- Replace "notification" with "used buffer notification" to be
  more consistent.
v3:
- Add an intro explaining the entire coalescing operation.
v4:
- Minor wording fixes.
- Rephrase the general note.

  device-types/net/description.tex | 69 +++-
  1 file changed, 41 insertions(+), 28 deletions(-)

diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index 1741c79..11760f3 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -1514,15 +1514,15 @@ \subsubsection{Control Virtqueue}\label{sec:Device 
Types / Network Device / Devi
  If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
  send control commands for dynamically changing the coalescing parameters.
  
-\begin{lstlisting}

-struct virtio_net_ctrl_coal_rx {
-le32 rx_max_packets;
-le32 rx_usecs;
-};
+\begin{note}
+The behavior of the device in response to these commands is best-effort:
+the device may generate notifications more or less frequently than specified.
+\end{note}
  
-struct virtio_net_ctrl_coal_tx {

-le32 tx_max_packets;
-le32 tx_usecs;
+\begin{lstlisting}
+struct virtio_net_ctrl_coal {
+le32 max_packets;
+le32 usecs;
  };
  
  #define VIRTIO_NET_CTRL_NOTF_COAL 6

@@ -1532,49 +1532,62 @@ \subsubsection{Control Virtqueue}\label{sec:Device 
Types / Network Device / Devi
  
  Coalescing parameters:

  \begin{itemize}
-\item \field{rx_usecs}: Maximum number of usecs to delay a RX notification.
-\item \field{tx_usecs}: Maximum number of usecs to delay a TX notification.
-\item \field{rx_max_packets}: Maximum number of packets to receive before a RX 
notification.
-\item \field{tx_max_packets}: Maximum number of packets to send before a TX 
notification.
+\item \field{usecs} for RX: Maximum number of usecs to delay a RX notification.
+\item \field{usecs} for TX: Maximum number of usecs to delay a TX notification.
+\item \field{max_packets} for RX: Maximum number of packets to receive before 
a RX notification.
+\item \field{max_packets} for TX: Maximum number of packets to send before a 
TX notification.
  \end{itemize}
  
-

  The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
  \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and 
\field{tx_max_packets} parameters.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and 
\field{rx_max_packets} parameters.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{usecs} and 
\field{max_packets} parameters for all the transmit virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{usecs} and 
\field{max_packets} parameters for all the receive virtqueues.
  \end{enumerate}
  
-\subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}

+\subparagraph{Operation}\label{sec:Device Types / Network Device / Device 
Operation / Control Virtqueue / Notifications Coalescing / Operation}
+
+The device sends a used buffer notification once the notification conditions 
are met, if the notifications are not suppressed as explained in \ref{sec:Basic 
Facilities of a Virtio Device / Virtqueues / Used Buffer Notification 
Suppression}.
+
+When the device has non-zero \field{usecs} and non-zero \field{max_packets}, 
it starts counting usecs and packets upon receiving/sending a packet.
+The device counts packets and usecs for each receive virtqueue and transmit 
virtqueue separately.
+In this case, the notification conditions are met when \field{usecs} usecs 
elapses, or upon sending/receiving \field{max_packets} packets, whichever 
happens first.
+


Hi, Alvaro.

"when \field{usecs} usecs elapses"  --> "when \field{usecs} elapses", right?

Thanks for your clear work.


+When the device has \field{usecs} = 0 or \field{max_packets} = 0, the 
notification conditions are met after every packet received/sent.
+
+\subparagraph{RX Example}\label{sec:Device Types / Network Device / Device 
Operation / Control Virtqueue / Notifications Coalescing / RX Example}
  
  If, for example:

  \begin{itemize}
-\item \field{rx

Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

On Fri, Feb 17, 2023 at 10:42:21AM +0200, Alvaro Karsz wrote:
> Hi Heng,
> 
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> > +notifications coalescing.
> > +
> >  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
> >
> >  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> > @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit 
> > requirements}\label{sec:Device Types / Network Device
> >  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> >  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or 
> > VIRTIO_NET_F_HOST_TSO6.
> >  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> >  \end{description}
> >
> >  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / 
> > Network Device / Feature bits / Legacy Interface: Feature bits}
> > @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device 
> > Types / Network Device / Devi
> >  };
> >
> >  #define VIRTIO_NET_CTRL_NOTF_COAL 6
> > - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> >   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> > +
> >  \end{lstlisting}
> >
> >  Coalescing parameters:
> > @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device 
> > Types / Network Device / Devi
> >  \end{itemize}
> >
> >
> > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> >  \begin{enumerate}
> >  \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and 
> > \field{tx_max_packets} parameters.
> >  \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and 
> > \field{rx_max_packets} parameters.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and 
> > \field{max_usecs} parameters for a enabled
> > +transmit/receive virtqueue whose 
> > number is \field{vqn}.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the 
> > \field{max_packets} and \field{max_usecs} parameters of
> > +a enabled transmit/receive 
> > virtqueue whose number is \field{vqn}, and then
> > +responds them to the driver.
> >  \end{enumerate}
> >
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> > +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the 
> > coalescing
> > +   parameters of a enabled transmit/receive virtqueue.
> > +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a 
> > device, and the device
> > +   responds to the driver with the coalescing parameters of a enabled 
> > transmit/receive virtqueue.
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_coal_vq {
> > +le16 vqn;
> > +le16 reserved;
> > +le32 max_packets;
> > +le32 max_usecs;
> > +};
> > +\end{lstlisting}
> > +
> 
> Maybe we can use struct virtio_net_ctrl_coal inside struct
> virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> max_packets?
> I'm not sure if it would be confusing, what do you think?
> 

Hi Alvaro.

I guess you mean one of the following two forms:

#1
struct virtio_net_ctrl_coal {
le32 max_packets;
le32 max_usecs;
};

struct virtio_net_ctrl_coal_vq {
le16 vqn;
le16 reserved;
struct virtio_net_ctrl_coal coal;
} coal_vq;

#2
struct virtio_net_ctrl_coal {
le32 max_packets;
le32 max_usecs;
le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
};

If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq 
to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the 
presence of vqn and reserved is awkward.
If it's #2, I think this is a bit like the v1 version, using 
virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be 
semantic, but it is indeed more unified in function.

I think we should hear from Michael and Parav.

> > +Virtqueue coalescing parameters:
> > +\begin{itemize}
> > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive 
> > virtqueue, excluding the control virtqueue.
> > +\item \field{max_packets}: The maximum number of packets sent/received by 
> > the specified virtqueue before a TX/RX notification.
> > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the 
> > specified virtqueue delays a TX/RX notification.
> > +\end{itemize}
> > +
> > +\field{reserved} is reserved and it is ignored by the device.
> > +
> 
> max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> ("Maximum number of packets to receive/send before a RX/TX notification").
> The fact that this is applied to all VQs or to a specific one 

Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

On Fri, Feb 17, 2023 at 11:40:15AM +0200, Alvaro Karsz wrote:
> > > Maybe we can use struct virtio_net_ctrl_coal inside struct
> > > virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> > > max_packets?
> > > I'm not sure if it would be confusing, what do you think?
> > >
> >
> > Hi Alvaro.
> >
> > I guess you mean one of the following two forms:
> >
> > #1
> > struct virtio_net_ctrl_coal {
> > le32 max_packets;
> > le32 max_usecs;
> > };
> >
> > struct virtio_net_ctrl_coal_vq {
> > le16 vqn;
> > le16 reserved;
> > struct virtio_net_ctrl_coal coal;
> > } coal_vq;
> >
> > #2
> > struct virtio_net_ctrl_coal {
> > le32 max_packets;
> > le32 max_usecs;
> > le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
> > le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
> > };
> >
> > If it's #1, I think the format is a bit ugly, it's not semantic to use 
> > coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and 
> > the presence of vqn and reserved is awkward.
> > If it's #2, I think this is a bit like the v1 version, using 
> > virtio_net_ctrl_coal as a virtual queue to send commands does not seem to 
> > be semantic, but it is indeed more unified in function.
> >
> > I think we should hear from Michael and Parav.
> >
> 
> I meant #1.
> We can see virtio_net_ctrl_coal as a struct holding coalescing
> parameters, regardless of the commands.
> Yes, let's wait for more comments on that.
> 
> > > > +Virtqueue coalescing parameters:
> > > > +\begin{itemize}
> > > > +\item \field{vqn}: The virtqueue number of the enabled transmit or 
> > > > receive virtqueue, excluding the control virtqueue.
> > > > +\item \field{max_packets}: The maximum number of packets sent/received 
> > > > by the specified virtqueue before a TX/RX notification.
> > > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the 
> > > > specified virtqueue delays a TX/RX notification.
> > > > +\end{itemize}
> > > > +
> > > > +\field{reserved} is reserved and it is ignored by the device.
> > > > +
> > >
> > > max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> > > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> > > ("Maximum number of packets to receive/send before a RX/TX notification").
> > > The fact that this is applied to all VQs or to a specific one is
> > > derived from the command.
> > > Same for max_usecs.
> > > Maybe we can join the coalescing parameters somehow instead of
> > > repeating the explanations?
> > >
> 
> Any thoughts on this part?

Good idea, and if so, is there a good way to expose vqn to the interpretation 
of max_packets ?

#1
\item \field{vqn}: The virtqueue number of the enabled transmit or receive 
virtqueue.
\item \field{max_packets}: The maximum number of packets sent/received by the 
specified virtqueue before a TX/RX notification.

#2
\item \field{max_packets}: Maximum number of packets to receive/send before a 
RX/TX notification.

Thanks.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [PATCH v8] virtio-net: support inner header hash

[Heng Qi]

在 2023/2/18 上午12:24, Parav Pandit 写道:

From: virtio-dev@lists.oasis-open.org  On
Behalf Of Heng Qi

[..]


We assume that hash_report_tunnel_types is still present in the next version,

I am little lost.


Hi, Parav.

You are not lost. I'm just answering some of Michael's questions and 
making assumptions. :)



I thought we all agreed that reporting just the tunnel type in data path path 
virtio_net_hdr is not useful to sw.
And hence we should omit in the virtio_net_hdr.

Did I miss the motivation to add it back?
If not, probably it is better to review and discuss v9 without it, which will 
be easier to discuss.


but it only exists in virtio net hdr and should be populated by the device after
the hash calculation. hash_tunnel_types already controls whether the device
computes internal header hashes.


I don't really follow this, hash_report_tunnel_type is better off
keeping it "report" literally.


Talking about VIRTIO_NET_F_HASH_TUNNEL here. Not
hash_report_tunnel_type.


-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

在 2023/2/17 下午6:07, Michael S. Tsirkin 写道:

On Fri, Feb 17, 2023 at 11:40:15AM +0200, Alvaro Karsz wrote:

Maybe we can use struct virtio_net_ctrl_coal inside struct
virtio_net_ctrl_coal_vq instead of repeating max_usecs and
max_packets?
I'm not sure if it would be confusing, what do you think?


Hi Alvaro.

I guess you mean one of the following two forms:

#1
struct virtio_net_ctrl_coal {
 le32 max_packets;
 le32 max_usecs;
};

struct virtio_net_ctrl_coal_vq {
 le16 vqn;
 le16 reserved;
 struct virtio_net_ctrl_coal coal;
} coal_vq;

#2
struct virtio_net_ctrl_coal {
 le32 max_packets;
 le32 max_usecs;
 le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
 le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
};

If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq 
to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the 
presence of vqn and reserved is awkward.
If it's #2, I think this is a bit like the v1 version, using 
virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be 
semantic, but it is indeed more unified in function.

I think we should hear from Michael and Parav.


I meant #1.
We can see virtio_net_ctrl_coal as a struct holding coalescing
parameters, regardless of the commands.
Yes, let's wait for more comments on that.

Reusing virtio_net_ctrl_coal is a nice thought. Makes it a bit clearer
these have exactly the same role.
Whether to put vqn first or last does not matter imho.


+Virtqueue coalescing parameters:
+\begin{itemize}
+\item \field{vqn}: The virtqueue number of the enabled transmit or receive 
virtqueue, excluding the control virtqueue.
+\item \field{max_packets}: The maximum number of packets sent/received by the 
specified virtqueue before a TX/RX notification.
+\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified 
virtqueue delays a TX/RX notification.
+\end{itemize}
+
+\field{reserved} is reserved and it is ignored by the device.
+

max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification").
The fact that this is applied to all VQs or to a specific one is
derived from the command.
Same for max_usecs.
Maybe we can join the coalescing parameters somehow instead of
repeating the explanations?


Any thoughts on this part?

I think I agree. Generally I think we should first of all describe the
new VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text
to that.

Then just explain that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET have the same effect
as VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET repeated for
all currently enabled tx/rx vqs.
Plus maybe a single annotated example where there's a mix of
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands. For example with 2 vq pairs:

1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets for vq 0 and 2, vq 1 and 3 retain 
reset value
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 0, vq 2 retains value from 1
3. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 1, vq 3 retains reset value
4. VIRTIO_NET_CTRL_NOTF_COAL_TX_SET overrides command 3

no need for many examples.



Good idea. This is a clear and comprehensive example.

Thanks.






-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation

[Heng Qi]

On Fri, Feb 17, 2023 at 04:12:34PM +, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin 
> > Sent: Friday, February 17, 2023 6:35 AM
> 
> > > We mention the device reset case, but nothing about VQ reset.
> > >
> > > I feel that no matter how we handle this, we break something.
> > >
> > > Having default coalescing values may collide with "Upon reset, a
> > > device MUST initialize all coalescing parameters to 0."
> > 
> > No this is after device reset.
> > 
> > > We can say that VQ reset doesn't affect the "global parameters" and a
> > > device reset does, but this collides with "Device Requirements:
> > > Virtqueue Reset".
> > >
> Not really.
> When the device resets, VQ objects are destroyed in the device.
> So VQ's notifications parameters doesn't exists on device reset.
> 
> And so the same case with VQ reset.
> When a VQ is reset (disabled), VQ's notifications configuration is removed in 
> the device too.
> Just like its desc ring and other addresses are invalid.

Yes, but there seems to be such a situation: when the device is reactivated, as 
the specification says,
all parameters are set to 0 (use parameters as the default configuration on the 
device).
When CTRL_COAL_SET and CTRL_COAL_VQ_SET are sent, the configuration is updated 
(the parameters of each vq may be different,
but the global parameter configuration may be recorded), at this time, if vq is 
reset,
should the parameters be 0 or a recorded global parameters after it is 
re-enabled?

> 
> > > In fact, resetting coalescing values after vq reset may be derived
> > > from "Upon reset, a device MUST initialize all coalescing parameters
> > > to 0".
> > > This is consistent with "Device Requirements: Virtqueue Reset".
> > >
> > > I can add in my patch some clarifications.
> > >
> > > This will break the linux virtio_net ethtool implementation a little,
> > > we'll need to fix it.
> > 
> > Not good. I feel we must come up with spec that is backwards compatible.
> > Hmm, maybe this is why Parav kept talking about modes.
> > I did not realize at the time, sorry Parav.
> > 
> > I still feel modes are not the best way to describe things so I propose 
> > this:
> > - in addition to per vq parameters, device that supports global TX/RX
> >   commands and ring reset maintains two sets of default parameters: for RX 
> > and
> > TX
> > - existing commands change default and change all enabled vqs
> >   of the correct type (RX/TX) to the same value
> > - vq reset changes a vq to the default
> > - device reset changes defaults to 0 and changes all vqs to  0
> > 
> > note how defaults are only used for ring reset.  is "vq reset parameter"
> > a better name? I feel we will repeat "reset" too many times in a sentence 
> > if we
> > call it that though.
> > 
> > So fundamentally the only change is with RING_RESET, then default is not
> > always 0, it can be set by the global command.
> 
> True default is not zero when global are configured.
> It is ok to report VQ parameters on GET command when globals are configured.

-
To unsubscribe, e-mail: virtio-dev-unsubscr...@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-h...@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v6] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature

[Heng Qi]

Hi, Alvaro.
Thanks for your work !
As suggested earlier in the 'virtqueue coalescing' thread, I will be 
working on top of your patch. :)



在 2023/2/19 下午5:03, Alvaro Karsz 写道:

This patch makes several improvements to the notification coalescing
feature, including:

- Consolidating virtio_net_ctrl_coal_tx and virtio_net_ctrl_coal_rx
   into a single struct, virtio_net_ctrl_coal, as they are identical.
- Emphasizing that the coalescing commands are best-effort.
- Defining the behavior of coalescing with regards to delivering
   notifications when a change occur.
- Stating that the commands should apply to all the receive/transmit
   virtqueues.
- Stating that every receive/transmit virtqueue should count it's own
   packets.
- A new intro explaining the entire coalescing operation.

Signed-off-by: Alvaro Karsz 
Reviewed-by: Parav Pandit 
---
v2:
- Add the last 2 points to the patch.
- Rephrase the "commands are best-effort" note.
- Replace "notification" with "used buffer notification" to be
  more consistent.
v3:
- Add an intro explaining the entire coalescing operation.
v4:
- Minor wording fixes.
- Rephrase the general note.
v5:
- Replace virtio_net_ctrl_coal->usecs with
  virtio_net_ctrl_coal->max_usecs
v6:
- Replace usecs with microseconds.
- Make the TX/RX examples relevant for MQ as well.
- Explain that upon meeting the notification condition, the
  device starts counting packets and microseconds again.
- Add Parav's Reviewed-by tag (Thanks!)

  device-types/net/description.tex | 74 +++-
  1 file changed, 44 insertions(+), 30 deletions(-)

diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index 1741c79..c7a8ca6 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -1514,15 +1514,15 @@ \subsubsection{Control Virtqueue}\label{sec:Device 
Types / Network Device / Devi
  If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
  send control commands for dynamically changing the coalescing parameters.
  
-\begin{lstlisting}

-struct virtio_net_ctrl_coal_rx {
-le32 rx_max_packets;
-le32 rx_usecs;
-};
+\begin{note}
+The behavior of the device in response to these commands is best-effort:
+the device may generate notifications more or less frequently than specified.
+\end{note}
  
-struct virtio_net_ctrl_coal_tx {

-le32 tx_max_packets;
-le32 tx_usecs;
+\begin{lstlisting}
+struct virtio_net_ctrl_coal {
+le32 max_packets;
+le32 max_usecs;
  };
  
  #define VIRTIO_NET_CTRL_NOTF_COAL 6

@@ -1532,49 +1532,63 @@ \subsubsection{Control Virtqueue}\label{sec:Device 
Types / Network Device / Devi
  
  Coalescing parameters:

  \begin{itemize}
-\item \field{rx_usecs}: Maximum number of usecs to delay a RX notification.
-\item \field{tx_usecs}: Maximum number of usecs to delay a TX notification.
-\item \field{rx_max_packets}: Maximum number of packets to receive before a RX 
notification.
-\item \field{tx_max_packets}: Maximum number of packets to send before a TX 
notification.
+\item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX 
notification.
+\item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX 
notification.
+\item \field{max_packets} for RX: Maximum number of packets to receive before 
a RX notification.
+\item \field{max_packets} for TX: Maximum number of packets to send before a 
TX notification.
  \end{itemize}
  
-

  The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
  \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and 
\field{tx_max_packets} parameters.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and 
\field{rx_max_packets} parameters.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and 
\field{max_packets} parameters for all the transmit virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and 
\field{max_packets} parameters for all the receive virtqueues.
  \end{enumerate}
  
-\subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}

+\subparagraph{Operation}\label{sec:Device Types / Network Device / Device 
Operation / Control Virtqueue / Notifications Coalescing / Operation}
+
+The device sends a used buffer notification once the notification conditions 
are met, if the notifications are not suppressed as explained in \ref{sec:Basic 
Facilities of a Virtio Device / Virtqueues / Used Buffer Notification 
Suppression}.
+
+When the device has non-zero \field{max_usecs} and non-zero 
\field{max_packets}, it starts counting microseconds and packets upon 
receiving/sending a packet.
+The device counts packets and microseconds for each receive virtqueue and 
transmit virtqueue separately.
+In this case, the notification c