Re: [ovs-dev] [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading feature on DPDK physical ports.
Regards _Sugesh > -Original Message- > From: Jesse Gross [mailto:je...@kernel.org] > Sent: Wednesday, August 24, 2016 3:59 AM > To: Chandran, Sugesh> Cc: ovs dev > Subject: Re: [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading > feature on DPDK physical ports. > > On Tue, Aug 23, 2016 at 2:57 AM, Chandran, Sugesh > wrote: > >> -Original Message- > >> From: Jesse Gross [mailto:je...@kernel.org] > >> Sent: Monday, August 22, 2016 7:50 PM > >> To: Chandran, Sugesh > >> Cc: ovs dev > >> Subject: Re: [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading > >> feature on DPDK physical ports. > >> > >> On Mon, Aug 22, 2016 at 6:40 AM, Sugesh Chandran > >> wrote: > >> > diff --git a/lib/netdev-native-tnl.c b/lib/netdev-native-tnl.c > >> > index > >> > ce2582f..78ce0c9 100644 > >> > --- a/lib/netdev-native-tnl.c > >> > +++ b/lib/netdev-native-tnl.c > >> > @@ -85,11 +85,17 @@ netdev_tnl_ip_extract_tnl_md(struct dp_packet > >> > *packet, struct flow_tnl *tnl, > >> > > >> > ovs_be32 ip_src, ip_dst; > >> > > >> > -if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { > >> > -VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); > >> > -return NULL; > >> > +if(OVS_UNLIKELY(!dp_packet_ip_checksum_valid(packet))) { > >> > +if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { > >> > +VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); > >> > +return NULL; > >> > +} > >> > } > >> > > >> > +/* Reset the IP checksum offload flags if present, to avoid > >> > wrong > >> > + * interpretation in the further packet processing when > >> recirculated.*/ > >> > +reset_dp_packet_ip_checksum_ol_flags(packet); > >> > + > >> > if (ntohs(ip->ip_tot_len) > l3_size) { > >> > VLOG_WARN_RL(_rl, "ip packet is truncated (IP > >> > length %d, > >> actual %d)", > >> > ntohs(ip->ip_tot_len), l3_size); @@ > >> > -179,20 > >> > +185,26 @@ udp_extract_tnl_md(struct dp_packet *packet, struct > >> flow_tnl *tnl, > >> > } > >> > > >> > if (udp->udp_csum) { > >> > -uint32_t csum; > >> > -if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { > >> > -csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); > >> > -} else { > >> > -csum = packet_csum_pseudoheader(dp_packet_l3(packet)); > >> > -} > >> > - > >> > -csum = csum_continue(csum, udp, dp_packet_size(packet) - > >> > - ((const unsigned char *)udp - > >> > - (const unsigned char > >> > *)dp_packet_l2(packet))); > >> > -if (csum_finish(csum)) { > >> > -return NULL; > >> > +if(OVS_UNLIKELY(!dp_packet_l4_checksum_valid(packet))) { > >> > +uint32_t csum; > >> > +if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { > >> > +csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); > >> > +} else { > >> > +csum = packet_csum_pseudoheader(dp_packet_l3(packet)); > >> > +} > >> > + > >> > +csum = csum_continue(csum, udp, dp_packet_size(packet) - > >> > + ((const unsigned char *)udp - > >> > + (const unsigned char > >> > *)dp_packet_l2(packet))); > >> > +if (csum_finish(csum)) { > >> > +return NULL; > >> > +} > >> > } > >> > tnl->flags |= FLOW_TNL_F_CSUM; > >> > + > >> > +/* Reset the udp checksum offload flags if present, to avoid > >> > wrong > >> > + * interpretation in the further packet processing when > >> recirculated.*/ > >> > +reset_dp_packet_l4_checksum_ol_flags(packet); > >> > } > >> > >> Just one final comment on this - I think it's probably better to > >> unconditionally clear the checksum offload flags (both L3 and L4) > >> whenever we pop off a tunnel. Those headers will no longer exist in > >> the packet and therefore those flags can't have any meaning. In > >> theory this shouldn't make any difference compared to what you have > here but it seems a little bit more robust. > > [Sugesh] I kept the reset logic separate for each layer intentionally > > to provide better modularity and maintainability. For eg: In future to > > implement IP-in-IP tunneling, the checksum flags has to be cleared in IP > layer, not in UDP. > > Also considering your previous comment, the chances of having > > different combination of checksum offload, its nice to keep the reset logic > specific to layers for better readability. > > What do you think? > > As you said, for all the existing tunneling implementation it should > > be fine to clear off the flags in one place
Re: [ovs-dev] [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading feature on DPDK physical ports.
On Tue, Aug 23, 2016 at 2:57 AM, Chandran, Sugeshwrote: >> -Original Message- >> From: Jesse Gross [mailto:je...@kernel.org] >> Sent: Monday, August 22, 2016 7:50 PM >> To: Chandran, Sugesh >> Cc: ovs dev >> Subject: Re: [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading >> feature on DPDK physical ports. >> >> On Mon, Aug 22, 2016 at 6:40 AM, Sugesh Chandran >> wrote: >> > diff --git a/lib/netdev-native-tnl.c b/lib/netdev-native-tnl.c index >> > ce2582f..78ce0c9 100644 >> > --- a/lib/netdev-native-tnl.c >> > +++ b/lib/netdev-native-tnl.c >> > @@ -85,11 +85,17 @@ netdev_tnl_ip_extract_tnl_md(struct dp_packet >> > *packet, struct flow_tnl *tnl, >> > >> > ovs_be32 ip_src, ip_dst; >> > >> > -if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { >> > -VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); >> > -return NULL; >> > +if(OVS_UNLIKELY(!dp_packet_ip_checksum_valid(packet))) { >> > +if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { >> > +VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); >> > +return NULL; >> > +} >> > } >> > >> > +/* Reset the IP checksum offload flags if present, to avoid wrong >> > + * interpretation in the further packet processing when >> recirculated.*/ >> > +reset_dp_packet_ip_checksum_ol_flags(packet); >> > + >> > if (ntohs(ip->ip_tot_len) > l3_size) { >> > VLOG_WARN_RL(_rl, "ip packet is truncated (IP length %d, >> actual %d)", >> > ntohs(ip->ip_tot_len), l3_size); @@ -179,20 >> > +185,26 @@ udp_extract_tnl_md(struct dp_packet *packet, struct >> flow_tnl *tnl, >> > } >> > >> > if (udp->udp_csum) { >> > -uint32_t csum; >> > -if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { >> > -csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); >> > -} else { >> > -csum = packet_csum_pseudoheader(dp_packet_l3(packet)); >> > -} >> > - >> > -csum = csum_continue(csum, udp, dp_packet_size(packet) - >> > - ((const unsigned char *)udp - >> > - (const unsigned char >> > *)dp_packet_l2(packet))); >> > -if (csum_finish(csum)) { >> > -return NULL; >> > +if(OVS_UNLIKELY(!dp_packet_l4_checksum_valid(packet))) { >> > +uint32_t csum; >> > +if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { >> > +csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); >> > +} else { >> > +csum = packet_csum_pseudoheader(dp_packet_l3(packet)); >> > +} >> > + >> > +csum = csum_continue(csum, udp, dp_packet_size(packet) - >> > + ((const unsigned char *)udp - >> > + (const unsigned char >> > *)dp_packet_l2(packet))); >> > +if (csum_finish(csum)) { >> > +return NULL; >> > +} >> > } >> > tnl->flags |= FLOW_TNL_F_CSUM; >> > + >> > +/* Reset the udp checksum offload flags if present, to avoid wrong >> > + * interpretation in the further packet processing when >> recirculated.*/ >> > +reset_dp_packet_l4_checksum_ol_flags(packet); >> > } >> >> Just one final comment on this - I think it's probably better to >> unconditionally >> clear the checksum offload flags (both L3 and L4) whenever we pop off a >> tunnel. Those headers will no longer exist in the packet and therefore those >> flags can't have any meaning. In theory this shouldn't make any difference >> compared to what you have here but it seems a little bit more robust. > [Sugesh] I kept the reset logic separate for each layer intentionally to > provide better modularity > and maintainability. For eg: In future to implement IP-in-IP tunneling, the > checksum > flags has to be cleared in IP layer, not in UDP. > Also considering your previous comment, the chances of having different > combination > of checksum offload, its nice to keep the reset logic specific to layers for > better readability. > What do you think? > As you said, for all the existing tunneling implementation it should be fine > to clear off the flags in one place > unconditionally. If you feel, it make more sense to separate the reset logic > at the time of need comes, I am OK > to send out next version with proposed changes. The IPIP tunnel case is actually the type of situation that makes me a little nervous - if the UDP checksum flag was set in that case, what does it refer to? I suppose it's possible that the driver is capable of understand IPIP tunnels and it's an inner L4 checksum and we could use that later. However, at the moment, the infrastructure isn't really there for us to track what the
Re: [ovs-dev] [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading feature on DPDK physical ports.
Hi Jesse, Please find my answers inline below. Regards _Sugesh > -Original Message- > From: Jesse Gross [mailto:je...@kernel.org] > Sent: Monday, August 22, 2016 7:50 PM > To: Chandran, Sugesh> Cc: ovs dev > Subject: Re: [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading > feature on DPDK physical ports. > > On Mon, Aug 22, 2016 at 6:40 AM, Sugesh Chandran > wrote: > > diff --git a/lib/netdev-native-tnl.c b/lib/netdev-native-tnl.c index > > ce2582f..78ce0c9 100644 > > --- a/lib/netdev-native-tnl.c > > +++ b/lib/netdev-native-tnl.c > > @@ -85,11 +85,17 @@ netdev_tnl_ip_extract_tnl_md(struct dp_packet > > *packet, struct flow_tnl *tnl, > > > > ovs_be32 ip_src, ip_dst; > > > > -if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { > > -VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); > > -return NULL; > > +if(OVS_UNLIKELY(!dp_packet_ip_checksum_valid(packet))) { > > +if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { > > +VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); > > +return NULL; > > +} > > } > > > > +/* Reset the IP checksum offload flags if present, to avoid wrong > > + * interpretation in the further packet processing when > recirculated.*/ > > +reset_dp_packet_ip_checksum_ol_flags(packet); > > + > > if (ntohs(ip->ip_tot_len) > l3_size) { > > VLOG_WARN_RL(_rl, "ip packet is truncated (IP length %d, > actual %d)", > > ntohs(ip->ip_tot_len), l3_size); @@ -179,20 > > +185,26 @@ udp_extract_tnl_md(struct dp_packet *packet, struct > flow_tnl *tnl, > > } > > > > if (udp->udp_csum) { > > -uint32_t csum; > > -if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { > > -csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); > > -} else { > > -csum = packet_csum_pseudoheader(dp_packet_l3(packet)); > > -} > > - > > -csum = csum_continue(csum, udp, dp_packet_size(packet) - > > - ((const unsigned char *)udp - > > - (const unsigned char > > *)dp_packet_l2(packet))); > > -if (csum_finish(csum)) { > > -return NULL; > > +if(OVS_UNLIKELY(!dp_packet_l4_checksum_valid(packet))) { > > +uint32_t csum; > > +if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { > > +csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); > > +} else { > > +csum = packet_csum_pseudoheader(dp_packet_l3(packet)); > > +} > > + > > +csum = csum_continue(csum, udp, dp_packet_size(packet) - > > + ((const unsigned char *)udp - > > + (const unsigned char > > *)dp_packet_l2(packet))); > > +if (csum_finish(csum)) { > > +return NULL; > > +} > > } > > tnl->flags |= FLOW_TNL_F_CSUM; > > + > > +/* Reset the udp checksum offload flags if present, to avoid wrong > > + * interpretation in the further packet processing when > recirculated.*/ > > +reset_dp_packet_l4_checksum_ol_flags(packet); > > } > > Just one final comment on this - I think it's probably better to > unconditionally > clear the checksum offload flags (both L3 and L4) whenever we pop off a > tunnel. Those headers will no longer exist in the packet and therefore those > flags can't have any meaning. In theory this shouldn't make any difference > compared to what you have here but it seems a little bit more robust. [Sugesh] I kept the reset logic separate for each layer intentionally to provide better modularity and maintainability. For eg: In future to implement IP-in-IP tunneling, the checksum flags has to be cleared in IP layer, not in UDP. Also considering your previous comment, the chances of having different combination of checksum offload, its nice to keep the reset logic specific to layers for better readability. What do you think? As you said, for all the existing tunneling implementation it should be fine to clear off the flags in one place unconditionally. If you feel, it make more sense to separate the reset logic at the time of need comes, I am OK to send out next version with proposed changes. Please let me know. > > Generally, this looks good to me though. Obviously, we won't be able to > apply it until there is support for DPDK 16.11 in OVS. Will you hold onto the > patch and resubmit it at that time? [Sugesh] Yes, that’s right. I will hold this patch and will send out again once the 16.11 is out. ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
Re: [ovs-dev] [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading feature on DPDK physical ports.
On Mon, Aug 22, 2016 at 6:40 AM, Sugesh Chandranwrote: > diff --git a/lib/netdev-native-tnl.c b/lib/netdev-native-tnl.c > index ce2582f..78ce0c9 100644 > --- a/lib/netdev-native-tnl.c > +++ b/lib/netdev-native-tnl.c > @@ -85,11 +85,17 @@ netdev_tnl_ip_extract_tnl_md(struct dp_packet *packet, > struct flow_tnl *tnl, > > ovs_be32 ip_src, ip_dst; > > -if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { > -VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); > -return NULL; > +if(OVS_UNLIKELY(!dp_packet_ip_checksum_valid(packet))) { > +if (csum(ip, IP_IHL(ip->ip_ihl_ver) * 4)) { > +VLOG_WARN_RL(_rl, "ip packet has invalid checksum"); > +return NULL; > +} > } > > +/* Reset the IP checksum offload flags if present, to avoid wrong > + * interpretation in the further packet processing when > recirculated.*/ > +reset_dp_packet_ip_checksum_ol_flags(packet); > + > if (ntohs(ip->ip_tot_len) > l3_size) { > VLOG_WARN_RL(_rl, "ip packet is truncated (IP length %d, > actual %d)", > ntohs(ip->ip_tot_len), l3_size); > @@ -179,20 +185,26 @@ udp_extract_tnl_md(struct dp_packet *packet, struct > flow_tnl *tnl, > } > > if (udp->udp_csum) { > -uint32_t csum; > -if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { > -csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); > -} else { > -csum = packet_csum_pseudoheader(dp_packet_l3(packet)); > -} > - > -csum = csum_continue(csum, udp, dp_packet_size(packet) - > - ((const unsigned char *)udp - > - (const unsigned char *)dp_packet_l2(packet))); > -if (csum_finish(csum)) { > -return NULL; > +if(OVS_UNLIKELY(!dp_packet_l4_checksum_valid(packet))) { > +uint32_t csum; > +if (netdev_tnl_is_header_ipv6(dp_packet_data(packet))) { > +csum = packet_csum_pseudoheader6(dp_packet_l3(packet)); > +} else { > +csum = packet_csum_pseudoheader(dp_packet_l3(packet)); > +} > + > +csum = csum_continue(csum, udp, dp_packet_size(packet) - > + ((const unsigned char *)udp - > + (const unsigned char > *)dp_packet_l2(packet))); > +if (csum_finish(csum)) { > +return NULL; > +} > } > tnl->flags |= FLOW_TNL_F_CSUM; > + > +/* Reset the udp checksum offload flags if present, to avoid wrong > + * interpretation in the further packet processing when > recirculated.*/ > +reset_dp_packet_l4_checksum_ol_flags(packet); > } Just one final comment on this - I think it's probably better to unconditionally clear the checksum offload flags (both L3 and L4) whenever we pop off a tunnel. Those headers will no longer exist in the packet and therefore those flags can't have any meaning. In theory this shouldn't make any difference compared to what you have here but it seems a little bit more robust. Generally, this looks good to me though. Obviously, we won't be able to apply it until there is support for DPDK 16.11 in OVS. Will you hold onto the patch and resubmit it at that time? ___ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
[ovs-dev] [RFC PATCHv3] netdev-dpdk: Enable Rx checksum offloading feature on DPDK physical ports.
Add Rx checksum offloading feature support on DPDK physical ports. By default, the Rx checksum offloading is enabled if NIC supports. However, the checksum offloading can be turned OFF either while adding a new DPDK physical port to OVS or at runtime. The rx checksum offloading can be turned off by setting the parameter to 'false'. For eg: To disable the rx checksum offloading when adding a port, 'ovs-vsctl add-port br0 dpdk0 -- \ set Interface dpdk0 type=dpdk options:rx-checksum-offload=false' OR (to disable at run time after port is being added to OVS) 'ovs-vsctl set Interface dpdk0 options:rx-checksum-offload=false' Similarly to turn ON rx checksum offloading at run time, 'ovs-vsctl set Interface dpdk0 options:rx-checksum-offload=true' This is a RFC patch as the new checksum offload flags 'PKT_RX_L4_CKSUM_GOOD' and 'PKT_RX_IP_CKSUM_GOOD' will be available only in DPDK 16.11 release. OVS must compile with DPDK 16.11 release to use the checksum offloading feature. The Tx checksum offloading support is not implemented due to the following reasons. 1) Checksum offloading and vectorization are mutually exclusive in DPDK poll mode driver. Vector packet processing is turned OFF when checksum offloading is enabled which causes significant performance drop at Tx side. 2) Normally, OVS generates checksum for tunnel packets in software at the 'tunnel push' operation, where the tunnel headers are created. However enabling Tx checksum offloading involves, *) Mark every packets for tx checksum offloading at 'tunnel_push' and recirculate. *) At the time of xmit, validate the same flag and instruct the NIC to do the checksum calculation. In case NIC doesnt support Tx checksum offloading, the checksum calculation has to be done in software before sending out the packets. No significant performance improvement noticed with Tx checksum offloading due to the e overhead of additional validations + non vector packet processing. In some test scenarios, it introduces performance drop too. Rx checksum offloading still offers 8-9% of improvement on VxLAN tunneling decapsulation even though the SSE vector Rx function is disabled in DPDK poll mode driver. Signed-off-by: Sugesh Chandran--- v3 - Reset the checksum offload flags in tunnel pop operation after the validation. - Reconfigure the dpdk port with rx checksum offload only if new configuration is different than current one. v2 - Set Rx checksum enabled by default. - Modified commit message, explaining the tradeoff with tx checksum offloading. - Use dpdk mbuf checksum offload flags instead of defining new metadata field in OVS dp_packet. - validate udp checksum mbuf flag only if the checksum present in the packet. - Doc update with Rx checksum offloading feature. --- INSTALL.DPDK-ADVANCED.md | 18 -- lib/dp-packet.h | 36 lib/netdev-dpdk.c| 46 ++ lib/netdev-native-tnl.c | 42 +++--- vswitchd/vswitch.xml | 13 + 5 files changed, 138 insertions(+), 17 deletions(-) diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md index 857c805..6cc42d9 100755 --- a/INSTALL.DPDK-ADVANCED.md +++ b/INSTALL.DPDK-ADVANCED.md @@ -14,7 +14,8 @@ OVS DPDK ADVANCED INSTALL GUIDE 9. [Flow Control](#fc) 10. [Pdump](#pdump) 11. [Jumbo Frames](#jumbo) -12. [Vsperf](#vsperf) +12. [Rx Checksum Offload](#rx_csum) +13. [Vsperf](#vsperf) ## 1. Overview @@ -834,7 +835,20 @@ vhost ports: ifconfig eth1 mtu 9000 ``` -## 12. Vsperf +## 12. Rx Checksum Offload +By default, DPDK physical ports are enabled with Rx checksum offload. Rx +checksum offload can be configured on a DPDK physical port either when adding +or at run time. + +e.g. To disable Rx checksum offload when adding a DPDK port dpdk0: + +`ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:rx-checksum-offload=false` + +e.g. To disable the Rx checksum offloading on a existing DPDK port dpdk0: + +`ovs-vsctl set Interface dpdk0 type=dpdk options:rx-checksum-offload=false` + +## 13. Vsperf Vsperf project goal is to develop vSwitch test framework that can be used to validate the suitability of different vSwitch implementations in a Telco deployment diff --git a/lib/dp-packet.h b/lib/dp-packet.h index 7c1e637..377572b 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -592,6 +592,42 @@ dp_packet_rss_invalidate(struct dp_packet *p) #endif } +static inline bool +dp_packet_ip_checksum_valid(struct dp_packet *p) +{ +#ifdef DPDK_NETDEV +return p->mbuf.ol_flags & PKT_RX_IP_CKSUM_GOOD; +#else +return 0; +#endif +} + +static inline bool +dp_packet_l4_checksum_valid(struct dp_packet *p) +{ +#ifdef DPDK_NETDEV +return p->mbuf.ol_flags & PKT_RX_L4_CKSUM_GOOD; +#else +return 0; +#endif +} + +static inline void +reset_dp_packet_l4_checksum_ol_flags(struct